Adds dependencies needed for Dataflow ML support

[chamikaramj]

Needed to add Dataflow ML support to the YAML template and Job builder.

These dependencies are used by following features.

Feature	Python module	Dependencies	Status in base container	Action needed
Vertex AI MLTransforms(VertexAITextEmbeddings, VertexAIImageEmbeddings)	apache_beam/ml/transforms/embeddings/vertex_ai.py	google-cloud-aiplatform (vertexai), pillow	✅ Already in base (google-cloud-aiplatform==1.164.0, pillow==12.3.0)	None
Vertex AI RunInference(VertexAIModelHandlerJSON)	apache_beam/ml/inference/vertex_ai_inference.py	google-cloud-aiplatform	✅ Already in base	None
Hugging Face MLTransform(SentenceTransformerEmbeddings)	apache_beam/ml/transforms/embeddings/huggingface.py	sentence-transformers, pillow	❌ Missing sentence-transformers (pillow is present)	Add sentence-transformers
Hugging Face RunInference(HuggingFacePipelineModelHandler)	apache_beam/ml/inference/huggingface_inference.py	tensorflow, torch, transformers (all imported unconditionally at module top-level)	❌ All three are missing from base	Add tensorflow(torch and transformers are pulled in automatically as transitive dependencies of sentence-transformers)

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces necessary dependencies to support Machine Learning workflows within Dataflow templates. By including 'sentence-transformers' and 'tensorflow', the changes enable users to leverage ML capabilities directly within their YAML-based pipeline configurations.

Highlights

• Dependency Updates: Added 'sentence-transformers' and 'tensorflow' to the project dependencies to enable Dataflow ML support.
• Configuration Alignment: Updated both the requirements file and the main YAML template script to ensure consistency in package availability.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds sentence-transformers and tensorflow to support Dataflow ML. Feedback highlights that these heavy packages should not be added to DEFAULT_DEPENDENCIES in main.py as they are already pre-installed in the base container, and listing them there increases worker startup times and risks breaking private network pipelines. Additionally, it is recommended to pin these dependencies to specific, compatible versions in default_base_yaml_requirements.txt to prevent dependency conflicts and ensure reproducible builds.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.05%. Comparing base (0351342) to head (8734439).
⚠️ Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (0351342) and HEAD (8734439). Click for more details.

HEAD has 10 uploads less than BASE

Flag	BASE (0351342)	HEAD (8734439)
	11	1

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #4310       +/-   ##
=============================================
- Coverage     56.34%   38.05%   -18.30%     
+ Complexity     7856      513     -7343     
=============================================
  Files          1155      195      -960     
  Lines         73226    11799    -61427     
  Branches       8585     1190     -7395     
=============================================
- Hits          41259     4490    -36769     
+ Misses        29183     6941    -22242     
+ Partials       2784      368     -2416     

Components	Coverage Δ
spanner-templates	∅ <ø> (∅)
spanner-import-export	∅ <ø> (∅)
spanner-live-forward-migration	∅ <ø> (∅)
spanner-live-reverse-replication	∅ <ø> (∅)
spanner-bulk-migration	∅ <ø> (∅)
gcs-spanner-dv	∅ <ø> (∅)
see 985 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.