Release 0.4.60

[luisremis]

This pull request introduces several improvements and updates across the development environment, CI workflows, and core connector logic. The main themes are: improved error handling and resource management in connectors, modernization and expansion of CI workflows (including nightly and GPU tests), and enhancements to the development container setup.

Development environment and CI/CD improvements:

• Added a new nightly GitHub Actions workflow to run scheduled tests, improving test coverage and reliability.
• Updated all CI workflows (pr.yaml, main.yml, develop.yml, release.yaml) to use the latest Google Cloud authentication actions and improved environment variable handling. Also, added a dedicated GPU test job in pr.yaml. [1] [2] [3] [4] [5]
• Enhanced the devcontainer setup by switching to a local Dockerfile, installing additional Python packages (awscli, openai-clip), and enabling the Pylance extension for better Python language support. [1] [2] [3]

Core connector and error handling enhancements:

• Improved resource management in both Connector and ConnectorRest classes by adding explicit close() methods and ensuring connections/sessions are properly closed, preventing resource leaks. [1] [2] [3] [4]
• Enhanced error handling in execute_query by adding an error_handler callback, improving logging for partial errors, and making the error reporting more robust and informative. [1] [2] [3] [4]

Connector logic and API improvements:

• Refined the check_status method in Connector to correctly identify and return the first negative status in nested responses, improving reliability of status checks.
• Improved the REST connector to support fallback imports for base64 decoding and to robustly initialize or re-initialize HTTP sessions as needed. [1] [2] [3]

These changes collectively improve the developer experience, testing reliability, and robustness of the core connector logic.

References:

• Nightly workflow:
• CI/CD updates and GPU testing: [1] [2] [3] [4] [5]
• Devcontainer enhancements: [1] [2] [3]
• Connector resource management: [1] [2] [3] [4]
• Error handling improvements: [1] [2] [3] [4]
• Status check and REST connector improvements: [1] [2] [3] [4]

[Copilot]

Pull request overview

This PR cuts the 0.4.60 release, combining connector robustness improvements (resource cleanup + richer error handling), new dataset utilities (PyTorch + new TensorFlow dataset support), and CI/dev-environment updates (new nightly workflow, GPU CI job, and devcontainer refresh).

Changes:

• Added/expanded dataset and operations utilities (PyTorch dataset now supports multiple blob-returning Find* commands; new TensorFlow dataset + tests; expanded Operations helpers).
• Improved connector lifecycle and error reporting (close() semantics, safer session management, error_handler callback, and better partial-error logging).
• Modernized and expanded CI/dev tooling (new nightly workflow, GPU PR job, safer env/log handling in test scripts, updated devcontainer).

Reviewed changes

Copilot reviewed 54 out of 55 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/test_UserConvenience.py	Adds coverage for ConnectorRest.close() and transparent session recreation.
test/test_torch_connector.py	Updates PyTorch dataset tests and adds coverage for non-image blobs (Blob/Video).
test/test_tf_connector.py	Introduces TensorFlow dataset integration tests (new file).
test/test_Sources.py	Adds unit tests for HTTP timeout handling and retry behavior.
test/test_ResponseHandler.py	Adds tests for new error_handler behavior under full/partial failures.
test/test_Parallel.py	Improves failure behavior assertions and adds coverage for dict responses + connection cleanup.
test/test_Operations.py	Adds tests for new/updated Operations helpers (resize scale, threshold, preview, etc.).
test/test_Images.py	Adds tests for resize-by-scale and ignored operations in point resolution.
test/test_Datawizard.py	Adds tests for QueryBuilder.get_query_cmd() parsing behavior.
test/test_Connector.py	Adds unit tests for improved Connector.check_status() and safe close() behavior.
test/test_ConnectionPool.py	Adds coverage for pool shutdown and post-close reconnect behavior.
test/run_test.sh	Makes CI/local test runs safer via env defaults, tmp credential file, and filter sanitization.
test/run_test_container.sh	Improves container/network cleanup, permissions handling, readiness checks, and skip-slow support.
test/generateInput.py	Switches image generation import to the new CLI module path.
robots.md	Updates guidance for skipping slow/external tests via SKIP_SLOW_TESTS.
pyproject.toml	Adds/clarifies optional speedups extra and includes ML deps in extras.
examples/README.md	Documents the new loader-hierarchy example script.
examples/loaders_101/data_loader_hierarchy.py	Adds a runnable script explaining loader hierarchy and usage.
docker/pytorch-gpu/Dockerfile	Updates GPU base image and installs CLIP for GPU testing.
docker/pytorch-gpu/build.sh	Adds cache control and --pull for more reproducible GPU image builds.
docker/notebook/Dockerfile.cpu	Switches CLIP install method to openai-clip.
docker/devcontainer/Dockerfile	Removes the old devcontainer Dockerfile (migrated to .devcontainer/).
aperturedb/VideoDownloader.py	Adds request timeout + error handling and fixes stats wording/empty case.
aperturedb/VideoDataCSV.py	Uses context manager for reading video files.
aperturedb/transformers/clip.py	Updates CLIP dependency/error message to openai-clip.
aperturedb/TensorFlowDataset.py	Adds TensorFlow dataset implementation (new feature).
aperturedb/Sources.py	Adds timeout support + fallback behavior and safer content-length validation.
aperturedb/Query.py	Adds QueryBuilder.get_query_cmd() helper for constraint/query parsing.
aperturedb/PyTorchDataset.py	Generalizes dataset to handle multiple Find* commands and improves error handling.
aperturedb/ParallelQuerySet.py	Uses QueryBuilder.get_query_cmd() and threads through error_handler support.
aperturedb/ParallelQuery.py	Adds thread-safe error counting, improved partial-error stats handling, and closes cloned clients.
aperturedb/ParallelLoader.py	Minor stats summation cleanup.
aperturedb/Operations.py	Adds resize-by-scale, threshold, and preview operations with validation.
aperturedb/Images.py	Improves rotate rounding and supports resize-by-scale in point resolution.
aperturedb/ImageDownloader.py	Adds request timeout, better retry logging, and clearer failure handling.
aperturedb/ConnectorRest.py	Adds session init/teardown, safer blob defaults, and optional pybase64 decode.
aperturedb/Connector.py	Introduces explicit close() and improves recursive status checking logic.
aperturedb/CommonLibrary.py	Adds error_handler callback support and improves logging for failures/partial errors.
aperturedb/cli/utilities.py	Adds generate-images CLI and disables Typer pretty exceptions in subcommands.
aperturedb/cli/transact.py	Disables Typer pretty exceptions for consistent CLI output.
aperturedb/cli/tokens.py	Disables Typer pretty exceptions for consistent CLI output.
aperturedb/cli/keys.py	Disables Typer pretty exceptions for consistent CLI output.
aperturedb/cli/ingest.py	Disables Typer pretty exceptions for consistent CLI output.
aperturedb/cli/generate_images.py	Adds image-generation implementation used by CLI/tests.
aperturedb/cli/configure.py	Disables Typer pretty exceptions for consistent CLI output.
aperturedb/cli/adb.py	Disables Typer pretty exceptions at the root CLI level.
aperturedb/init.py	Bumps version to 0.4.60.
.github/workflows/release.yaml	Migrates to google-github-actions/auth@v2 + setup-gcloud@v2.
.github/workflows/pr.yaml	Updates PR workflow and adds a GPU CI job.
.github/workflows/nightly.yml	Adds a scheduled nightly test workflow.
.github/workflows/main.yml	Migrates to google-github-actions/auth@v2 + setup-gcloud@v2.
.github/workflows/develop.yml	Migrates to google-github-actions/auth@v2 + setup-gcloud@v2.
.devcontainer/Dockerfile	Adds a local devcontainer Dockerfile with needed Python tools.
.devcontainer/docker-compose.yml	Switches devcontainer build to local Dockerfile context.
.devcontainer/devcontainer.json	Enables Pylance extension for improved Python language support.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.