Add caption-image app by gsaluja9 · Pull Request #204 · aperture-data/workflows

gsaluja9 · 2025-09-15T20:33:29Z

Adds auto generation of image captions using BLIP.
https://huggingface.co/docs/transformers/main/en/model_doc/blip#transformers.BlipForConditionalGeneration

ad-claw000

This workflow looks like a great addition. LGTM!

ad-claw000

Great addition! However, I noticed a few issues that should be addressed before this is merged:

Hardcoded batch_size: In apps/caption-image/app/images.py, self.batch_size = 32 is hardcoded inside FindImageQueryGenerator.__init__. It ignores the batch_size parameter passed from the CLI in caption_images.py. You should pass batch_size into the generator constructor and use it there.
Pagination logic during updates: The query generator fetches batches using batch_id: idx while filtering on wf_caption_image == None. Because the response handler updates these entities and removes the None condition, the total number of matching images changes dynamically. Depending on how ApertureDB evaluates batch_id, this could lead to skipping images (e.g. batch 1 shifts into batch 0's place after batch 0 is updated). A safer approach might be to not rely on batch_id and instead repeatedly request the first N items (e.g., limit), or use a stable identifier for pagination.
Module-level Model Loading: The AutoProcessor and BlipForConditionalGeneration are loaded at the module level in images.py. This means they are loaded into memory as soon as the module is imported, even if just running --help. Consider lazy-loading them inside the class or function, or at least only when the command is actually executed.

Please let me know when these are updated!

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

ad-claw000 · 2026-05-24T18:14:24Z

I've pushed a commit to address the review comments:

batch_size is now correctly propagated from the CLI to the query generator.
Switched from batch_id pagination to using limit since wf_caption_image gets updated dynamically.
The BLIP model and processor are now lazy-loaded on demand to speed up script initialization (e.g. when just running --help).

Let me know if this looks good to go!

Copilot

Pull request overview

This PR introduces a new caption-image workflow that auto-generates image captions using the HuggingFace BLIP model, adds container/build plumbing for it, and adds devcontainer configurations to run workflows against a local ApertureDB stack.

Changes:

Added a new apps/caption-image workflow that finds uncaptioned images and writes captions back to ApertureDB.
Integrated the new app into CI builds and added a Docker build-time “warmup/validation” step.
Added devcontainer configs (compose + scripts) to spin up a local ApertureDB + Lenz + WebUI stack for multiple workflows.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
workflows-devcontainer.code-workspace	Workspace config to open workflows repo alongside `../app`.
postinstall.sh	Devcontainer post-create script to configure `adb` and install shell completion.
initcommand.sh	Devcontainer init script (build base image + generate `.env`).
configuration_params.py	Emits `ADB_PORT` based on platform (mac vs others).
base/docker/scripts/sitecustomize.py	Improves global exception hook (docstring + avoids shadowing `type`).
apps/caption-image/requirements.txt	Adds Python dependency on `transformers`.
apps/caption-image/README.md	Documents the new caption-image workflow and usage.
apps/caption-image/Dockerfile	Builds caption-image image; installs torch/torchvision + transformers; runs validation.
apps/caption-image/app/warmup_validate.py	Build-time BLIP warmup + caption assertion.
apps/caption-image/app/images.py	QueryGenerator implementation: fetches images, runs BLIP, updates captions.
apps/caption-image/app/caption_images.py	Typer-based entrypoint for running the caption workflow.
apps/caption-image/app/app.sh	Container entrypoint script that runs `caption_images.py`.
.vscode/launch.json	Debug configuration for Python.
.gitignore	Ignores `aperturedb/` directories (local dev data).
.github/workflows/main.yml	Adds `caption-image` to the CI build matrix.
.devcontainer/dataset-ingestion/docker-compose.yml	Local stack compose file for dataset-ingestion devcontainer.
.devcontainer/dataset-ingestion/devcontainer.json	VS Code devcontainer definition for dataset-ingestion.
.devcontainer/crawl-website/docker-compose.yml	Local stack compose file for crawl-website devcontainer.
.devcontainer/crawl-website/devcontainer.json	VS Code devcontainer definition for crawl-website.
.devcontainer/caption-image/docker-compose.yml	Local stack compose file for caption-image devcontainer.
.devcontainer/caption-image/devcontainer.json	VS Code devcontainer definition for caption-image.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Moved configuration_params.py to .devcontainer - Updated initcommand.sh to loop over all devcontainers - Provided ADB_PORT default in docker-compose.yml files - Updated images.py to correctly use batching, add PyTorch inference context, fix missing DONE state, handle execution query errors - Fixes to warmup_validate.py to avoid external network request - Replaced PIP commands with requirements.txt - Addressed logging and env var issues in caption_images.py

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

- Add threading.Lock to get_model_and_processor lazy init - Remove unused desc_blobs variable - Change PRELOAD_MODEL default to false in Dockerfile - Replace torchvision with pillow in requirements.txt - Fix WORKFLOW_VERSION quoting in initcommand.sh - Add caption-image service to docker-compose.yml

…of batch_id

ad-claw000 · 2026-05-25T02:10:27Z

Replaced the batch_id pagination logic with a stable identifier (fetching all uncaptioned _uniqueids upfront) to prevent skipping images as their _done status is updated. This addresses point 2 from @ad-claw000's review. See commit 7725c94.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 12 comments.

- Use count query + server side batch in FindImageQueryGenerator - Add validation for batch_size > 0 - Switch to RGB and handle decode exceptions - Simplify uniqueids-captions zipping to skip failed - Update log levels to check WF_LOG_LEVEL - Add PRELOAD_MODEL flag to docker-compose.yml - Add proper evaluation and no_grad to warmup_validate - Clean up docs to reflect python None - Add set -euo pipefail to bash scripts

Copilot

Pull request overview

Copilot reviewed 30 out of 31 changed files in this pull request and generated 6 comments.

luisremis

this pr does 2 things:

add dev containers and configs for some of the apps
introduce a new app, caption-image

Please remove everything related to 1, and just focus this PR on 2.

…rs for pagination Removes devcontainer files as requested to focus solely on the caption-image app. Restores stable identifier (_uniqueid) for pagination to address the remaining comment regarding skipping images during updates.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

- do not delete huggingface cache if PRELOAD_MODEL=true - restore batch processing with count query in FindImage - do not enforce PRELOAD_MODEL=true in test script

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

…arness

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

- Surface count-query parsing errors instead of swallowing them in images.py - Remove unused create_connector import in test/seed.py to reduce noise

Addresses review feedback regarding pagination logic during updates. Using a pre-fetched list of unique IDs prevents skipping images when the result set dynamically mutates.

ad-claw000 · 2026-07-02T05:02:19Z

Addressed the pagination logic issue from the review. I restored the use of a stable identifier (_uniqueid) to safely paginate over images without risking skips when the dataset mutates. See commit c5fdae8.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

…size

This resolves a review comment regarding items skipping when using batch_id with a dynamic result set. A previous commit inadvertently reverted this fix. It now correctly pre-fetches the uniqueids to provide stable pagination.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

…o prevent skipping images

ad-claw000 · 2026-07-02T13:58:22Z

Restored stable identifier pagination (pre-fetching _uniqueids) with a limit of 100,000 to prevent skipping images during updates, addressing the feedback. See commit 272414b.

Copilot

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

luisremis · 2026-07-02T17:09:01Z

Docker images for version v2026.7.1 were built and pushed after this PR was merged. View workflow run

image captions usign blip.

38084c4

gsaluja9 requested review from bovlb and drewaogle September 15, 2025 22:42

gsaluja9 marked this pull request as ready for review September 17, 2025 13:52

Adding devcontainers. (#208)

2dfb0b8

gsaluja9 requested a review from luisremis September 17, 2025 18:22

stray file.

e3b5994

bovlb reviewed Sep 18, 2025

View reviewed changes

gsaluja9 and others added 2 commits September 19, 2025 09:29

Some review feedback

4a3651f

Merge branch 'main' into image_captions

5516ccd

ad-claw000 approved these changes May 19, 2026

View reviewed changes

ad-claw000 previously requested changes May 21, 2026

View reviewed changes

luisremis assigned ad-claw000 May 24, 2026

Address review comments for image captions

c1baf46

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

Copilot AI review requested due to automatic review settings May 24, 2026 18:14

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 18:14 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 20:21

merge: resolve conflicts with main

324e1be

Copilot AI review requested due to automatic review settings May 24, 2026 20:25

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 20:25 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 22:12

Fix pagination logic to use a stable identifier (unique IDs) instead …

7725c94

…of batch_id

Copilot AI review requested due to automatic review settings May 25, 2026 02:10

Copilot started reviewing on behalf of ad-claw000 May 25, 2026 02:10 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Copilot AI reviewed Jun 27, 2026

View reviewed changes

fix(caption-image): address PR review comments

de07d26

luisremis reviewed Jul 1, 2026

View reviewed changes

Comment thread base/docker/scripts/sitecustomize.py Outdated

luisremis requested changes Jul 1, 2026

View reviewed changes

luisremis changed the title ~~image captions using blip.~~ Add caption-image app Jul 1, 2026

Copilot AI review requested due to automatic review settings July 2, 2026 00:45

Copilot started reviewing on behalf of ad-claw000 July 2, 2026 00:45 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread apps/caption-image/Dockerfile Outdated

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/test.sh Outdated

OpenClaw Bot added 2 commits July 2, 2026 01:09

Merge remote-tracking branch 'origin/main' into image_captions

4807ef7

fix(caption-image): address PR review comments

59ac20e

- do not delete huggingface cache if PRELOAD_MODEL=true - restore batch processing with count query in FindImage - do not enforce PRELOAD_MODEL=true in test script

Copilot AI review requested due to automatic review settings July 2, 2026 01:11

Copilot started reviewing on behalf of ad-claw000 July 2, 2026 01:12 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread apps/caption-image/README.md

Comment thread apps/caption-image/requirements.txt Outdated

Comment thread apps/caption-image/test.sh Outdated

Address review comments: README fix, requirements version, and test h…

59acb41

…arness

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py Outdated

Comment thread apps/caption-image/test/seed.py Outdated

OpenClaw Bot added 2 commits July 2, 2026 02:19

fix: surface parsing error and remove unused import

5055fc4

- Surface count-query parsing errors instead of swallowing them in images.py - Remove unused create_connector import in test/seed.py to reduce noise

fix(caption-image): use _uniqueid list for stable pagination

c5fdae8

Addresses review feedback regarding pagination logic during updates. Using a pre-fetched list of unique IDs prevents skipping images when the result set dynamically mutates.

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/test/Dockerfile Outdated

Comment thread apps/caption-image/README.md

OpenClaw Bot added 2 commits July 2, 2026 05:33

Address review comments regarding image batching and test Dockerfile …

a8faa53

…size

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread apps/mcp-server/test/docker-compose.yml Outdated

Comment thread apps/caption-image/app/images.py

OpenClaw Bot added 2 commits July 2, 2026 08:17

Address review comments on seed retry and FindImage query batching

2217068

fix(caption-image): use stable identifier with limit for pagination t…

272414b

…o prevent skipping images

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread docker-compose.yml Outdated

Comment thread apps/caption-image/app/images.py

Address review comments regarding batching and healthcheck port

4baf502

luisremis approved these changes Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

gsaluja9 commented Sep 15, 2025 • edited by luisremis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 commented May 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luisremis left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

gsaluja9 commented Sep 15, 2025 •

edited by luisremis

Loading

ad-claw000 commented May 24, 2026 •

edited

Loading