Skip to content

Add consistent uploadId/batchId log prefix on the worker's call path - #114

Merged
DaxServer merged 1 commit into
mainfrom
feat/worker-log-id-prefix
Jul 5, 2026
Merged

DaxServer merged 1 commit into
mainfrom
feat/worker-log-id-prefix

Conversation

@DaxServer

Copy link
Copy Markdown
Owner

Log lines reachable from the upload worker's job processing identified the upload/batch inconsistently — some interpolated [uploadId/batchId] into the message, some duplicated it as a redundant structured pino field, and several call sites (tokenRetry.ts, most of mediawiki/client.ts, handlers/mapillary.ts) had no id at all.

  • Added an idTag(uploadId, batchId) helper in core/logger.ts
  • upload.worker.ts and workers/queue.ts now build the tag once and use it consistently, dropping redundant structured id duplicates
  • MediaWikiClient.applySdc/.nullEdit and MapillaryHandler.fetchImagesBatch/fetchImagesByIds now accept the tag so their log lines (chunk upload, SDC apply, null edit, retries) carry it too
  • fetchImagesBatch/fetchImagesByIds's tag param is optional (default '') since they're also called from a WS collection-browsing flow with no upload/batch context
  • tokenRetry.ts needed no changes — callers now embed the tag in the label string they already pass in

— Claude Sonnet 5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Confidence Score: 5/5

This looks safe to merge after a small hardening cleanup in queue removal logging.

  • No blocking issues found in the changed code.
  • The only noted issue is conditional on an existing queued job with missing stored upload data.
  • The main worker and MediaWiki call paths pass the new tag consistently.

backend/src/workers/queue.ts

Important Files Changed

Filename Overview
backend/src/core/logger.ts Adds a pure helper for formatting upload and batch IDs.
backend/src/mediawiki/client.ts Adds tag-prefixed logging to upload, SDC, null edit, and retry labels.
backend/src/handlers/mapillary.ts Adds optional tag propagation for Mapillary image-fetch logging.
backend/src/workers/upload.worker.ts Builds one tag per upload job and passes it to downstream logging paths.
backend/src/workers/queue.ts Uses idTag in enqueue and remove logging; stale or malformed stored jobs can make remove logging fail.
backend/src/tests/uploadClient.test.ts Updates MediaWiki client tests to pass explicit tag strings.

Reviews (1): Last reviewed commit: "feat: add consistent uploadId/batchId lo..." | Re-trigger Greptile

Comment on lines +54 to +56
logger.info(
`[worker] ${idTag(job.data.uploadId, job.data.batchId)} job ${jobId} removed from queue`,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Removal Log Reads Job Data

When removeUploadJob removes an existing BullMQ job whose stored data is missing or from an older shape, the new log line reads job.data.uploadId and job.data.batchId after the remove. The old path could remove the job using only jobId; this path can now reject during cleanup or emit [undefined/undefined], which can break callers that cancel or requeue stale jobs.

@DaxServer
DaxServer merged commit ff8e891 into main Jul 5, 2026
5 checks passed
@DaxServer
DaxServer deleted the feat/worker-log-id-prefix branch July 5, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant