Skip to content

feat: jh job result list and jh job result download - #74

Merged
lamdor merged 2 commits into
mainfrom
la/job-results
Oct 16, 2026
Merged

lamdor merged 2 commits into
mainfrom
la/job-results

Conversation

@lamdor

@lamdor lamdor commented Oct 15, 2026

Copy link
Copy Markdown
Member

Adds jh job result to list and download a job's result files.


What

New jh job result subcommands for the files a job uploads via RESULTS_FILE, e.g. to pull a previous batch run's output into a new job without writing Julia against JuliaHub.jl.

  • jh job result list <job-name> lists result files with size and upload time. --all shows every job file (input, source, project, result) with its type; --json prints {"job", "status", "files"}.
  • jh job result download <job-name> [file-name] downloads one file, or every result file when no name is given. -o is the destination file or directory, --force overwrites existing files, --type selects another job file type.
$ jh job result list --all jr-g3yakikz3l
jr-g3yakikz3l Completed (7 files)
TYPE      NAME                 SIZE  UPLOADED
input     Manifest.toml     7.8 KiB  Oct 14, 2026 21:38:18 -0500
...
result    outputs.txt       1.0 MiB  Oct 14, 2026 21:38:45 -0500

$ jh job result download jr-g3yakikz3l outputs.txt -o previous-run.txt
Downloaded outputs.txt (1.0 MiB) to previous-run.txt

How

  • Job details: GET /api/rest/jobs/{jobname} (Hasura REST, X-Hasura-Role: jhuser), the endpoint JuliaHub.jl's JuliaHub.job uses. An empty details list means the job doesn't exist or isn't visible.
  • File download: GET /jobs/{jobname}/files/{filename}?filetype=..., which 302-redirects to a short-lived presigned object-store URL. The client drops Authorization on every redirect. Go only does that cross-domain, and the object store rejects a second auth mechanism (and shouldn't see the token).
  • Downloads stream to a temp file next to the destination, get checked against the recorded sha2_256 hash, then get renamed into place. A failed or corrupted download leaves nothing behind.
  • JuliaHub records that hash as hex. JuliaHub.jl's FileHash base64-decodes it, which yields garbage, though nothing there compares it. Verified against nightly: the downloaded bytes' SHA-256 equals the hex value. Hex is tried first, with base64 as a fallback.
  • Output file names use only the base name of the server-supplied name, so they can't escape the target directory.

The existing jh job list / jh job start placeholders are unchanged.

Notes

  • Tested live against nightly: list, --all, --json, single and all-file download, -o file and directory, the no-overwrite and --force paths, --type source, a missing file name, and an unknown job. Also ran the new e2e test.
  • Running this inside a JuliaHub job still needs jh installed in the image and a ~/.juliahub login. jh doesn't yet read the job's injected credentials, so that's a possible follow-up.

🤖 Posted by an AI agent · harness: Claude Code 2.1.301 · model: claude-opus-5-5
Review: reviewed by @lamdor
Session: 1ce5aeb7-9a97-4eb5-aa37-46b1100cfa21

Adds `jh job result` for working with the files a job uploads via `RESULTS_FILE`, e.g. to pull a previous batch run's output into a new job.

- `jh job result list <job-name>` lists result files with size and upload time; `--all` shows every job file (input, source, project, result) with its type, `--json` prints `{"job", "status", "files"}`.
- `jh job result download <job-name> [file-name]` downloads one or all result files; `-o` sets the destination file or directory, `--force` overwrites, `--type` selects another job file type.

Job details come from `GET /api/rest/jobs/{jobname}` (the endpoint JuliaHub.jl's `JuliaHub.job` uses). Files are fetched from `GET /jobs/{jobname}/files/{filename}?filetype=...`, which 302-redirects to a presigned object-store URL; the client drops `Authorization` on every redirect since Go only does so cross-domain. Downloads stream to a temp file, are checked against the recorded `sha2_256` hash, and are then renamed into place. JuliaHub records that hash as hex (verified against a live instance), not base64 as JuliaHub.jl's `FileHash` assumes, so hex is tried first.

Agent-Harness: Claude Code 2.1.301
Agent-Model: claude-opus-5-5
Agent-Session: 1ce5aeb7-9a97-4eb5-aa37-46b1100cfa21
Agent-Review: reviewed by @lamdor
@lamdor
lamdor requested review from a team and pankgeorg October 15, 2026 20:52
@lamdor
lamdor marked this pull request as ready for review October 15, 2026 20:52
pankgeorg
pankgeorg previously approved these changes Oct 15, 2026

@pankgeorg pankgeorg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the security-sensitive parts hold up, and the inline comments are edge cases and suggestions, not blockers.

Checked and fine

  • Redirects (against the go1.24.4 net/http/client.go source): on every hop, Client.do re-copies the initial request's headers into a fresh map, then calls CheckRedirect last. So Del("Authorization") keeps the token on hop 0 only, which goes to https://<server>. That covers chains like API → API → store and API → store → API, and an https→http downgrade. The download request carries no Cookie, Proxy-Authorization or X-Hasura-Role, and the hook lives on a per-command client, so other commands aren't affected. The test is meaningful: Go compares hostnames without the port, so 127.0.0.1:A → 127.0.0.1:B counts as same-domain, and a plain client does forward the token there. I checked that with a throwaway probe.
  • Path safety: .., ., empty names, absolute paths, \ and C: volumes all reduce to a safe base name. See the inline comment for Windows device names.
  • Atomic download: the temp file is removed on every error path, and nothing is left at the destination after a mismatch. os.Rename replaces existing files on Windows too, so --force works there.
  • go build, go vet and the targeted tests pass locally.

Tests that would pin the edge cases: -o as a function of file count, pending (size: null) files in download-all, duplicate destinations, and a multi-hop redirect chain.

Nits not anchored inline

  • main.go: the jobCmd Long text still says "Job functionality is currently in development"; only list/start are placeholders now.
  • jobs.go:152: nameWidth counts bytes while pad/truncate count runes, so non-ASCII names widen the column. Cosmetic.
  • jobs.go:341: MkdirAll runs before verification, so -o newdir/ plus a checksum mismatch leaves an empty newdir/. Ctrl-C leaves a hidden .name.*.tmp file. Both are acceptable.
  • formatTokenDate now formats job timestamps too; a neutral name would read better.

🤖 Posted for @pankgeorg by an AI agent — harness: Claude Code 2.1.301 · model: claude-opus-5-5
Session: local Claude Code session 9b44d1cd-8141-494c-9383-8742eb5d0444

Comment thread jobs.go Outdated
Comment thread jobs.go
Comment thread jobs.go Outdated
Comment thread jobs.go Outdated
Comment thread jobs.go Outdated
Comment thread jobs.go Outdated
Comment thread jobs.go
Comment thread jobs.go
Comment thread jobs.go
Comment thread main.go Outdated
- Download-all always treats `-o` as a directory, so the on-disk shape no longer depends on how many result files a job has; a named download keeps the file-or-directory rule.
- Download-all skips files whose upload is still in progress (with a stderr notice); naming a pending file is an error.
- `--type` is validated against input/source/project/result, and every destination is resolved before anything is downloaded, so two files flattening to the same name fail up front instead of failing partway or overwriting each other.
- The download client bounds only the wait for response headers (60s) instead of the whole request, so large downloads over slow links are not aborted.
- Without `--force`, the finished download is hard-linked into place, so a file created at the destination during the transfer is not replaced (falling back to re-check + rename where hard links are unsupported).
- Downloads are checked against the recorded size; an empty recorded hash is treated as missing, and unverified downloads say "(not verified)".
- Destination names must pass `filepath.IsLocal`, which also rejects Windows device names.
- `Referer` is dropped on redirects along with `Authorization`.
- `--force` loses its `-f` shorthand, which means `--file` elsewhere in the CLI.
- Nits: the `jh job` help no longer calls all job functionality in development, the name column width counts runes, and `formatTokenDate` is renamed `formatTimestamp`.

Agent-Harness: Claude Code 2.1.301
Agent-Model: claude-opus-5-5
Agent-Session: 29b26359-89b6-4a2c-bc20-668092480740
Agent-Review: reviewed by @lamdor
@lamdor

lamdor commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Thanks for the thorough review. b7a8f44 addresses every inline comment, plus these nits:

  • jh job help: it no longer calls all job functionality in development, only list/start.
  • The name column width counts runes.
  • formatTokenDate is renamed formatTimestamp.
  • New tests cover -o as a function of the request, pending files, duplicate destinations, an invalid --type, size mismatch, an empty hash, the no-overwrite link, and an API → API → store → store redirect chain.

I left the MkdirAll-before-verify and Ctrl-C temp-file nits as is, as you suggested. I re-ran the download paths against nightly (download-all into a new -o dir, --force, named -o file, bad --type).

🤖 Posted by an AI agent · harness: Claude Code 2.1.301 · model: claude-opus-5-5
Review: reviewed by @lamdor
Session: 29b26359-89b6-4a2c-bc20-668092480740

@lamdor
lamdor requested a review from pankgeorg October 16, 2026 15:48
@pankgeorg

Copy link
Copy Markdown
Member

Is this compatible with the current juliahub? If so, let's backport to release-0.4, merge and tag 0.4.3?

@devin-ai-integration

Copy link
Copy Markdown

Works with JuliaHub 26.4.2. I checked every server API this PR calls against the v26.4.2 tag of JuliaHub. I did not run it against a live 26.4.2 instance.

  • GET /api/rest/jobs/{jobname}: Hasura REST endpoint GET job details (hasura/metadata/rest_endpoints.yaml, url: jobs/:jobname). It returns details: job_details_vw(...) and selects jobname, status, files and jobname_alias. The jhuser role can read all four columns, filtered to userid = X-Hasura-JuliaHub-Id. That means another user's job shows up as an empty list, which the CLI reports as "not found".
  • files shape: job_details_vw (migration 000420) builds each entry as {name, size, type, upload_timestamp, hash: {algorithm, value}}, with type limited to input/result/source/project. These are the same fields and types as JobFile/jobFileTypes.
  • Hash encoding: validate_jobfilehash only accepts ^\w{64}$ with sha2_256, and the hashes JuliaHub computes itself use bytes2hex(sha2_256(...)). So the hex-first decode is correct.
  • GET /jobs/{jobname}/files/{filename}?filetype=: route JobInfoImpl.job_file_download in services/JobManagement/src/JobInfoEndpoints.jl, proxied by the websrvr location ~ ^/jobs/.*/(files|outputs)/.*$ block, which accepts a bearer token through authenticate_token_or_session. The handler checks that the caller owns the job (business admins are allowed too), filters by filetype, and returns HttpRedirect. LegacyApiUtils turns that into a 302 with a location header pointing at a presigned S3/MinIO URL. Job files get a 60s expiry, so stripping Authorization on redirect is needed and is enough.

On the client side, go build ./... and go test ./... pass on b7a8f44.

Written by Devin

@lamdor
lamdor merged commit 2be53c0 into main Oct 16, 2026
1 check passed
@lamdor
lamdor deleted the la/job-results branch October 16, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants