feat(svc-datasets): record dataset file write time and serve the full File model in v1 and v2

[Przyval]

Intent

Kapten mengembangkan OpenFoundry - emulator Palantir Foundry buatannya sendiri - menuju kesetaraan dengan Foundry, sebagai alat bantu belajar dan riset.

Temuan ini muncul saat menyajikan /api/v1. Kontrak Foundry menuntut File.updatedTime, tetapi tidak ada penyimpanan berkas di OpenFoundry yang mencatat waktu tulis. Akibatnya dua hal:

1. Rute v2 GET /datasets/{rid}/files dan /files/{filePath} SUDAH menyajikan bentuk yang tidak lengkap hari ini - serializeFile menghilangkan field itu.
2. Kedua rute v1 yang setara sengaja TIDAK disajikan, karena firstmate menolak menerbitkan bentuk yang tidak lengkap hanya dengan alasan "v2 juga begitu". Cacat yang sudah ada bukan izin menambah cacat kedua yang sama.

Jadi v1 menolak menyajikan bentuk yang salah sementara v2 tetap menyajikannya - ketidakkonsistenan yang hanya bisa ditutup dengan memperbaiki akarnya.

What Changed

• Both file stores now record a write time: FileStore stamps updatedTime on every putFile, and PgFileStore inserts and returns updated_at (new migration db/migrations/010_dataset_file_updated_at.sql plus scripts/migrate.sql add the column, backfilled from created_at) - which also fixes overwrites of an existing path failing with 42703 undefined_column.
• /api/v2 file routes now serve Foundry's complete File shape (path, transactionRid, sizeBytes, updatedTime) instead of dropping updatedTime, and the previously unserved v1 equivalents - list dataset files and read file metadata - are now implemented with their own v1 serializer.
• DatasetFile in packages/api-types, the conjure dataset-service.yml definition, README/GUIDE/AGENTS notes, and the dataset lifecycle integration test are updated to the served shape, with a new services/svc-datasets/tests/pg-file-write-time.test.ts asserting the Pg write SQL against both schema sources.

Risk Assessment

✅ Low: The change fixes the root cause the intent names (no recorded write time), serves the identical complete File model in both versions, keeps the Pg store's SQL and both schema sources in agreement, and is covered by behavioral route and store tests with no in-repo consumer of the removed size/contentType fields.

Testing

Targeted unit, api-types and dataset-lifecycle integration suites all pass, and I backed them with two product-level demonstrations: a curl transcript against the running svc-datasets service showing v1 and v2 serving the identical complete File model with updatedTime (and v1's file routes reachable at all, which they previously were not), and a real-Postgres run reproducing the 42703 undefined_column overwrite failure before migration 010 and showing it fixed with the write time advancing after. No screenshot or rendered-UI artifact applies: the change is confined to HTTP API serialization, a file store and a SQL migration, with no console or other rendered surface touched, so the HTTP response bodies and persisted database rows are the end-user surface.

Evidence: v1 + v2 file API transcript against the running service

$ # dataset: ri.datasets.main.dataset.2b6bbf2a-b8a0-4342-9317-d85dd0c61871

$ curl -X PUT .../api/v2/datasets/$RID/files/raw/q1.csv  (upload)
{
    "path": "raw/q1.csv",
    "transactionRid": "",
    "sizeBytes": 19,
    "updatedTime": "2026-09-11T04:17:14.350Z"
}

$ curl .../api/v2/datasets/$RID/files   (v2 list - now carries updatedTime)
{
    "data": [
        {
            "path": "raw/q1.csv",
            "transactionRid": "",
            "sizeBytes": 19,
            "updatedTime": "2026-09-11T04:17:14.350Z"
        }
    ]
}

$ curl .../api/v1/datasets/$RID/files   (v1 list - now SERVED, was unserved)
{
    "data": [
        {
            "path": "raw/q1.csv",
            "transactionRid": "",
            "sizeBytes": 19,
            "updatedTime": "2026-09-11T04:17:14.350Z"
        }
    ]
}

$ curl .../api/v1/datasets/$RID/files/raw/q1.csv   (v1 file metadata - now SERVED)
{
    "path": "raw/q1.csv",
    "transactionRid": "",
    "sizeBytes": 19,
    "updatedTime": "2026-09-11T04:17:14.350Z"
}

$ curl '.../api/v1/datasets/$RID/files?branchId=experiment'   (unservable scoping refused)
{"errorCode":"INVALID_ARGUMENT","errorName":"InvalidArgument","errorInstanceId":"5f714247-67ca-4074-bc76-9498e37174d0","parameters":{"param":"branchId","reason":"is not supported: file records carry no branch or transaction dimension, so a scoped request cannot be answered from the single copy of a path"},"statusCode":400} [HTTP 400]

$ curl -D - .../api/v2/datasets/$RID/files/raw/q1.csv   (download: contentType lives in the header, not the body)
HTTP/1.1 200 OK
content-type: text/csv
region,amount
EU,10

$ # overwrite the same path one second later -> updatedTime advances
{
    "path": "raw/q1.csv",
    "transactionRid": "",
    "sizeBytes": 22,
    "updatedTime": "2026-09-11T04:17:15.624Z"
}

$ curl .../api/v1/datasets/$RID/files/raw/q1.csv   (v1 reflects the new write time)
{
    "path": "raw/q1.csv",
    "transactionRid": "",
    "sizeBytes": 22,
    "updatedTime": "2026-09-11T04:17:15.624Z"
}

Evidence: Postgres: 42703 overwrite failure before migration 010, write time recorded after

# Real Postgres 16 in docker; schema from db/migrations 001-009, i.e. dataset_files without updated_at

$ node <PgFileStore: write raw/q1.csv, overwrite it ~1s later, read it back>
error: column "updated_at" of relation "dataset_files" does not exist
  code: '42703',

$ psql -f db/migrations/010_dataset_file_updated_at.sql
applied

$ node <the same script again>
first write -> {"path":"raw/q1.csv","size":7,"updatedTime":"2026-09-11T04:19:19.848Z"}
overwrite   -> {"path":"raw/q1.csv","size":11,"updatedTime":"2026-09-11T04:19:20.953Z"}
getFile     -> {"path":"raw/q1.csv","size":"11","updatedTime":"2026-09-11T04:19:20.953Z"}

$ psql -c 'SELECT path, created_at, updated_at, updated_at > created_at AS write_time_advanced FROM dataset_files'
    path    |          created_at           |          updated_at           | write_time_advanced 
------------+-------------------------------+-------------------------------+---------------------
 raw/q1.csv | 2026-09-11 04:19:19.848817+00 | 2026-09-11 04:19:20.953841+00 | t
(1 row)

Evidence: v1 file routes now served with the full File model (excerpt)

$ curl .../api/v1/datasets/$RID/files (v1 list - now SERVED, was unserved)
{
"data": [
{
"path": "raw/q1.csv",
"transactionRid": "",
"sizeBytes": 19,
"updatedTime": "2026-09-11T04:17:14.350Z"
}
]
}

$ curl '.../api/v1/datasets/$RID/files?branchId=experiment'
{"errorCode":"INVALID_ARGUMENT","parameters":{"param":"branchId","reason":"is not supported: file records carry no branch or transaction dimension..."},"statusCode":400} [HTTP 400]

$ # overwrite the same path one second later -> updatedTime advances
{
"path": "raw/q1.csv",
"sizeBytes": 22,
"updatedTime": "2026-09-11T04:17:15.624Z"
}

- Outcome: ⚠️ 1 info across 1 run (3m59s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos

• ⚠️ services/svc-datasets/tests/pg-file-write-time.test.ts:121 - The schema-source test asserts only a loose regex over raw SQL file text (/dataset_files[\s\S]*updated_at/), which is the source-content-only anti-pattern and is additionally too weak to prove anything: [\s\S]* spans the whole file, so it would pass if updated_at belonged to any later, unrelated table, or if it appeared only inside a comment. Replace with an assertion that has semantic meaning (e.g. apply both schema sources to a real/parsed schema and assert dataset_files has an updated_at column), or drop these two cases and rely on the SQL-statement and route-level behavior tests.
• ℹ️ services/svc-datasets/src/store/pg-file-store.ts:26 - toIsoInstant throws a RangeError (a 500) if updated_at ever comes back unparseable; a stricter guard or an explicit error would degrade more clearly. Low likelihood given the column is NOT NULL TIMESTAMPTZ.
• ℹ️ services/svc-datasets/src/routes/v2/files.ts:49 - The v2 File payload now drops size and contentType (renamed/removed), which is a wire-shape breaking change for any existing v2 client. Nothing in this repo consumes those fields and the change is explicitly justified by the Foundry model in AGENTS.md/GUIDE.md, so noting only.

🔧 Fix: assert parsed schema model instead of raw SQL regex
3 issues (1 error, 2 warnings) still open:

• 🚨 tests/integration/dataset-lifecycle.test.ts:99 - The v2 upload response no longer carries contentType or size (serializeFile now emits path/transactionRid/sizeBytes/updatedTime), but this integration test still asserts body.contentType === "text/csv" and body.size === byteLength. It will fail on pnpm run test:integration, which is a separate command from the unit suite and so was not covered by the unit tests added in this change. Update the assertions to the new File shape (sizeBytes, plus a parseable updatedTime, and drop contentType).
• ⚠️ packages/api-types/src/datasets.ts:86 - DatasetFile in the shared api-types package still declares { path, size, contentType, transactionRid }, which is now a shape no route emits. Any consumer typed against it reads size/contentType as undefined at runtime with no type error, and ListFilesResponse inherits the same stale shape. Align it with the served model (path, transactionRid, sizeBytes, updatedTime) and update the type-level assertion at packages/api-types/tests/api-types.test.ts:205.
• ⚠️ scripts/migrate.sql:412 - The new comment claims the ALTER statements added here also fix "an existing database created before updated_at existed". They cannot: docker-compose.yml mounts this file into /docker-entrypoint-initdb.d, which Postgres executes only when the data directory is empty, so an existing Compose volume never runs these statements and its dataset_files still lacks the column (the 42703 overwrite failure remains reachable there until pnpm db:migrate is run). The statements themselves are harmless and idempotent on a fresh init; the comment is what is wrong - correct it to say the ALTERs keep this file in step with db/migrations/010 and that an already-initialised volume must be migrated with pnpm db:migrate.

🔧 Fix: align DatasetFile type, integration test, and migrate.sql comment
2 issues (1 warning, 1 info) still open:

• ⚠️ conjure/dataset-service.yml:83 - DatasetFile in the conjure IDL still declares {path, size: integer, contentType: string, transactionRid} and has no updatedTime. This file is the declared source of the shape that was changed: packages/api-types/src/datasets.ts opens with "Derived from conjure/dataset-service.yml", and packages/sdk/sdk-cli/src/commands/generate.ts compiles conjure/ into generated SDK clients. The fix round updated the derivative (api-types) but not the source, so the two now disagree, and anyone running sdk-cli generate produces a client whose DatasetFile.size/.contentType are always undefined at runtime and which cannot see updatedTime at all - the exact class of silent wrong-shape defect this change set out to close. Update the DatasetFile definition (and its docs) to path, transactionRid, sizeBytes: integer, updatedTime: datetime to match what both API versions now serve.
• ℹ️ services/svc-datasets/src/routes/v1/files.ts:58 - rejectUnservableScoping treats an empty-string value (?branchId=) as absent and silently serves the unscoped answer, where a non-empty value is a 400. That is a defensible reading (an empty scope asks for nothing), and the Foundry clients drop None parameters rather than sending empties, so no real caller hits it. Noting only.

🔧 Fix: update conjure DatasetFile to the served File shape
2 infos still open:

• ℹ️ scripts/migrate.sql:412 - The four ALTER/UPDATE statements added after CREATE TABLE IF NOT EXISTS dataset_files are inert on every path they can actually execute: the CREATE above already declares updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), and the file's own (now correct) comment states Postgres runs /docker-entrypoint-initdb.d only on an empty data directory, so an existing volume never reaches them. They are harmless but permanently dead, and services/svc-datasets/tests/pg-file-write-time.test.ts now asserts the compose source contains exactly one updated_at backfill, which pins them in place. Removing them (and relaxing that assertion to the migrations source only) would be simpler; keeping them as deliberate drift-parity with db/migrations/010 is also defensible, which is why this is the author's call rather than an auto-fix.
• ℹ️ services/svc-datasets/src/routes/v2/files.ts:49 - Re-confirming after the fix rounds: v2 upload/list/metadata responses now emit sizeBytes/updatedTime and no longer emit size/contentType. I grepped apps/, packages/ and services/ - no in-repo consumer reads either field, and conjure, api-types and both route layers now agree field for field, so the rename is consistent end to end. Noted only (the equivalent finding was already reviewed and accepted in round 1).

🔧 Fix: drop unreachable updated_at ALTERs from compose schema
2 infos still open:

• ℹ️ services/svc-datasets/tests/pg-file-write-time.test.ts:227 - The overwrite guard asserts the emitted SQL text matches /DO UPDATE SET[\s\S]*updated_at = NOW()/ against a FakePool. It is an assertion about the statement OpenFoundry sends (a serialized protocol, so not the source-grep anti-pattern), but it cannot show that a second putFile to the same path actually advances updatedTime on Postgres - the canned row is fixed. The behavioral proof exists only for the in-memory store (v1-api.test.ts 'moves the write time when an existing path is overwritten'). Acceptable without a live Postgres in unit tests; noting the residual gap.
• ℹ️ services/svc-datasets/src/routes/v1/files.ts:150 - rejectUnservableScoping runs before datasetStore.getDataset, so a request for a nonexistent dataset carrying ?branchId=x answers 400 InvalidArgument rather than 404. Foundry validates arguments before resource existence too, so this ordering is defensible and no caller is misled; noting only.

⚠️ **Test** - 1 info

• ℹ️ services/svc-datasets/src/store/pg-file-store.ts - Against Postgres, PgFileStore.getFile returns size as a string ("11"), because dataset_files.size is bigint and node-pg parses bigint to string. Both toV1File and serializeFile pass it straight through, so in DATABASE_URL mode a read of file metadata emits "sizeBytes": "11" instead of a number, diverging from the Foundry model this change is otherwise aligning. Pre-existing (the old size field had the same problem) and orthogonal to updatedTime, so not fixed here; noting only. Observed in the pg evidence transcript: getFile -> {"size":"11"} where the putFile return is {"size":11}.
• npx vitest run --root services/svc-datasets (69 tests: v1-api, datasets-service, pg-file-write-time, list-files-query-params)
• npx vitest run --root packages/api-types
• npx vitest run --config tests/vitest.config.ts tests/integration/dataset-lifecycle.test.ts
• Manual E2E: started the real service (node services/svc-datasets/dist/index.js, PORT=8385) and drove v1/v2 file upload, list, metadata, download-header and overwrite via curl
• Manual E2E: Postgres 16 in docker, applied db/migrations/001-009, ran PgFileStore putFile/putFile/getFile (reproduced 42703), applied db/migrations/010_dataset_file_updated_at.sql, re-ran, then queried dataset_files for created_at vs updated_at

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.