Skip to content

Fix #100 - Flesh out the metadata model and add structured field types - #103

Open
jbuckner wants to merge 1 commit into
28-swift-formatfrom
100-metadata-model
Open

Fix #100 - Flesh out the metadata model and add structured field types#103
jbuckner wants to merge 1 commit into
28-swift-formatfrom
100-metadata-model

Conversation

@jbuckner

Copy link
Copy Markdown
Owner

Closes #100.

Stacked on #101, which has to merge first.

ItemMetadata goes from 62 fields to 154, and the fields that have real structure inside them stop being strings.

Breaking. This is 2.0.

Enums

Fields with a documented value set now use IAEnum. The Archive adds values without notice, so an unrecognized one is kept rather than dropped:

metadata.mediatype?.value == .etree     // compare against the case
metadata.mediatype?.value == "etree"    // or the wire string, so old call sites still compile
metadata.mediatype?.value?.known        // MediaType?, nil if the Archive sent something new
metadata.mediatype?.value?.rawValue     // String, always

Vocabularies: MediaType, FileSource, FileFormat, PageProgression, Sound, Condition, ConditionVisual, BookReaderDefaults, SortBy, CurationState.

One gotcha worth knowing: XCTAssertEqual(x, .etree) does not compile, because it unifies both sides to one type. Use XCTAssertTrue(x == .etree).

New field types

  • IAEpochDate for created, item_last_updated, mtime, viruscheck
  • IAAspectRatio for "16:9". It needs its own type: IATimeInterval reads that as 969 seconds
  • IACuration for [curator]…[/curator][date]…[/date][state]…[/state]
  • IAExternalIdentifier for urn:isbn:…, splitting on the first two colons so the value keeps its own

They encode back to the string they were parsed from, so a cached item stays the shape the Archive serves.

IABool now accepts yes/no and 1/0, which is what closed_captioning and several flags actually send. DateParser handles 20241031063000 and 20151231, so scandate, republisher_date, sponsordate, firstfiledate and lastfiledate stop decoding to nil.

Retyped

mediatype, format, File.source, File.format to enums. created, File.mtime, Item.itemLastUpdated to IAEpochDate. curation to IACuration.

runtime stays a String, which is a change from the plan. Uploaders type it by hand and the Archive doesn't normalize it, so a sample of real values has 245 Mins., 2hr, 07min, 3'19'58 and 60min sitting next to 59:32 and 2:29:30. Typing it as an interval would have silently nulled about half of them. There's a metadata.runtimeInterval for the subset that is a duration, and a file's length is the one to trust.

Hyphenated keys

ItemMetadata and File get explicit CodingKeys. The decoder runs .convertFromSnakeCase, which only rewrites underscores, so identifier-access, page-progression, sort-by, external-identifier and the rest reach the key lookup unchanged and never match a synthesized key. CodingKeys is all-or-nothing, hence the full list.

Tests

The ones that matter decode four unedited /metadata responses (etree, texts, movies, collection) and check values against the raw JSON. A hand-written fixture can't catch the failure that matters here: a field whose name doesn't line up with its wire key decodes fine, it just comes back nil forever. One test walks every key in those payloads and reports anything the model drops; it's currently empty.

Two bugs the tests caught while writing this:

  • Adding the compact date formats made the empty string parse to 2000-01-01, because an all-numeric DateFormatter format matches "". DateParser now rejects blank input.
  • IAEpochDate originally used Date as its field type, so ModelField.encode handed it to the encoder's dateEncodingStrategy. Under the default that round-tripped 31 years off; under .iso8601 it came back nil. EpochDate now encodes the number itself, and there's a test across all three strategies.

QA

swift test - 169 tests pass
swift format lint --strict - exits 0
xcodebuild build -scheme InternetArchiveKit -destination 'generic/platform=iOS Simulator' - succeeds
✅ Decoded live responses for sci2007-07-28.Schoeps, goody, FRANCE24_20241031_063000 and etree, asserting the new fields land where the raw JSON has them

Unrelated: testScrapeTotal and testScrapeTotalAsyncThrows hit live archive.org and fail when the index moves. Reproduces on a clean main. Filed as #102.

🤖 Generated with Claude Code

https://claude.ai/code/session_014yqNSBpEowza5teukE2JJL

ItemMetadata goes from 62 fields to 154, covering the public schema plus the
internal fields that actually come back from a GET. Fields with a documented
value set become IAEnum, which keeps an unrecognized value as a raw string
rather than dropping it, since the Archive extends its vocabularies without
notice.

New field types for the values that carry structure inside a string:
IAEpochDate, IAAspectRatio, IACuration and IAExternalIdentifier. They encode
back to the string they were parsed from so a cached item stays the shape the
Archive serves. IABool now takes yes/no and 1/0, and DateParser handles the
compact yyyyMMddHHmmss and yyyyMMdd stamps that scandate and friends use, so
those stop decoding to nil.

ItemMetadata and File need explicit CodingKeys: the decoder's
convertFromSnakeCase only rewrites underscores, so hyphenated keys like
identifier-access reach the lookup unchanged and never match a synthesized key.

runtime stays a String. Uploaders type it by hand and the Archive doesn't
normalize it, so real values include "245 Mins." and "2hr, 07min" alongside
"59:32". Typing it as an interval would have nulled about half of them.
runtimeInterval parses the subset that is a duration.

Two things the tests caught. The compact date formats made an empty string
parse to 2000-01-01, because an all-numeric DateFormatter format matches "",
so DateParser now rejects blank input. And IAEpochDate first used Date as its
field type, which let ModelField.encode hand it to the encoder's
dateEncodingStrategy: the default round-tripped 31 years off and .iso8601 came
back nil. EpochDate encodes the number itself now.

Breaking, so this is 2.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014yqNSBpEowza5teukE2JJL
@jbuckner

Copy link
Copy Markdown
Owner Author

CI red on Build and Test is not from this branch: testScrapeTotal and testScrapeTotalAsyncThrows hit live archive.org, and the Scrape API is currently ignoring the q parameter (every query returns the same 9294 results, total_only=true returns 0). Reproduces on a clean main. Details in #102.

All 25 new tests pass, along with the rest of the offline suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flesh out the metadata model and add structured field types

1 participant