Fix #100 - Flesh out the metadata model and add structured field types - #103
Open
jbuckner wants to merge 1 commit into
Open
Fix #100 - Flesh out the metadata model and add structured field types#103jbuckner wants to merge 1 commit into
jbuckner wants to merge 1 commit into
Conversation
ItemMetadata goes from 62 fields to 154, covering the public schema plus the internal fields that actually come back from a GET. Fields with a documented value set become IAEnum, which keeps an unrecognized value as a raw string rather than dropping it, since the Archive extends its vocabularies without notice. New field types for the values that carry structure inside a string: IAEpochDate, IAAspectRatio, IACuration and IAExternalIdentifier. They encode back to the string they were parsed from so a cached item stays the shape the Archive serves. IABool now takes yes/no and 1/0, and DateParser handles the compact yyyyMMddHHmmss and yyyyMMdd stamps that scandate and friends use, so those stop decoding to nil. ItemMetadata and File need explicit CodingKeys: the decoder's convertFromSnakeCase only rewrites underscores, so hyphenated keys like identifier-access reach the lookup unchanged and never match a synthesized key. runtime stays a String. Uploaders type it by hand and the Archive doesn't normalize it, so real values include "245 Mins." and "2hr, 07min" alongside "59:32". Typing it as an interval would have nulled about half of them. runtimeInterval parses the subset that is a duration. Two things the tests caught. The compact date formats made an empty string parse to 2000-01-01, because an all-numeric DateFormatter format matches "", so DateParser now rejects blank input. And IAEpochDate first used Date as its field type, which let ModelField.encode hand it to the encoder's dateEncodingStrategy: the default round-tripped 31 years off and .iso8601 came back nil. EpochDate encodes the number itself now. Breaking, so this is 2.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yqNSBpEowza5teukE2JJL
Owner
Author
|
CI red on All 25 new tests pass, along with the rest of the offline suite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100.
Stacked on #101, which has to merge first.
ItemMetadatagoes from 62 fields to 154, and the fields that have real structure inside them stop being strings.Breaking. This is 2.0.
Enums
Fields with a documented value set now use
IAEnum. The Archive adds values without notice, so an unrecognized one is kept rather than dropped:Vocabularies:
MediaType,FileSource,FileFormat,PageProgression,Sound,Condition,ConditionVisual,BookReaderDefaults,SortBy,CurationState.One gotcha worth knowing:
XCTAssertEqual(x, .etree)does not compile, because it unifies both sides to one type. UseXCTAssertTrue(x == .etree).New field types
IAEpochDateforcreated,item_last_updated,mtime,viruscheckIAAspectRatiofor"16:9". It needs its own type:IATimeIntervalreads that as 969 secondsIACurationfor[curator]…[/curator][date]…[/date][state]…[/state]IAExternalIdentifierforurn:isbn:…, splitting on the first two colons so the value keeps its ownThey encode back to the string they were parsed from, so a cached item stays the shape the Archive serves.
IABoolnow acceptsyes/noand1/0, which is whatclosed_captioningand several flags actually send.DateParserhandles20241031063000and20151231, soscandate,republisher_date,sponsordate,firstfiledateandlastfiledatestop decoding to nil.Retyped
mediatype,format,File.source,File.formatto enums.created,File.mtime,Item.itemLastUpdatedtoIAEpochDate.curationtoIACuration.runtimestays a String, which is a change from the plan. Uploaders type it by hand and the Archive doesn't normalize it, so a sample of real values has245 Mins.,2hr, 07min,3'19'58and60minsitting next to59:32and2:29:30. Typing it as an interval would have silently nulled about half of them. There's ametadata.runtimeIntervalfor the subset that is a duration, and a file'slengthis the one to trust.Hyphenated keys
ItemMetadataandFileget explicitCodingKeys. The decoder runs.convertFromSnakeCase, which only rewrites underscores, soidentifier-access,page-progression,sort-by,external-identifierand the rest reach the key lookup unchanged and never match a synthesized key.CodingKeysis all-or-nothing, hence the full list.Tests
The ones that matter decode four unedited
/metadataresponses (etree, texts, movies, collection) and check values against the raw JSON. A hand-written fixture can't catch the failure that matters here: a field whose name doesn't line up with its wire key decodes fine, it just comes back nil forever. One test walks every key in those payloads and reports anything the model drops; it's currently empty.Two bugs the tests caught while writing this:
DateFormatterformat matches"".DateParsernow rejects blank input.IAEpochDateoriginally usedDateas its field type, soModelField.encodehanded it to the encoder'sdateEncodingStrategy. Under the default that round-tripped 31 years off; under.iso8601it came back nil.EpochDatenow encodes the number itself, and there's a test across all three strategies.QA
✅
swift test- 169 tests pass✅
swift format lint --strict- exits 0✅
xcodebuild build -scheme InternetArchiveKit -destination 'generic/platform=iOS Simulator'- succeeds✅ Decoded live responses for
sci2007-07-28.Schoeps,goody,FRANCE24_20241031_063000andetree, asserting the new fields land where the raw JSON has themUnrelated:
testScrapeTotalandtestScrapeTotalAsyncThrowshit live archive.org and fail when the index moves. Reproduces on a cleanmain. Filed as #102.🤖 Generated with Claude Code
https://claude.ai/code/session_014yqNSBpEowza5teukE2JJL