SPIKE: YAML-LD manifests in place of the authoritative Turtle

[ericprud]

Beside each */manifest.ttl, a manifest.yaml that says the same thing:

• bin/ttl2yamlld.js writes it from the Turtle heuristically -- a transliteration of the token stream (a comment-keeping lexer, a parser for the shape a manifest has, an emitter driven by the context), not a parse to a graph and back -- so every comment comes across, where it was. The mf:entries collection and the definitions become one list of the tests themselves, in the collection's order.
• manifest-context.jsonld is the context under which that YAML is, as JSON-LD, the same RDF graph as the Turtle. context.jsonld is not: it maps schema, data, shape and focus into sx: where the Turtle says sht:, and has no term for trait.
• bin/yaml2jsonld.js writes from the YAML the manifest.jsonld that bin/genJSON.js writes from the Turtle, byte for byte, for all five manifests.

shex.js (branch manifest-refactor) holds the evidence: packages/shex-manifest/test/TestSuiteManifest-spike-test.js checks, per manifest, that the YAML is what the converter writes, carries every comment line, is the Turtle's graph (canonical N-Quads equal), and derives the committed manifest.jsonld; and TEST_yaml_manifest=true runs its validation and parser suites (5708 tests) from the YAML.

js-yaml joins the devDependencies for yaml2jsonld's command line.

vocab: mk_vocab.js maintains a second vocabulary, the ShEx manifest vocabulary

http://www.w3.org/ns/shex-manifest#, from vocab/manifest-vocab.csv: what a manifest of ShEx validations is written in -- Manifest and Entry, entries (an ordered list), name, an entry's schema, data and queryMap with their ...URL (by reference) and ...Label spellings -- with a context that also has comment (rdfs:comment) and node, shape and status aliased to the ShEx vocabulary's own shape-map terms. The test suite's manifests and implementations' example manifests share it.

mk_vocab.js takes --vocab shex|shex-manifest; what was hard-wired to the ShEx vocabulary (prefix, title, description, file names, the page's abstract) is per-vocabulary configuration. The ShEx outputs are byte-identical before and after.

SPIKE: the manifests in the ShEx manifest vocabulary, the legacy Turtle generated from them

manifest.yaml (beside each */manifest.ttl) is no longer the Turtle transliterated: it is the manifest in the format implementations use for their own examples -- the ShEx manifest vocabulary (https://www.w3.org/ns/shex-manifest, vocab/manifest-vocab.csv) plus this suite's terms (manifest-context.jsonld) -- so the legacy expression goes:

• no test is typed: a validation test is an entry with a status, conformant or nonconformant; a schema that must be rejected says how (schemaError: syntax | structure); a representation test names its schema's three serializations;
• an entry is named (name), its inputs are its own (no mf:action node), and approval is what mf:status was;
• entries is one list of the tests themselves, in mf:entries order;
• no @base: a reference is relative to the manifest.

bin/manifest-terms.js is the one table of which key is which legacy predicate. bin/ttl2yamlld.js (the migration, comments carried across) reads it one way; bin/yaml2ttl.js, new, and bin/yaml2jsonld.js read it the other way and write the legacy manifest.ttl and manifest.jsonld from the YAML. For all five manifests the generated Turtle's graph is the original's and the generated manifest.jsonld is the committed file byte for byte, so nothing was lost; shex.js's spike test checks it.

Quirks of the Turtle carried so that it can be written back: one test whose mf:name is not its fragment keeps both; mf:proposed/mf:Proposed stay distinct approval values; negativeStructure's mf:comment arcs stay mf:comment.

SPIKE: a blank-node or literal focus is a query map, not a node

node and shape are IRI references. A focus node that is a blank node or a literal (31 validation tests), or a shape that is a blank node, is now written as a ShapeMap association -- queryMap: "_:abcd@http://a.example/S1" -- because that is text. As a JSON-LD node reference a blank node's label does not survive (a processor relabels it), and the label is what those tests are about; a literal is no reference at all.

bin/manifest-terms.js parses and writes the association; yaml2ttl and yaml2jsonld put sht:focus and sht:shape back from it. The generated Turtle and manifest.jsonld are still the originals, graph for graph and byte for byte.

SPIKE: README says what shex.js checks of the YAML manifests, and what a reader of their graph should know

The claim that shex.js's validate runs validation/manifest.yaml as it runs its own examples was loose: validate runs the entries but does not compare them with their status. What is checked (shex.js, branch manifest-refactor, packages/shex-manifest/test/TestSuiteManifest-spike-test.js): each manifest read as JSON-LD drops nothing; @shexjs/manifest reads all five, plainly and as JSON-LD, into the same entries; and shex.js's examples runner gets the stated status for the 1239 of 1309 validation entries that need only what an example does.

Also notes that an entry is identified by its name, so the 23 test-to-test references are IRIs the YAML's own graph does not describe (the generated Turtle does), and that the contexts are published only once this is.