One identifier label for the citation model - #699
Merged
Conversation
The citation model's generated training data labelled DOIs <idno> and every other identifier <pubnum>. GROBID's corpus has no <idno> label, and CitationSemanticExtractor had no mapping for it, so a DOI predicted by a model trained on generated data became a note: never a SemanticExternalIdentifier, never scored as reference_doi, and not counted towards is_reference_valid. Over the committed corpus that was 5431 of 5446 identifier tokens, and 583 of 589 identifier elements. models/citation/labels.py now states the label set, the identifier label and the labels that legitimately stay notes; the training TEI paths, the JATS sub-field map, the extractor and the benchmark's reference_doi analysis all read from it, and a test fails if they drift apart. Type detection stays at extraction, so no training label depends on a regex having matched - which is what gave those five truncated DOIs a type="DOI" they could not earn. CitationTrainingTeiParser maps any idno element back to the identifier label rather than enumerating the types get_post_processed_xml_root can write, so a sixth type cannot make the existing corpus unparseable. SimpleModelSemanticExtractor warns once per label when a model emits something outside its expected note set; only citation opts in. Over the same corpus, typed DOI identifiers rise from 38 - all of them <web>-derived - to 616 across 3304 references.
TEICitationSaxParser maps both idno and pubnum to <pubnum> and leaves the
type unused ("TBD: keep the idno type for further exploitation");
CitationParser types the prediction afterwards through
BiblioItem.checkIdentifier(). Its training TEI spells the kinds
type="arXiv" and type="PMC" and covers ISSN and ISBN, none of which
SemanticExternalIdentifierTypes has - so an enumerated type list would
raise on GROBID's own corpus.
One label for every identifier leaves nothing to separate a DOI from a PMID that follows it directly: the JATS label function returns bare labels, so the generator kept writing into the open <idno> and produced a single element whose value carried both identifiers, typed as a DOI. The previous split avoided that only because the two kinds had different element paths. Emitting B-<pubnum> at a sub-field change within a reference gives each identifier its own element again. Same shape as the reference segmenter's instance transitions. Two identifiers of one kind in a reference stay undistinguished - the annotation carries sub-field kind and reference instance, not sub-field instance. The delft conversion still merges the two elements into one training span, because the training TEI parser rebuilds B-/I- from label changes rather than element boundaries. That is a shared-parser defect, recorded separately, and it has to land before the artifact is regenerated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of https://github.com/eLifePathways/ScienceBeam2.0/issues/145
Why this matters: a citation model trained on our generated data labels DOIs
<idno>, and theextractor has no mapping for that label — so every DOI it predicts becomes an anonymous note. It never
becomes a
SemanticExternalIdentifier, never reaches the TEI asreference_doi, and doesn't counttowards
is_reference_valid. Over the committed corpus that was 583 of 589 identifier elements(5431 of 5446 tokens): the field the benchmark scores was being thrown away after training had already
paid for it.
Measured over the same corpus, before and after, across 3304 references:
<web>-derived, since<idno>produced noneThis is GROBID's convention, not a new one.
TEICitationSaxParsermaps bothidnoandpubnumelements to a single
<pubnum>label and leaves the type unused (// TBD: keep the idno type for further exploitation);CitationParsertypes the prediction afterwards viaBiblioItem.checkIdentifier(). DOI and PMID stay separate where it counts — as typed values,<idno type="…">in the output TEI, and separately scorable fields. Dropping the extra label alsobrings the generated label set to GROBID's 19, so the two corpora can finally be trained together
without carrying two conventions.
What changes
models/citation/labels.pystates the label set, the identifier label and the labels thatlegitimately stay notes. The training TEI paths, the JATS sub-field map, the extractor and the
benchmark's
reference_doianalysis all read from it, and a test fails if they drift apart.idnoelement back to the identifier label instead ofenumerating the five types our detection can produce. GROBID's own training TEI spells them
type="arXiv"andtype="PMC"and includes ISSN and ISBN — an enumerated list would raise on itscorpus.
notes and warns once for anything else, and a test asserts every label in the set is either mapped or
declared.
followed by a PMID is no longer written as one
<idno>whose value carries both — that produced acorrupted DOI, which scores worse than a missing one.
No serving change
GROBID's shipped models never emit
<idno>, so extraction behaviour for current models is unchanged.AbstractTrainingTeiParserhas one production caller,generate_delft_data— nothing in the inferencepath.
Verified
122 citation tests including a DOI/arXiv/PMID/PMCID round trip through generation → parsing →
extraction, and a coverage test over the whole label set; 948 tests across
tests/models,tests/training,tests/document; flake8, pylint and mypy clean. All 88 committed TEI files parsewithout exception, and
<idno>disappears from the generated label set.