Skip to content

fix(parser): ignore language proficiency rows #865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
qtjg wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(parser): ignore language proficiency rows #865

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1b083e3
fix(parser): ignore language proficiency rows
qtjg Aug 18, 2026
616c09c
test(parser): cover language proficiency rows
qtjg Aug 18, 2026
b20c622
test(fixtures): update language proficiency expectations
qtjg Aug 18, 2026
8eaae56
test(fixtures): refresh Word skill snapshot
qtjg Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions src/lib/heuristics/extract/skills.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -835,6 +835,39 @@ describe("extractSkills — category capture (#473)", () => {
]);
});

it("drops language proficiency rows without dropping language lists", () => {
const section = skillsLines([
[...BULLET_RUNS, { x: 74, str: "Languages: Python, Go, TypeScript", w: 180 }],
[...BULLET_RUNS, { x: 74, str: "Language: Fluent in Spanish", w: 160 }],
[...BULLET_RUNS, { x: 74, str: "Languages: Spanish, French, Mandarin", w: 180 }],
[
...BULLET_RUNS,
{ x: 74, str: "Certifications: Certified in AWS Solutions Architecture", w: 220 },
],
]);
const { value, categories } = extractSkills(section);

expect(value).toEqual([
"Python",
"Go",
"TypeScript",
"Spanish",
"French",
"Mandarin",
"Certified in AWS Solutions Architecture",
]);
expect(categories).toEqual([
{ label: "Languages", skills: ["Python", "Go", "TypeScript"] },
{ label: "Languages", skills: ["Spanish", "French", "Mandarin"] },
{ label: "Certifications", skills: ["Certified in AWS Solutions Architecture"] },
]);
expect(value).not.toContain("Fluent in Spanish");
expect(categories).not.toContainEqual({
label: "Language",
skills: ["Fluent in Spanish"],
});
});

it("INVARIANT 1: `skills` deep-equals `categories.flatMap((c) => c.skills)`", () => {
// A wrapped Frontend list whose continuation flushed as its own bare cell
// ("… HTML5," ⏎ "CSS3, JavaScript") must fold back into Frontend, not become
Expand Down
32 changes: 30 additions & 2 deletions src/lib/heuristics/extract/skills.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,25 @@ const SUBLABEL_PREFIX_RE = new RegExp(`^(${SUBLABEL_BODY}):\\s*`);
* tore off its body. See the rejoin in `splitColumnCells`. */
const BARE_SUBLABEL_RE = new RegExp(`^${SUBLABEL_BODY}:$`);

/** A Skills sub-label that MAY head a spoken-language row. Deliberately NOT
* added to NON_SKILL_SUBLABEL_RE: on most engineering résumés `Languages:`
* heads the programming-language row, so the label is ambiguous and the body
* must decide whether this is a proficiency statement. */
const LANGUAGE_LABEL_RE = /^languages?$/i;

@s-annam s-annam Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondary. Stricter than its sibling NON_SKILL_SUBLABEL_RE, so #833's defect survives on common variants. All still admitted on this branch:

  • Foreign Languages: Fluent in Spanish → ["Fluent in Spanish"]
  • Spoken Languages: Native German, conversational French → ["Native German","conversational French"]
  • Languages : Fluent in Spanish → ["Fluent in Spanish"]

The third is a capture artifact: SUBLABEL_BODY ([A-Z][A-Za-z &/]+) admits a space, so the capture is "Languages ". NON_SKILL_SUBLABEL_RE carries a deliberate \s*$ for exactly this and matchCellLabel .trim()s the capture — this regex does neither, so Interests : Tennis is dropped while Languages : Fluent in Spanish escapes.

/^(?:foreign\s+|spoken\s+|other\s+)?languages?\s*$/i mirrors the sibling's leading-qualifier tolerance and covers all three.


/** A spoken-language proficiency predication, distinguished from a delimited
* programming-language list by its proficiency wording rather than by the
* label. */
const LANGUAGE_PROFICIENCY_BODY_RE =
/\b(fluent|native|bilingual|conversational|proficient|intermediate|beginner|basic|working\s+proficiency|mother\s+tongue)\b/i;

@s-annam s-annam Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondary. The vocabulary misses the standard scale wordings: \bproficient\b does not match the noun proficiency, and only working\s+proficiency is enumerated. Still admitted:

  • Languages: Full professional proficiency in German
  • Languages: Elementary proficiency in French
  • Languages: JLPT N2 Japanese Proficiency

The last is drawn by two committed fixtures — google-docs/google-docs-skia-proxy-multiline-bullets-coursework.pdf and unknown/student-projects-activities-singlecol.pdf (verified with pdftotext) — and both baselines are untouched by this PR, confirming the row still reaches skills.

Adding proficienc(y|ies) plus elementary / limited closes it. Worth doing here rather than as a follow-up, since the fixtures already exist.


function isLanguageProficiencyCell(cell: string): boolean {
const debulleted = stripBullet(cell);
const match = debulleted.match(SUBLABEL_PREFIX_RE);
if (!match || !LANGUAGE_LABEL_RE.test(match[1])) return false;
return LANGUAGE_PROFICIENCY_BODY_RE.test(debulleted.slice(match[0].length));

@s-annam s-annam Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. This tests the proficiency vocabulary against the whole body, so a delimited programming-language row containing one matching word is dropped entirely — and tokenizeCell drops the cell before the split, so every sibling token on the row dies too.

Measured on this branch vs origin/main:

Input origin/main this branch
Languages: C, C++, Visual Basic ["C","C++","Visual Basic"] []
Languages: Kotlin, Swift, React Native 3 tokens []
Languages: Proficient in Java, Python, Go ["Proficient in Java","Python","Go"] []
Languages: Java, Python (proficient), Go ["Java","Python (proficient)","Go"] []

\bbasic\b catches Visual Basic, \bnative\b catches React Native, \bproficient\b catches the common Proficient in <list> phrasing. The label is effectively deciding, which is what #833 ruled out.

#833 step 1 specifies the body conjunct as "a proficiency predication rather than a delimited list" — the delimited-list half is missing. Requiring every delimited fragment to be a proficiency predication satisfies all four ACs and rejects all four rows above:

const fragments = debulleted
  .slice(match[0].length)
  .split(SKILL_SPLIT_RE)
  .map((f) => f.trim())
  .filter((f) => f !== "");
return fragments.length > 0 && fragments.every((f) => LANGUAGE_PROFICIENCY_BODY_RE.test(f));

(splitRespectingParens is what handles commas — worth reusing the existing splitter.) Please pin the four rows as unit assertions; the new test covers only clean rows.

}

/**
* Tokenizes a single column cell into valid skill tokens and adds them to
* `out`. Drops the cell entirely when it looks like a contact/profile link —
Expand All @@ -317,7 +336,11 @@ function tokenizeCell(cell: string, out: Set<string>): void {
// leading `Label:` prefix and drop the whole cell when it names a
// hobbies/interests list — before the label is stripped and the items split.
const labelMatch = debulleted.match(SUBLABEL_PREFIX_RE);
if (labelMatch && NON_SKILL_SUBLABEL_RE.test(labelMatch[1])) return;
if (
labelMatch &&
(NON_SKILL_SUBLABEL_RE.test(labelMatch[1]) || isLanguageProficiencyCell(debulleted))

@s-annam s-annam Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. Returning here yields zero tokens, so extractSkills continues at skills.ts:680 before matchCellLabel runs and no Languages category is ever opened. A soft-wrapped continuation then falls into the "bare cell extends the last category" branch at skills.ts:691 and is filed under the previous label.

Frameworks: React, Vue / Languages: English (native), Spanish (fluent), / French:

  • origin/main → Frameworks:[React,Vue], Languages:[English (native),Spanish (fluent),French]
  • this branch → Frameworks:[React,Vue,French]

French becomes a Framework — silently wrong data rather than missing data, on a surface that feeds JD-match and job-search keywords.

Note the all-fragments fix for the other blocker does not resolve this: English (native), Spanish (fluent), is still legitimately dropped. Either register the category label before the zero-token continue, or drop the offending fragments rather than the whole cell — the latter makes both blockers fall out of one change.

)
return;
const clean = debulleted.replace(SUBLABEL_PREFIX_RE, "");
// A whole cell that is a profile link ("github.com/janesmith") must be
// dropped before splitting — a path slash would otherwise leave the path
Expand Down Expand Up @@ -616,7 +639,12 @@ function upcomingContinuationTexts(lineCells: string[][], from: number): string[
*/
function matchCellLabel(cell: string): string | undefined {
const m = stripBullet(cell).match(SUBLABEL_PREFIX_RE);
if (!m || NON_SKILL_SUBLABEL_RE.test(m[1])) return undefined;
if (
!m ||
NON_SKILL_SUBLABEL_RE.test(m[1]) ||
isLanguageProficiencyCell(cell)

@s-annam s-annam Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Unreachable: extractSkills:680 continues on a zero-token cell and tokenizeCell already returned early for exactly these, so matchCellLabel — which has no other callers — can never be handed one. Fine to keep as defensive alignment with the NON_SKILL_SUBLABEL_RE mirror, but the docblock just above (skills.ts:632) explains only the Interests/Hobbies mirror and should name this guard too.

Related: this passes the raw cell while skills.ts:341 passes the already-stripBulleted text. Correct either way since stripBullet is idempotent, but taking the captured label and body as parameters would make the "two decisions stay aligned" contract structural rather than by-convention.

)
return undefined;
return m[1].trim();
}

Expand Down
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"phoneIsValid",
"skills"
],
"skillsCount": 12,
"skillsCount": 11,
"experienceCount": 2,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "regex",
"pageCount": 1,
"rawCharCount": 1428,
"extractedCharCount": 1147,
"extractedCharCount": 1130,
"sections": [
{
"name": "profile",
Expand All @@ -101,7 +101,7 @@
"hasSummary": false,
"experienceCount": 2,
"educationCount": 1,
"skillsCount": 12
"skillsCount": 11
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down
5 changes: 0 additions & 5 deletions tests/fixtures/pdfs/google-docs/google-docs-skia-proxy-role-first-experience.truth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,6 @@
"issue": null,
"status": "unfiled",
"note": "Role 2's employer line reads “Multicultural Engineering Program – State Polytechnic University”; `company` comes back as just the university — the program half is not lost, the parser puts it on `team`, but `experience.company` scores the `company` field alone. The identical shape is measured on unknown/single-column-title-below-anchor. Possibly a defensible org/team split rather than a defect — recorded rather than assumed, because ground truth's job is to state what the page says and let a human adjudicate."
},
"skills": {
"issue": 833,
"status": "open",
"note": "“Fluent in Spanish” is admitted as a skill from the “Language:” row; the Programming Languages row is now correct after #832."
}
}
}
6 changes: 3 additions & 3 deletions tests/fixtures/pdfs/word/openresume-laverne-word-quartz.expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"skills",
"summary"
],
"skillsCount": 8,
"skillsCount": 7,
"experienceCount": 3,
"educationCount": 1,
"projectsCount": 0,
Expand Down Expand Up @@ -74,7 +74,7 @@
"sectionSource": "regex",
"pageCount": 1,
"rawCharCount": 1336,
"extractedCharCount": 856,
"extractedCharCount": 839,
"sections": [
{
"name": "profile",
Expand Down Expand Up @@ -105,7 +105,7 @@
"hasSummary": true,
"experienceCount": 3,
"educationCount": 1,
"skillsCount": 8
"skillsCount": 7
},
"linkAnnotationCount": 0,
"disagreements": []
Expand Down
Loading