Skip to content

fix(from_file): convert registered vocabularies in the browser; keep quoted labels #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jstet merged 1 commit into from
Sep 24, 2026
Merged

fix(from_file): convert registered vocabularies in the browser; keep quoted labels #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,10 @@ Kobo or LimeSurvey is the platform's own export step (see
Node-only, and never imported by `src/index.ts`: `src/cli.ts` (the
`formtransform` binary), `src/cliShared.ts` (its argument parsing and file
I/O), `src/fileChoices.ts` (loads `select_*_from_file` CSVs from disk) and
`src/generateFixtures.ts`.
`src/generateFixtures.ts`. The registered vocabularies don't need
`fileChoices.ts`: codegen emits their options to
`src/generated/VocabularyOptions.ts`, and the converter inlines them in the
browser too.

### Code Generation (`codegen/`)

Expand Down
4 changes: 4 additions & 0 deletions codegen/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
generate_question_types,
generate_typescript,
generate_typescript_ddi,
generate_vocabulary_options,
)
from .examples import build_example_artifacts
from .loader import load_registry
Expand Down Expand Up @@ -66,6 +67,9 @@ def main(argv: list[str] | None = None) -> None:
generate_question_types(registry, ts_out / "QuestionTypes.ts")
print(" ✅ src/generated/QuestionTypes.ts (labelled catalogue for consumers)")

generate_vocabulary_options(registry, base_dir / "registry" / "vocab", ts_out / "VocabularyOptions.ts")
print(" ✅ src/generated/VocabularyOptions.ts (registered vocabularies for select_*_from_file)")

generate_schematron(registry, base_dir / "ddi-validation" / "schematron" / "ddi_custom_rules.sch")
print(" ✅ ddi-validation/schematron/ddi_custom_rules.sch (DDI validation rules)")

Expand Down
9 changes: 8 additions & 1 deletion codegen/emit_skill.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,14 @@ def _type_section(d: dict[str, Any], variants: list[dict], base_dir: Path) -> li
aliases = d["xlsform"].get("aliases")
if aliases:
out.append(f"**Accepted aliases:** {', '.join(f'`{a}`' for a in aliases)}\n")
if d["xlsform"].get("requiresListName"):
if d.get("ddi", {}).get("externalCodeList"):
out.append(
"**Requires a registered vocabulary:** write the type cell as "
f"`{ts} <file>.csv` with a file from the registered-vocabularies table in "
"[xlsform-syntax.md](xlsform-syntax.md). The options come from that "
"vocabulary; add no rows to the `choices` sheet.\n"
)
elif d["xlsform"].get("requiresListName"):
out.append(
"**Requires a choice list:** write the type cell as "
f"`{ts} <list_name>` and define the options on the `choices` sheet under "
Expand Down
41 changes: 41 additions & 0 deletions codegen/emit_ts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Generators that emit TypeScript / JSON artifacts consumed by the in-repo transformer."""

import csv
import json
from collections import OrderedDict
from pathlib import Path
Expand Down Expand Up @@ -353,6 +354,46 @@ def generate_conventions_ts(registry: dict[str, Any], output: Path):
)


def generate_vocabulary_options(registry: dict[str, Any], vocab_dir: Path, output: Path):
"""Emit every registered vocabulary's options as a TypeScript module.

`select_*_from_file` inlines the vocabulary's options into LimeSurvey TSV.
The browser has no filesystem to read `registry/vocab/<file>.csv` from, so
the registered vocabularies ship as data (formtransform#24). Parsed with the
`csv` module: labels such as "Korea, Republic of" are quoted.
"""
lines = [
"/**",
" * GENERATED CODE - DO NOT EDIT",
" * Source: registry/root.jsonld + registry/vocab/*.csv",
" * Generated by: codegen",
" */",
"",
"/** Registered vocabulary options as `[code, label]`, keyed by xlsform filename. */",
"export const VOCABULARY_OPTIONS: Record<string, readonly (readonly [string, string])[]> = {",
]
vocabs = sorted(
(
d["xlsformFilename"]
for d in registry.values()
if d.get("@type") == "Vocabulary" and d.get("xlsformFilename")
),
)
for filename in vocabs:
with open(vocab_dir / filename, encoding="utf-8-sig", newline="") as f:
rows = list(csv.DictReader(f))
lines.append(f" {_ts_key(filename)}: [")
for row in rows:
code = (row.get("code") or "").strip()
if code:
label = (row.get("label") or "").strip()
lines.append(f" [{json.dumps(code, ensure_ascii=False)}, {json.dumps(label, ensure_ascii=False)}],")
lines.append(" ],")
lines.append("};")
lines.append("")
output.write_text("\n".join(lines))


def _entry_lines(
key: str,
*,
Expand Down
5 changes: 2 additions & 3 deletions codegen/loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,8 @@ def _validate_vocab_files(registry: dict[str, Any], vocab_dir: Path) -> None:
Vocabularies are open-ended (see convention:externalCodeList's
``vocabularyDeclaration``): declaring one is the only step, so this is the
check that keeps a declaration honest. Without it a typo'd or missing CSV
surfaces much later — the forward converter silently falls back to erroring
on an "unimplemented" from_file type when `resolveFileChoices` finds no
options to inline.
surfaces much later, as a missing file when `VocabularyOptions.ts` is
generated.
"""
errors: list[str] = []
for at_id, data in registry.items():
Expand Down
28 changes: 14 additions & 14 deletions registry/entities/select_multiple_long_list/tsv.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ SQ BZ Belize en
SQ BJ Benin en
SQ BM Bermuda en
SQ BT Bhutan en
SQ BO &quot;Bolivia en
SQ BQ &quot;Bonaire en
SQ BO Bolivia, Plurinational State of en
SQ BQ Bonaire, Sint Eustatius and Saba en
SQ BA Bosnia and Herzegovina en
SQ BW Botswana en
SQ BV Bouvet Island en
Expand All @@ -56,7 +56,7 @@ SQ CC Cocos (Keeling) Islands en
SQ CO Colombia en
SQ KM Comoros en
SQ CG Congo en
SQ CD &quot;Congo en
SQ CD Congo, Democratic Republic of the en
SQ CK Cook Islands en
SQ CR Costa Rica en
SQ CI Côte d&#39;Ivoire en
Expand Down Expand Up @@ -110,7 +110,7 @@ SQ HU Hungary en
SQ IS Iceland en
SQ IN India en
SQ ID Indonesia en
SQ IR &quot;Iran en
SQ IR Iran, Islamic Republic of en
SQ IQ Iraq en
SQ IE Ireland en
SQ IM Isle of Man en
Expand All @@ -123,8 +123,8 @@ SQ JO Jordan en
SQ KZ Kazakhstan en
SQ KE Kenya en
SQ KI Kiribati en
SQ KP &quot;Korea en
SQ KR &quot;Korea en
SQ KP Korea, Democratic People&#39;s Republic of en
SQ KR Korea, Republic of en
SQ KW Kuwait en
SQ KG Kyrgyzstan en
SQ LA Lao People&#39;s Democratic Republic en
Expand All @@ -149,8 +149,8 @@ SQ MR Mauritania en
SQ MU Mauritius en
SQ YT Mayotte en
SQ MX Mexico en
SQ FM &quot;Micronesia en
SQ MD &quot;Moldova en
SQ FM Micronesia, Federated States of en
SQ MD Moldova, Republic of en
SQ MC Monaco en
SQ MN Mongolia en
SQ ME Montenegro en
Expand All @@ -161,7 +161,7 @@ SQ MM Myanmar en
SQ NA Namibia en
SQ NR Nauru en
SQ NP Nepal en
SQ NL &quot;Netherlands en
SQ NL Netherlands, Kingdom of the en
SQ NC New Caledonia en
SQ NZ New Zealand en
SQ NI Nicaragua en
Expand All @@ -175,7 +175,7 @@ SQ NO Norway en
SQ OM Oman en
SQ PK Pakistan en
SQ PW Palau en
SQ PS &quot;Palestine en
SQ PS Palestine, State of en
SQ PA Panama en
SQ PG Papua New Guinea en
SQ PY Paraguay en
Expand All @@ -191,7 +191,7 @@ SQ RO Romania en
SQ RU Russian Federation en
SQ RW Rwanda en
SQ BL Saint Barthélemy en
SQ SH &quot;Saint Helena en
SQ SH Saint Helena, Ascension and Tristan da Cunha en
SQ KN Saint Kitts and Nevis en
SQ LC Saint Lucia en
SQ MF Saint Martin (French part) en
Expand Down Expand Up @@ -222,9 +222,9 @@ SQ SJ Svalbard and Jan Mayen en
SQ SE Sweden en
SQ CH Switzerland en
SQ SY Syrian Arab Republic en
SQ TW &quot;Taiwan en
SQ TW Taiwan, Province of China en
SQ TJ Tajikistan en
SQ TZ &quot;Tanzania en
SQ TZ Tanzania, United Republic of en
SQ TH Thailand en
SQ TL Timor-Leste en
SQ TG Togo en
Expand All @@ -245,7 +245,7 @@ SQ UM United States Minor Outlying Islands en
SQ UY Uruguay en
SQ UZ Uzbekistan en
SQ VU Vanuatu en
SQ VE &quot;Venezuela en
SQ VE Venezuela, Bolivarian Republic of en
SQ VN Viet Nam en
SQ VG Virgin Islands (British) en
SQ VI Virgin Islands (U.S.) en
Expand Down
28 changes: 14 additions & 14 deletions registry/entities/select_one_long_list/tsv.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ A BZ Belize en
A BJ Benin en
A BM Bermuda en
A BT Bhutan en
A BO &quot;Bolivia en
A BQ &quot;Bonaire en
A BO Bolivia, Plurinational State of en
A BQ Bonaire, Sint Eustatius and Saba en
A BA Bosnia and Herzegovina en
A BW Botswana en
A BV Bouvet Island en
Expand All @@ -56,7 +56,7 @@ A CC Cocos (Keeling) Islands en
A CO Colombia en
A KM Comoros en
A CG Congo en
A CD &quot;Congo en
A CD Congo, Democratic Republic of the en
A CK Cook Islands en
A CR Costa Rica en
A CI Côte d&#39;Ivoire en
Expand Down Expand Up @@ -110,7 +110,7 @@ A HU Hungary en
A IS Iceland en
A IN India en
A ID Indonesia en
A IR &quot;Iran en
A IR Iran, Islamic Republic of en
A IQ Iraq en
A IE Ireland en
A IM Isle of Man en
Expand All @@ -123,8 +123,8 @@ A JO Jordan en
A KZ Kazakhstan en
A KE Kenya en
A KI Kiribati en
A KP &quot;Korea en
A KR &quot;Korea en
A KP Korea, Democratic People&#39;s Republic of en
A KR Korea, Republic of en
A KW Kuwait en
A KG Kyrgyzstan en
A LA Lao People&#39;s Democratic Republic en
Expand All @@ -149,8 +149,8 @@ A MR Mauritania en
A MU Mauritius en
A YT Mayotte en
A MX Mexico en
A FM &quot;Micronesia en
A MD &quot;Moldova en
A FM Micronesia, Federated States of en
A MD Moldova, Republic of en
A MC Monaco en
A MN Mongolia en
A ME Montenegro en
Expand All @@ -161,7 +161,7 @@ A MM Myanmar en
A NA Namibia en
A NR Nauru en
A NP Nepal en
A NL &quot;Netherlands en
A NL Netherlands, Kingdom of the en
A NC New Caledonia en
A NZ New Zealand en
A NI Nicaragua en
Expand All @@ -175,7 +175,7 @@ A NO Norway en
A OM Oman en
A PK Pakistan en
A PW Palau en
A PS &quot;Palestine en
A PS Palestine, State of en
A PA Panama en
A PG Papua New Guinea en
A PY Paraguay en
Expand All @@ -191,7 +191,7 @@ A RO Romania en
A RU Russian Federation en
A RW Rwanda en
A BL Saint Barthélemy en
A SH &quot;Saint Helena en
A SH Saint Helena, Ascension and Tristan da Cunha en
A KN Saint Kitts and Nevis en
A LC Saint Lucia en
A MF Saint Martin (French part) en
Expand Down Expand Up @@ -222,9 +222,9 @@ A SJ Svalbard and Jan Mayen en
A SE Sweden en
A CH Switzerland en
A SY Syrian Arab Republic en
A TW &quot;Taiwan en
A TW Taiwan, Province of China en
A TJ Tajikistan en
A TZ &quot;Tanzania en
A TZ Tanzania, United Republic of en
A TH Thailand en
A TL Timor-Leste en
A TG Togo en
Expand All @@ -245,7 +245,7 @@ A UM United States Minor Outlying Islands en
A UY Uruguay en
A UZ Uzbekistan en
A VU Vanuatu en
A VE &quot;Venezuela en
A VE Venezuela, Bolivarian Republic of en
A VN Viet Nam en
A VG Virgin Islands (British) en
A VI Virgin Islands (U.S.) en
Expand Down
5 changes: 0 additions & 5 deletions scripts/bless-tsv-snapshots.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,6 @@ if (!fs.existsSync(entry)) {
process.exit(1);
}
const { XLSFormToTSVConverter } = await import(pathToFileURL(entry).href);
const { resolveFileChoices } = await import(
pathToFileURL(path.join(REPO_ROOT, 'dist', 'fileChoices.js')).href
);
const VOCAB_DIR = path.join(REPO_ROOT, 'registry', 'vocab');

// Same registry walk as codegen.load_registry: root graph + every split source
// (schema.jsonld + conventions/*.jsonld + registry/<slug>/definition.jsonld).
Expand Down Expand Up @@ -68,7 +64,6 @@ for (const variant of variants) {
survey,
xlsform.choices ?? [],
xlsform.settings ?? [],
resolveFileChoices(survey, VOCAB_DIR),
);
fs.writeFileSync(path.join(exDir, 'tsv.tsv'), tsv);
console.log(
Expand Down
4 changes: 2 additions & 2 deletions skills/cdl-survey-types/references/question-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@

**Use when:** When selecting from a long non-exclusive list of options maintained as a controlled vocabulary, allowing multiple selections.

**Requires a choice list:** write the type cell as `select_multiple_from_file <list_name>` and define the options on the `choices` sheet under that `list_name`.
**Requires a registered vocabulary:** write the type cell as `select_multiple_from_file <file>.csv` with a file from the registered-vocabularies table in [xlsform-syntax.md](xlsform-syntax.md). The options come from that vocabulary; add no rows to the `choices` sheet.

**Constraints:** variable `name` ≤ 20 chars, `^[a-zA-Z0-9]+$`

Expand All @@ -72,7 +72,7 @@

**Use when:** When selecting from a long closed list of exhaustive, mutually exclusive options maintained as a controlled vocabulary (e.g. country, occupation), presented via dropdown or autocomplete.

**Requires a choice list:** write the type cell as `select_one_from_file <list_name>` and define the options on the `choices` sheet under that `list_name`.
**Requires a registered vocabulary:** write the type cell as `select_one_from_file <file>.csv` with a file from the registered-vocabularies table in [xlsform-syntax.md](xlsform-syntax.md). The options come from that vocabulary; add no rows to the `choices` sheet.

**Constraints:** variable `name` ≤ 20 chars, `^[a-zA-Z0-9]+$`

Expand Down
Loading
Loading