Repository navigation
refator: metadata validate - #346
Open
nikbpetrov wants to merge 4 commits into
Open
nikbpetrov wants to merge 4 commits into
nikbpetrov wants to merge 4 commits into
Conversation
nikbpetrov
marked this pull request as ready for review
October 30, 2026 08:54
nikbpetrov
force-pushed
the
p1-metadata-validate
branch
from
October 30, 2026 08:54
b371076 to
504a586
Compare
nikbpetrov
force-pushed
the
p1-metadata-validate
branch
from
November 9, 2026 09:56
504a586 to
61d86b4
Compare
pyfixest 0.70.0 (released 2026-11-08) rejects a fixed-effect column of object dtype, and
`two_way_fixed_effects` passes `question_pk` and `model_pk` as plain object columns:
FixedEffectEvaluationError: Fixed-effect column `model_pk` has unsupported object dtype.
The requirement had no upper bound, so every fresh install, CI and the next leaderboard deploy
alike, picks up 0.70.0 and the fit fails. Cap it at the last version the code was run with; casting
the key columns to `string` and lifting the cap is a separate change.
Port `src/metadata/tag_questions/main.py` to `src/metadata/_tag.py`, exported as `metadata.tag` (the domain logic: category from the metadata model or fixed per source, English flag from Jev, merge into the metadata frame; function bodies verbatim) and `src/orchestration/func_metadata_tag/` (the Cloud Run job `func-metadata-tag-questions`: download the metadata, then one `tag()` call and one upload per sampled source with the 2 s GCS write-rate sleep). `orchestration._io` gains `load_question_metadata()` and `upload_question_metadata()`; `_schemas` gains `MetadataFrame`, which types `valid_question` and `english` as bool or "" because the file carries the placeholder until validate runs or when a Jev call failed. `tag()` and the two `_io` helpers take and return `DataFrame[MetadataFrame]` / `DataFrame[QuestionFrame]` under `@pa.check_types`, so a malformed metadata or question file fails the job instead of being written; `load_question_metadata()` fills the columns a file written before they existed lacks. The module is private (`_tag.py`) because a public `tag.py` would be shadowed by the re-exported function. The package exports `tag` through a module `__getattr__` rather than an eager import, so a job that imports `metadata` for another function does not load `_tag`'s Jev SDK, which only the tag job ships; a contract test pins that. `src/metadata/tag_questions/` is removed and the root `Makefile` target `tag-questions` repointed. Requirements drop `functions-framework` (the job runs `python3 main.py`) and `scipy` (nothing in the staged import chain loads it). Tests pin the two production fixes the job carries, the 512-token category budget (`c0d475e`) and the one-time key fetch before the concurrent calls (`1fb7ddd`), and the (source, id) merge key. Parity: on a mirror of the dev question bank, with the same deterministic model and Jev fakes on both sides, the old and new jobs write a byte-identical `question_metadata.jsonl` (final and every per-source upload) and send the same rows to the model and to Jev, in the as-is, perturbed and empty-file scenarios. Tests: `unit/metadata/test_tag.py`, `unit/orchestration/test_metadata_tag_job.py` and `integration/test_metadata_io.py`, superseding `unit/metadata/test_tag_questions.py` and `test_tag_questions_english.py`; `integration/test_metadata_drivers.py`, `e2e/test_question_set_pipeline.py`, `contract/test_offline_imports.py` and `contract/test_runtime_requirements.py` follow the moved modules.
Port `src/metadata/validate_questions/main.py` to `src/metadata/_validate.py`, exported as
`metadata.validate` (the domain logic: dataset sources and Metaculus are valid without a call, the
other sampled sources are judged by the metadata model with the legacy verdict parsing; function
bodies verbatim) and `src/orchestration/func_metadata_validate/` (the Cloud Run job
`func-metadata-validate-questions`: download the metadata, then per sampled source one `validate()`
call, one upload and the 2 s GCS write-rate sleep). Job name, resources, timeout, service account
and env vars are unchanged.
* `metadata.validate(dfmeta, dfq, source)` validates the source's questions and returns the
metadata with that source's rows replaced, sorted by source then id, the same shape as `tag()`.
It is `@pa.check_types` on `DataFrame[MetadataFrame]` (`dfq` a `DataFrame[QuestionFrame]`), so
`valid_question` is the schema's object column of True/False, not dtype bool, and the job
counts invalid rows with `.eq(False)` on the source's slice. The module is private
(`_validate.py`) so that the re-exported function does not shadow it.
* The job uses `orchestration._io.load_question_metadata()`, `upload_question_metadata()` and
`_schemas.MetadataFrame` from the metadata tag commit; `load_question_metadata()` fills the
columns an old file lacks, so the job needs no `valid_question` guard.
* Requirements drop `scipy`: nothing in the staged import chain loads it now that
`helpers.question_curation` no longer imports the wikipedia helper.
* Remove `src/metadata/validate_questions/` and repoint the root `Makefile` target
`validate-questions`.
Parity: legacy and new job on a mirror of the dev question bank, with the same deterministic fake
model on both sides, write byte-identical per-source uploads and final `question_metadata.jsonl`
and send the same prompts, in the as-is, placeholder and missing-row scenarios.
Tests: `unit/metadata/test_validate.py`, `unit/orchestration/test_metadata_validate_job.py` and
three assertions in `integration/test_metadata_io.py`; `integration/test_metadata_drivers.py`,
`e2e/test_question_set_pipeline.py` and `contract/test_{offline_imports,runtime_requirements}.py`
are repointed, and `unit/metadata/test_validate_questions.py` is deleted as superseded.
…)` everywhere The question-set sampler and the nightly question-bank summary read `question_metadata.jsonl` on their own: the sampler through `data_utils.download_and_read` with `META_DATA_FILE_COLUMN_DTYPE`, which declared `valid_question` and `english` as `bool`, and the summary with a bare `pd.read_json` on the bucket URL. The file is three-valued: both flags hold `True`, `False` or `""` for a question the tag or validate job has not classified yet. The sampler only excluded those questions because the `bool` cast turned `""` into `False`; the summary masked with the raw column and would raise on a placeholder row. Both now call `_io.load_question_metadata()`, the typed `MetadataFrame` reader the metadata jobs use, and test for `True` explicitly, so a question without a verdict is excluded by a rule the reader can see rather than by a dtype coercion. The behaviour is unchanged: a question not yet classified is not sampled and not counted as valid until the next metadata run fills it in. `META_DATA_FILE_COLUMN_DTYPE` and `META_DATA_FILE_COLUMNS` are removed; `MetadataFrame` is the single description of the file, and the tests take the column list from its schema.
nikbpetrov
force-pushed
the
p1-metadata-validate
branch
from
November 9, 2026 12:07
61d86b4 to
b6c9723
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On top of #345