Repository navigation
Conversation
Including the columns we want to fill out. If the mapping csv already exists, it checks for new fields and append them (if any).
signekb
commented
Oct 29, 2026
Comment on lines
+22
to
+24
| FIELD_RESOURCE_MAPPING_PATH = ( | ||
| SRC / "questionnaire_data" / "metadata" / "redcap" / "mapping-field-to-resource.csv" | ||
| ) |
Contributor
Author
There was a problem hiding this comment.
I don't particularly like this path, but this is the current location.
signekb
commented
Oct 29, 2026
| df.write_csv(raw_data_path, compression="gzip") | ||
|
|
||
|
|
||
| def read_newest_raw(raw_data_paths: Path): |
Contributor
Author
There was a problem hiding this comment.
Since I need to read the newest raw data in order to split it for staging, I thought this could be reused. Although this also results in the raw data being read twice 🤔
signekb
commented
Oct 29, 2026
Comment on lines
+52
to
+65
| if mapping_csv_path.exists(): | ||
| mapping_csv = pl.read_csv(mapping_csv_path, separator=";") | ||
| existing_field_names = mapping_csv["field_name"].to_list() | ||
| new_fields = [ | ||
| field_name | ||
| for field_name in field_names | ||
| if field_name not in existing_field_names | ||
| ] | ||
| if len(new_fields) != 0: | ||
| pl.concat( | ||
| [mapping_csv, pl.DataFrame({"field_name": new_fields})], | ||
| # Fills in missing columns with None. | ||
| how="diagonal", | ||
| ).write_csv(mapping_csv_path, separator=";") |
Contributor
Author
There was a problem hiding this comment.
Not sure if this is overdone, but this allows us to add any new fields if they appear without overwriting the existing when adding this as a task.
lwjohnst86
requested changes
Oct 29, 2026
github-project-automation
Bot
moved this from In review
to In progress
in Platform development
lwjohnst86
requested changes
Oct 30, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Including the columns (
new_field_name,resource_title,resource_name,comments) that we want to fill out. If the mapping csv already exists, it checks for new fields and append them (if any).Needs a thorough review.
Checklist
just build-raw(but didn't change the mapping csv yet)