Docs/: Update school year description in all prod_wh tables

[htymeson]

Description & motivation

Motivation

This is a first pass at SafeInsights work to improve existing DBT docs while writing documentation for the SCDE enclave, as documented in #SI-108 (here).

Description

Fill in a description for every school_year column in prod_wh schema, so that a range of audiences understand both
the sourcing mechanism and format of school_year variables within each table. This involved:

• Identifying the sourcing mechanism of the school_year variable for each specific core_wh table based on model lineage, and categorizing those into four main categories of school_year description
• Using docs blocks to store those four standard descriptions
• Applying those descriptions to every school_year reference in the core_warehouse/ models, which involved the following changes from existing documentation (note - there are 57 .yml files in there, and 11 validly have no school_year column):
  • Filling out a new description when none existed (30 files)
  • Updating a description from an existing value to one of the new descriptions (14 files)
  • Adding a variable when school_year was present in the model but not in the docs (2 files)

Breaking changes introduced by this PR:

I reeally doubt these are breaking changes, but will note just in case. I added school_year variable and descriptions to the following yml files:

• dim_class_period.yml
• fct_student_discipline_actions.yml

PR Merge Priority:

• [ x] Low
• Medium
• High

Changes to existing files:

Changes to every yml file in the models/core_warehouse/ folder. To help organize, there are four broad categories that were used to update or fill in (for the first time) the school_year description.

Category 1 — ODS Pull (24 files)

Description: school_year_ods_pull

dim_student.yml, dim_learning_standard.yml, dim_parent.yml, dim_program.yml, fct_student_migrant_education_program_associations.yml, fct_student_program_association.yml, fct_student_special_education_program_association.yml, fct_student_title_i_part_a_program_association.yml, fct_student_homeless_program_association.yml, fct_student_school_food_service_program_associations.yml, fct_student_language_instruction_program_association.yml, fct_student_cte_program_associations.yml, fct_student_subgroup.yml, fct_student_parent_association.yml, dim_course.yml, dim_graduation_plan.yml, dim_cohort.yml, fct_student_cohort_association.yml, dim_discipline_incident.yml, fct_student_discipline_incident_behaviors.yml, fct_student_discipline_incident_summary.yml, fct_student_discipline_actions.yml, fct_student_discipline_actions_summary.yml, dim_class_period.yml

Category 2

Version A: School Year Source Reference, no fallback (12 files)

Description: school_year_source_reference

fct_student_school_attendance_event.yml, fct_student_section_association.yml, fct_student_section_attendance_event.yml, fct_staff_section_association.yml, dim_school_calendar.yml, dim_calendar_date.yml, dim_course_section.yml, dim_session.yml, dim_grading_period.yml, fct_student_grades.yml, fct_student_daily_attendance.yml, msr_student_cumulative_attendance.yml

Version B: School Year Source Reference, with fallback (2 files, unique text each)

fct_staff_school_association.yml:

Description: school_year_source_reference + "If that reference is missing, school_year falls back to the source ODS pull year."

fct_student_school_association.yml:

Description: school_year_source_reference + "If that reference is missing, school_year falls back to a value inferred from entry_date using a school-year cutoff date that can vary by implementation."

Category 3 — Assessment-derived (4 files, identical text)

Description: school_year_assessment

fct_student_assessment.yml, dim_assessment.yml, dim_objective_assessment.yml, fct_student_objective_assessment.yml

Category 4 — Academic Record, Required source field (4 files)

Description: school_year_academic_record

fct_student_academic_record.yml, fct_student_diploma.yml, fct_student_gpa.yml, fct_course_transcripts.yml

Changes of particular note

The following files had prior descriptions that were unusual:

• dim_course: ""The school year passed to the Ed-Fi API when pulling this record from Course". When I traced back, I didn't see anything to distinguish it from the typical api_year based on single-year ODS sourcing.
• dim_discipline_incident: "Sourced from the year of the ODS, since Ed-Fi does not include school year in the model for discipline incidents". I'm not sure why this was particularly noted here? Is this an important differentiation from all the edfi models that just infer school_year from the ODS?
• dim_grading_period: "The identifier for the grading period school year." Real vague. I didn't see any differences from just inferred ODS year in the lineage here, so unsure why this had its own anomalous description.
• dim_graduation_plan: "The identifier for the grading period school year.". Yeah, I think this one was just copy-pasted from grading period. I also traced school_year back to the api_year in raw edfi models (graduation_school_year is different, though).
• fct_student_assessment: "The school year for which the assessment was administered to a student.". That got overwritten with the complicated assessment assignment pathway.
• fct_student_diploma: "The school year for which the diploma was awarded to a student.". I think the general noting that school_year here does not come from API year is well-represented from the updated description.

New files created:

Added a new sub-folder in /models/core_warehouse/ called docs/.

Within that new sub-folder, added a new file school_year_docs.md containing the doc blocks definitions for the four categories of school year descriptions.

Tests and QC done:

NOTE: I used Claude Code to do the majority of the categorization and lineage mapping, so that likely merits extra review on that alone.

• I traced the lineage and confirmed the categorization for a small sub-sample of these pathways.
• I manually verified that the documented categorization in this PR description matched the description implemented for each yml.

edu_wh PR Review Checklist:

Make sure the following have been completed before approving this PR:

• [x ] Description of changes has been added to Unreleased section of CHANGELOG.md. Add under ## New Features for features, etc.
• [ x] Code has been tested/checked for Databricks and Snowflake compatibility - EA engineers see Databricks checklist here
• [x ] Reviewer confirms the grain of all tables are unchanged, OR any changes are expected, communicated, and this PR is flagged as a breaking change (not for patch release)
• If a new configuration xwalk was added:
  • The code is written such that the xwalk is optional (preferred), and this behavior was tested, OR
  • The code is written such that the xwalk is required, and the required xwalk is added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new xwalk has been added to EDU documentation site here
• If a new configuration variable was added:
  • The code is written such that the variable is optional (preferred), and this behavior was tested, OR
  • The code is written such that the variable is required, and a default value was added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new variable has been added to EDU documentation site here

Important

Questions

1. Is the derive_school_year thing that will infer school_year from administration date for test-related tables (and entry date in student-school assocation) optional for implementations? If so, I will update language to note optional configuration.
2. Any suggested changes to the categories or standard descriptions?
3. I'm wondering about the "{{ doc('{docblock') }}" format for referencing the doc blocks, specifically - does that exact quotation format work with our DBT system? It does not exactly match the DBT docs blocks documentation, which uses the format '{{ doc("docblock") }}' (single quotes outside, double quotes inside) in their example reference. I do not have DBT access, and this clearly uses some DBT placement, so I would definitely test that the format is working.

[htymeson]

okay @rlittle08 updated to use docs blocks and adjusted PR description accordingly! I think this is ready for review, lmk if anything needs changing.

[rlittle08]

i worry that the 'unlike most other tables' is just adding confusion.. do most users want the verbose explanation of the ODS magic behind the scenes, or do they just want to know it's a school year?

I'm thinking that this detail on school years is a better fit for a supplementary document, than a dbt docs column description