TDOE#174: Fix Student Program Dedupe, and add Program Status staging models.

[AngelicaLastra]

Description & motivation

This PR is a subset of TN's enhancements PR #174, pulled forward separately because this piece is ready to ship while the rest of that work is still being scoped.

It adds a k_student_program surrogate key to all 8 student program association staging tables and modifies the dedupe CTE on each to partition by k_student_program. This fixes a bug in how these models are deduped: they were previously deduplicating without the full PK, specifically missing ed_org_id. This surfaces when a Program's ed org is defined at a different level than the Association's ed org and a student is associated with the same Program at two different ed orgs. For example, if a Program is defined at the District level but a student within that District is associated with it at two different Schools (say, they switched schools midyear), the current implementation dedupes those two Association rows down to one, even though they represent enrollment at two distinct Schools.

Main highlights:

• k_student_program surrogate key added to all 8 parent student program association stage models, with dedupe updated to partition on it. Key components: tenant_code, api_year, student_unique_id, ed_org_id, program_ed_org_id, program_name, program_type, program_enroll_begin_date.
• ed_org_id and k_student_program added to all existing program sub-table models, since ed_org_id is part of the PK.
• 8 new _program_participation_statuses sub-tables, one per program association type, unpacking v_program_participation_statuses into flat rows across all Student Program Association models (consistent with the existing pattern for Program Services sub-tables).

Warning

This PR is a direct sibling of edu_wh's PR: edanalytics/edu_wh#249
They must be merged in unison.

PR Merge Priority:

• Low
• Medium
• High

Changes to existing files:

Parent program association stage models:

• base_ef3__student_migrant_education_program_associations.sql: Added v:programReference:educationOrganizationId::integer as program_ed_org_id. This field was never extracted for migrant education.

• Added k_student_program via dbt_utils.generate_surrogate_key and updated dbt_utils.deduplicate to partition by it, replacing the old k_student, k_program, program_enroll_begin_date, school_year partition. The old partition omitted ed_org_id, which is part of the Ed-Fi natural key for all student program associations.

  • stg_ef3__student_program_associations.sql, stg_ef3__student_cte_program_associations.sql
  • stg_ef3__student_homeless_program_associations.sql
  • stg_ef3__student_language_instruction_program_associations.sql
  • stg_ef3__student_migrant_education_program_associations.sql
  • stg_ef3__student_school_food_service_program_association.sql
  • stg_ef3__student_special_education_program_associations.sql
  • stg_ef3__student_title_i_part_a_program_associations.sql

Existing program sub-table models:

• Added k_student_program and ed_org_id to each SELECT list.
  • stg_ef3__stu_cte__program_services.sql
  • stg_ef3__stu_homeless__program_services.sql
  • stg_ef3__stu_lang_instr__program_services.sql
  • stg_ef3__stu_lang_instr__proficiency_assessments.sql
  • stg_ef3__stu_migrant_edu__program_services.sql
  • stg_ef3__stu_school_food_service__program_services.sql
  • stg_ef3__stu_spec_ed__program_services.sql
  • stg_ef3__stu_title_i_part_a__program_services.sql
  • stg_ef3__stu_title_i_part_a__services.sql

Bug fixes caught in review:

• stg_ef3__stu_program__program_participation_statuses.sql: Removed school_year from the SELECT. The generic student_program_associations parent exposes api_year as school_year and an earlier draft included it, but none of the other 7 _program_participation_statuses sub-tables do (or sub-tables in this repo in general), so it was dropped for consistency.

New files created:

All 8 new sub-tables follow the same pattern as the existing _program_services models: they ref() the corresponding parent stage model, laterally flatten v_program_participation_statuses via json_flatten, and select k_student_program, the standard key columns (k_student, k_student_xyear, k_program, k_lea, k_school, ed_org_id), the enrollment dates, and the status fields (participation_status, status_begin_date, designated_by, status_end_date). The spec-ed variant adds an extended CTE to call extract_extension after the flatten step, mirroring the same pattern in stg_ef3__stu_spec_ed__program_services (necessary because the spec-ed array nests its own v_ext, creating an ambiguity if extensions are applied before the flatten).

• stg_ef3__stu_cte__program_participation_statuses.sql
• stg_ef3__stu_homeless__program_participation_statuses.sql
• stg_ef3__stu_lang_instr__program_participation_statuses.sql
• stg_ef3__stu_migrant_edu__program_participation_statuses.sql
• stg_ef3__stu_program__program_participation_statuses.sql
• stg_ef3__stu_school_food_service__program_participation_statuses.sql
• stg_ef3__stu_spec_ed__program_participation_statuses.sql
• stg_ef3__stu_title_i_part_a__program_participation_statuses.sql

Tests and QC done:

• dbt compile clean
• `dbt run is successful
• Row counts on parent program association models verified before/after dedupe change
• k_student_program confirmed unique per parent model