Skip to content

fix(idvs): sum IDV child outlays in the child join and fix them for integer IDs - #4856

Open
BMDan wants to merge 1 commit into
fedspendingtransparency:qatfrom
BMDan:mod/idv-amounts-large-tree
Open

BMDan wants to merge 1 commit into
fedspendingtransparency:qatfrom
BMDan:mod/idv-amounts-large-tree

Conversation

@BMDan

@BMDan BMDan commented Nov 10, 2026 •

Copy link
Copy Markdown

Description:

/api/v2/idvs/amounts/<award_id>/ returns null for child_award_total_outlay and grandchild_award_total_outlay when the IDV is requested by its internal (integer) award id. The same IDV requested by its generated unique award id returns the correct totals. This PR fixes that, and makes the outlay calculation do about half the work, which matters for very large IDVs such as GSA schedules.

Reproduction against production (same IDV both ways; 351035969 is the internal id of CONT_IDV_75P00123A00008_7570):

$ curl -s https://api.usaspending.gov/api/v2/idvs/amounts/CONT_IDV_75P00123A00008_7570/ | \
  jq -r .child_award_total_outlay
7911077.1
$ curl -s https://api.usaspending.gov/api/v2/idvs/amounts/351035969/ | \
  jq -r .child_award_total_outlay
null

CONT_IDV_75A50223D00005_7505 (internal id 351030267) shows the same thing: 1015100.0 by generated id, null by integer id. With this change both forms return the same values.

Technical Details:

fetch_idv_child_outlays in usaspending_api/idvs/v2/views/amounts.py used to:

  1. find the child (or grandchild) contracts with child_award_sql / grandchild_award_sql, wrapped in a CTE;
  2. join that CTE back to rpt.award_search by award_id, only to read total_outlays;
  3. filter that join again with parent_award_piid = '<piid>' OR parent_award_piid IN (SELECT piid ... category = 'idv'), where <piid> came from splitting the request's award id on _.

Step 3 caused the bug. For an integer award id, award_id_piid was the integer itself, so the filter compared parent_award_piid with something like '351035969' and matched nothing. For a generated id whose PIID contains an underscore, the split also cut the PIID short. The filter was redundant in every case: the join in step 1 already matches parent_award_piid to the parent's PIID, together with the agency.

The outlays are now summed in the join that finds the children (child_outlay_sql, grandchild_outlay_sql). The child side of that join reads rpt.award_search directly instead of vw_awards, because vw_awards (a view over rpt.award_search) does not expose total_outlays. Each child row is now visited once, and the result no longer depends on the form of the request's id.

fetch_account_details_idv is unchanged apart from type annotations. I tried passing its child and grandchild sets to the DEFC query as subqueries instead of literal IN (...) lists. It was slower, because the planner estimates the grandchild join at 1 row; a pre-aggregated variant tied. Neither is included.

Other changes in the touched files: imports sorted and type annotations added so that ruff check and ruff format --check pass on both files with the pinned ruff 0.14.14 (CI checks whole changed files), and raise NotFound(...) from None in the view's except block.

Requirements for PR Merge:

  1. Unit & integration tests updated
  2. [N/A] API documentation updated (examples listed below)
  3. Data validation completed (examples listed below)
  4. [N/A] Appropriate Operations ticket(s) created
  5. [N/A] Jira Ticket(s)

Explain N/A in above checklist:

  1. Tests: test_awards_idvs_amounts_v2.py gains test_child_and_grandchild_outlays_and_account_amounts, parametrized over the integer and the generated award id. It builds an IDV with two child contracts and a child IDV with two contracts of its own, plus account rows under a DEFC and a decoy contract under another agency's PIID. On qat the integer-id case fails (child_award_total_outlay is None, expected 15); with this change all four tests in the file pass.
  2. API documentation: N/A. The response shape and field meanings are unchanged; integer-id requests now return the values the contract already describes.
  3. Data validation:
    1. Frontend: no change needed; same fields, same format.
    2. Performance: measured on the 2026-10-06 database download, for CONT_IDV_47QSEA18D008C_4732 (259,535 grandchild orders since FY2024). The grandchild outlay query takes 0.54 s instead of 1.1 s with a warm cache. Child and grandchild outlay totals were identical between the old and new queries (822,974.03 and 112,719,237.77).
    3. Format: unchanged; values are the same as before for generated ids and now correct for integer ids.
  4. Operations ticket: N/A. No migration, configuration or deployment step.
  5. Jira: N/A. I have no Jira access and found no existing issue for this.

Out-of-scope -- what this PR does not fix

For IDVs with very large grandchild trees, idvs/amounts (and idvs/funding_rollup, idvs/activity, idvs/accounts, and idvs/awards with type=grandchild_awards) still return 502 after about 60 seconds. See #4861 . This fix is restricted to the other elements mentioned above.

…nteger ids

- fetch_idv_child_outlays read each child contract twice: once in the join that finds
  the children, then again from rpt.award_search by award_id to get total_outlays,
  behind a parent_award_piid filter with an OR and an IN subquery
- the outlays are now summed in the join itself (vw_awards is a view of
  rpt.award_search, which carries total_outlays); the extra filter repeated what the
  join already matches on
- the old filter compared parent_award_piid to the request's integer id when the
  endpoint was called with one, so child and grandchild outlays came back null for
  /api/v2/idvs/amounts/<int>/; it also cut the PIID at the first underscore of a
  generated id
- measured on the 2026-10-06 database download, for a GSA schedule IDV with 259,535
  grandchild orders: the grandchild outlay query takes 0.54 s instead of 1.1 s warm,
  with identical results
- test: an IDV tree with child and grandchild contracts, account rows under a DEFC,
  and a contract under another agency's PIID, requested by both id forms
@BMDan BMDan changed the title fix(idvs): sum IDV child outlays in the child join and fix them for i… fix(idvs): sum IDV child outlays in the child join and fix them for integer IDs Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant