Skip to content

fix(data): stop fundamentals being visible before their publish_date (BUG-22) - #18

Merged
sijie-Z merged 2 commits into
mainfrom
fix/fundamental-pit-leak
Sep 13, 2026
Merged

sijie-Z merged 2 commits into
mainfrom
fix/fundamental-pit-leak

Conversation

@sijie-Z

@sijie-Z sijie-Z commented Sep 13, 2026

Copy link
Copy Markdown
Owner

P0. First of the four research-integrity defects. NO_LOOKAHEAD_CONTRACT 第三条 requires exactly this behaviour, and the point-in-time accessors already existed — neither was wired to the panel the factors actually read.

What was wrong

The daily financial panel was forward-filled from the fiscal period end, then backward-filled:

df_full = df.reindex(full_idx)              # df still keyed on qdate
df_full = df_full.groupby("asset").ffill()
df_full = df_full.groupby("asset").bfill()  # "fill initial NaN"

A-share reports land 40–50 days after the period closes, so this published figures weeks before anyone could have seen them. Measured on the provider's own output (seed 7, 000001.SH):

Date vs publish_date Value shown
2021-04-10 40 days before visible ❌ should be unknown
2021-05-10 10 days before visible ❌ should be unknown
2021-05-20 publication day visible ✅

(fiscal_period_end 2021-03-31 → publish_date 2021-05-20, lag 50 days.)

The bfill() had a second effect: it pushed a company's first report back to the first date of the sample, so fundamentals were "known" months before anything was published.

Fix

Re-index each report to its publish_date before forward-filling, so a value first appears on the day it became public. Drop the bfill — before a company's first report there is genuinely nothing to know, and filling it backwards is itself a look-ahead.

The fiscal period moves to a new fiscal_period_end column rather than overloading the index, which now carries the visibility date.

Verified

before publish_date      -> nan
on / after publish_date  -> the published value
before the first report  -> nan

Two regression tests added, and both were checked against the old implementation — they fail there:

  • test_no_value_is_visible_before_its_publish_date
  • test_nothing_is_backfilled_before_the_first_report

The pre-existing test_publish_date_after_report_date asserted the old model (that index dates which happen to be quarter-ends are the original report rows). That assumption no longer holds, so it now compares publish_date against the new fiscal_period_end column — the same property, checked against the new shape.

1315 passed, 1 skipped.

Impact — this invalidates prior fundamental results

Everything computed through the daily panel is affected, so prior fundamental-factor ICs need re-running. That includes the M1 roe run and the M3 factor-zoo entries for roe / pb_ratio / pe_ratio / log_market_cap / asset_growth.

Per the contract's own remediation procedure, those results should be marked affected rather than deleted. The validity metadata to do that does not exist yet — tracked separately.

🤖 Generated with Claude Code

…(BUG-22)

The daily financial panel was forward-filled from the *fiscal period end* and
then backward-filled:

    df_full = df.reindex(full_idx)            # df still keyed on qdate
    df_full = df_full.groupby("asset").ffill()
    df_full = df_full.groupby("asset").bfill()   # fill initial NaN

Since A-share reports land 40-50 days after the period closes, this published
figures weeks before anyone could have seen them, and the bfill pushed a
company's first report back to the first date of the sample.

Measured on the provider's own output (seed 7, 000001.SH):

    fiscal_period_end 2021-03-31, publish_date 2021-05-20, lag 50 days
    2021-04-10 (40d before)  -> visible     <- should be unknown
    2021-05-10 (10d before)  -> visible     <- should be unknown
    2021-05-20 (publish day) -> visible     ok

Fix: re-index each report to its publish_date before forward-filling, so a
value first appears on the day it became public, and drop the bfill -- before
a company's first report there is genuinely nothing to know, and filling it
backwards is itself a look-ahead.

The fiscal period is preserved in a new `fiscal_period_end` column rather than
overloading the index, which now carries the visibility date.

This matters because NO_LOOKAHEAD_CONTRACT 第三条 requires exactly this
(`mask = financials['publish_date'] <= signal_date`). The rule and the
point-in-time accessors both already existed; neither was wired to the panel
the factors actually read. Every fundamental-factor result computed through
the daily panel is affected, so prior fundamental ICs need re-running.

After the fix:
    before publish_date -> nan (unknown)
    on/after            -> the published value
    before the first report -> nan

Two regression tests added, both checked against the old implementation:
test_no_value_is_visible_before_its_publish_date and
test_nothing_is_backfilled_before_the_first_report -- both fail there.

The pre-existing test_publish_date_after_report_date asserted the old model
(that index dates which are quarter-ends are the original report rows). That
assumption no longer holds, so it now compares publish_date against the new
fiscal_period_end column, which checks the same property against the new shape.

1315 passed, 1 skipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b64f7af8-d5b6-4406-adc9-19fc5c30bb8c


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sijie-Z
sijie-Z merged commit 07982a5 into main Sep 13, 2026
9 checks passed
@sijie-Z
sijie-Z deleted the fix/fundamental-pit-leak branch September 13, 2026 04:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant