Repository navigation
fix(data): stop fundamentals being visible before their publish_date (BUG-22) - #18
Merged
Merged
Conversation
…(BUG-22)
The daily financial panel was forward-filled from the *fiscal period end* and
then backward-filled:
df_full = df.reindex(full_idx) # df still keyed on qdate
df_full = df_full.groupby("asset").ffill()
df_full = df_full.groupby("asset").bfill() # fill initial NaN
Since A-share reports land 40-50 days after the period closes, this published
figures weeks before anyone could have seen them, and the bfill pushed a
company's first report back to the first date of the sample.
Measured on the provider's own output (seed 7, 000001.SH):
fiscal_period_end 2021-03-31, publish_date 2021-05-20, lag 50 days
2021-04-10 (40d before) -> visible <- should be unknown
2021-05-10 (10d before) -> visible <- should be unknown
2021-05-20 (publish day) -> visible ok
Fix: re-index each report to its publish_date before forward-filling, so a
value first appears on the day it became public, and drop the bfill -- before
a company's first report there is genuinely nothing to know, and filling it
backwards is itself a look-ahead.
The fiscal period is preserved in a new `fiscal_period_end` column rather than
overloading the index, which now carries the visibility date.
This matters because NO_LOOKAHEAD_CONTRACT 第三条 requires exactly this
(`mask = financials['publish_date'] <= signal_date`). The rule and the
point-in-time accessors both already existed; neither was wired to the panel
the factors actually read. Every fundamental-factor result computed through
the daily panel is affected, so prior fundamental ICs need re-running.
After the fix:
before publish_date -> nan (unknown)
on/after -> the published value
before the first report -> nan
Two regression tests added, both checked against the old implementation:
test_no_value_is_visible_before_its_publish_date and
test_nothing_is_backfilled_before_the_first_report -- both fail there.
The pre-existing test_publish_date_after_report_date asserted the old model
(that index dates which are quarter-ends are the original report rows). That
assumption no longer holds, so it now compares publish_date against the new
fiscal_period_end column, which checks the same property against the new shape.
1315 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P0. First of the four research-integrity defects.
NO_LOOKAHEAD_CONTRACT第三条 requires exactly this behaviour, and the point-in-time accessors already existed — neither was wired to the panel the factors actually read.What was wrong
The daily financial panel was forward-filled from the fiscal period end, then backward-filled:
A-share reports land 40–50 days after the period closes, so this published figures weeks before anyone could have seen them. Measured on the provider's own output (seed 7,
000001.SH):(
fiscal_period_end2021-03-31 →publish_date2021-05-20, lag 50 days.)The
bfill()had a second effect: it pushed a company's first report back to the first date of the sample, so fundamentals were "known" months before anything was published.Fix
Re-index each report to its
publish_datebefore forward-filling, so a value first appears on the day it became public. Drop the bfill — before a company's first report there is genuinely nothing to know, and filling it backwards is itself a look-ahead.The fiscal period moves to a new
fiscal_period_endcolumn rather than overloading the index, which now carries the visibility date.Verified
Two regression tests added, and both were checked against the old implementation — they fail there:
test_no_value_is_visible_before_its_publish_datetest_nothing_is_backfilled_before_the_first_reportThe pre-existing
test_publish_date_after_report_dateasserted the old model (that index dates which happen to be quarter-ends are the original report rows). That assumption no longer holds, so it now comparespublish_dateagainst the newfiscal_period_endcolumn — the same property, checked against the new shape.1315 passed, 1 skipped.Impact — this invalidates prior fundamental results
Everything computed through the daily panel is affected, so prior fundamental-factor ICs need re-running. That includes the M1
roerun and the M3 factor-zoo entries forroe/pb_ratio/pe_ratio/log_market_cap/asset_growth.Per the contract's own remediation procedure, those results should be marked
affectedrather than deleted. The validity metadata to do that does not exist yet — tracked separately.🤖 Generated with Claude Code