ref(reprocessing): optionally write unprocessed payload to nodestore instead of redis

[matt-codecov]

Ref FS-533

backs up unprocessed event copies in Nodestore with a 24h TTL during processing instead of Redis. when saving the event after processing, event_manager reads the unprocessed backup from either nodestore or redis (whichever has it) and re-uploads it as a subkey in the processed event's row in nodestore

there are two reasons we do it this way instead of just durably storing the unprocessed copy right away:

• we initially store it with 24h TTL because there's no systematic way to ensure the many places where the pipeline early-exits all clean up the orphaned unprocessed copy. we would leak data in Nodestore for 90d when the current pipeline just leaks it in Redis for 24h. and extending the expiry on pipeline completion requires a full reupload
• since we have to reupload anyway, we reupload as a subkey because a) that's where it was persisted before so the change is more minimal, and b) the subkey structure supposedly compresses better because the processed and unprocessed copies are highly similar

TBD: whether the nodestore write during preprocessing is too much of a latency hit for the ingest consumer to swallow

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[matt-codecov]

@sentry review
bugbot run

[linear-code]

FS-533

[matt-codecov]

@sentry review
bugbot run