feat(service): Instrument the remaining backends with the change stream

[matt-codecov]

• document the ChangeStream abstraction in objectstore-service/docs/architecture.md
• wire up LocalFsBackend, InMemoryBackend, and S3CompatibleBackend with a ChangeStreamFactory / self.change_stream handle
• make the above backends emit write/update/delete messages to self.change_stream

Closes FS-489

also includes a drive-by fix: previously the S3 backend would return Err(...) when deleting a non-existent object. the rest of our backends return Ok(()) in that case. so this PR makes the S3 backend match the other backends' behavior

[linear-code]

FS-489

[codecov]

Codecov Report

❌ Patch coverage is 92.64706% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.64%. Comparing base (cd1aca6) to head (df0a423).

Files with missing lines	Patch %	Lines
objectstore-service/src/backend/s3_compatible.rs	78.26%	10 Missing ⚠️
objectstore-service/src/backend/mod.rs	16.66%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   88.54%   88.64%   +0.09%     
==========================================
  Files         105      105              
  Lines       17289    17449     +160     
==========================================
+ Hits        15308    15467     +159     
- Misses       1981     1982       +1     

Components	Coverage Δ
Rust Backend	92.60% <92.64%> (+0.08%)	⬆️
Rust Client	81.97% <ø> (ø)
Python Client	93.56% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aldy505]

Not sure what you meant by this

[lcian]

If you want to add a new ChangeStream implementor to certain backends, you need to change:

• The Service config to configure where the changes get written for that particular change stream (for COGS this is the storage_cogs top-level config key in the yaml -- you do this at the Service level so you can share a single Kafka producer/db connection/etc.)
• The backend config for the backends that should support such change stream implementation (in this PR we needed to add the cogs field to FileSystemConfig)
• ChangeStreamFactory::new and ::build, to teach the factory how to use those new configs.

[aldy505]

Why are we using &ChangeStreamFactory here instead of Arc<dyn ChangeStream>? Shouldn't ChangeStream implementation is dynamically switchable, in SaaS we can use the cost tracker, in self-hosted we can use the garbage collector?

[lcian]

I was also thinking about this. See https://github.com/getsentry/objectstore/pull/607/changes#r3952214185 for my though process. Matt will probably have more to add to this as well.

[lcian]

I understand the design with the ChangeStreamFactory now.
That allows us to define a pub cleanup: Option<CleanupStreamConfig> for expiry cleanup just here but not on e.g. the GCS backend, making it impossible to construct a GCS backend with a CleanupChangeStream.

If we were passing in Arc<dyn ChangeStream> to each backend instead, we wouldn't be able to enforce that as easily.

[lcian]

Nit: should we say that size should be an estimate or proportional to the bytes that you're storing/billed for in/by the backend, rather than literally the size?
Why: looking at S3 docs it seems that the storage bytes you're charged for has some nuances that depend on things such as the storage tier: https://aws.amazon.com/s3/pricing/#:~:text=*%20S3%20Intelligent%2DTiering,pricing%20page.

[lcian]

Looks good to me.