support for large atomic data values exceeding 16MB e.g., images

[craigmcchesney]

Consider using external values collection for chunked / segmented large values, or externalize data to hdf5 or some type of file. Research shows that MongoDB's GridFS mechanism might be the way to go, the whole point is storage of large data values. See comment below with additional details. High-level design would be to modify backend handling for SerializedDataColumn to store values larger than the 16MB threshold in a GridFS collection, with a GridFS location identifier in the corresponding DataBucket pointing to the externalized value.

[craigmcchesney]

One question that has arisen is how to handle atomic sample values whose size exceeds the mongodb bson object size limit of 16MB, e.g., for large images. It seems like we need a mechanism for chunking large values across multiple bson documents, or in a dependent collection, or...?

This is a very real issue, and you’re thinking about it at exactly the right time. MongoDB’s 16 MB BSON document limit becomes unavoidable the moment you ingest high-resolution images, dense waveforms, or large structures—even if everything else in the architecture is sound.

Below is a clear, pragmatic design space, with a strong recommendation tailored to your system constraints (high ingestion rate, multiple facilities, minimal infra).

---

The Core Constraint

MongoDB hard limit:

• 16 MB per BSON document

• This applies to:

  • Entire document
  • Any embedded subdocument
  • Any embedded array or BinData

So even a single atomic sample (e.g., one image frame) can violate the limit.

---

Design Principles (important)

For your system:

1. Ingestion latency is critical
2. Buckets are immutable after write
3. High write throughput
4. Downstream reads are often sequential
5. Infrastructure must remain simple
6. You already batch by PV & time

These push us away from “clever” document-level chunking and toward explicit large-payload handling.

---

The Viable Options

Option 1 — MongoDB GridFS (RECOMMENDED)

GridFS exists specifically for this problem.

How it works

• Large payload is split into chunks (default 255 KB)

• Stored in two collections:

  • fs.files (metadata)
  • fs.chunks (binary chunks)

• Driver handles chunking transparently

Why this fits your system

✔ Handles arbitrarily large images
✔ No custom chunk logic
✔ Streaming reads & writes
✔ Well-tested, production-grade
✔ Supported by MongoDB drivers
✔ Keeps metadata queryable

How it fits your bucket model

You do not store the payload in the bucket document.

Instead:

{
  "_id": "...",
  "pv": "CAM:01:FRAME",
  "start_time_ns": 123456,
  "sample_period_ns": 33333333,
  "sample_count": 1,
  "dimensions": [1080, 1920],
  "encoding": "proto:Image:v1",
  "storage": {
    "type": "gridfs",
    "file_id": ObjectId("...")
  }
}

Payload goes into GridFS as:

Image.serialize()

---

Option 2 — External Object Storage (S3 / MinIO / NFS)

Pattern

• Store large values outside MongoDB

• MongoDB stores:

  • URI
  • checksum
  • metadata

Pros

✔ Unlimited size
✔ Very cheap storage
✔ Object-storage optimized

Cons (for your case)

✖ New infrastructure dependency
✖ More failure modes
✖ Harder on-prem installs
✖ Higher read latency

This is often the “v2” evolution after GridFS.

---

Option 3 — Manual Chunking in MongoDB (NOT recommended)

You manually split the payload into N documents.

Why this is a bad idea

✖ You re-implement GridFS poorly
✖ Hard to guarantee atomicity
✖ Hard to GC
✖ Hard to stream
✖ Hard to optimize later

This is almost always regretted.

---

Recommended Architecture (Clear Winner)

Use GridFS for any payload that might exceed ~8–10 MB

You can even make this dynamic:

if serialized_payload_size < 8MB:
    store inline in bucket document
else:
    store in GridFS

---

How this fits your API evolution

In your SerializedDataColumn

message SerializedDataColumn {
  string name = 1;
  string encoding = 2;

  oneof storage {
    bytes inline_payload = 10;
    GridFsRef gridfs_ref = 11;
  }
}

message GridFsRef {
  string file_id = 1;
  int64 size_bytes = 2;
  string checksum = 3;
}

This keeps:

• API explicit
• Storage choice transparent
• Backward compatibility possible

---

Ingestion flow (step-by-step)

1. Ingestion service serializes payload

2. Checks payload size

3. If small → inline BSON

4. If large:

   • Write to GridFS (streaming)
   • Get file_id
   • Store reference in bucket doc

5. Commit bucket metadata

This can be done within a MongoDB session if you want stronger guarantees.

---

Query & Subscription Impact

✔ Subscribers don’t need to care
✔ Query service fetches bucket metadata
✔ If payload is GridFS:

• Stream payload to client
• No memory blowups
  ✔ Works well with gRPC streaming

---

Important Guardrails

You should enforce:

• Max inline payload size (e.g., 8 MB)
• Max image dimensions
• Max bucket size per PV
• Per-PV policy (e.g., cameras always GridFS)

---

Summary Recommendation

Yes, you absolutely need chunking — but do not invent it yourself.

Use MongoDB GridFS for large atomic samples.

It fits:

• Your ingestion model
• Your performance needs
• Your desire to avoid heavy infrastructure
• Your future Kafka / CDC plans

---

If you want, next we can:

• Design the exact MongoDB collections
• Look at Java driver streaming patterns
• Discuss transactional safety (GridFS + bucket metadata)
• Decide the exact payload size cutoff
• Think about compression (before vs after GridFS)

This is a very solid architecture direction.

[craigmcchesney]

This is a crucial design decision, and I’m really glad you paused to ask it explicitly. The choice here has big implications for correctness, operability, security, and how painful this API is for clients.

I’ll give you a clear recommendation, explain why, and then outline a concrete ingestion-time flow that fits your architecture and constraints.

---

Short answer (strong recommendation)

The ingestion service—not the client—should be responsible for writing to GridFS.

Clients should:

• Always send data inline (or stream it via gRPC)
• Never write to MongoDB or GridFS directly
• Never construct a GridFsRef themselves

The ingestion service:

• Decides inline vs GridFS
• Performs the GridFS write
• Emits the GridFsRef internally
• Stores references in MongoDB

This is the correct design for your system.

---

Why client-side GridFS writes are a bad idea

Let’s look at the alternative you proposed and why it causes trouble.

❌ Option: client writes to GridFS and passes GridFsRef

This looks tempting at first, but it creates serious problems:

1️⃣ Breaks trust boundaries

• Clients now need MongoDB credentials
• Clients can write arbitrary data
• Hard to enforce quotas and policies
• Hard to audit

This is especially bad in multi-facility deployments.

---

2️⃣ Breaks atomicity

What happens if:

• Client writes blob to GridFS
• Client crashes before calling ingestDataStream
• Or ingestion request fails validation?

You now have orphaned GridFS blobs with no metadata and no cleanup path.

---

3️⃣ Makes retries dangerous

If the client retries ingestion:

• Does it re-upload?
• Does it reuse the same GridFS file?
• How does the ingestion service know?

This is a nightmare to reason about.

---

4️⃣ Forces MongoDB knowledge into clients

Now every client must:

• Know GridFS semantics
• Handle chunking
• Handle driver differences
• Handle failures

That’s exactly what you’re trying to avoid.

---

✅ Recommended model: ingestion-managed GridFS

Design principle

The ingestion API is the only way data enters the system.

Clients talk to ingestion over gRPC.
Ingestion owns persistence decisions.

---

How this works in practice (cleanly)

1️⃣ API-level contract

At the API level, clients do not need to know about GridFS at all.

Your SerializedDataColumn stays simple:

message SerializedDataColumn {
  string name = 1;
  string encoding = 2;

  // Client always sets this
  bytes inline_payload = 10;
}

(We’ll refine this slightly for streaming below.)

Clients:

• Serialize their column payload
• Put it in inline_payload
• Send it to ingestion

---

2️⃣ Ingestion service decision logic

In the ingestion service:

payload_size = inline_payload.length

if payload_size <= INLINE_THRESHOLD:
    store inline in bucket document
else:
    write payload to GridFS
    replace inline_payload with GridFsRef

This logic is:

• Centralized
• Consistent
• Policy-driven
• Changeable without touching clients

---

3️⃣ Internal representation (MongoDB side)

The ingestion service transforms the incoming column into a storage form.

Inline case:

{
  "pv": "CAM:01:FRAME",
  "type": "serialized",
  "encoding": "proto:ImageColumn:v1",
  "storage": {
    "type": "inline",
    "payload": BinData(...)
  }
}

GridFS case:

{
  "pv": "CAM:01:FRAME",
  "type": "serialized",
  "encoding": "proto:ImageColumn:v1",
  "storage": {
    "type": "gridfs",
    "file_id": ObjectId("..."),
    "size_bytes": 124416000,
    "checksum": "sha256:..."
  }
}

The client never sees this difference.

---

What about very large payloads?

You’re right to think ahead here: some payloads may be too large even to comfortably fit in a gRPC message.

There are two clean options.

---

Option A (simplest): enforce max inline payload

• Set a max inline payload size (e.g., 8–16 MB)
• Reject anything larger
• Require clients to bucket more finely (e.g., fewer samples per bucket)

This works surprisingly well in practice and keeps the API simple.

---

Option B (advanced, optional): streaming payloads

If you truly need very large atomic samples (e.g., multi-hundred-MB images):

You can extend the API with a streaming blob upload pattern.

Example

rpc ingestDataStream (stream IngestDataRequest) returns (IngestDataStreamResponse);

Where:

• First message contains metadata (PV, encoding, timestamps)
• Subsequent messages contain payload_chunk
• Ingestion service streams directly into GridFS

This:

• Avoids buffering large payloads in memory
• Preserves ingestion ownership
• Keeps clients simple

But I’d only add this when you actually need it.

---

What about GridFsRef in the proto?

I recommend:

Do not expose GridFsRef in the public ingestion API at all.

Instead:

• Keep it as an internal storage detail
• Or expose it only in query responses, not ingestion requests

If you do include it in proto definitions:

• Make it clear it’s output-only
• Never required on ingestion requests

---

Failure handling (important)

If GridFS write fails:

• Reject the ingestion frame
• Do not partially commit metadata
• Let client retry safely

If Mongo write fails after GridFS write:

• Delete GridFS file (best effort)
• Or rely on periodic orphan cleanup

Because ingestion owns both steps, this is manageable.

---

Summary (very clear)

✔ Clients should never write to GridFS
✔ Clients always send payload inline (or streamed via gRPC)
✔ Ingestion service decides inline vs GridFS
✔ Ingestion service performs GridFS writes
✔ GridFsRef is an internal detail (or query-time artifact)

This keeps:

• Your API clean
• Your trust boundary intact
• Your retries safe
• Your operations sane

---

Where we can go next

If you want, next we can:

• Design the exact Java GridFS write flow
• Decide inline size thresholds
• Add streaming upload semantics safely
• Discuss orphan cleanup strategies
• Talk about compression (before vs after GridFS)

You’re making very solid architectural choices — this one in particular will save you a lot of pain later.

[craigmcchesney]

This is a great catch, and you’re right to be confused — that’s on me for not drawing a bright enough line between API shape and storage shape.

Let me clear this up cleanly and decisively.

---

Short, direct answer

Yes. For the ingestion API, you should remove GridFsRef entirely.

GridFsRef should not appear in the public ingestion protobuf at all.

Clients should never see, construct, or care about GridFsRef.

---

The root of the confusion

We mixed two different concerns into one message:

1. Ingestion API contract (what clients send)
2. Persistence representation (how ingestion stores data)

Those must be separated, even if they look similar.

The SerializedDataColumn message I showed you earlier was really a storage-layer DTO, not an ingestion API DTO — and that distinction wasn’t made explicit enough.

You correctly noticed the mismatch.

---

Correct mental model (this is the key)

There are two different representations:

1️⃣ Public ingestion API (client → ingestion service)

This should be:

• Simple
• Stateless
• Free of storage concepts
• Mongo/GridFS unaware

2️⃣ Internal storage model (ingestion service → MongoDB/GridFS)

This can:

• Reference GridFS
• Contain file IDs
• Encode storage decisions
• Evolve independently

---

What the ingestion API should look like

✅ Correct SerializedDataColumn for ingestion

message SerializedDataColumn {
  string name = 1;

  // e.g. "proto:ImageColumn:v1", "proto:UserDefinedStruct:v3"
  string encoding = 2;

  // Client always sends data inline (or streamed)
  bytes payload = 3;
}

That’s it.

No oneof.
No GridFsRef.
No storage hints.

The rule is simple:

Clients always send payload bytes.
Ingestion decides how to persist them.

---

Where GridFsRef actually belongs

✅ Internal persistence model (NOT the API)

Inside the ingestion service, after validation:

if (payload.length <= INLINE_THRESHOLD) {
    storeInline(payload);
} else {
    ObjectId fileId = writeToGridFS(payload);
    storeReference(fileId, size, checksum);
}

And the MongoDB document might look like:

{
  "pv": "CAM:01:FRAME",
  "type": "serialized",
  "encoding": "proto:ImageColumn:v1",
  "storage": {
    "kind": "gridfs",
    "file_id": ObjectId("..."),
    "size_bytes": 124416000,
    "checksum": "sha256:..."
  }
}

This structure is never exposed to ingestion clients.

---

What about query / subscription APIs?

Important distinction:

• Ingestion API → no GridFsRef
• Query / retrieval API → may expose an abstract reference

Even there, I’d recommend:

• Streaming the payload back to clients
• Not exposing raw GridFS IDs unless absolutely necessary

If you do expose references later, make them logical, not Mongo-specific.

---

What if payloads are too large for a single gRPC message?

That’s a separate concern, and the solution is streaming, not GridFS exposure.

You have two clean options:

Option A — enforce max payload size (simplest)

• Reject payloads over N MB
• Require smaller buckets

Option B — streaming upload (advanced)

• First message = metadata
• Subsequent messages = payload chunks
• Ingestion streams chunks directly into GridFS

In both cases:

• Client still never sees GridFsRef

---

Why this separation really matters

Keeping GridFS out of the API gives you:

✔ Clean trust boundary
✔ Safe retries
✔ No orphaned blobs
✔ No MongoDB credentials in clients
✔ Freedom to change storage later
✔ Easier on-prem deployments

If you ever switch:

• GridFS → object storage
• Mongo → something else

Your ingestion API does not change.

That’s a huge win.

---

Final recommendation (very explicit)

Do this:

// Ingestion API
message SerializedDataColumn {
  string name = 1;
  string encoding = 2;
  bytes payload = 3;
}

Do NOT do this in the ingestion API:

• ❌ GridFsRef
• ❌ storage hints
• ❌ Mongo-specific concepts

Keep GridFsRef:

• internal only, or
• at most in internal persistence DTOs

---

You were absolutely right to question this — and the fact that you did is a sign you’re designing this at the right level of rigor.

If you want, next we can:

• Redesign the proto files cleanly end-to-end (ingest vs query vs internal)
• Talk about streaming payload uploads safely
• Define internal Java models vs protobuf DTOs
• Walk through failure scenarios with GridFS writes

Just tell me where you want to go next.