design: unbounded storage backend for gantry - #330
design: unbounded storage backend for gantry#330Vaibhav Patel (vpatelsj) wants to merge 2 commits into
Conversation
|
|
||
| ## Approach | ||
|
|
||
| Introduce a **priority chain** of cache origins that gantry consults before falling |
There was a problem hiding this comment.
How would unbounded-storage's cache get filled in this case? It needs an upstream backend to fill from.
Maybe unbounded storage should implement an OCI backend that knows how to pull from registries. Or maybe Gantry exposes an HTTP endpoint on the loopback for storage to pull from on cache miss.
There was a problem hiding this comment.
For this initial proposal, I am treating unbounded-storage as a readonly cache ie. if the image is present in unbounded-storage then use it but if not then just pull it from internet.
There was a problem hiding this comment.
But how will they get into unbounded-storage in that case?
Users don't write to unbounded storage - they read from a configured frontend, and it pulls from the corresponding backend impl if the value isn't in cache. So it's always a pull through cache. In order for Gantry to use unbounded-storage it needs both a way to pull values from unbounded AND the ability for unboudned to fetch those values on misses.
There was a problem hiding this comment.
Can you show me an example of how it is done today?
There was a problem hiding this comment.
This integration test is a pretty good example: https://github.com/Azure/unbounded/blob/main/internal/orca/inttest/storageboundary_test.go
It spins up a chain of unbounded-storage -> Orca -> Garage (for fake S3 origin), then downloads some blobs through the chain by hitting the unbounded-storage S3 frontend.
|
|
||
| ## Non-Goals | ||
|
|
||
| - Writing blobs into unbounded-storage. Gantry only reads from it; cache population |
There was a problem hiding this comment.
IMO this one defeats the purpose of the integration. The cache should be transparent to users
There was a problem hiding this comment.
Cache is indeed transparent to users in my mind too. For gantry, transparency is that users specify a container image in pod definition. Now whether the container image is pulled from origin from internet, or a gantry peer or unbounded storage is not visible to user nor their concern. Hydrating the unbounded-storage on cache miss is not a goal for this initial proposal. We can definitely that functionality add that later on and it will still remain transparent to user. Gantry will do the hydration behind the scenes.
| is the operator's concern. | ||
| - OCI Distribution Spec support in unbounded-storage. It is treated as a plain HTTP | ||
| cache keyed by URL path. | ||
| - Circuit breaker or availability tracking in v1. Fall-through on every failure is |
There was a problem hiding this comment.
How important is it to support multiple origins? I'm trying to understand the use case. It would be much simpler to just support one: no need to think about circuit breakers
There was a problem hiding this comment.
Currently, I am positioning gantry as a component that can also run on non-unbounded clusters and in that spirit that shouldnt be necessary need unbounded-storage but could use it if its available.
There was a problem hiding this comment.
I understand needing swappable origins, my question is about why you are proposing support for multiple chained origins. It would be simpler to fall through to a single configured origin, and I can't think of a use case where multiple are needed (maybe I'm missing something there)
There was a problem hiding this comment.
Let's say a container image acr.foo.bar:latest manifest requires a blob with sha1234 needs to be pulled on a node A. This blob is not present on the node but it could be at multiple locations:
1). it can be on the another node's containerd cache.
2). it can be on unbounded storage - chunked across various nodes.
3). it can be on a harbor registry that is local to the datacenter or cluster
4). it can be on acr.foo.bar over the internet.
5). it can be just not present at all.
A single configured origin ie. any one option of above doesnt make sense to me. Chaining helps in hydrating the cache as well. Gantry can detect that its mirror failed across all the caches and went all the way to internet to pull the blob but now it push the downloaded to harbor or unbounded storage etc..
There was a problem hiding this comment.
(1) and (2) should be the only cases where it's possible to fall through e.g. cache misses or unavailability. But for 3-5 we should be able to try exactly one upstream server: local harbor would have a unique prefix in the image tag, same for public registries.
Conceptually, I think it makes sense to decouple the concept of pull through caching from the concept of origins/upstreams. Configure any trusted origins, then configure the cache layer as a separate concern. Every image resolves back to a single origin, and is cached by gantry's p2p and/or the configured external cache e.g. unbounded storage.
2 participants
No description provided.