benchmarks: durdir benchmark that evaluates the efficacy and performance of DurDir - #907
Conversation
Max Smythe (maxsmythe)
left a comment
There was a problem hiding this comment.
Thank you for this.
Left some comments.
Also, we should avoid sending 4k line PRs where possible -- they take multiple hours to review closely.
Can we avoid refactoring anything under internal/benchmarking/boomer for now? I don't think the abstractions are quite right and the refactoring is a large portion of the delta. We can implement ~3-4 tests and then figure out what the best refactor is.
| help="Size in bytes written to /var/lib/glutton/bench-data (default 8 MiB)", | ||
| include_in_web_ui=True, | ||
| ) | ||
| parser.add_argument( |
There was a problem hiding this comment.
If this applies to any workload, it should probably live in its own Python file
There was a problem hiding this comment.
Split out. --resume-mode now lives in common/resume_mode.py
| # if those disagree the stats rows describe traffic that was never sent. | ||
| # The value must name a stub in locust/tests/, a --workload case in | ||
| # cmd/benchmarking/boomer-glutton/main.go, and an entry in runner.py's | ||
| # BOOMER_WORKLOADS dict. |
There was a problem hiding this comment.
TODO: look to reduce # of sources of truth
There was a problem hiding this comment.
Addressed
| return args | ||
|
|
||
|
|
||
| def boomer_workload(test_file: str) -> str | None: |
There was a problem hiding this comment.
It would be better to test for Python workloads... I suspect we will no longer be adding those, so it will be a static list. A cleanup work item will be to remove those tests.
This will also remove one place workload lists need to be kept in alignment
There was a problem hiding this comment.
Went with a single static map from stub filename to user class:
BOOMER_USER_CLASSES = {"glutton.py": "GluttonUser", "durdir.py": "DurDirUser"}
needs_boomer is membership in it and the same entry supplies --user-class, so the filename and the class name stay in one place rather than two.
Kept the check on the boomer side rather than the Python side because runner.py needs the class name string either way
| python3 runner.py -f tests/durdir.py -t 1m -u 1 --name durdir_run --dest /tmp/bench \ | ||
| --durdir-template glutton-durdir-data --resume-mode explicit --durdir-file-size-bytes 8388608 | ||
| ``` | ||
| * **Interactive Web UI**: The stack runs one workload per deployment, chosen at deploy time: |
There was a problem hiding this comment.
I don't think "workload" is correct... that is the name given to the ActorTemplate and workers.
One "test type"? "Virtual user type"?
There was a problem hiding this comment.
UserClass?
There was a problem hiding this comment.
Fixed.
| * `glutton-durdir-data` (default): Attaches a durable data directory under `/var/lib/glutton` without memory snapshot restore. | ||
| * `glutton-durdir-full`: Attaches a durable data directory and performs a full memory snapshot restore. | ||
|
|
||
| #### File-Size Sweeps |
There was a problem hiding this comment.
I don't think we need to document that we can run multiple tests by adding multiple tests. Also, I think setting the read mode to digest would be orthogonal to "sweep".
I'd remove this section.
There was a problem hiding this comment.
Removed.
| cfg *Config | ||
| actorName string | ||
| hostHeader string | ||
| userClass string |
There was a problem hiding this comment.
I like this idea, though we will probably need some design/refactoring to make this truly generic across all users. We'll tackle that more on a later test.
|
|
||
| // resume brings the actor up before the next request. | ||
| // | ||
| // The first resume is always explicit, whatever the mode. It is the only |
There was a problem hiding this comment.
This is far too much context for a comment. Documents the code and likely to rot quickly.
There was a problem hiding this comment.
Addressed
| user *gluttonUser | ||
| expectedDigest string | ||
| fileSize int64 | ||
| readMode gluttonpb.ReadMode |
There was a problem hiding this comment.
Things that are stored as dynamic config should not be persisted to the class, they should be read from the dynamic config, so behavior can change dynamically (might need to have some read atomicity to avoid breaking behavior due to time aliasing)
There was a problem hiding this comment.
Fixed, including the aliasing half.
| slog.Info("configured durdir file size", slog.Int64("bytes", fileSize), slog.String("template", tmpl)) | ||
|
|
||
| if readMode == gluttonpb.ReadMode_READ_MODE_DATA && fileSize > 64*1024*1024 { | ||
| slog.Warn("durdir_file_size_bytes > 64MB with durdir_read_mode=data may hit router or Envoy timeouts; set durdir_read_mode=digest", |
There was a problem hiding this comment.
No need to warn about this... if we are hitting timeouts that would be the point of the benchmark, to find out.
There was a problem hiding this comment.
Addressed
| # its own per-worker diagnostics at /metrics on :8001 (aggregate stats | ||
| # flow through the master via boomer.RecordSuccess). | ||
| # | ||
| # ${BENCHMARK_WORKLOAD} is substituted by benchmarking/locust/deploy.sh |
There was a problem hiding this comment.
Comment describes too many internal details
There was a problem hiding this comment.
Addressed
|
let's squash this on merge (and more PRs, generally, unless there's really useful commit history) |
b7e91e5 to
98916bc
Compare
Max Smythe (maxsmythe)
left a comment
There was a problem hiding this comment.
Thanks! Couple of comments
| _FLAGS = ("--trace-probability", "--min-wait-time", "--max-wait-time") | ||
| # Boomer-tunable flags and their types. CLI form ("--foo-bar") is converted | ||
| # to the attribute / JSON-key form ("foo_bar") by _attr(). | ||
| _FLAGS = { |
There was a problem hiding this comment.
Not a thing for this review, but probably the correct way to do this is to add each flag to a unified registry upon import.
Again, not a thing for this PR but a cleanup for later to help remove duplicate sources of truth.
| echo " -h|--help Show this help message" | ||
| echo " --deploy Deploy the locust workers" | ||
| echo " --delete Delete the locust workers" | ||
| echo " --user-class NAME User class to deploy: GluttonUser | DurDirUser (default: GluttonUser)" |
There was a problem hiding this comment.
let's not use case-sensitive arguments (just do lowercase). Also, we can name the class after the file.
Let's assume a convention of /tests/{user_class}.py (e.g. GluttonUser becomes glutton).
That also removes the need to do mapping and argument correctness check is just looking for the existence of the correct file.
There was a problem hiding this comment.
Adopted. --user-class is lowercase and case-insensitive, the file is
tests/<value>.py, and the class is <Value>User — so glutton ->
GluttonUser, durdir -> DurdirUser. Renamed the Python class to match the
convention rather than renaming the file.
deploy.sh validation is now just a file-existence check, BENCHMARK_USER_FILE
is gone from the manifest, and runner.py's filename->class map is gone
entirely — the flag value is the test file's stem.
|
|
||
|
|
||
| # Maps locust test filename to boomer's --user-class flag. | ||
| BOOMER_USER_CLASSES: dict[str, str] = { |
There was a problem hiding this comment.
Adopting the filename == user + .py convention also removes the need for this code.
There was a problem hiding this comment.
addressed
| """Return True if the test file is the glutton stub; the real GluttonUser | ||
| implementation lives in the boomer-glutton binary.""" | ||
| return os.path.basename(test_file) == "glutton.py" | ||
| """Return True if the test file's User class runs on boomer.""" |
There was a problem hiding this comment.
Per previous review, let's flip this to test whether the user class needs Python. membership of that set should be static, reducing brittleness.
There was a problem hiding this comment.
Flipped. PYTHON_TESTS is the static set and anything not in it routes to
boomer.
| ### DurDir Benchmark | ||
|
|
||
| The DurDir benchmark evaluates actor suspend/resume performance, disk persistence overhead, | ||
| and state restoration latency when durable actor directories (`/var/lib/glutton`) are attached. |
There was a problem hiding this comment.
nit: remove reference to "var/lib/glutton" -- too much implementation detail.
There was a problem hiding this comment.
Done
| // docs/dev/best-practices/tracing.md: extract incoming context, | ||
| // then name the span after the operation in each handler. | ||
| handler = otelhttp.NewHandler(mux, "/") | ||
| handler = otelhttp.NewHandler(newMux(svc), "glutton-http") |
There was a problem hiding this comment.
Why are we renaming "/" "glutton-http"?
There was a problem hiding this comment.
Not intentional, leftover from when the mux had a single route. Reverted to
"/", which is what docs/dev/best-practices/tracing.md prescribes: the root
mux gets "/" and the operation name comes from the per-handler span. That's
already what protoRoute does, so Ping/WriteDisk/ReadDisk show up as
children.
| if _, err := f.Seek(0, io.SeekStart); err != nil { | ||
| return nil, status.Errorf(codes.Internal, "seek %s: %v", path, err) | ||
| } | ||
| h := sha256.New() |
There was a problem hiding this comment.
I'm not sure we want to always pair reads with writes.
It is necessary if we want the full SHA for a file that was only partially overwritten, otherwise, can we calculate SHA as data is written to disk?
that way people can benchmark pure writes.
There was a problem hiding this comment.
Good Catch. Fixed by branching on the write mode. Under TRUNCATE the file is exactly N
bytes by construction, so the digest streams through an io.MultiWriter during
the write and there is no read-back at all. OVERWRITE is the only mode that
can leave bytes past N from a larger earlier write, so that branch keeps the
re-hash. Both existing tests pin the two paths, unchanged.
| resumeBoots []bool | ||
| } | ||
|
|
||
| func (f *fakeControlClient) CreateAtespace(ctx context.Context, in *ateapipb.CreateAtespaceRequest, opts ...grpc.CallOption) (*ateapipb.Atespace, error) { |
There was a problem hiding this comment.
Is there no other place where API server has been faked, where we can use that? Can we centralize this so only one fake needs to be created?
There was a problem hiding this comment.
Went looking — five packages already do this, and every one is test-local, so
none of them is importable:
cmd/atecontroller/internal/controllers/actortemplate_controller_test.go:98cmd/atenet/internal/router/health_test.go:42cmd/atenet/internal/router/ingress/ingress_test.go:40cmd/atenet/internal/router/ingress/resumer_test.go:33cmd/atenet/internal/router/egress/egress_test.go:198
(internal/testenv is envtest running a real Kubernetes apiserver, so it's a
different thing.)
So the duplication is real, but centralizing means a new non-test package plus
migrating five call sites across atecontroller and atenet. Happy to take it as a
follow-up.
There was a problem hiding this comment.
Let's file it as an issue.
| // The data slice is the source of truth: both routes report len(data) and | ||
| // sha256(data), and /readdisk serves data as payload. | ||
| // Each override field makes the actor lie about exactly one property. | ||
| type diskServer struct { |
There was a problem hiding this comment.
We should just create a mock glutton server as part of glutton, vs. relying on each test to make its own mock.
bonus... can probably do this by just creating shims in actual server code for anything that has a side effect (like writing to disk), then use dependency injection to differentiate between a real and fake server.
There was a problem hiding this comment.
main.go:18 is package main, so the service isn't importable, a shared mock
means moving those 698 lines into a real package first.
The in-memory store would also have to model on-disk residue (main.go:457, the
OVERWRITE branch) or it goes green while the server is wrong, the bug from
the thread above. glutton's own tests use t.TempDir() for that reason.
Narrower point: the override fields at fixture_test.go:99-106 exist to make the
actor lie so the client's verification path can be proven to catch it, so some
lying variant stays here regardless.
Follow-up alongside the apiserver fake, one PR or two?
There was a problem hiding this comment.
Lets move the fake to internal/benchmarking/glutton/fake
We should factor glutton's core logic from main.go into internal/benchmarking/glutton/..., but that can be done later.
Override fields on fakes can be public, no need to localize that code (indeed, the point of centralized fakes is to make it easy to override behavior in multiple places).
Also, to be sure, there is no need to write tests for the test fixture, so this should minimally impact line count.
There was a problem hiding this comment.
Moved to internal/benchmarking/glutton/fake; diskServer → fake.Server with the
override fields exported. No tests for the fixture.
One deliberate duplication: the fake declares its own WriteDiskRoute/ReadDiskRoute
instead of importing the client's (durdir.go:54-55), so it depends on nothing. Both
collapse onto one definition when glutton's core moves into internal/benchmarking/glutton/....
| WRITE_MODE_OVERWRITE = 1; | ||
| } | ||
|
|
||
| // ReadMode selects how much of the file ReadDisk sends back. Digest-only |
There was a problem hiding this comment.
Shorten this doc string, assume a user knows why they would want functionality, they just need to know what a thing does.
There was a problem hiding this comment.
Done
a17cb0f to
8f3d41d
Compare
Max Smythe (maxsmythe)
left a comment
There was a problem hiding this comment.
Getting close, a few more comments to sort out, then the tests are left for reviewing.
| `--dest`. This is what the nightly automation uses. | ||
|
|
||
| ```bash | ||
| python3 runner.py -f tests/<name>.py -t 1m -u 1 --name <run-name> --dest /tmp/bench |
There was a problem hiding this comment.
name => user class
There was a problem hiding this comment.
Done
| ) | ||
|
|
||
| switch class { | ||
| case "glutton": |
There was a problem hiding this comment.
General pattern for things like this (unknown set of extensible dependencies) is to declare the linkage next to the code that implements it and add it to a registry. See the type code for KRM objects:
substrate/pkg/api/v1alpha1/actortemplate_types.go
Lines 433 to 435 in 8541d6d
in our case, something like:
registry.go
type Entry struct {
name string
locustFile string
userClass string
initFunc func(Config) (func(), func())
}
var reg map[string]entry
func Add(entry Entry) {
reg[name] := entry{locustFile: locustFile, userClass: userClass, initFunc: initFunc}
}
func Init(name string, cfg Config) (func(), Func()) {
return reg[name].initFunc(cfg)
}
// etc...userclass.go
import registry // if not in same package (which it normally wouldn't be)
entry := registry.Entry{
name: "something",
locustFile: "something.py",
userClass: "SomethingUser",
initFunc: initFunc,
}
registry.Add(entry)etc.
Registry class should live inside internal/benchmarking directory tree.
This will make it so newer entries do not need to update main.go
I realize that this will require a (hopefully minor) update to the glutton user class file.
Also, the term "register" is now overloaded... original usage of the term should probably be updated to "initBoomer" etc. anyway... "register" is a misnomer.
There was a problem hiding this comment.
Done, as internal/benchmarking/boomer/userclass/. Entry{Name, LocustFile, UserClass, Init}, with Add/Lookup/Names; Add panics on a duplicate name.
Both classes register from func init() next to their implementation - lifecycle.go:62 and durdir.go:62 - matching the SchemeBuilder.Register pattern you linked. main.go no longer has a per-class branch; adding a class touches only its own file.
Also renamed the overloaded term: RegisterPing is now initPing.
| mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) { | ||
| w.WriteHeader(http.StatusOK) | ||
| }) | ||
| mux := readyzMux() |
There was a problem hiding this comment.
this mux is only used in the gRPC case and should only be defined there.
There was a problem hiding this comment.
Split into two constructors, each called at its use site. readyzMux() is just the probe, and gRPC mode passes it to splitGRPC as the non-gRPC handler (main.go:135). HTTP mode calls newMux(svc) (:184), which layers the three proto routes on top of the same probe. Nothing is built for a mode that doesn't use it.
| // OVERWRITE leaves any bytes past size from a larger earlier write, so the | ||
| // streamed digest covers only a prefix. Re-hash from disk to get the real one. | ||
| if req.GetWriteMode() == glutton.WriteMode_WRITE_MODE_OVERWRITE { | ||
| if _, err := f.Seek(0, io.SeekStart); err != nil { |
There was a problem hiding this comment.
Is it not possible to continue reading from the pre-existing cursor location until the end of the file, continuing to compute the SHA?
We should avoid double-reading. I believe I had suggested this in a previous comment.
There was a problem hiding this comment.
Added, no re-read of the prefix in either mode. The digest streams through io.MultiWriter(f, h) as the bytes are generated (main.go:451). TRUNCATE is complete at that point. OVERWRITE is the only mode that can leave a tail from a larger earlier write, and the cursor is already sitting at size, so it just continues io.Copy(h, f) to EOF and folds that into the same digest. The response also returns the real on-disk size rather than the requested one, so it lines up with ReadDisk.
| func (u *durDirUser) bootstrap(ctx context.Context, dynCfg dynconfig.Config) error { | ||
| fileSize, readMode := u.params(dynCfg) | ||
|
|
||
| // First resume: always explicit to warm up actor from golden snapshot |
There was a problem hiding this comment.
I'm not sure explicit resume is necessary on first boot as long as actor exists?
There was a problem hiding this comment.
Bootstrap now uses the configured ResumeMode like every other iteration, so implicit mode issues no ResumeActor at all; the forced-explicit special case and its test are deleted.
8f3d41d to
5830eb0
Compare
Max Smythe (maxsmythe)
left a comment
There was a problem hiding this comment.
LGTM after two nits.
Thank you for the PR!
| return nil, status.Errorf(codes.Internal, "write %s: %v", path, err) | ||
| } | ||
|
|
||
| // OVERWRITE has no O_TRUNC, so bytes past size can survive a larger earlier |
There was a problem hiding this comment.
nit:
"bytes past size can survive a larger earlier write" -> "bytes from a larger, earlier write will persist"
There was a problem hiding this comment.
Addressed
|
|
||
| ### Headless | ||
|
|
||
| `runner.py` runs a test without the web UI, writing CSVs, logs and traces to |
There was a problem hiding this comment.
nit: clarify that this is a command that is invoked by automation as a job on workload cluster. It is not meant to be run on a local machine to run tests in a headless mode.
Parallel documentation with deploy.sh makes it seem like the two approaches have equivalent UX.
There was a problem hiding this comment.
Addressed
DeleteActor requires SUSPENDED or CRASHED; the bootstrap-failure path deleted a running actor, leaking it and eventually exhausting workers. Observed in the microvm sweep as 10× not in a deletable status (status: STATUS_RUNNING) followed by no free workers available.
Move the httptest-backed fake glutton server out of boomer/glutton into internal/benchmarking/glutton/fake so it can be shared by other benchmark tests. The WriteDiskRoute and ReadDiskRoute constants are duplicated in the fake package deliberately so it has no dependency on the boomer client; both sides will collapse onto a single definition when glutton's core is factored out.
b092ce2 to
2249813
Compare
|
LGTM, thank you for all the work! |
ba45517
into
agent-substrate:main
Benchmark DurDir
Fixes #673
Adds a
DurDirUserload-generation workload that exercises the DurableDirsuspend/resume loop end to end, verifies every served byte against a SHA-256
digest, and emits separable latency percentiles for each step.
The loop
Under
onCommit: Datathe container cold-boots from the OCI image and processmemory is discarded, so a matching digest after resume can only have come from
the restored DurableDir. That is the durability assertion.
Results
All six scenarios ran on GKE/gvisor at 1 VU for 1m each:
DatavsFullsnapshot scope, explicit vs implicit resume, and a 5/10/64 MiB size sweep.
Zero failures across every run, so every served byte matched its digest in
all six.
Snapshot growth over repeated overwrites:
SuspendActorlatency stayed flatacross consecutive overwrite-and-suspend cycles on the same DurableDir volume.
The loop overwrites with
WRITE_MODE_TRUNCATE, so the file is exactly X bytesafter every write and the captured directory contents are the same size every
cycle. Nothing accumulates across suspends. That is the growth question the
issue asks about.
Latency percentiles per step are in the run artifacts.
Change surface
WriteDiskreturns size + sha256; newReadDiskwith aREAD_MODE_DIGEST_ONLYmode for measuring restore cost without paying wiretransfer; disk RPCs exposed over HTTP mode.
glutton-durdir-{data,full}, with adurableDirvolume andonPause: Full/onCommit: {Data,Full}.a
DurDirUsertask on top of it; a generalresume_modeknob.--workloadselects the task at deploy time;durdir.pystub,typed dynconfig flags, six nightly scenarios.
The two boomer commits above are incremental extractions made as the second
workload landed. The final package layout for
internal/benchmarking/boomerlands as a follow-up PR.
Testing
go build ./...,go test -race ./cmd/benchmarking/... ./internal/benchmarking/...clean, no race warnings.
hack/verify-all.sh: all nine checks pass. Python protos regenerated, treeclean.
Ready; golden snapshots confirmed in the bucket.re-read with matching sha256 across every read. Process memory discarded and
container cold-booted in between.
failures.
glutton_baseline_5_usersran 919 requests with 0failures and 7 ms ping latency post-rebase. No regression on the existing
benchmark.
Deliberately out of scope
ActorSnapshothas no size field, so there isnothing for a client to read. The issue permits deferring this.
SuspendActorlatency is the available proxy;
atelet.snapshot.sizeis the server-side one.reviewable in place.
WriteRAM; small follow-up.