feat(file-service): add model upload limits and the models_enabled flag - #8000
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8000 +/- ##
=========================================
Coverage 93.42% 93.42%
- Complexity 4703 4706 +3
=========================================
Files 1180 1181 +1
Lines 47766 47787 +21
Branches 5321 5322 +1
=========================================
+ Hits 44623 44647 +24
+ Misses 1699 1697 -2
+ Partials 1444 1443 -1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 379 | 0.231 | 25,208/39,244/39,244 us | 🔴 +24.0% / 🔴 +180.1% |
| 🟢 | bs=100 sw=10 sl=64 | 951 | 0.581 | 104,363/124,466/124,466 us | 🟢 -6.6% / 🔴 +30.9% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,086 | 0.663 | 922,409/997,301/997,301 us | ⚪ within ±5% / 🔴 +8.7% |
Baseline details
Latest main 646e460 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 379 tuples/sec | 444 tuples/sec | 893.99 tuples/sec | -14.6% | -57.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.231 MB/s | 0.271 MB/s | 0.546 MB/s | -14.8% | -57.7% |
| bs=10 sw=10 sl=64 | p50 | 25,208 us | 20,322 us | 11,176 us | +24.0% | +125.5% |
| bs=10 sw=10 sl=64 | p95 | 39,244 us | 31,641 us | 14,009 us | +24.0% | +180.1% |
| bs=10 sw=10 sl=64 | p99 | 39,244 us | 31,641 us | 16,898 us | +24.0% | +132.2% |
| bs=100 sw=10 sl=64 | throughput | 951 tuples/sec | 931 tuples/sec | 1,157 tuples/sec | +2.1% | -17.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.581 MB/s | 0.568 MB/s | 0.706 MB/s | +2.3% | -17.8% |
| bs=100 sw=10 sl=64 | p50 | 104,363 us | 108,327 us | 88,763 us | -3.7% | +17.6% |
| bs=100 sw=10 sl=64 | p95 | 124,466 us | 133,236 us | 95,096 us | -6.6% | +30.9% |
| bs=100 sw=10 sl=64 | p99 | 124,466 us | 133,236 us | 102,854 us | -6.6% | +21.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,086 tuples/sec | 1,089 tuples/sec | 1,187 tuples/sec | -0.3% | -8.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.663 MB/s | 0.665 MB/s | 0.724 MB/s | -0.3% | -8.5% |
| bs=1000 sw=10 sl=64 | p50 | 922,409 us | 915,733 us | 874,968 us | +0.7% | +5.4% |
| bs=1000 sw=10 sl=64 | p95 | 997,301 us | 1,015,283 us | 917,815 us | -1.8% | +8.7% |
| bs=1000 sw=10 sl=64 | p99 | 997,301 us | 1,015,283 us | 949,868 us | -1.8% | +5.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,527.74,200,128000,379,0.231,25207.62,39243.89,39243.89
1,100,10,64,20,2102.39,2000,1280000,951,0.581,104362.56,124465.94,124465.94
2,1000,10,64,20,18419.04,20000,12800000,1086,0.663,922408.94,997300.72,997300.72|
/request-review @aicam |
aicam
left a comment
There was a problem hiding this comment.
Add enabling model flag to this PR
aicam
left a comment
There was a problem hiding this comment.
Please follow the naming convention we have.
87008c1 to
a6565fb
Compare
What changes were proposed in this PR?
gui.tabs.models_enabled, defaultfalse— the flag the model UI will gate on. Nothing reads it yet; it hides the sidebar entry, not the route (same asdatasets_enabled).single_file_upload_max_size_mibkey literally, so both types resolved the same site setting.UploadLimitsdescriptor — ties a resource type to its upload keys and exposessingleFileUploadMaxBytes.ResourceStoragegains anuploadLimitsslot, so both call sites read the limit through the descriptor instead of naming a key. The same helper also existed inDatasetResourceandModelResource, where it was dead code; both copies removed.Naming (review note).
site_settingsrows are keyed by a leaf's last path segment, sodataset { }/model { }are flattened away before storage and two bare leaves would collide —DefaultsConfig's uniquenessrequirefails at load. So the model leaves need a prefix, and per review the dataset leaves got one too:Clean rename, no alias. Also fixes
MAX_NUMBER_OF_CONCURRENT_UPLOADING_FILE->DATASET_MAX_....That rename is why this touches the frontend and ships a migration:
admin-settings,files-uploaderanddataset-detail.sql/updates/42.sqlrenames the four existingsite_settingsrows — otherwise the seeder recreates them at defaults and an admin's raised limit is silently lost. Idempotent.ConfigResourcederives its public whitelist fromgui/dataset/model, which is why the pinned set inConfigResourceSpecchanged.Any related issues, documentation, discussions?
Backend prerequisite for the model dashboard UI — #6499 and #6501.
How was this PR tested?
New
UploadLimitsSpec(5 tests, no DB harness): pins each descriptor's keys againstdefault.conf, asserts the dataset and model key sets are disjoint, and asserts the model ceiling resolves to 2 GiB.42.sqlrun against a scratch Postgres: values preserved, unrelated rows untouched, second run a no-op, no PK violation when both old and new rows exist.Existing suites pass unmodified except the two pinned config assertions:
DefaultsConfigSpec5,ConfigResourceSpec33,UploadLimitsSpec5,DatasetResourceSpec152,ModelUploadResourceSpec13,ModelResourceSpec22,admin-settings45,dataset-detail203,files-uploader30. scalafmt + prettier clean.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)