Skip to content

feat(file-service): add model upload limits and the models_enabled flag - #8000

Merged
aicam merged 3 commits into
apache:mainfrom
tanishqgandhi1908:refactor/resource-upload-limits
Aug 27, 2026
Merged

feat(file-service): add model upload limits and the models_enabled flag#8000
aicam merged 3 commits into
apache:mainfrom
tanishqgandhi1908:refactor/resource-upload-limits

Conversation

@tanishqgandhi1908

@tanishqgandhi1908 tanishqgandhi1908 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

  1. gui.tabs.models_enabled, default false — the flag the model UI will gate on. Nothing reads it yet; it hides the sidebar entry, not the route (same as datasets_enabled).
  2. Models get their own upload ceiling: 2 GiB, up from the 20 MiB they inherited from datasets. The shared upload engine read the dataset's single_file_upload_max_size_mib key literally, so both types resolved the same site setting.
  3. An UploadLimits descriptor — ties a resource type to its upload keys and exposes singleFileUploadMaxBytes. ResourceStorage gains an uploadLimits slot, so both call sites read the limit through the descriptor instead of naming a key. The same helper also existed in DatasetResource and ModelResource, where it was dead code; both copies removed.

Naming (review note). site_settings rows are keyed by a leaf's last path segment, so dataset { } / model { } are flattened away before storage and two bare leaves would collide — DefaultsConfig's uniqueness require fails at load. So the model leaves need a prefix, and per review the dataset leaves got one too:

dataset { dataset_single_file_upload_max_size_mib = 20   }
model   { model_single_file_upload_max_size_mib   = 2048 }

Clean rename, no alias. Also fixes MAX_NUMBER_OF_CONCURRENT_UPLOADING_FILE -> DATASET_MAX_....

That rename is why this touches the frontend and ships a migration:

  • The key strings are hardcoded in admin-settings, files-uploader and dataset-detail.
  • sql/updates/42.sql renames the four existing site_settings rows — otherwise the seeder recreates them at defaults and an admin's raised limit is silently lost. Idempotent.

ConfigResource derives its public whitelist from gui/dataset/model, which is why the pinned set in ConfigResourceSpec changed.

Any related issues, documentation, discussions?

Backend prerequisite for the model dashboard UI — #6499 and #6501.

How was this PR tested?

New UploadLimitsSpec (5 tests, no DB harness): pins each descriptor's keys against default.conf, asserts the dataset and model key sets are disjoint, and asserts the model ceiling resolves to 2 GiB.

42.sql run against a scratch Postgres: values preserved, unrelated rows untouched, second run a no-op, no PK violation when both old and new rows exist.

Existing suites pass unmodified except the two pinned config assertions:
DefaultsConfigSpec 5, ConfigResourceSpec 33, UploadLimitsSpec 5, DatasetResourceSpec 152, ModelUploadResourceSpec 13, ModelResourceSpec 22, admin-settings 45, dataset-detail 203, files-uploader 30. scalafmt + prettier clean.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

@github-actions github-actions Bot added refactor Refactor the code common platform Non-amber Scala service paths labels Aug 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @zyratlo, @Neilk1021
    You can notify them by mentioning @Yicong-Huang, @zyratlo, @Neilk1021 in a comment.

@codecov-commenter

codecov-commenter commented Aug 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.42%. Comparing base (646e460) to head (5e38344).

Files with missing lines Patch % Lines
.../apache/texera/service/resource/UploadLimits.scala 91.66% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8000   +/-   ##
=========================================
  Coverage     93.42%   93.42%           
- Complexity     4703     4706    +3     
=========================================
  Files          1180     1181    +1     
  Lines         47766    47787   +21     
  Branches       5321     5322    +1     
=========================================
+ Hits          44623    44647   +24     
+ Misses         1699     1697    -2     
+ Partials       1444     1443    -1     
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø)
agent-service 99.32% <ø> (ø) Carriedforward from a6565fb
amber 89.58% <ø> (+<0.01%) ⬆️
computing-unit-managing-service 73.67% <ø> (ø)
config-service 86.73% <100.00%> (ø)
file-service 87.89% <92.59%> (+0.42%) ⬆️
frontend 95.89% <100.00%> (ø)
notebook-migration-service 79.31% <ø> (ø)
pyamber 97.78% <ø> (ø) Carriedforward from a6565fb
workflow-compiling-service 77.19% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 646e460 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 379 0.231 25,208/39,244/39,244 us 🔴 +24.0% / 🔴 +180.1%
🟢 bs=100 sw=10 sl=64 951 0.581 104,363/124,466/124,466 us 🟢 -6.6% / 🔴 +30.9%
bs=1000 sw=10 sl=64 1,086 0.663 922,409/997,301/997,301 us ⚪ within ±5% / 🔴 +8.7%
Baseline details

Latest main 646e460 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 379 tuples/sec 444 tuples/sec 893.99 tuples/sec -14.6% -57.6%
bs=10 sw=10 sl=64 MB/s 0.231 MB/s 0.271 MB/s 0.546 MB/s -14.8% -57.7%
bs=10 sw=10 sl=64 p50 25,208 us 20,322 us 11,176 us +24.0% +125.5%
bs=10 sw=10 sl=64 p95 39,244 us 31,641 us 14,009 us +24.0% +180.1%
bs=10 sw=10 sl=64 p99 39,244 us 31,641 us 16,898 us +24.0% +132.2%
bs=100 sw=10 sl=64 throughput 951 tuples/sec 931 tuples/sec 1,157 tuples/sec +2.1% -17.8%
bs=100 sw=10 sl=64 MB/s 0.581 MB/s 0.568 MB/s 0.706 MB/s +2.3% -17.8%
bs=100 sw=10 sl=64 p50 104,363 us 108,327 us 88,763 us -3.7% +17.6%
bs=100 sw=10 sl=64 p95 124,466 us 133,236 us 95,096 us -6.6% +30.9%
bs=100 sw=10 sl=64 p99 124,466 us 133,236 us 102,854 us -6.6% +21.0%
bs=1000 sw=10 sl=64 throughput 1,086 tuples/sec 1,089 tuples/sec 1,187 tuples/sec -0.3% -8.5%
bs=1000 sw=10 sl=64 MB/s 0.663 MB/s 0.665 MB/s 0.724 MB/s -0.3% -8.5%
bs=1000 sw=10 sl=64 p50 922,409 us 915,733 us 874,968 us +0.7% +5.4%
bs=1000 sw=10 sl=64 p95 997,301 us 1,015,283 us 917,815 us -1.8% +8.7%
bs=1000 sw=10 sl=64 p99 997,301 us 1,015,283 us 949,868 us -1.8% +5.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,527.74,200,128000,379,0.231,25207.62,39243.89,39243.89
1,100,10,64,20,2102.39,2000,1280000,951,0.581,104362.56,124465.94,124465.94
2,1000,10,64,20,18419.04,20000,12800000,1086,0.663,922408.94,997300.72,997300.72

@tanishqgandhi1908

Copy link
Copy Markdown
Contributor Author

/request-review @aicam

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add enabling model flag to this PR

@github-actions
github-actions Bot requested a review from aicam August 27, 2026 17:45
@tanishqgandhi1908 tanishqgandhi1908 changed the title feat(file-service): give models their own upload limits feat(file-service): add model upload limits and the models_enabled flag Aug 27, 2026

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please follow the naming convention we have.

Comment thread common/config/src/main/resources/default.conf
@github-actions github-actions Bot added ddl-change Changes to the TexeraDB DDL frontend Changes related to the frontend GUI labels Aug 27, 2026
@tanishqgandhi1908
tanishqgandhi1908 force-pushed the refactor/resource-upload-limits branch from 87008c1 to a6565fb Compare August 27, 2026 21:41

@aicam aicam left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aicam
aicam added this pull request to the merge queue Aug 27, 2026
Merged via the queue into apache:main with commit 516087e Aug 27, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common ddl-change Changes to the TexeraDB DDL frontend Changes related to the frontend GUI platform Non-amber Scala service paths refactor Refactor the code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants