Skip to content

env.set refused while tasks or I/O run; one cap for collection growth - #135

Merged
rendro merged 5 commits into
mainfrom
w1/env-and-caps
Oct 23, 2026
Merged

env.set refused while tasks or I/O run; one cap for collection growth#135
rendro merged 5 commits into
mainfrom
w1/env-and-caps

Conversation

@rendro

@rendro rendro commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Wave 1, lane S, step 1: decided by the owner on 2026-10-22 and 2026-10-23.

  • env.set and env.remove work until the program starts its first task or I/O operation, and are refused from then on: env.set cannot change the environment once tasks or I/O have started: set it at the start of main. Before any I/O no thread exists that could read the environment, so changing it is safe, and the answer never depends on timing. Per session in the REPL, per run in silt test; a listener and timers do not count.
  • One limit for growth: lists, maps, sets, list literals, stream.collect, stream.chunks and channel capacities stop at 10,000,000 elements, with one message form (list.append: list exceeds maximum size of 10000000 elements). Strings and Bytes stop at 10,000,000 bytes, now also for interpolation, string.join, bytes.concat, the regex replacers, encoding.form_encode, json/toml writers and string.from. The HTTP body limit is the same 10,000,000 bytes (was 10 MiB).

Verified by an independent verifier; its findings are fixed. The owner approved every new message text on 2026-10-23.

rendro added 3 commits October 23, 2026 03:54
@rendro
env.set and env.remove are refused while other tasks or I/O are running
94bfe93 
The environment is the process's, and a change to it is not
synchronized with a thread that reads it through the C library (a name
lookup, a TLS library): std's own lock does not cover those readers.
Until now only a call from a spawned task was refused; the program's
own thread could change the environment while a task or an I/O
operation of the pool read it.

Now both builtins are a runtime error while the program is not alone:

  env.set cannot change the environment while other tasks or I/O are
  running: set it before spawning tasks

(and `env.remove ...` likewise). Not alone means: a spawned task has
not ended (one that waits counts), another thread runs silt code (a
postgres stream worker), an I/O operation somebody waits for is in
flight, or an operation of the pool is queued or still running whether
or not anybody waits for it (one a `task.deadline` gave up on). The
last needs a new count in the pool: an operation whose waiter left and
that was told to stop was in neither `busy` nor `unheard` until its
thread returned.

A spawned task is never alone (the program's thread is live), so the
old check and its text "cannot be called from a spawned task" are gone:
one rule, one text.

- src/scheduler.rs: Scheduler::alone, over every owner.
- src/vm/io_op.rs: PoolState::running, IoPool::quiet, a unit test.
- src/builtins/io.rs: the refusal, SAFETY comments that say why.
- docs/stdlib/io-fs.md: the rule and its reason; docs/ffi.md: what an
  embedder's own threads must do (no switch exists).
- The REPL tests that changed the environment while a worker task
  waited now pass their stamps through files.
- Goldens lang/stdlib/env_change__*: refused while a task waits, while
  a task runs (env.remove), from a spawned task; allowed once the
  tasks and the I/O have ended.
@rendro
Every builtin that grows a collection stops at 10,000,000
c593240 
A list or a set holds at most ten million elements, a map at most ten
million entries. Until now some builtins checked that bound, each with
its own words, and the commonest growers did not: a loop of
`list.append`, `map.set` or `set.insert` grew until the machine's
memory ran out.

One function, `collections::fits`, now says it, in one form:

  <function>: <list|set|map> exceeds maximum size of 10000000 <elements|entries>

New caps: list.append, list.prepend, map.set and map.update (a new
key), map.merge, set.insert (a new element), set.union,
set.symmetric_difference, stream.collect, and stream.chunks, whose `n`
over the cap is refused at the call
(`stream.chunks: chunk exceeds maximum size of 10000000 elements`).
Merges count the new keys only when the two together could be too
many, so the common case costs nothing.

The capped builtins keep their bound and take the one form:
  concatenated list exceeds maximum size of N elements
    -> list.concat: list exceeds ..., and for `[..xs, ..ys]`
       list literal: list exceeds ...
  list.flat_map/flatten/unfold/scan: accumulated result exceeds
    maximum list length of N elements -> list.X: list exceeds ...
  list.zip/intersperse: result length L exceeds maximum materialized
    length N -> list.X: list exceeds ...

One-shot conversions of an input already in memory (string.split,
json.parse_list, regex.find_all, bytes.to_list, ...) stay uncapped:
they make as many elements as the input holds.

The bound is one constant, MAX_RANGE_MATERIALIZE renamed
MAX_COLLECTION_LEN (it also bounds writing out a range). Under
cfg(test) a thread may lower the cap the builtins see: the unit tests
of every capped builtin run at a cap of 3, since a map of ten million
entries takes seconds to make; `fits` itself is tested at the real cap.
Goldens lang/limits/collection_cap__* (list.append, list.prepend,
set.insert at the real cap) fail on the base; the goldens and the Rust
test of the old texts are moved to the new ones.
docs/language/collections.md says how large a collection can be.
@rendro
stream.chunks takes any n again: a chunk stops growing at the cap
60fb146 
The cap commit refused an `n` over ten million at the call. That broke
stage 7's rule that a chunk size sets no memory aside: a size larger
than the stream is one chunk of all of it
(concurrency/streams/stage7_d2__a_chunk_size_sets_no_memory_aside,
whose oracle run caught it). Now the chunk is capped where it grows:
the stage ends with

  stream.chunks: chunk exceeds maximum size of 10000000 elements

when a chunk would get its ten-million-and-first value, whatever `n`
is. The unit test of the cap cannot reach it (a stage runs on a worker,
which keeps the real cap); `fits` itself is tested at the real cap.
@rendro rendro mentioned this pull request Oct 23, 2026
Merged
@rendro
rendro added this pull request to stack #141 October 23, 2026 06:42
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) io_op test doc: env.set is refused while an operation runs
7ee787f 
Fix round 1, N1: the doc of an_operation_is_running_until_it_returns_whoever_waits said env.set waits for running operations; it refuses.
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) env.set and env.remove work until the first task or I/O, never…
7db5b1c 
… after

Owner decision V1 (2026-10-23), replacing the rule of 94bfe93. That
rule refused a change while another task was live or an I/O operation
ran, so whether a program could change its environment depended on
when its tasks ended (the verifier's B1/B2/G3). Now it does not depend
on timing: the environment can be changed until the program starts its
first task or I/O operation, and is refused from then on, also once
they have ended:

  env.set cannot change the environment once tasks or I/O have
  started: set it at the start of main

(env.remove likewise). A task is anything the scheduler runs
(task.spawn, task.deadline, a stream's stages, a server's handlers); an
I/O operation is one handed to the I/O pool (files, stdin, connections,
requests). Printing, env.get, fs.exists and tcp.listen start nothing.
The REPL keeps the rule per session; `silt test` per run.

The scheduler keeps one flag, set by its first task and by the pool's
first operation. The counters 94bfe93 added for the old rule
(Scheduler::alone, the pool's `running` count and IoPool::quiet, with
their unit test) are gone: nothing else used them. The SAFETY comments,
docs/stdlib/io-fs.md and docs/ffi.md give the new reason: before the
first task or I/O no thread of the runtime that may read the
environment exists.

Goldens lang/stdlib/env_change__*: works at the start of main; refused
after a joined task, a finished stream (env.remove), a file read, a
task.deadline body, a cancelled server, from a spawned task, and in a
REPL session after its first task. They replace the goldens of the old
rule.
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) Strings, Bytes and channel buffers stop at 10,000,000 too
a79b272 
Owner decision V2 (2026-10-23), the verifier's G1. Three growers had no
bound; a loop that doubles one took the machine's memory (the 0.23
binary is killed at a 2 GB cap by each of the doubling goldens):

  string interpolation: result would exceed maximum string size
    (N bytes > 10000000 limit)
  bytes.concat: result would exceed maximum size (N bytes > 10000000 limit)
  bytes.concat_all: result would exceed maximum size (N bytes > 10000000 limit)
  channel.new: capacity exceeds maximum size of 10000000 elements

Found with the same eye and capped in the form string.repeat and
string.replace already have (a new text for the owner's list):

  string.join: result would exceed maximum string size
    (N bytes > 10000000 limit)

bytes.concat_all and string.join count the size before they copy a
byte: their items can be one value many times, so their result was not
bounded by the memory the program already held. An interpolation is
checked after it is written, as its parts are in memory already.

Goldens lang/limits/collection_cap__{string_interpolation,
bytes_concat, bytes_concat_all, string_join, channel_capacity}_past_the_cap
fail on the base; the concat_all, join and channel ones also show that
exactly ten million is allowed. docs/language/collections.md names them.
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) An HTTP body is at most 10,000,000 bytes, the most a Bytes holds
1c90284 
Ruling on the V2 collision (provisional, for the owner's page). The
body limit of the HTTP server (requests it reads) and client
(responses it reads) was 10 MiB, 10,485,760 bytes: more than the
10,000,000 a Bytes may now hold, so a program could be given a body it
could not make. BODY_MAX is now MAX_COLLECTION_LEN, one bound:

  the body is larger than 10000000 bytes   (was: ... than 10 MiB)

docs/stdlib/http.md and errors.md say 10,000,000 bytes (the LSP
hover golden that shows errors.md follows). The golden
http_bodies__a_body_over_the_limit_is_an_error_not_a_cut_body tests
10,000,000 and 10,000,001: a handler sends the body at the limit, and a
server written by hand sends the larger ones in parts, since no program
can make them.

Two server tests answered with a 64 MB body built by bytes.concat_all
(more than the system takes in for a client that reads nothing); they
now read it from a file the test writes (Server::big_answer), as a
program still can: a string read from a file is not capped.
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) The text a builtin writes is no longer than a string may be
ad43ff2 
Ruling (3) on the growers found with V2 (provisional, for the owner's
page). These could make far more than the program holds, from one long
string used many times; each now stops at the 10,000,000-byte limit,
in the form string.repeat and string.replace have:

  <fn>: result would exceed maximum string size (N bytes > 10000000 limit)

for regex.replace, regex.replace_all, regex.replace_all_with,
encoding.form_encode, json.stringify, json.pretty, toml.stringify,
toml.pretty, string.from, and the text of a value shown in a string
interpolation (which already capped what it joins).

How: regex.replace(_all) build their text match by match (as
Regex::replacen does) and stop at the limit; replace_all_with and
form_encode check as they append; JSON is written into a Bounded writer
(builtins::common) that refuses the write that would pass the limit;
string.from and interpolation show the value into one
(Vm::shown_within); TOML, whose writer cannot be stopped, is refused on
a count of the fewest bytes its data takes, then on its length. Where a
text is stopped as it is written, N is as far as it had come (it would
have been longer); println, panic and `.display()` are not bounded.

Goldens lang/limits/collection_cap__{regex_replace_all,
regex_replace_all_with, form_encode, json_stringify, toml_pretty,
string_from}_past_the_cap. docs/language/collections.md lists them.
builtins::common is pub(crate) now, for the VM's use of Bounded.
@rendro
rendro marked this pull request as ready for review October 23, 2026 14:56
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
(#135) The deep-value test shows what a string can hold, and the rest…
7352f03 
… is refused

CI's release-only job: deep_values_are_shown_inspected_and_dropped
showed a value 1,500,000 levels deep in a string interpolation, a text
of 12,000,004 bytes, which the string cap of ad43ff2 refuses. The test
guards that showing, inspecting and dropping a deep value goes down no
native stack. It now:
- shows `a` at the deepest level whose text a string holds (1,249,999
  levels in a release build, 9,999,996 bytes; the debug depth of
  150,000 as before), still far beyond any native stack;
- inspects `b`, and a tuple holding both, at the full depth (io.inspect
  is not capped);
- shows `b` at the full depth and asserts the cap's error, exit 1: the
  text is stopped as it is written, not by an overflow (in a debug
  build, whose depth fits, it asserts the length).
Passes in both builds (cargo nextest --release / debug, -E
'test(deep_value)').
@rendro
Merge origin/main (lane L, fc55fc2) into w1/env-and-caps
4faccfc 
One conflict, src/builtins/collections.rs: lane L added map.is_empty
beside map.merge, which this branch caps; both are kept (is_empty, then
the capped merge). Lane L's other new collection builtins (is_empty of
list and set, sort_with, unique_by) make no collection larger than the
one they are given and need no cap.
@rendro
rendro merged commit b724305 into main Oct 23, 2026
23 checks passed
rendro added a commit that referenced this pull request Oct 23, 2026
@rendro
Docs fixes from the 0.21 review; error codes page; channel self-refer…
8e6882d 
…ence warning (#144)

* Docs: the faults the 0.21 review found

- loops-and-pipes.md: "loop() works inside closures" said more than is
  true. A closure can hold a loop of its own, but a `loop(...)` inside
  a closure cannot go on with a loop outside it (that is an error);
  the paragraph now says both.
- bindings-and-functions.md: `return` in a closure ends the closure's
  call, not the function around it, so a callback cannot end a fold
  early; the page says so and shows list.fold_until doing it.
- toml.md: the toml.pretty section showed the error text of
  toml.stringify ("toml.pretty: unsupported value kind Fn" is what it
  prints).
- io-fs.md: fs.copy and fs.remove said `Err(message)`; they give
  `Err(IoError)`.
- getting-started.md: the read_file block used `io` and `string`
  without importing them and called a character count "bytes"; it is
  now a whole program that runs. Section 5 says what a library package
  is (its src/lib.silt is what `import <name>` gives a dependent).
- modules.md: `task` was labelled "Structured concurrency", which silt
  does not have (a cancel does not reach a task's children): now
  "Tasks".
- New page docs/language/error-codes.md: every diagnostic code with
  what it means, by phase, linked from the language guide. A meta test
  checks that it lists each code of `Code::ALL` once and no other.

* Concurrency docs: a channel whose buffer holds the channel stays until the end

A value in a channel's buffer that holds that same channel (in a record
field, or captured by a closure) keeps the channel alive: when the
program drops the channel while the value is still buffered, the
channel and its buffer stay in memory until the program ends. The
limitations section of docs/concurrency.md now says so, and what to do
(receive what is buffered first, or send a value that does not hold
the channel).

* (#140) PEM errors read as they did with rustls-pemfile

Fix round 1, B3. rustls-pki-types prints its PEM errors in its own
words: "missing section end marker: [80, 82, ...]" for a section that
does not end, and "base64 decode error: InvalidCharacter(33)". The
texts are mapped back to the ones 0.23 printed:

  parse private key: section end "PRIVATE KEY" missing
  parse cert chain: InvalidCharacter(33)
  (and `illegal section start: "..."`)

Every case of the verifier's probe (lsv/p/tls_probe.silt, 18 cert/key
pairs) now prints what the 0.23 binary prints. New golden
lang/tcp_tls/tls_server__a_pem_that_does_not_parse_says_why (a cert
whose section does not end, one with a non-base64 character): it holds
on the base and fails on eff35f1.

* (#143) json.Syntax at a raw newline in a string is the newline's offset

Fix round 1, B4. serde reports a raw newline inside a string at
column 0 of the next line; the offset was the first byte of that
line, one past the newline. Column 0 now means the newline that ends
the line before: `{"a": "<newline>x"}` gives 7. All 19 inputs of the
verifier's probe (lsv/p/js) give the offset their name says, but one:
04_unicode_x_at_17, whose `x` is at byte 18 (the name counts é and €
as one byte too few; the output 18 is right).
The golden json_syntax__the_offset_counts_bytes_from_the_start_of_the_
text gets the case.

* (#144) Docs: a git dependency names its ref; error-code rows say what they are for

Fix round 1, B5 and G2.
- getting-started.md and modules.md wrote `silt add <name> --git <url>`,
  which `silt add` refuses ("--git requires exactly one of --rev,
  --branch, or --tag"). Both now write `--tag <tag>` and name the other
  two. The command as written gets past argument checking and fails
  only for the network (probed with an unreachable URL).
- error-codes.md, checked against `silt check --format json` on small
  programs for 37 of the codes:
  E0305 is a field or method the type at hand lacks (`p.b` on a record
  without `b`, a literal or update field its type does not declare),
  not "a field no record in scope declares" (the enum's comment in
  src/diagnostic.rs says the same now);
  E0302 is a name in a pin pattern that nothing binds (a bare unknown
  name is E0201);
  E0303 and E0304 say where they are reported (an anonymous record's
  alias as a literal; a record literal or pattern whose type name names
  no record);
  E0321 is a `_` arm that is not last in a `match` without a scrutinee;
  E0315 is reported by nothing today, and its row says so.

* (#135) io_op test doc: env.set is refused while an operation runs

Fix round 1, N1: the doc of an_operation_is_running_until_it_returns_whoever_waits said env.set waits for running operations; it refuses.

* (#140) The release job asks for no actions: read

Fix round 1, N3. The job checks out the tag, downloads this run's own artifacts, writes SHA256SUMS, attests and makes the release: none of that reads another run's artifacts, which is what actions: read is for. It keeps contents: write, id-token: write and attestations: write. (The top-level permissions, which the other jobs use, are unchanged.)

* (#135) env.set and env.remove work until the first task or I/O, never after

Owner decision V1 (2026-10-23), replacing the rule of 94bfe93. That
rule refused a change while another task was live or an I/O operation
ran, so whether a program could change its environment depended on
when its tasks ended (the verifier's B1/B2/G3). Now it does not depend
on timing: the environment can be changed until the program starts its
first task or I/O operation, and is refused from then on, also once
they have ended:

  env.set cannot change the environment once tasks or I/O have
  started: set it at the start of main

(env.remove likewise). A task is anything the scheduler runs
(task.spawn, task.deadline, a stream's stages, a server's handlers); an
I/O operation is one handed to the I/O pool (files, stdin, connections,
requests). Printing, env.get, fs.exists and tcp.listen start nothing.
The REPL keeps the rule per session; `silt test` per run.

The scheduler keeps one flag, set by its first task and by the pool's
first operation. The counters 94bfe93 added for the old rule
(Scheduler::alone, the pool's `running` count and IoPool::quiet, with
their unit test) are gone: nothing else used them. The SAFETY comments,
docs/stdlib/io-fs.md and docs/ffi.md give the new reason: before the
first task or I/O no thread of the runtime that may read the
environment exists.

Goldens lang/stdlib/env_change__*: works at the start of main; refused
after a joined task, a finished stream (env.remove), a file read, a
task.deadline body, a cancelled server, from a spawned task, and in a
REPL session after its first task. They replace the goldens of the old
rule.

* (#135) Strings, Bytes and channel buffers stop at 10,000,000 too

Owner decision V2 (2026-10-23), the verifier's G1. Three growers had no
bound; a loop that doubles one took the machine's memory (the 0.23
binary is killed at a 2 GB cap by each of the doubling goldens):

  string interpolation: result would exceed maximum string size
    (N bytes > 10000000 limit)
  bytes.concat: result would exceed maximum size (N bytes > 10000000 limit)
  bytes.concat_all: result would exceed maximum size (N bytes > 10000000 limit)
  channel.new: capacity exceeds maximum size of 10000000 elements

Found with the same eye and capped in the form string.repeat and
string.replace already have (a new text for the owner's list):

  string.join: result would exceed maximum string size
    (N bytes > 10000000 limit)

bytes.concat_all and string.join count the size before they copy a
byte: their items can be one value many times, so their result was not
bounded by the memory the program already held. An interpolation is
checked after it is written, as its parts are in memory already.

Goldens lang/limits/collection_cap__{string_interpolation,
bytes_concat, bytes_concat_all, string_join, channel_capacity}_past_the_cap
fail on the base; the concat_all, join and channel ones also show that
exactly ten million is allowed. docs/language/collections.md names them.

* (#144) The usage of silt add shows the git ref as required

Ruling on fix-round finding (1). `silt --help`, README.md,
docs/getting-started.md and the "missing source flag" error showed
`--git <url> [--rev|--branch|--tag <ref>]`, brackets that say the ref
may be left out; `silt add` refuses a git dependency without one. They
now read `--git <url> (--rev|--branch|--tag) <ref>` (the same width, so
the help's columns stay where they were). The comments in
src/cli/add.rs, src/cli/help.rs and src/main.rs that quote it follow.

* (#135) An HTTP body is at most 10,000,000 bytes, the most a Bytes holds

Ruling on the V2 collision (provisional, for the owner's page). The
body limit of the HTTP server (requests it reads) and client
(responses it reads) was 10 MiB, 10,485,760 bytes: more than the
10,000,000 a Bytes may now hold, so a program could be given a body it
could not make. BODY_MAX is now MAX_COLLECTION_LEN, one bound:

  the body is larger than 10000000 bytes   (was: ... than 10 MiB)

docs/stdlib/http.md and errors.md say 10,000,000 bytes (the LSP
hover golden that shows errors.md follows). The golden
http_bodies__a_body_over_the_limit_is_an_error_not_a_cut_body tests
10,000,000 and 10,000,001: a handler sends the body at the limit, and a
server written by hand sends the larger ones in parts, since no program
can make them.

Two server tests answered with a 64 MB body built by bytes.concat_all
(more than the system takes in for a client that reads nothing); they
now read it from a file the test writes (Server::big_answer), as a
program still can: a string read from a file is not capped.

* (#135) The text a builtin writes is no longer than a string may be

Ruling (3) on the growers found with V2 (provisional, for the owner's
page). These could make far more than the program holds, from one long
string used many times; each now stops at the 10,000,000-byte limit,
in the form string.repeat and string.replace have:

  <fn>: result would exceed maximum string size (N bytes > 10000000 limit)

for regex.replace, regex.replace_all, regex.replace_all_with,
encoding.form_encode, json.stringify, json.pretty, toml.stringify,
toml.pretty, string.from, and the text of a value shown in a string
interpolation (which already capped what it joins).

How: regex.replace(_all) build their text match by match (as
Regex::replacen does) and stop at the limit; replace_all_with and
form_encode check as they append; JSON is written into a Bounded writer
(builtins::common) that refuses the write that would pass the limit;
string.from and interpolation show the value into one
(Vm::shown_within); TOML, whose writer cannot be stopped, is refused on
a count of the fewest bytes its data takes, then on its length. Where a
text is stopped as it is written, N is as far as it had come (it would
have been longer); println, panic and `.display()` are not bounded.

Goldens lang/limits/collection_cap__{regex_replace_all,
regex_replace_all_with, form_encode, json_stringify, toml_pretty,
string_from}_past_the_cap. docs/language/collections.md lists them.
builtins::common is pub(crate) now, for the VM's use of Bounded.

* (#135) The deep-value test shows what a string can hold, and the rest is refused

CI's release-only job: deep_values_are_shown_inspected_and_dropped
showed a value 1,500,000 levels deep in a string interpolation, a text
of 12,000,004 bytes, which the string cap of ad43ff2 refuses. The test
guards that showing, inspecting and dropping a deep value goes down no
native stack. It now:
- shows `a` at the deepest level whose text a string holds (1,249,999
  levels in a release build, 9,999,996 bytes; the debug depth of
  150,000 as before), still far beyond any native stack;
- inspects `b`, and a tuple holding both, at the full depth (io.inspect
  is not capped);
- shows `b` at the full depth and asserts the cap's error, exit 1: the
  text is stopped as it is written, not by an overflow (in a debug
  build, whose depth fits, it asserts the length).
Passes in both builds (cargo nextest --release / debug, -E
'test(deep_value)').
@rendro
rendro deleted the w1/env-and-caps branch October 23, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rendro