Repository navigation
chore: sync upstream oven-sh/bun main (2026-10-18) - #191
Merged
Merged
Conversation
…p using a connection that went silent after an abort (oven-sh#45422) ### What does this PR do? Two fixes for `fetch()` over HTTP/2 (`BUN_FEATURE_FLAG_EXPERIMENTAL_HTTP2_CLIENT`). Both are cases where a request hangs, and later requests to the same origin hang with it. #### 1. `fetch` rejects when a fatal TLS alert arrives after the handshake A TLS 1.3 client finishes its handshake before the server checks the client certificate. When the server then sends a fatal alert (for example `certificate_required`), uSockets reports it as a failed handshake on a socket that the HTTP/2 session already owns. `on_handshake` only handled sockets owned by an `HTTPClient`, so it marked the socket dead and closed it without telling the session. The request never settled. Its abort-tracker entry was dropped, so `abort()` had no effect. The session stayed registered, so later requests to the origin joined it and hung as well. The session now fails with the same error as HTTP/1.1, `EPROTO`. The other handshake event such a socket can get, a TLS 1.2 renegotiation that finished, took the same path to the same hang. It now fails the session with `HTTP2ProtocolError`, since RFC 9113 §9.2.1 makes a renegotiation a connection error of type `PROTOCOL_ERROR`. Bun 1.4.3 does not hang here (it rejects with `ECONNRESET`), so the hang has not been in a release. #### 2. `fetch` stops using a connection that went silent after an abort When a connection dies with no FIN or RST (a network change, a dropped NAT entry), the kernel keeps accepting writes for many minutes. An abort only sent `RST_STREAM` and kept the session, so each later request to the origin joined it, or took it from the keep-alive pool, and hung until its own timeout. The client never sent a PING, and with `timeout: false` on every stream no timer ran at all. HTTP/1.1 does not have this problem, because an abort closes the socket. - When the session has read nothing for a second, and the aborted request's stream (if it has one yet) is at least a second old, a PING goes out with the `RST_STREAM`. - Until the ACK arrives the session takes no new stream and is not pooled, so new requests use another connection. The ACK puts the session back in use. - While the PING is unanswered the session's idle timer is capped at 15 seconds (16 to 20 on the 4-second timer wheel), which is Go's default `PingTimeout`, or at `BUN_CONFIG_HTTP_IDLE_TIMEOUT` if that is shorter. What re-arms the idle timer re-arms the cap: progress on a stream, a writable event, a request body chunk, another abort. So does DATA for a stream that was reset, which can sit in front of the ACK. - When that timer fires, the connection is closed and the streams still on it fail with `ECONNRESET`. A request whose own shorter `timeout` runs out first still fails with a `TimeoutError`. - gRPC servers close a connection that sends several PINGs with no HEADERS or DATA from the server in between (golang/go#70575), which would take the other streams down. So only one liveness PING goes out between responses. The server counts per connection, so the keep-alive pool does not give the PING back; HEADERS or DATA do, also those for a stream that was already reset. A second such abort retires the session as a GOAWAY would: no new stream, and closed when its last stream ends. - `FetchSession.close()` also closes a connection that has no request left and only waits for the ACK. Any other abort changes nothing. ### How did you verify your code works? 12 new tests in `test/js/web/fetch/fetch-http2-client.test.ts`, run locally on Linux x64 only. | | `main` (CI's release build of 116dd64) | this branch (`release-asan` build) | | --- | --- | --- | | TLS 1.3 fatal alert after the handshake | hangs | rejects with `EPROTO` | | TLS 1.2 renegotiation on an h2 connection (Node is the peer) | hangs | rejects with `HTTP2ProtocolError` | | all 12 new tests | | 12 pass | The first push (69ee953) was green on every lane that finished in build 126227, apart from `bun-install-windows-rename-retry.test.ts` on Windows aarch64. That test also fails on `main` (build 126172) and on 8 more of the 30 builds before this one. The third commit fixes what the review found. Each fix has a test that I ran before and after it: | | 69ee953 | c79a222 | | --- | --- | --- | | `FetchSession.close()` with a PING unanswered | the connection stays open | closed | | a request's own `timeout` runs out with a PING unanswered | `ECONNRESET` (`main`: `TimeoutError`) | `TimeoutError` | | silent aborts in a row, with the pool in between | a PING each time | one PING, then a new connection | | the test of the unanswered PING | 20 s | 9 s | Also green on this branch's ASAN build, with no ASAN report: the rest of `fetch-http2-client.test.ts` (88 tests in all), `fetch-session`, `fetch-http2-adversarial`, `fetch-http2-leak`, `fetch.tls`, `fetch-keepalive`, `fetch-abort-socket-close-race`, `fetch-abort-queued`. `fetch-tls-abortsignal-timeout` has a 50 ms wall-clock limit and fails on my loaded machine with and without the third commit (11 failures in 6 runs without, 4 with). It does not turn the HTTP/2 client on. I used a `release-asan` build and not `bun bd`. The liveness tests sleep past the one-second grace period because nothing observable marks that moment. They start the sleep when the test server has seen the request.
…t side (oven-sh#45104) ### What does this PR do? Replaces bindgen v1 (`src/codegen/bindgen.ts`, `bindgen-lib*.ts`) and bindgen v2 (`src/codegen/bindgenv2/`) with one generator that also does classes, and ports every user of the two. It is meant to be the one the remaining Rust binding mechanisms (`.classes.ts`, `#[host_fn]`, `$rust` / `$newRustFunction`, hand-declared externs) move to. The body of a binding is always Rust: functions written in C++ (`$cpp` / `$newCppFunction`, hand-written host functions) are not covered. A `.bind.ts` schema generates the C++ that checks and converts arguments with JSC's own API, the Rust that receives them already converted (v1 and v2 generated no Rust: the Rust side was copied by hand), and the TypeScript types `src/js` sees. `docs/project/bindgen.mdx` is rewritten and describes all of it. ```ts export const Glob = b.class({ rustPath: "crate::api::glob::Glob", construct: { args: { pattern: b.string.utf8.owned } }, proto: { match: { args: { str: b.string.utf8 }, returns: b.bool }, }, }); ``` ```rust impl GlobClass for Glob { fn constructor(pattern: bun_core::Utf8Bytes<'static>) -> Self { Glob { pattern } } fn r#match<'a>(&self, str: &'a [u8]) -> bool { bun_glob::r#match(self.pattern.slice(), str).matches() } } ``` The type vocabulary is v2's (`b.union`, `b.enumeration`, `.loose`, `.nullable`, `altNames`) and the class vocabulary is `.classes.ts`'s (`construct`, `proto`, `cache`, `passThis`, `privateSymbol`, `builtin`), except: | Was | Is | Why | | --- | --- | --- | | `klass` | `static` | it means the static members | | `values`, `x_get_cached` / `x_set_cached` | `internalFields`, `x_field` / `set_x_field` | nothing is cached; JSC's word, and distinct from `cache: true` | | `export const Foo = b.enumeration("Foo", ..)` | `export const Foo = b.enumeration(..)` | a named type is called what it is exported as | | `.nullable.loose` | `.falsyAsNull` | `.loose` means "coerce" everywhere else | | `b.String` / `t.UTF8String`, `b.ArrayBuffer` | `b.string` / `b.string.utf8`, `b.bufferSource` / `.bytes` / `.arrayBuffer` | one name per JS type; a modifier picks the Rust form | | `builtin: "globScanCodeGenerator"` | `builtin: true` | derived from the class and the property | | `internalName: "onOpen"` | `rustName: "on_open"` | it names the Rust field; the sibling key is `rustPath` | | `SSLConfig` + `userFacingName: "TLSOptions"` | `TLSOptions` | a dictionary is exported under its JS name (`bun.d.ts`), which is what errors call it | | `$bindgenFn` | `$bindgen` | it gives classes too | #### Bugs fixed Replacing hand-written argument handling meant comparing old and new behaviour on hostile inputs, which is how these turned up. Each reproduces on CI's release build of the merge base (`116dd64fd`) and has a test in the API's own file. | Call | On `main` | With this PR | | --- | --- | --- | | `Bun.listen({ tls: { key: new Uint8Array(new SharedArrayBuffer(8)) } })`; the same for a `DataView`, a `Buffer` or an array of them, as `key`, `cert`, `ca`, `crl` or `ALPNProtocols` | **the process aborts** | throws the `ERR_INVALID_ARG_TYPE` a bare `SharedArrayBuffer` gets | | `tls: { key: both.subarray(n), cert }`: a view that shows part of its buffer, for the same five options | the whole buffer is read, so whatever comes first in `both` is loaded | the view is read | | `Object.prototype.passphrase = 5`, then any `Bun.listen({ tls })` | throws `TLSOptions.passphrase must be a string`: `TLSOptions` and `SocketOptions` members are found on `Object.prototype` | listens | | `glob.scanSync({ dot: 1 })`, or any truthy non-boolean for a boolean option | `false`: dotfiles are missing, `onlyFiles: 1` lists directories | `ToBoolean` | | `glob.scanSync({ onlyFiles: "" })` | the default, `true` | `false` | | `glob.scanSync(new String(cwd))` | read as an options object with no options | the `cwd` | | `os.userInfo(5)`, or any other non-object | throws `TypeError: Type error` | returns the info, as in Node | | `os.userInfo(fn)` | `fn` is read as the options, so a throwing `encoding` getter on it throws | returns the info, as in Node | | `os.setPriority(priority, undefined)` | throws | `setPriority(priority)`, as in Node | | `os.setPriority()` | `Not enough arguments` | Node's `The "priority" argument must be of type number. Received undefined` | | `os.getPriority(Infinity)`, `os.setPriority(Infinity)` | `It must be >= -2147483648 and <= 2147483647` | `It must be an integer`, as in Node | | `os.setPriority(20)`, `os.getPriority(2 ** 40)` | `It must be >= -20 and <= 19` | `It must be >= -20 && <= 19`, as in Node: `.validateInteger` is the tree's `Bun::V::validateInteger` | | `.length` of seven `node:os` functions | `getPriority`, `userInfo` 2; `homedir`, `hostname`, `loadavg`, `networkInterfaces`, `uptime` 1 | 1 and 0, as in Node (v26.7.0) | | `new os.freemem()`, `totalmem`, `release`, `version` | returns the number or string | `is not a constructor`, as in Node | | `new os.getPriority()`, `homedir`, `hostname`, `uptime` | returns the number or string | an object, as in Node | | `Bun.$.braces(str, 5)` | `TypeError: Type error`, no code | `ERR_INVALID_ARG_TYPE` `BracesOptions must be an object` | The `Object.prototype` fix covers the realm the API belongs to. One case is as on `main`, neither fixed nor made worse: an options object created in another realm (`node:vm`) inherits from that realm's `Object.prototype`, and members planted there are still found. The lookup is the existing `getIfPropertyExistsPrototypePollutionMitigation`, which stops at the calling realm's `Object.prototype` only, for every caller it has. Two more are in v2's converter for strict integers, which every schema shares. A BigInt out of the type's range but inside 64 bits threw twice. NaN reached a cast from `double` to `int64_t`, which is undefined. The generator's own rules fix the `Object.prototype` lookups, the `Glob` booleans, the `.length`s and the error codes. The shared-memory abort, the partial view, the `Infinity` message and the two above are fixed in the shared converters, so every schema that uses those types has the fix. `new` on the `node:os` functions is fixed in `src/js/node/os.ts`, where eight of them are plain functions as in Node. #### Generated code, before and after The diff shows the generators, not what they write. Below is the output for one function (was bindgen v1), one dictionary and one union (were v2) and one class method (was `.classes.ts`), then for how a type is chosen (5 and 6), for defaults (7) and for the receiver check (8). Before is `main` at 347794754fd, after is this branch at 994b62d4e38; both are what `bun scripts/build.ts --profile=debug --mode=codegen` writes to `build/*/codegen`, copied as is. `// …` marks a cut, and runs of blank lines are collapsed. **1. A function: `os.getPriority(pid)`.** The conversion is the same one. The argument now crosses by value and the result comes back as the return value, instead of both through pointers to C++ locals. The Rust that receives the call was written by hand, found by a script that scrapes `// HOST_EXPORT` comments; it is generated now, and 378 lines of it leave `src/runtime/hw_exports.rs`. <details><summary>Schema, C++, Rust</summary> Schema, `src/runtime/node/node_os.bind.ts`. Before: ```ts export const getPriority = fn({ args: { global: t.globalObject, pid: t.i32.validateInt32().default(0).nonNull, }, ret: t.i32, }); ``` After: ```ts export const getPriority = b.fn({ args: { pid: b.i32.validateInteger().default(0), }, returns: b.i32, capability: "js", throws: true, }); ``` Generated C++. Before, in `GeneratedBindings.cpp`: ```cpp extern "C" bool bindgen_Node_os_dispatchGetPriority1(JSC::JSGlobalObject*, const int32_t*, int32_t*); extern "C" SYSV_ABI JSC::EncodedJSValue bindgen_Node_os_jsGetPriority(JSC::JSGlobalObject* global, JSC::CallFrame* callFrame) { auto& vm = JSC::getVM(global); auto throwScope = DECLARE_THROW_SCOPE(vm); JSC::EnsureStillAliveScope arg0 = callFrame->argument(0); int32_t argPid; if (!arg0.value().isUndefined()) { argPid = WebCore::convert<Bun::BindgenCustomEnforceRange<int32_t, -2147483648, 2147483647, Bun::BindgenCustomEnforceRangeKind::Node>>(*global, arg0.value(), []() { return "pid"_s; }); RETURN_IF_EXCEPTION(throwScope, {}); } else { argPid = 0; } int32_t out; if (!bindgen_Node_os_dispatchGetPriority1( global, &argPid, &out )) { return {}; } return JSC::JSValue::encode(WebCore::toJS<WebCore::IDLLong>(*global, out)); } ``` After, in `GeneratedNodeOs.cpp`: ```cpp extern "C" JSC::EncodedJSValue Bun__Bindgen__NodeOs__getPriority(JSC::JSGlobalObject*, Bun::Bindgen::ExternOf<Bun::Bindgen::IDLValidateInteger<int32_t, -2147483648, 2147483647>>); namespace Bun::Bindgen::Generated::NodeOs { JSC_DEFINE_HOST_FUNCTION(jsGetPriority, (JSC::JSGlobalObject * globalObject, JSC::CallFrame* callFrame)) { auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); JSC::EnsureStillAliveScope argument0 = callFrame->argument(0); [[maybe_unused]] auto ctx0 = Bun::Bindgen::LiteralConversionContext { "pid"_s }; auto converted0 = argument0.value().isUndefined() ? int32_t { 0 } : WebCore::Converter<Bun::Bindgen::IDLValidateInteger<int32_t, -2147483648, 2147483647>>::convert(*globalObject, argument0.value(), "pid"_s); RETURN_IF_EXCEPTION(scope, {}); RELEASE_AND_RETURN(scope, Bun__Bindgen__NodeOs__getPriority(globalObject, Bun::Bindgen::toExtern(converted0))); } ``` Rust that C++ calls. Before, by hand in `src/runtime/hw_exports.rs`: ```rust /// # Safety /// `arg_pid` and `out` must be valid C++ stack locals. // HOST_EXPORT(bindgen_Node_os_dispatchGetPriority1, c) // Called only from the generated `extern "C"` thunk; C++ guarantees non-null stack locals. pub(crate) fn bindgen_node_os_dispatch_get_priority( global: &JSGlobalObject, arg_pid: *const i32, out: *mut i32, ) -> bool { // SAFETY: `arg_pid`/`out` are valid C++ stack locals. bindgen_out( global, out, node_os::get_priority(global, unsafe { *arg_pid }), ) } ``` After, generated in `generated_bindgen_bun_runtime.rs`: ```rust #[unsafe(no_mangle)] pub unsafe extern "C" fn Bun__Bindgen__NodeOs__getPriority(global: *mut JSGlobalObject, pid: i32) -> JSValue { // SAFETY: called by generated C++, whose frame holds what it passes until this returns. unsafe { bindgen::JsScope::with(global, |cx| { let result = crate::node::os::get_priority(cx, pid); bindgen::host_fn_return(cx, result, |heap, value| bindgen::to_js::i32(heap, value)) }) } } ``` The body in `node_os.rs` changes in its first two lines only. Before: ```rust pub(crate) fn get_priority(global: &JSGlobalObject, pid: i32) -> JsResult<i32> { let result = get_process_priority(pid); // … ``` After: ```rust pub(crate) fn get_priority(cx: &mut JsScope<'_>, pid: i32) -> JsResult<i32> { let global = cx.global(); let result = get_process_priority(pid); // … ``` </details> **2. A dictionary: `TLSOptions`.** Each member was looked up with `object->get` and an `Identifier` made from a string on every call. It is now `members.get(i)`: the names are made once per VM, a plain object is read with `Structure::get` + `getDirect`, and the lookup stops before `Object.prototype`. The Rust mirror of the C++ structs was kept in step by hand in `src/jsc/generated.rs`, with nothing on either side checking that the two layouts agree. It is generated (701 lines leave that file), and both sides assert the size and alignment the generator computed. <details><summary>Schema, C++, Rust</summary> Schema. Before, `SSLConfig.bindv2.ts`: ```ts export const SSLConfig = b.dictionary( { name: "SSLConfig", userFacingName: "TLSOptions", generateConversionFunction: true, }, { passphrase: b.String.nullable, dhParamsFile: { type: b.String.nullable, internalName: "dh_params_file", }, serverName: { type: b.String.nullable, internalName: "server_name", altNames: ["servername"], }, lowMemoryMode: { type: b.bool, default: false, internalName: "low_memory_mode", }, // … ``` After, `SSLConfig.bind.ts`: ```ts export const TLSOptions = b.dictionary( { passphrase: b.string.nullable, dhParamsFile: b.string.nullable, serverName: { type: b.string.nullable, altNames: ["servername"] }, lowMemoryMode: b.bool.default(false), // … }, { fromJS: true }, ); ``` Generated C++, `GeneratedSSLConfig.cpp`. Before: ```cpp template<> Bun::Bindgen::Generated::SSLConfig WebCore::convertDictionary<Bun::Bindgen::Generated::SSLConfig>( JSC::JSGlobalObject& globalObject, JSC::JSValue value) { ::JSC::VM& vm = globalObject.vm(); auto throwScope = DECLARE_THROW_SCOPE(vm); auto ctx = Bun::Bindgen::LiteralConversionContext { "TLSOptions"_s }; auto* object = value.getObject(); if (!object) [[unlikely]] { ctx.throwNotObject(globalObject, throwScope); return {}; } bool isArray = JSC::isArray(&globalObject, value); RETURN_IF_EXCEPTION(throwScope, {}); if (isArray) [[unlikely]] { ctx.throwNotObject(globalObject, throwScope); return {}; } ::Bun::Bindgen::Generated::SSLConfig result; { ::JSC::JSValue value0; auto ctx0 = Bun::Bindgen::LiteralConversionContext { "TLSOptions.passphrase"_s }; do { value0 = object->get( &globalObject, ::JSC::Identifier::fromString(vm, "passphrase"_s)); RETURN_IF_EXCEPTION(throwScope, {}); } while (false); result.passphrase = Bun::convertIDL<::WebCore::IDLNullable<::Bun::IDLStrictString>>(globalObject, value0, ctx0); RETURN_IF_EXCEPTION(throwScope, {}); } // … { ::JSC::JSValue value2; auto ctx2 = Bun::Bindgen::LiteralConversionContext { "TLSOptions.serverName"_s }; do { value2 = object->get( &globalObject, ::JSC::Identifier::fromString(vm, "serverName"_s)); RETURN_IF_EXCEPTION(throwScope, {}); if (!value2.isUndefined()) break; value2 = object->get( &globalObject, ::JSC::Identifier::fromString(vm, "servername"_s)); RETURN_IF_EXCEPTION(throwScope, {}); } while (false); result.server_name = Bun::convertIDL<::WebCore::IDLNullable<::Bun::IDLStrictString>>(globalObject, value2, ctx2); RETURN_IF_EXCEPTION(throwScope, {}); } // … ``` After: ```cpp template<> void Bun::Bindgen::convertDictionaryInto<Bun::Bindgen::Generated::TLSOptions>(JSC::JSGlobalObject& lexicalGlobalObject, JSC::JSValue value, Bun::Bindgen::Generated::TLSOptions& result) { auto* globalObject = &lexicalGlobalObject; auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); [[maybe_unused]] auto ctx = Bun::Bindgen::LiteralConversionContext { "TLSOptions"_s }; auto* object = value.getObject(); if (!object) [[unlikely]] { ctx.throwNotObject(*globalObject, scope); return; } bool isArray = JSC::isArray(globalObject, value); RETURN_IF_EXCEPTION(scope, void()); if (isArray) [[unlikely]] { ctx.throwNotObject(*globalObject, scope); return; } Bun::Bindgen::DictionaryMemberReader members(vm, globalObject, object, 35, 25); { [[maybe_unused]] auto ctx0 = Bun::Bindgen::LiteralConversionContext { "TLSOptions.passphrase"_s }; JSC::JSValue value0 = members.get(0); RETURN_IF_EXCEPTION(scope, void()); result.passphrase = Bun::convertIDL<WebCore::IDLNullable<Bun::Bindgen::IDLStrictString>>(*globalObject, value0, ctx0); RETURN_IF_EXCEPTION(scope, void()); } // … { [[maybe_unused]] auto ctx2 = Bun::Bindgen::LiteralConversionContext { "TLSOptions.serverName"_s }; JSC::JSValue value2 = members.get(2); RETURN_IF_EXCEPTION(scope, void()); if (value2.isUndefined()) { value2 = members.get(3); RETURN_IF_EXCEPTION(scope, void()); } result.server_name = Bun::convertIDL<WebCore::IDLNullable<Bun::Bindgen::IDLStrictString>>(*globalObject, value2, ctx2); RETURN_IF_EXCEPTION(scope, void()); } // … ``` Rust. Before, by hand in `src/jsc/generated.rs`: ```rust #[repr(C)] #[derive(Clone, Copy)] struct ExternSSLConfig { passphrase: RawWTFStringImpl, dh_params_file: RawWTFStringImpl, server_name: RawWTFStringImpl, low_memory_mode: bool, // … } impl SSLConfig { fn convert_from_extern(ext: &ExternSSLConfig) -> Self { Self { passphrase: adopt_opt_string(ext.passphrase), dh_params_file: adopt_opt_string(ext.dh_params_file), server_name: adopt_opt_string(ext.server_name), low_memory_mode: ext.low_memory_mode, // … ``` After, generated: ```rust #[repr(C)] #[derive(Clone, Copy)] pub struct ExternTLSOptions { passphrase: bindgen::extern_types::ExternString, dh_params_file: bindgen::extern_types::ExternString, server_name: bindgen::extern_types::ExternString, low_memory_mode: bool, // … } const _: () = assert!(size_of::<ExternTLSOptions>() == 224 && align_of::<ExternTLSOptions>() == 8); impl TLSOptions { pub unsafe fn from_extern(value: &ExternTLSOptions) -> Self { // SAFETY: as above, member by member. unsafe { Self { passphrase: { let value = value.passphrase; if value.is_null() { None } else { Some(bindgen::extern_types::adopt_string(value)) } }, dh_params_file: { let value = value.dh_params_file; if value.is_null() { None } else { Some(bindgen::extern_types::adopt_string(value)) } }, server_name: { let value = value.server_name; if value.is_null() { None } else { Some(bindgen::extern_types::adopt_string(value)) } }, low_memory_mode: value.low_memory_mode, // … ``` and in `GeneratedSSLConfig.h`: ```cpp static_assert(sizeof(ExternTLSOptions) == 224 && alignof(ExternTLSOptions) == 8); ``` </details> **3. A class method: `Glob#match(str)`.** `.classes.ts` generated the receiver check and handed Rust the raw `CallFrame`; counting the arguments, checking their types and converting the string were written by hand in each method. The check and the conversion are generated from the types in the schema, nothing counts the arguments (one that was not passed is `undefined`, which is not a string), and the body is left with what the method does. The object also changes shape: it held a `void*` to a separately allocated Rust struct, freed from the destructor, and a `JSC::Weak` handle to itself, which every collection walked, calling Rust's `hasPendingActivity` for each object not marked yet. The struct is now inside the JS object and there is no handle. The `#if BUN_DEBUG` block of the old callback (the caller's file name for a debugger, an audit of the returned cell) is not generated any more; its assertion that an empty result comes with an exception is in `bindgen::host_fn_return`. `Glob`'s share of the generated C++ goes from 420 lines of `ZigGeneratedClasses.cpp`, plus its class declaration in the header, to the 171 lines of `GeneratedGlob.cpp` and 138 of `GeneratedGlob.h`, which now include the options dictionary and the string-or-options union that were parsed by hand. <details><summary>Schema, C++, Rust</summary> Schema. Before, `Glob.classes.ts`: ```ts define({ name: "Glob", construct: true, finalize: true, hasPendingActivity: true, configurable: false, klass: {}, JSType: "0b11101110", proto: { // … match: { fn: "match", length: 1, }, }, }), ``` After, `Glob.bind.ts`: ```ts export const Glob = b.class({ rustPath: "crate::api::glob::Glob", configurable: false, construct: { args: { pattern: b.string.utf8.owned }, }, proto: { // … match: { args: { str: b.string.utf8 }, returns: b.bool, }, }, }); ``` Generated C++. Before, in `ZigGeneratedClasses.cpp`: ```cpp JSC_DEFINE_HOST_FUNCTION(GlobPrototype__matchCallback, (JSGlobalObject * lexicalGlobalObject, CallFrame* callFrame)) { auto& vm = JSC::getVM(lexicalGlobalObject); auto scope = DECLARE_THROW_SCOPE(vm); JSGlob* thisObject = dynamicDowncast<JSGlob>(callFrame->thisValue()); if (!thisObject) [[unlikely]] { RELEASE_AND_RETURN(scope, Bun::throwInvalidThisCallError(lexicalGlobalObject, callFrame, "Glob"_s)); } JSC::EnsureStillAliveScope thisArg = JSC::EnsureStillAliveScope(thisObject); #if BUN_DEBUG /** View the file name of the JS file that called this function * from a debugger */ SourceOrigin sourceOrigin = callFrame->callerSourceOrigin(vm); auto fileNameUTF8 = sourceOrigin.string().utf8(); const char* fileName = fileNameUTF8.legacyCStringPointer(); static const char* lastFileName = nullptr; if (lastFileName != fileName) { lastFileName = fileName; } JSC::EncodedJSValue result = GlobPrototype__match(thisObject->wrapped(), lexicalGlobalObject, callFrame); ASSERT_WITH_MESSAGE(!JSValue::decode(result).isEmpty() or DECLARE_TOP_EXCEPTION_SCOPE(vm).exception() != 0, "Glob.match returned an empty value without an exception"); #if ASSERT_ENABLED JSValue decodedValue = JSValue::decode(result); if (!decodedValue.isEmpty() && decodedValue.isCell()) { JSC::Integrity::auditCellFully(vm, decodedValue.asCell()); } #endif return result; #endif return GlobPrototype__match(thisObject->wrapped(), lexicalGlobalObject, callFrame); } ``` After, in `GeneratedGlob.cpp`: ```cpp JSC_DEFINE_HOST_FUNCTION(jsGlobPrototypeFunction_match, (JSC::JSGlobalObject * globalObject, JSC::CallFrame* callFrame)) { auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); auto* thisObject = dynamicDowncast<Bun::Bindgen::Generated::JSGlob>(callFrame->thisValue()); if (!thisObject) [[unlikely]] RELEASE_AND_RETURN(scope, Bun::throwInvalidThisCallError(globalObject, callFrame, "Glob"_s)); JSC::EnsureStillAliveScope argument0 = callFrame->argument(0); [[maybe_unused]] auto ctx0 = Bun::Bindgen::LiteralConversionContext { "str"_s }; Bun::Bindgen::IDLStrictUTF8String::ImplementationType converted0; WebCore::Converter<Bun::Bindgen::IDLStrictUTF8String>::convertInto(*globalObject, argument0.value(), ctx0, converted0); RETURN_IF_EXCEPTION(scope, {}); RELEASE_AND_RETURN(scope, Bun__Bindgen__GlobPrototype__match(globalObject, thisObject, Bun::Bindgen::toExtern(converted0))); } ``` Generated Rust. Before, in `generated_classes.rs`: ```rust bun_jsc::jsc_host_abi! { #[allow(dead_code, unreachable_pub, unused)] #[unsafe(no_mangle)] pub unsafe fn GlobPrototype__match(this: &Glob, global: &JSGlobalObject, callframe: &CallFrame) -> JSValue { host_fn::host_fn_this_shared(this, global, callframe, |t, g, c| Glob::r#match(t, g, c)) } } ``` After, a trait the struct must implement, so a missing or mistyped method is a compile error, and the function C++ calls: ```rust pub trait GlobClass: bindgen::JsPayload { /// What the members declared `throws` fail with. type Error<'a>: bindgen::IntoError<'a>; fn constructor(pattern: bun_core::Utf8Bytes<'static>) -> Self; /// The constructor function, `Glob` in JavaScript. fn get_constructor<'a>(heap: &bindgen::JsHeap<'a>) -> bindgen::JsLocal<'a> { // SAFETY: cannot throw; makes the class's objects on first use. unsafe { bindgen::JsLocal::from_extern(Bun__Bindgen__Glob__getConstructor(heap.global_ptr()), heap) } } fn scan<'a>(this: bindgen::JsWrapper<'a, Self>, cx: &mut bindgen::JsScope<'a>, options: Option<GlobScanArgument<'a>>) -> Result<bindgen::JsLocal<'a>, Self::Error<'a>>; fn scan_sync<'a>(&self, heap: &bindgen::JsHeap<'a>, options: Option<GlobScanArgument<'a>>) -> Result<bindgen::JsLocal<'a>, Self::Error<'a>>; fn r#match<'a>(&self, str: &'a [u8]) -> bool; } #[unsafe(no_mangle)] pub unsafe extern "C" fn Bun__Bindgen__GlobPrototype__match(global: *mut JSGlobalObject, this: *mut core::ffi::c_void, str: bindgen::extern_types::ExternBytes) -> JSValue { // SAFETY: called by generated C++, whose frame holds what it passes until this returns. unsafe { bindgen::JsScope::with(global, |cx| { let this = bindgen::JsWrapper::<crate::api::glob::Glob>::from_extern(core::ptr::NonNull::new_unchecked(this), &cx.heap); let heap = &cx.heap; let str = str.get(heap); let result = Ok::<_, core::convert::Infallible>(<crate::api::glob::Glob as GlobClass>::r#match(this.payload_ref(), str)); bindgen::host_fn_return(cx, result, |heap, value| bindgen::to_js::bool(heap, value)) }) } } ``` The body, in `src/runtime/api/glob.rs`. Before: ```rust #[bun_jsc::host_fn(method)] pub(crate) fn r#match( &self, global_this: &JSGlobalObject, callframe: &CallFrame, ) -> JsResult<JSValue> { // SAFETY: bun_vm() returns a non-null *mut to the live VirtualMachine for this global. let mut arguments = ArgumentsSlice::init(global_this.bun_vm(), callframe.arguments()); let Some(str_arg) = arguments.next_eat() else { return Err(global_this.throw(format_args!( "Glob.matchString: expected 1 arguments, got 0" ))); }; if !str_arg.is_string() { return Err(global_this.throw(format_args!( "Glob.matchString: first argument is not a string" ))); } let str = str_arg.to_utf8(global_this)?; let matched = bun_glob::matcher::match_prepared::<{ cfg!(windows) }>( self.pattern.slice(), &self.shape, str.slice(), bun_glob::Dots::Match, ); Ok(JSValue::from(matched.matches())) } ``` After: ```rust fn r#match<'a>(&self, str: &'a [u8]) -> bool { bun_glob::matcher::match_prepared::<{ cfg!(windows) }>( self.pattern.slice(), &self.shape, str, bun_glob::Dots::Match, ) .matches() } ``` Where the Rust struct lives. Before, in `ZigGeneratedClasses.h` and `.cpp`: ```cpp class JSGlob final : public JSC::JSDestructibleObject { // … void* m_ctx { nullptr }; // … }; JSGlob::~JSGlob() { if (m_ctx) [[likely]] { GlobClass__finalize(m_ctx); } } ``` After, in `GeneratedGlob.h`; `JSWrapper` is one hand-written template in `src/jsc/bindings/Bindgen/JSWrapper.h`, shared by every class: ```cpp struct GlobTraits { static constexpr auto className = "Glob"_s; static constexpr unsigned classIndex = 3; static constexpr unsigned numberOfInternalFields = 0; static constexpr unsigned numberOfCachedGetters = 0; static size_t payloadSize() { return Bun__Bindgen__Glob__payloadSize; } static void finalize(void* payload) { Bun__Bindgen__GlobClass__finalize(payload); } }; using JSGlob = JSWrapper<GlobTraits>; ``` </details> **4. A union: `TLSOptions.key`, `null | string | buffer | Blob | array`.** v2 generated nothing for a union: a variadic template tried each alternative's `tryConvert` in turn, each returning a `std::optional`, into a `std::variant`, which `std::visit` then lowered twice. The generator now knows the kinds of value each alternative takes, so it writes one decision, with JSC's own predicates: whether the value is a cell, once; the tests on the type byte, which the compiler loads once; the class check last. The chosen alternative is converted into its own member of a plain struct. <details><summary>C++</summary> Before, hand-written in `src/jsc/bindings/BunIDLConvert.h`, for every union: ```cpp template<Bun::IDLConversionContext Ctx> static std::optional<ReturnType> tryConvert( JSC::JSGlobalObject& globalObject, JSC::JSValue value, Ctx& ctx) { auto& vm = JSC::getVM(&globalObject); auto scope = DECLARE_THROW_SCOPE(vm); std::optional<ReturnType> result; auto tryAlternative = [&]<typename T>() -> bool { auto alternativeResult = Bun::tryConvertIDL<T>(globalObject, value, ctx); RETURN_IF_EXCEPTION(scope, true); if (!alternativeResult.has_value()) { return false; } result = ReturnType { std::move(*alternativeResult) }; return true; }; (tryAlternative.template operator()<IDL>() || ...); return result; } ``` After, generated in `GeneratedSSLConfig.h`: ```cpp struct TLSOptionsFile { TLSOptionsFile() {} Bun::Bindgen::IDLStrictString::ImplementationType string {}; Bun::Bindgen::IDLArrayBufferRef::ImplementationType buffer {}; Bun::Bindgen::IDLBlobRef::ImplementationType file {}; Bun::Bindgen::IDLArray<Bun::Bindgen::IDLTaggedUnion<Bun::Bindgen::Generated::TLSOptionsSingleFile>>::ImplementationType array {}; uint8_t tag; }; template<Bun::IDLConversionContext Ctx> static void convertInto(JSC::JSGlobalObject& lexicalGlobalObject, JSC::JSValue value, Ctx& ctx, ReturnType& result) { auto* globalObject = &lexicalGlobalObject; auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); unsigned choice = 0; if (value.isCell()) { auto* cell = value.asCell(); if (cell->isString()) choice = 2; else if (cell->inherits<JSC::JSArrayBuffer>() || cell->inherits<JSC::JSArrayBufferView>()) choice = 3; else if (JSC::isJSArray(cell)) choice = 5; else if (cell->inherits<WebCore::JSBlob>()) choice = 4; } else if (value.isUndefinedOrNull()) choice = 1; switch (choice) { case 1: { result.tag = 0; return; } case 2: { result.tag = 1; scope.release(); result.string = Bun::convertIDL<Bun::Bindgen::IDLStrictString>(*globalObject, value, ctx); return; } case 3: { if (auto converted = Bun::tryConvertIDL<Bun::Bindgen::IDLArrayBufferRef>(*globalObject, value, ctx)) { result.buffer = WTF::move(*converted); result.tag = 2; return; } RETURN_IF_EXCEPTION(scope, void()); break; } case 4: { if (auto converted = Bun::tryConvertIDL<Bun::Bindgen::IDLBlobRef>(*globalObject, value, ctx)) { result.file = WTF::move(*converted); result.tag = 3; return; } RETURN_IF_EXCEPTION(scope, void()); break; } case 5: { result.tag = 4; scope.release(); result.array = Bun::convertIDL<Bun::Bindgen::IDLArray<Bun::Bindgen::IDLTaggedUnion<Bun::Bindgen::Generated::TLSOptionsSingleFile>>>(*globalObject, value, ctx); return; } } ctx.template throwNoMatchInUnion<Bun::Bindgen::IDLStrictString, Bun::Bindgen::IDLArrayBufferRef, Bun::Bindgen::IDLBlobRef, Bun::Bindgen::IDLArray<Bun::Bindgen::IDLTaggedUnion<Bun::Bindgen::Generated::TLSOptionsSingleFile>>, Bun::Bindgen::IDLStrictNull>(*globalObject, scope); } ``` </details> **5. The decision, worked through.** `TestAnything` in the test schema has an alternative of every kind, so its converter shows the whole rule: a value goes to the first alternative, in written order, that takes its kind. ```ts export const TestAnything = b.union({ nothing: b.null, flag: b.bool, integer: b.i32, text: b.string, bytes: b.bufferSource.arrayBuffer, blob: b.blob, counter: TestCounter, token: TestToken, list: b.array(b.f64), options: TestOptions, }); ``` | Alternative | Kinds it takes | Told by | | --- | --- | --- | | `nothing` | `undefined`, `null` | the bits of the value | | `flag` | boolean | the bits of the value | | `integer` | number, BigInt | the bits of the value; the type byte for a BigInt on the heap | | `text` | string | the type byte | | `bytes` | `ArrayBuffer`, view; refuses a shared one | the type byte | | `list` | array | the type byte | | `blob` | `Blob`; refuses a detached one | its `ClassInfo` | | `counter`, `token` | their instances | the type byte and the `ClassInfo` in the cell | | `options` | every kind: a dictionary reports its own error | nothing: it gets what is left | The written order decides who wins a kind. The order of the tests is the generator's: cheapest first, whatever the schema's order. `options` costs no test. `bytes` can refuse, so its case goes on to the next alternative that takes the kind, which is `options`. <details><summary>C++, generated in <code>GeneratedBindgenTest.h</code> (cases 1 to 4, 6, 8 and 9 left out)</summary> ```cpp unsigned choice = 0; if (value.isCell()) { auto* cell = value.asCell(); if (cell->isHeapBigInt()) choice = 3; else if (cell->isString()) choice = 4; else if (cell->inherits<JSC::JSArrayBuffer>() || cell->inherits<JSC::JSArrayBufferView>()) choice = 5; else if (JSC::isJSArray(cell)) choice = 9; else if (cell->inherits<WebCore::JSBlob>()) choice = 6; else if (cell->inherits<Bun::Bindgen::Generated::JSTestCounter>()) choice = 7; else if (cell->inherits<Bun::Bindgen::Generated::JSTestToken>()) choice = 8; } else if (value.isUndefinedOrNull()) choice = 1; else if (value.isBoolean()) choice = 2; else if (value.isNumber() || value.isBigInt32()) choice = 3; switch (choice) { case 0: { result.tag = 9; scope.release(); auto& resultOptionsValue = result.options.emplace(); Bun::Bindgen::convertDictionaryInto(*globalObject, value, resultOptionsValue); return; } case 5: { if (auto converted = Bun::tryConvertIDL<Bun::Bindgen::IDLArrayBufferRef>(*globalObject, value, ctx)) { result.bytes = WTF::move(*converted); result.tag = 4; return; } RETURN_IF_EXCEPTION(scope, void()); result.tag = 9; scope.release(); auto& resultOptionsValue = result.options.emplace(); Bun::Bindgen::convertDictionaryInto(*globalObject, value, resultOptionsValue); return; } case 7: { result.tag = 6; scope.release(); result.counter = Bun::convertIDL<Bun::Bindgen::IDLRawInterface<Bun::Bindgen::Generated::JSTestCounter>>(*globalObject, value, ctx); return; } } ``` </details> | Value | Goes to | What ran | | --- | --- | --- | | `5` | `integer` | `isCell`, then three tests on the bits | | `"x"` | `text` | `isCell`, one load of the type byte, two compares | | `new Uint8Array(4)` | `bytes` | the same load, four tests | | `new Uint8Array(new SharedArrayBuffer(2))` | `options` | as above; `bytes` refuses it, `options` reads it as an object | | `new TestCounter(3)` | `counter` | the byte compares, the `Blob` check, then a byte and a pointer compare | | `{ count: 8 }` | `options` | every test fails; no conversion was tried | **6. Overloads.** The same rule, argument by argument: each keeps the signatures that take its kind. An argument that was not passed is `undefined`, so nothing looks at how many there are, and `f(x)` and `f(x, undefined)` are the same call. A signature that has no such argument ignores it. For `os.setPriority` that comes to the test Node itself makes, `priority === undefined`: <details><summary>C++</summary> Before: ```cpp extern "C" SYSV_ABI JSC::EncodedJSValue bindgen_Node_os_jsSetPriority(JSC::JSGlobalObject* global, JSC::CallFrame* callFrame) { auto& vm = JSC::getVM(global); auto throwScope = DECLARE_THROW_SCOPE(vm); size_t argumentCount = std::min<size_t>(callFrame->argumentCount(), 3); if (argumentCount < 1) { return JSC::throwVMError(global, throwScope, createNotEnoughArgumentsError(global)); } if (argumentCount >= 2) { RELEASE_AND_RETURN(throwScope, bindgen_Node_os_jsSetPriority_v1(global, callFrame)); } RELEASE_AND_RETURN(throwScope, bindgen_Node_os_jsSetPriority_v2(global, callFrame)); } ``` After: ```cpp JSC_DEFINE_HOST_FUNCTION(jsSetPriority, (JSC::JSGlobalObject * globalObject, JSC::CallFrame* callFrame)) { auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); JSC::JSValue argument1 = callFrame->argument(1); if (argument1.isUndefined()) RELEASE_AND_RETURN(scope, jsSetPriority2(globalObject, callFrame)); RELEASE_AND_RETURN(scope, jsSetPriority1(globalObject, callFrame)); } ``` </details> `overloaded` in the test schema needs two arguments to tell its six signatures apart: ```ts 1 (text: b.string.utf8, times: b.u32.default(1)) 2 (text: b.string.utf8, upper: b.bool) 3 (a: b.f64, b: b.f64) 4 (number: b.f64) 5 (list: b.array(b.f64)) 6 (counter: TestCounter) ``` A string leaves 1 and 2, and the second argument picks between them: a boolean is 2, anything else is 1. A number leaves 3 and 4, and 3 wins if the second is a number too. An argument that none of them takes rules nothing out. If several are left at the end, the one written first wins, as in a union, and its conversion reports what is wrong: `overloaded({})` throws `text must be a string`. So of two signatures that start the same, the longer goes first. A signature that no call can reach is an error when the schema is loaded. <details><summary>C++, generated in <code>GeneratedBindgenTest.cpp</code></summary> ```cpp JSC_DEFINE_HOST_FUNCTION(jsOverloaded, (JSC::JSGlobalObject * globalObject, JSC::CallFrame* callFrame)) { auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); JSC::JSValue argument0 = callFrame->argument(0); unsigned choice0 = 0; if (argument0.isCell()) { auto* cell = argument0.asCell(); if (cell->isString() || cell->inherits<JSC::StringObject>()) choice0 = 1; else if (JSC::isJSArray(cell)) choice0 = 3; else if (cell->inherits<Bun::Bindgen::Generated::JSTestCounter>()) choice0 = 4; } else if (argument0.isNumber()) choice0 = 2; switch (choice0) { case 1: { JSC::JSValue argument1 = callFrame->argument(1); if (argument1.isBoolean()) RELEASE_AND_RETURN(scope, jsOverloaded2(globalObject, callFrame)); RELEASE_AND_RETURN(scope, jsOverloaded1(globalObject, callFrame)); } case 2: { JSC::JSValue argument1 = callFrame->argument(1); if (argument1.isNumber()) RELEASE_AND_RETURN(scope, jsOverloaded3(globalObject, callFrame)); RELEASE_AND_RETURN(scope, jsOverloaded4(globalObject, callFrame)); } case 3: RELEASE_AND_RETURN(scope, jsOverloaded5(globalObject, callFrame)); case 4: RELEASE_AND_RETURN(scope, jsOverloaded6(globalObject, callFrame)); default: { JSC::JSValue argument1 = callFrame->argument(1); unsigned choice1 = 0; if (argument1.isCell()) { auto* cell = argument1.asCell(); if (cell->isHeapBigInt()) choice1 = 1; } else if (argument1.isUndefined() || argument1.isNumber() || argument1.isBigInt32()) choice1 = 1; else if (argument1.isBoolean()) choice1 = 2; switch (choice1) { case 1: RELEASE_AND_RETURN(scope, jsOverloaded1(globalObject, callFrame)); case 2: RELEASE_AND_RETURN(scope, jsOverloaded2(globalObject, callFrame)); default: RELEASE_AND_RETURN(scope, jsOverloaded4(globalObject, callFrame)); } } } } ``` </details> **7. Defaults.** A default written in the schema is assigned member by member to the object the conversion would have filled, nested dictionaries, unions and `b.any` included. Nothing is built and then moved. <details><summary>Schema and C++</summary> ```ts export const TestInner = b.dictionary({ a: b.bool.default(true), b: b.i32.default(7), name: b.string.utf8.default("inner"), }); export const TestChoice = b.union({ none: b.null, flag: b.bool }); /** A string is a color. What `bytes` refuses, and everything else, is kept as it is. */ export const TestColorOrRest = b.union({ color: TestColor, bytes: b.bufferSource, rest: b.any }); export const TestOuter = b.dictionary({ inner: TestInner.nullable, value: b.any, choice: TestChoice, rest: TestColorOrRest, counter: TestCounter.nullable, }); /** What a default written in the schema gives each member, the nested ones included. */ export const describeDefaults = b.fn({ args: { options: TestOuter.default({ inner: {} }) }, returns: b.string.utf8, capability: "heap", }); ``` ```cpp JSC_DEFINE_HOST_FUNCTION(jsDescribeDefaults, (JSC::JSGlobalObject * globalObject, JSC::CallFrame* callFrame)) { auto& vm = JSC::getVM(globalObject); auto scope = DECLARE_THROW_SCOPE(vm); JSC::EnsureStillAliveScope argument0 = callFrame->argument(0); [[maybe_unused]] auto ctx0 = Bun::Bindgen::LiteralConversionContext { "options"_s }; WebCore::IDLDictionary<Bun::Bindgen::Generated::TestOuter>::ImplementationType converted0; if (argument0.value().isUndefined()) { auto& converted0InnerValue = converted0.inner.emplace(); converted0InnerValue.a = true; converted0InnerValue.b = int32_t { 7 }; converted0InnerValue.name.initialize("inner"_s); converted0.value = JSC::jsUndefined(); converted0.choice.tag = 0; converted0.rest.rest = JSC::jsUndefined(); converted0.rest.tag = 2; converted0.counter = WebCore::IDLNullable<Bun::Bindgen::IDLRawInterface<Bun::Bindgen::Generated::JSTestCounter>>::nullValue(); } else { Bun::Bindgen::convertDictionaryInto(*globalObject, argument0.value(), converted0); } RETURN_IF_EXCEPTION(scope, {}); RELEASE_AND_RETURN(scope, Bun__Bindgen__BindgenTest__describeDefaults(globalObject, Bun::Bindgen::toExtern(converted0))); } ``` </details> **8. The receiver check, as machine code.** The start of `Glob#match` on x64. `rbx` / `r15` is `this`. Before: the `Structure`'s id out of the cell, the base of the structure heap out of `g_config`, then the `ClassInfo` out of the `Structure`, each load waiting for the one before. After: two compares against the cell. <details><summary>x64</summary> Before, CI's release build of `7583003d8`: ```asm test rbx, rcx ; a cell? jne <throw> mov ecx, dword ptr [rbx] ; structure id and ecx, -0x2 mov rsi, qword ptr [rip + 0x3551d62] ; g_config: base of the structure heap cmp qword ptr [rsi + rcx + 0x50], 0xf81c28 ; Structure::classInfo == JSGlob::info() jne <throw> ``` After, a local release build: ```asm test r15, rax ; a cell? jne <throw> cmp byte ptr [r15 + 0x5], -0x12 ; type byte == JSDOMWrapperType jne <throw> cmp qword ptr [r15 + 0x10], 0xf90b78 ; JSDestructibleObject::classInfo == JSGlob::info() jne <throw> ``` </details> #### Safety The Rust a binding is written against is `bun_jsc::bindgen` (`src/jsc/bindgen.rs`). Three rules, enforced by the borrow checker: 1. Running JavaScript, or anything that can call back into Rust, takes `&mut JsScope<'a>`. 2. A pointer into memory JavaScript can free, detach, resize or alias (typed array bytes, another object's Rust payload) is only reachable inside a closure that borrows a field of `JsScope`, so rule 1 cannot happen while it is live. 3. A JS value is a `JsLocal<'a>`, a `JSValue` tied to the host call. It cannot be stored in a struct, captured by a task or sent to a thread. A `Strong`, or one of the class's `internalFields`, which the GC visits, holds one longer. Everything a body is lent takes its lifetime from the call: the generated thunk makes each `&[u8]`, `JsLocal` and `JsWrapper` with the call's `&JsHeap<'a>`, so a body that declares `&'static [u8]` does not compile. Three known gaps: `value.to_js()` gives back a plain `JSValue`, which is as storable as it is today; `Vec<JsLocal>` compiles although the GC does not look inside it; and a capability limits what goes through this API, while the older one has other ways to a `&JSGlobalObject` (`VirtualMachine::get().global()`). What the borrow checker cannot see is what happens in the generated C++ between converting an argument and entering Rust, where a later conversion can run JavaScript. There, each dictionary member is read right before it is converted; the pointer and length of `b.bufferSource.bytes` are read after every other argument is converted; and `b.string.utf8` holds the `GCOwnedDataScope` that `JSString::view()` returns, so JSC keeps the characters alive, the base's for a substring. What a function may do is a parameter it is given (`capability` in the schema): nothing, `&JsHeap<'a>` (allocate, describe an error) or `&mut JsScope<'a>` (run JS). The less it can do, the more it is handed directly: a method that cannot run JS gets `&self` / `&mut self` and `&[u8]`; one that can gets a `JsWrapper<'a, Self>` and borrows the payload for as long as nothing else can run. ```rust this.with_mut(&mut cx.payloads, |socket| { callback.call(cx, ..)?; // error[E0500]: closure requires unique access to `*cx` socket.bytes_written += 1; }); ``` `.withValue` on an argument gives both forms, the converted one for now and the value itself for whatever holds it longer. #### Choosing between types A union and a set of overloads follow one rule: a value goes to the first alternative, in written order, that takes its *kind*. The kinds are what JSC can tell apart without running JavaScript, cheapest first: the bits of the value (`undefined`, `null`, boolean, number), the type byte in the cell's header (string, `String` object, array, `ArrayBuffer`, view, BigInt), then the `ClassInfo` of a class. A type that coerces or reports its own error (a dictionary, an enumeration, `.loose`, a range, `b.any`) takes every kind, so it goes last. A type that refuses some values of its kinds (a shared buffer, a detached `Blob`) passes those on to the next alternative. From that the generator writes one decision per union or per overloaded argument (examples 4 to 6 above): whether the value is a cell, once; tests on the type byte, then class checks; no test for a kind that is treated as everything else is. An alternative or a signature that no value can reach is an error when the schema is loaded. Nothing looks at the number of arguments: one that was not passed is `undefined`, for a conversion and for choosing a signature alike. `b.overloads` is for functions, not yet for the methods of a class. Strings as UTF-8, dictionaries and unions are converted in the place they end up in. `JSStringUTF8View` is neither copyable nor movable, so the compiler holds every schema to that. #### Classes The Rust struct is stored inside the JS cell (`Bun::Bindgen::JSWrapper<Traits>`), after its internal fields, so a wrapper and its payload have one lifetime and one allocation. The Rust constructor runs before the cell exists, and nothing runs between the allocation and the write of the payload. The collector reads the internal fields and nothing else: no Rust runs on its threads, so there is no `hasPendingActivity` or `estimatedSize` callback to get wrong. The receiver check reads the cell and nothing else. A wrapper has the type byte `Glob` already had (`JSDOMWrapperType`), every cell with that byte is a `JSDestructibleObject`, and that holds its `ClassInfo`, so `dynamicDowncast` is a byte compare and a pointer compare with no `Structure` load. #### Performance Measured on CI's own release binaries (`bun-linux-x64`): the merge base `f2405c382` (build 126282) against `52f5c7a0e` (build 126343). Each operation runs N and 4N times in fresh processes under `perf stat`; the difference divided by 3N is the cost per call, without startup or JIT warm-up. Minimum of 7 interleaved rounds, pinned to a core reserved with a systemd cpuset, its SMT sibling idle, `BUN_JSC_useConcurrentJIT=0`. The control is the merge base against a copy of itself: within 0.1% on instructions except the two `Bun.gc` rows (0.4% and 2.3%), within 3.3% on cycles. Every operation printed the same result on both binaries. | Operation | Instructions | Cycles | Time per call | | --- | --- | --- | --- | | `server.reload({ socket })`, 7 handlers and `binaryType` | −56.4% | **−61.3%** | 310 → 121 ns | | `glob.match`, 14 ASCII characters | −7.4% | **−54.0%** | 44 → 20 ns | | `new Bun.Glob(pattern)` | −52.2% | **−50.9%** | 46 → 23 ns | | `glob.match`, UTF-16 | −36.4% | **−49.7%** | 68 → 35 ns | | `glob.match`, Latin-1 | −36.7% | **−45.9%** | 61 → 33 ns | | `glob.scanSync({ 6 options })`, empty directory | −39.2% | **−37.7%** | 1,282 → 1,023 ns | | `Bun.listen(..).stop()` | −45.1% | **−33.8%** | 3,320 → 3,012 ns | | `Module._stat`, UTF-16 path | −33.5% | −32.2% | 343 → 330 ns | | `glob.scanSync({ cwd })`, 4 files | −26.2% | −29.8% | 2,294 → 1,992 ns | | `os.homedir()` | +12.8% | −23.0% | 23 → 18 ns | | `glob.scanSync(cwd)`, empty directory | −16.9% | −21.5% | 1,084 → 977 ns | | `os.getPriority(pid)` | −19.3% | −15.8% | 37 → 36 ns | | `Module._stat`, ASCII path | +2.0% | −14.6% | 152 → 149 ns | | `Bun.gc(true)` with 200,000 live `Glob`s | −23.6% | −13.1% | 4.04 → 3.48 ms | | `os.version()` | −4.6% | −12.9% | 118 → 110 ns | | `Bun.$.braces(str, options)` | −9.1% | −11.2% | 279 → 247 ns | | `os.setPriority(priority)` | −7.7% | −8.2% | 79 → 78 ns | | `glob.match`, 200 ASCII characters | −0.1% | −8.1% | 279 → 258 ns | | `os.userInfo({ encoding })` | −3.5% | −7.6% | 326 → 306 ns | **No different from the merge base**, that is, inside the control's spread on cycles: `os.freemem`, `totalmem`, `loadavg`, `uptime`, `hostname`, `release`, `getPriority()`, `userInfo()`, `networkInterfaces`; `braces` without options; `tls.createSecureContext({ key, cert })` and with `ca` as an array of three (−0.6% and −0.5% instructions: parsing the options is nothing next to building the context); creating a `Worker` (+0.0% instructions); `Bun.gc(true)` on a small heap (+0.4% instructions, −0.4% cycles). No row is slower. The most instructions added are `os.homedir()`'s 53 and `Module._stat`'s 6, both faster in cycles, and `os.totalmem()`'s 2. Memory: a live `Glob` takes 143 to 144 bytes where it took 203 to 205 (500,000 of them, 7 runs of each binary). The anonymous memory of an empty script is the same, 6,132 to 6,136 kB on all three binaries. Where it comes from: - **Member names** are `Identifier`s made once per VM. v1 and v2 called `Identifier::fromString` per member per call; `Glob` looked each option up by name from Rust. - **Members of a plain object** are read with `JSObject::getDirect`, which is the whole answer because the lookup stops before `Object.prototype`. Each is read right before it is converted, and whether the shortcut still holds is asked each time. - **`b.string.utf8`** is the string's own bytes when it is 8-bit ASCII, otherwise a 128-byte buffer in the frame. A body that stores the string asks for `b.string.utf8.owned` instead and gets a `Utf8Bytes<'static>` that shares the buffer. The view is built in its final slot and cannot be moved. - **The payload is inside the cell**: one allocation, no finalizer indirection, and no `JSC::Weak` handle per object for every collection to walk, with a `hasPendingActivity` call into Rust for each object not marked yet. - **The right type costs its test.** The fast path of every strict conversion is inlined into the host function, and reporting a failure is out of line, in the base template all of them share. - **Nothing is built to be thrown away.** A union is a struct with a member per alternative, the large ones made when chosen; a default is assigned to the member it belongs to. - **The layers vanish**: with LTO the C++ host function, the Rust thunk and the body of `os.totalmem` are one function with no calls between them, as they were with v1. - Returned dictionaries and returned enumeration values (per-VM strings) have no user outside the test schema yet, so they are not in the table. What it costs: | What | How much | | --- | --- | | Binary size | smaller on 9 of the 12 targets, by 7.9 to 32.2 KB: −12.0 KB on linux-x64, −32.2 KB on darwin-aarch64, −20.0 KB on windows-x64. The three aarch64 Linux targets are within 0.1 KB (CI's report, build 126343 against the merge base's 126282) | | Memory per VM | one pointer per member name (61 in a release build) and per value of a returned enumeration (none yet) | | Memory per global object | a `LazyClassStructure` per class (1: `Glob`'s, moved here from `.classes.ts`) | | Stack | the 128-byte buffer and its bookkeeping per `b.string.utf8` argument or member | The tables are indexed by positions fixed at build time, filled once and never invalidated. Nothing is keyed on a runtime value. #### Ported - v1: `node:os` (13 functions, and `hostCpuCount`, which was a field of the `$rust` binding object), `Bun.$.braces`, `Bun.gc`, `fmtString`, `Module._stat`, `getDeinitCountForTesting`, the bindgen test functions. The `$rust("node_os.rs", "createNodeOsBinding")` and `$rust("bindgen_test.rs", ..)` entries are gone with them. - v2: `SSLConfig.bind.ts` (`TLSOptions`), `SocketConfig.bind.ts` (`SocketOptions`, `SocketHandler`). `FakeTimersConfig.bindv2.ts` had no consumer and is removed. - `.classes.ts`: `Glob`, as the first class. Deleted with them: the hand-copied mirror of v2's types in `src/jsc/generated.rs`, the hand-written dispatch shims in `src/runtime/hw_exports.rs`, `GeneratedBindings.rs`, and what only they called (`bun_collections::array_list`, `bun_alloc::realloc_raw`, `JSArray::create`, `host_fn::new_runtime_function`). Every generated struct that crosses the ABI has its size and alignment asserted on both sides. bindgen v2's C++ converter library is kept: it is what fixes the error messages. What only generated code uses is in `src/jsc/bindings/Bindgen/`, in `Bun::Bindgen`; the three headers `JSDOMConvert*.h` builds on (`BunIDLConvertBase.h`, `BunIDLConvertContext.h`, `BunIDLHumanReadable.h`) stay beside it. #### Behaviour changes What else differs from `main`. Each has a test in the API's own file. - **An argument that was not passed is `undefined`.** `new Bun.Glob()` and `glob.match()` throw what `new Bun.Glob(undefined)` and `glob.match(undefined)` do; they had a message of their own, `expected 1 arguments, got 0`. `Bun.$.braces()` is `braces(undefined)`, which returns `["undefined"]`, and `Module._stat()` is `_stat(undefined)`, which returns `-1`; both threw `ERR_MISSING_ARGS`. - **`Bun.Glob` errors.** Wrong arguments throw `TypeError` with a code instead of `Error`, and the messages name the argument, not the function: `ERR_INVALID_ARG_TYPE` `pattern must be a string`, `str must be a string`, `GlobScanOptions must be an object`, `GlobScanOptions.cwd must be a string`. - All `Glob` scan options are read before a `cwd` that is too long is reported, so a `dot` getter runs first. - `Bun.Glob.length` is 1 (was 0). #### Not in this PR The other `.classes.ts` classes and the hand-declared externs. Bodies written in C++. Mutable byte views, external memory accounting, typed callbacks and promises, namespaces. A lint against `Vec<JsLocal>` (`MarkedArgumentBuffer` is the right container). ### How did you verify your code works? - `test/internal/bindgen.test.ts`, through `src/jsc/bindgen_test.bind.ts`. A `*_test.bind.ts` schema is only part of a build with assertions (a debug build, CI's ASAN lane), so none of it ships: a release binary has no symbol and no string of it, and the file skips there. Strings (ASCII, Latin-1, UTF-16, ropes, unpaired surrogates, `String` objects), 19 kinds of object as a dictionary that must all give the same answer, a polluted `Object.prototype`, mutation between reads, by a getter and by a conversion, order and count of getter calls, buffers that are detached, shrunk or grown by a callback or by a later argument's conversion, shared memory, unions, arrays, thrown values. For classes: receiver checks, subclassing, `Reflect.construct`, a listener that re-enters the object, cached getters, a class without `construct`, a payload that holds a `Strong` on another object, and that the Rust value is dropped on collection. What that schema does not use (`.falsyAsNull`, `.validateInteger`, `fromJS`, `privateSymbol`, `builtin`) is covered by the suites of the APIs that do. - The late-buffer test was checked against the final code with its fix undone: it reads stale bytes. - The bugs fixed and the behaviour changes above: `test/js/node/os/os.test.js`, `test/js/bun/shell/brace.test.ts`, `test/js/bun/glob/proto.test.ts`, `test/js/bun/net/socket.test.ts`, `test/js/node/module/node-module-module.test.js`. They fail on a build of the merge base. - The lifetimes: `bindgen.rs` as it is, in a scratch crate with the rest of `bun_jsc` stubbed, against 22 attempts in safe Rust to get around them (a `'static` parameter, a static, `dyn Any`, a thread, a payload field, a closure that returns its argument, two `with_mut`, JavaScript inside `with` or `with_bytes`). rustc refuses each. - `getPriority` and `setPriority` with 18 bad arguments give Node v26.7.0's errors, word for word. - The generator writes byte-identical files under node and bun. - Existing suites that cover the ported code: `test/js/node/os`, `test/js/bun/glob`, `test/js/bun/shell/brace.test.ts`, `test/internal/highlighter.test.ts`, `test/js/bun/net/{socket,tcp-server,socket-retention,tls-fatal-error-closes}.test.ts`, `test/js/node/tls/node-tls-{connect,server}.test.ts`, `test/js/bun/http/bun-serve-ssl.test.ts`, `test/internal/build-codegen-declared-outputs.test.ts`. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…er consolidation (oven-sh#45307) ### What does this PR do? Follow-up to #44968. It fixes the regressions that showed up after that PR landed, plus older package manager bugs found while checking them, and makes `bun.lock` independent of network timing. Almost every fix comes with a test that fails on `main`. Some earlier changes caused regressions of their own and were taken out again; see "Known to be still open". Fixes #24190 Fixes #20015 Fixes #45070 Fixes #29944 #### Behavior changes to sign off One line each. Details are in the groups below. "Reversible" means the change can be backed out on its own, in the place named. - Dependencies are resolved in one fixed order, not as their manifests arrive (see "Resolution and lockfile"). With every manifest cached, that is the order `main` uses. Not a small revert. - Both linkers install through a `node_modules` that is a symlink (the project's or a workspace's), remove nothing behind it in their cleanup passes, and warn; `bun prune` stops with an error. Hoisted followed such links before #44968; isolated replaced the project's with a real folder (npm replaces, pnpm follows). Not a small revert. - Behind such a link, a file or folder in `.bin` named like a package's command stays, and the install fails with `EEXIST`. Reversible: `is_behind_link` in `bin.rs`. - Switching linkers: an entry that can't be renamed aside stays whole, with a warning, and the install goes on. It used to be deleted, or the install stopped. Reversible: `move_or_delete` in `isolated_install.rs`. - Hoisted layout: a package's location depends on `bun.lock` only, not on `--production` / `--omit` / `--filter` / `--os` / `--cpu`. Older versions worked the places out anew for such a run, so an install that lays something over an existing tree also removes the folders at places only the older layout fills. With those flags that costs one more hoist (about 5 % of such a run); fresh, warm and nearly all plain installs pay nothing. Not a small revert. - `bun.lock` gets three optional additions with no `lockfileVersion` bump: `"unresolved"`, `"hasInstallScript"` and `"./<name>/<dep>"` entries. Reversible: one writer and one parser block each in `bun.lock.rs`. - Every install checks `bun.lock` against the `overrides` that name a version or a range: a locked version that a rule forbids is resolved again, and `--frozen-lockfile` fails. 29 lockfile shapes written fresh by 1.4 pass unchanged; a 1.4 lockfile that disobeys its rules (half a merge, a migration, a rule added later) is rewritten. Reversible: `rows_outside_overrides`. - `overrides` for the name of a workspace apply to dependencies of the root and of other workspaces again, as in 1.4, whose `bun.lock` failed `--frozen-lockfile` and was rewritten. This undoes #37271, which came with #44968. - Security scanner: `bun add <pkg>`, `bun install <pkg>` and `bun update <pkg>` send every package when there is no lockfile, only another package manager's, or no `node_modules` folder, otherwise what the command adds or moves. Reversible: `|| !loaded_lockfile` and `|| !has_node_modules` in `security_scanner.rs`, and the `narrow` line in the runner. - An unreadable `bunfig.toml` or `.npmrc` fails closed: commands that install or publish stop (a directory there only warns). npm ignores an unreadable `.npmrc`, so a root-owned `~/.npmrc` from `sudo npm login` now blocks them. Reversible: the two calls of `refuse_unreadable_config`. - An `XDG_CONFIG_HOME` that is empty or not absolute is ignored, and `$HOME` is used. Before, `~/.bunfig.toml` wasn't read at all then. - Auto-install reads `.npmrc` and bunfig's `[install]` in every folder from the project root down to the importing file (the nearer wins). It turns itself off when one of them, or a `package.json` on the way up to the workspace root, can't be read or parsed; a broken `~/package.json` does that for everything below it. In a monorepo without a lockfile it reads every workspace's `package.json` once. - A workspace outside the project (`../x`, an absolute path, a link that leaves the root's folder) is linked to and stays in `bun.lock`, but its dependencies aren't installed and its scripts not run, as in npm; a `note:` says so. Its own `bin` is still made executable. Commands that install, typed in one that is a link, end with an error and change nothing; on `main` they installed there. This affects layouts like #25801 under the isolated linker, and #29598 asks for the opposite; both stay open. Reversible: `Lockfile::is_member_outside` returns false. - An `npm:` alias in a dependency's manifest applies to that dependency only; the project's own aliases are still followed everywhere. - Git dependencies are checked out without symbolic links: a link to a file in the repository becomes a copy, other links aren't installed. On Linux, files appear where nothing was installed before; on macOS, links used to be installed; on Windows, they were small text files. - Git dependencies need git 2.24 or newer (`git restore` is from 2.23; `--end-of-options`, which `main` already passes for a `#ref`, from 2.24). - Every git command that Bun runs for a git dependency has hooks, `submodule.recurse` and replace objects off, and a cached clone's `config` is rewritten with Bun's own text before each use. - `gitlab:`, `bitbucket:`, `gist:` and `sourcehut:` dependencies are written to `bun.lock` and printed as `git+ssh://git@gitlab.com/user/repo#<sha>` (npm's form); they failed to install from `bun.lock` on a clean machine. Old entries still install and aren't rewritten by a plain install. - Isolated: a dependency on the project itself (`file:.`, `workspace:*` on the root) is a live link to the project folder, not a hard-linked copy of the repository; the old copy is removed on the next install. - Vendored libarchive gets one more patch, `patches/libarchive/tar-npm-compat.patch`: a reader option `npm` that only package tarballs set. Every other tar reader, `Bun.Archive` included, reads as before. Reversible: remove it from `scripts/build/deps/libarchive.ts` and drop the option. - The global store rebuilds its entries once after this upgrade; `bun pm cache rm` frees the old ones. Reversible: drop `GLOBAL_STORE_LAYOUT` from the entry hash. - Lifecycle scripts see `PATH` entries of the form `X/node_modules/.bin` by their real paths, so a dependency's script no longer finds the project's shared `.bin` ahead of the system `PATH` through a link. Reversible: `resolve_bin_dirs_in_path`. - `--exact` / `exact = true` also pins a typed range, also with `--catalog` and for an alias: `bun add -E react@18` writes `18.3.1`, not `18`, as npm does. Reversible: one condition in `PackageJSONEditor::edit`. - `bun add pkg@2.0.0` without a group flag rewrites every entry of a name that is listed in several dependency groups, unless the install leaves one of those groups out (`--omit`, `--production`). It wrote an entry that no install reads, so nothing changed. - A `package-lock.json` entry from another host without an `integrity` keeps its version and takes its tarball and integrity from the configured registry, with a warning; the whole lockfile used to be ignored. One older test pinned that. - `bun pm cache rm` empties the cache directory and keeps it, so one that is a symlink, a mount point or in a read-only parent goes on working (5 expectations changed). `bun pm cache rm <word>`, which cleared everything, and unknown `bun pm cache` sub-commands are errors. - `bun info`, `bun pm view`, `whoami`, `diff` and `bun audit` retry a failed request like `bun install`; one 503 or dropped connection used to end them. With the registry unreachable they fail after about 8 s, not at once; a timeout or a 501 isn't retried. `bun publish` still uploads once. - `bunx`'s daily refresh costs one manifest request per package. `bunx` installs into the cache directory that `~/.bunfig.toml` or `~/.npmrc` names (macOS, Linux); it ignored both. - `bun publish` refuses a prerelease version without `--tag` or `publishConfig.tag`, like npm 11 (reversible: the second `if` in `refuse_tag`), and a tag that is a version or a range. - `bun publish` reads `publishConfig` again after `prepublishOnly`, `prepack` and `prepare`, like the name and version (flags still win). Of two equal keys in `package.json` the last one counts, as in `JSON.parse`. A symlink at the name Bun gives the tarball is replaced, not written through. - `bun audit` fails on an empty answer (empty body, 204, 304). One test from #44968 pinned the old result. - pnpm migration: the `patchedDependencies` key of a git-hosted package is rewritten in `package.json` to Bun's own key (`name@github:owner/repo#<sha>`); with the old key the next install dropped the patch silently. One expectation of two older tests changed. - `bun pm version` follows node-semver's `inc`. Four existing tests pinned other values. - Piped `bun pm diff` prints `diff --git` for files whose contents are in the patch (reversible: `patch_name` in `pm_diff_command.rs`). `--json` has two new keys for binary files, `sha256Before` and `sha256After`. - `warn: incorrect peer dependency ...` is printed only with `--verbose`. - Isolated: a link that the user put in `node_modules/.bun` in place of a package is replaced by every install; `main` kept it. - With a `package-lock.json` that can't be used, `bun ci`, `--frozen-lockfile` and `bun pm migrate` warn and try a `yarn.lock`; they failed. - A dangling link in `.bin` stays while the package it leads into is there: its build makes the file. - Beyond the package manager: the arguments of `Output::err`, the path in a system error, the `$ script` line of `bun run` and the log printer's source excerpt (bundler, `bun lint`, `bun fmt`, `bun check`) spell out control and bidirectional characters, and a path or line that isn't UTF-8 prints U+FFFD. In an excerpt a NUL stays. Reversible: one place each in `fmt.rs`, `output.rs`, `sys/lib.rs`, `ast/lib.rs`. - HTTP client: a request can carry a flag that lets other requests start first while every connection is taken. Only package tarball downloads set it. Reversible: `yields_to_other_requests`. - Runtime: on Alpine (musl), `Bun.which` no longer takes a file without the execute bit for a command. Reversible: one `#if` in `c-bindings.cpp`. #### Regressions from #44968 Resolution and lockfile: - A `package.json` with both `overrides` and `resolutions`: #44968 applied both, so the first `bun install` after the upgrade rewrote `bun.lock` and could move versions. As in every earlier release, `resolutions` is read only when there is no `overrides`, and a `bun.lock` written by 1.4 installs unchanged, frozen or not. - `bun install` rejected its own fresh `bun.lock` with `dependency cycle cannot be laid out in node_modules` when an optional dependency has no published version that fits, or when a cycle is closed only through an optional peer. - A second `bun install` added a nested copy, and a `bun.lock` entry for it: of a package whose name an unresolvable optional dependency has, and of an unmet peer whose dependent is installed in two places. - A fresh `bun.lock` could fail `--frozen-lockfile` right away, or change on the next install with nothing edited: with optional peers, and with a workspace and the published package of its name both installed and a peer dependency on that name. - `bun.lockb`: after a peer moved to another version, the old one stayed in the file with nothing depending on it. `--frozen-lockfile` failed, and `bun audit`, `bun why` and `bun pm hash` saw it. (A peer that ends at a copy outside its range still leaves one.) - `--frozen-lockfile` and `bun ci` reported an out-of-date lockfile when a workspace has a peer dependency on another workspace's name but gets it from its own dependency on the published version. - `bun update` was held back by the peer range of a package that loads another copy. - `packageExtensions`: a fresh `bun.lock` failed its own `--frozen-lockfile` when a rule matched a version the lockfile doesn't list. Removing a rule that had added two or more dependencies could crash this and every later install, and what it had added stayed loadable. - `bun add <name>` at the root of a monorepo that has a workspace of that name wrote the registry's latest version but installed the workspace. It writes `workspace:*` again. A root dependency on a dist-tag of such a name links the workspace from the first install. - A tarball, folder or git dependency whose own `package.json` has a free-text `name` (`"Project: X"`, `"foo@1"`) failed to install. It installs under the fallback name a nameless package gets, and `bun.lock` loads afterwards. - A git, GitHub or tarball dependency that depends on a `file:` folder inside itself was installed without that folder's dependencies. A folder that isn't in the package is now an error. `workspace:`, `link:`, `catalog:` and tarball-path entries of such a folder are left out: they mean something in the author's project only. - `touch package.json && bun install` exits 0 again. An empty `package.json` next to a lockfile is still an error. - Ranges such as `>=0.0.0-a *` stopped matching prereleases of `0.0.0`. A comparator that means `>=0.0.0` admits them, as in npm. Git dependencies: - `bun install` could run `git fetch --prune` on your project's own repository, and delete local branches and tags, when the cached copy of a git dependency was no longer a repository and the cache sits inside the project's work tree. Every git call now names its repository with `--git-dir`. - Several installs that share a cache failed with `cannot lock ref` when a git dependency had new commits (9 of 20 runs). Such a fetch runs once more. - After upstream renamed its default branch, a git dependency without a `#ref` failed on every warm cache. A cached repository follows the remote's current default branch, also for `#HEAD`. Linkers: - With `node_modules` a symlink, an isolated install deleted everything in the folder behind the link. Hoisted did the same when that folder holds a `.bun` folder, deleted a folder there named like a dependency that had left `package.json`, and replaced a workspace's linked `node_modules` (on a volume, a cache disk) with a real directory. - Hoisted: a `bun remove` or `bun install` that failed to save `bun.lock` or `package.json` still deleted packages. - Hoisted: `bun add`, `bun install` and `bun remove` deleted the copy of a dependency that a package's new version bundles, or ships undeclared in its `node_modules`. - Replacing a tree (switching linkers, or an isolated install after `rm -rf node_modules` at the root) deleted `.cache`, `.vite` and other tools' dot entries in every `node_modules`. They stay. What belongs to a package manager (`.pnpm*`, `.modules.yaml`, `.package-lock.json`, `.yarn-integrity`, `.yarn-state.yml`) goes with the old tree. - Isolated: a project path close to `PATH_MAX` crashed on the new staging directory, and `bun pm untrusted` / `bun pm trust` didn't see `link:` dependencies. - "Blocked N postinstalls" was no longer printed for Linux platform packages, because the registry's full metadata has `scripts` but no `hasInstallScript`. Bun derives it from `scripts`, as npm does. That also fixes the older gap under `minimumReleaseAge`. Speed: - Resolving got slower with many versions of one name (11x more instructions with 320 versions each of 5 names) and many auto-installed peers (4.8x with 2,400 plugins). Both loops no longer rescan everything per step. - Isolated installs and `bun why` no longer slow down with the square of the number of direct dependencies (an old cost). - `bun file.js` parsed every `dependencies` entry of each package it loaded from `node_modules` (up to 2x the instructions and +50 MB with 2,000 packages, about +7 % on a realistic tree). A range is parsed only when auto-install asks for that name. Commands: - `bun install`, `ci`, `add`, `remove` and `update` accept the common npm, Yarn and pnpm flags again. `--save-dev`, `--save-exact`, `--no-optional`, `--only=production`, `--immutable`, `--loglevel=...` and others map to Bun's flags (`bun remove --save-dev` drops the flag). `--no-audit`, `--no-fund`, `--legacy-peer-deps` and a few more are ignored. Other unknown flags stay errors. - `bun pm pack` held the whole tarball in memory (734 MB and 52 s for a 400 MB tarball). It streams to a temporary file, and the destination changes only once the tarball is whole: a failed pack keeps an earlier tarball. - `bun publish` refused a publish that turns provenance off (`--no-provenance`, `NPM_CONFIG_PROVENANCE=0`, `publishConfig.provenance: null`). Like npm, the command line beats `publishConfig`, then the environment, then `.npmrc`. - `bun publish -r` crashed or published the wrong `workspace:` version for a package that follows one with a `prepack`, `prepare` or `prepublishOnly` script. With npm's web login, Enter could open an earlier package's login page. - `bun pm pkg set --json x=010` stored 8. Values that aren't JSON (`010`, `0x10`, `1_000`, `.5`, `'text'`) are rejected, also inside arrays and objects. - `bun update --latest` with a name in two dependency lists rewrote one entry while `bun install` used the other. `bun update --latest <name>` rewrites the entry that `bun install` uses; the other stays as written. - `bun update --interactive` crashed in monorepos whose packages write `catalog:default` (or `catalog:` over `catalogs.default`). It updates the catalog entry where it's defined. - `bun create` into an existing folder replaced your `.gitignore` with the template's `gitignore` file. Yours stays unless you pass `--force`; a template's own `.gitignore` is a listed conflict. - After a request had failed for good, `bun install` still asked the registry for the peer dependencies, with a second round of retries. It stops there. A 404 or "no matching version" below the first level still asks for the peers. - A failed `bunx tool@latest` broke the copy that `bunx tool` runs, and `bunx tool` waited for a running `bunx tool@latest`. A failure before linking leaves the tree as it was. - `bun info pkg` failed on a registry document without a `latest` tag. `bun info pkg@*` names the version `bun add` installs. - Migrating a `yarn.lock` or `pnpm-lock.yaml` waited about 8 s when the registry couldn't be reached. It asks once, and uses cached manifests of any age. - Yarn Berry projects whose `package.json` writes ranges as `npm:^1.0.0` lost the versions `yarn.lock` pins. `bun install` reads `npm:<range>` as a range of the dependency's own name when it has a range character or is a complete version; `npm:bar`, `npm:latest`, `npm:1` and `npm:x` stay aliases. - `bun install` on a `yarn.lock` with a git dependency outside GitHub failed with `invalid git commit ""`, and `--frozen-lockfile` straight on a `yarn.lock` failed when the root has a required peer that it doesn't list. - The warning about an unreadable `bunfig.toml` printed up to three times. It prints once, and `--silent` hides it. `bun a.js` without a bunfig does one failed `open` instead of three. - Auto-install: a version written in an import (`import "foo@^2"`) was overridden by the range in `package.json`. - `bun patch --commit` and `bun install` rejected patches that delete the last lines of a file with no trailing newline. - `bun patch` and `--commit` failed when the project's path contains `"`, a tab or a backslash. A failed `--commit` with the global store deleted the edited folder. - Undoing every edit and running `bun patch --commit` no longer removed the patch. It removes the `patchedDependencies` entry, reinstalls the unpatched package and deletes the patch file if it's a regular file inside the project and no other entry names it. #### Resolution and lockfile - New `bun.lock` keys: - `"unresolved": [...]`, next to `optionalPeers`: optional dependencies that couldn't be resolved, listed only when another package of that name is in the tree. - `"hasInstallScript": true` in a package's info object (npm's key). Both are in `bun.d.ts`. - `"./<name>/<dep>"` entries: the dependencies of a workspace whose name the root gives to another package. - Older versions load such lockfiles, pass `--frozen-lockfile` (2 of 900 dense random projects didn't) and drop the additions on their next save, so a team on mixed versions sees them come and go. A plain install doesn't rewrite a lockfile just to add the first two. - A lockfile that lists a workspace next to a registry package of its name is rejected by Bun <= 1.4 ("Duplicate package path": the lockfile is ignored, frozen fails), as it already was when another workspace depended on that workspace. - An optional dependency that can't be resolved (no matching version, a missing tarball, folder, git repository, alias or catalog entry) was bound on the second or a clean frozen install to another package that happens to have its name. It stays unbound, with both linkers. This holds for lockfiles this version saves; an older one binds by name as before until the entry is resolved again. - A package installed in several places gets one stable binding for each optional peer. Upgrades no longer leave a copy that `bun.lock` doesn't list, and a committed lockfile that frozen kept rejecting passes. - A required peer that the root's own dependency, or the package's own dependency of that name, provides within its range counts as that package for `bun update`, `outdated`, `prune`, `pm licenses`, `audit`, `add --trust`, the security scanner, the install summary and `--production` / `--omit` / `--filter`. They went by the highest version that fits. Other providers are as before. - A workspace or package that lists a name both as its own dependency and as a peer always loads the version its own entry asks for. With the hoisted linker, `bun add` or `bun update` in another workspace could put a newer copy in front of it, outside that entry's range. - `bun install` could write a `bun.lock` it couldn't read back (`Failed to resolve prod dependency`, `InvalidLockfile`) when a package bundles a package that another one names as an optional peer (in 1.4.3 too). Such a peer, if nothing provides it, is no longer installed at the top of `node_modules`. - **A fresh `bun install` writes the same `bun.lock` whichever response arrives first, with a cold or a warm cache.** Each dependency used to be resolved when its manifest arrived, so network timing decided whether a range shared a version. - Dependencies are resolved in one fixed order, and only the first unresolved one waits for its manifest. - The ones behind it start their requests and downloads at once, and Bun requests ahead the manifests that their most likely versions depend on. That never touches the lockfile, and a request made on a wrong guess can't fail the install. - What a git, GitHub or tarball package depends on is only known after its download, so these are added one at a time after the registry packages, in the order of the dependencies that ask for them. - While every connection is taken, manifests go before tarballs, and peers don't wait for tarballs in flight. - No measurable cost: see the numbers at the end. - A range that allows a version listed in `patchedDependencies` resolves to that version. - A 404 or "no matching version" about a version that deduplication later drops no longer fails the install: it is a warning. - The `overrides` check under frozen prints `note: bun.lock has b@1.2.9, and overrides in package.json asks for 1.2.0`. It leaves alone bundled dependencies, optional peers, and a peer or optional dependency that shares its version with a dependency the rule doesn't apply to. - An override added to an installed project reaches a package that is only there as one package's peer dependency. If no version fits, that is a warning, as on a fresh install. - `bun add foo` and `bun update --latest` installed a version that a version-scoped override (`"foo@>=2.0.0": "1.2.0"`) forbids. They write `^2.0.0` and install `1.2.0`, like a clean install. - An override set to `catalog:` whose package isn't in the catalog gets a warning that points at the rule. - `bun install` and `Bun.semver` read build metadata after a bare major (`2+1`, `^2+x`) as part of that version, not as a second range that let another major through. `Bun.semver.order("2+1", ...)` no longer throws. #### Hoisted linker - What a self-contained workspace drops from its `package.json` leaves its `node_modules`. - The command of a workspace that isn't built yet stays in `.bin` when another dependency leaves `package.json` (#44968). - `--production`, `--omit=<type>`, `--filter`, `--os` and `--cpu` used to re-hoist, so a package could move between runs and an old nested copy could shadow the right version. They now skip packages but never move the rest. (#45070, #29944) Such a tree nests the way the `bun.lock` keys do: +0 % to +1.8 % folders on real lockfiles. - An install removes a folder from `node_modules` only if the `bun.lock` it started from placed a package there, or Bun up to 1.4 would have for the same lockfile and flags. A nested copy from before the upgrade used to stay and go on being loaded after the next version bump. A folder that a package ships in its tarball (the cache says; without the cache entry the folder stays), that a script can have laid or that somebody put there stays. What a flag merely leaves out stays too; `bun prune` removes that. - A warm install reads each installed `package.json` with 3 system calls instead of 4. - After only the project's `node_modules` was deleted, a package that changed version was linked over the old folder in a workspace's own `node_modules`, so files of the old version stayed (in 1.4.3 too). Such a folder is treated as in a warm install. - `bun prune --production` deleted nested packages below a package that a full install had hoisted to a different folder than a production install would. - Removing the last dependency from `package.json` removed nothing from `node_modules`. - A workspace that switches a dependency from a registry package to the workspace of that name kept loading the old nested copy. The install removes it, and `bun prune` sees it. - A command whose file is written by the package's own install script (a workspace CLI built in `prepare`) is linked by the same `bun install`, before the root's scripts run. - Both linkers: symlinks inside a `file:` folder dependency were dropped silently. Relative links that stay inside the folder are kept, links to files elsewhere in the project (by real paths) are installed as files, and anything else is skipped with a warning. - A package whose `package.json` or `node_modules/.bin` path doesn't fit in `PATH_MAX` fails its lifecycle scripts with `ENAMETOOLONG` instead of crashing. #### Isolated linker and global store - A dependency removed from a `package.json` stayed linked in `node_modules`, with its `.bin` entries. It's unlinked, also when another package still needs it, inside a workspace (#26305), and in the hidden `node_modules/.bun/node_modules`. Only that package's own `node_modules` is touched, and only a link that leads where Bun had linked it: your own links, `bun link` and other workspaces' links stay. - A workspace whose folder, or a parent of it, is a symlink got dangling dependency links. They're computed from the real folder (not on Windows). Workspace links behind a linked root `node_modules` no longer dangle either. - A registry package that depends on, or peers on, a workspace that has peer dependencies was stored under one name on the first install and another on the second. - A `hoistPattern` added to an installed project takes effect on the next `bun install`: fallback links it doesn't match are removed. (`publicHoistPattern` still only grows.) - After `bun prune --production`, a command or a fallback link that a removed package held goes to the package that stays, as in a fresh `bun install --production`. - `globalStore = true`: the first install against a warm store printed nothing useful. It lists what it linked and which postinstall scripts were blocked. - `globalStore = true`: a no-op install deleted and re-created every `node_modules/.bun/<pkg>` link (4 system calls per package, now 1). Two installs at once no longer fail with `EEXIST`, and a running program no longer loses a package for a moment. - `bun install --force` replaces a global store entry in one step on macOS and Linux, so other projects never see it missing. - `BUN_INSTALL_GLOBAL_STORE=false` (also `no`, `off` and an empty value) turned the store on. #### Workspaces - A workspace vanished from the install when the root depends on a registry version (or any other package) of the same name. It keeps its `bun.lock` entry and gets its own dependencies installed, while the root's `node_modules/<name>` holds what the root asked for. - A `bun.lock` or `bun.lockb` from Bun 1.4 or older that leaves such a workspace out still passes `--frozen-lockfile` and isn't rewritten by `bun install`. The workspace gets its entry the next time the dependencies change, or with a fresh lockfile. - `bun link <package>` inside a workspace treated the workspace as its own project. It installs from the workspace root like `bun add`: sibling `workspace:` and `catalog:` dependencies resolve, and `--save` updates the workspace's `package.json` and the root's `bun.lock` (#24190, #20015). - `bun add <name>` inside a workspace, where `<name>` is a workspace that is also published, links the workspace and writes `workspace:*`. - `bun add <name>@<tag>` and `bun update --latest` for such a name left a workspace entry bound to the registry package. They install and lock what a clean install of the written range gives. - `bun add alias@npm:<workspace>` wrote `"alias": "workspace:*"`, which every later install rejected. It keeps the `npm:` form. - `bun add lib@npm:other` at the root, with a workspace named `lib`, installs `other` rather than linking the workspace. - `workspace:../lib`, `workspace:lib@^` and `workspace:lib@~` rejected a workspace with a prerelease version or no version. - A `workspace:` range on the root package that the root's version doesn't satisfy is an error, as for any other workspace. #### Lifecycle scripts and trust - Isolated: a trusted `link:` dependency runs its lifecycle scripts when it's linked, as with the hoisted linker. An untrusted one is reported as blocked. - A trusted `link:` folder without a `package.json` no longer fails an install from `bun.lock`. - An install from a checked-in `bun.lock` reports blocked scripts, because the lockfile records which packages have install scripts. - `bun pm trust` refuses to run with a frozen lockfile. - A fork, folder or workspace named `sharp` runs its install scripts like any other package. `"ignoreScripts": []` in `package.json` turns the built-in skip list off. - `bun pm pack`, `bun publish` and `bun pm version` run their lifecycle scripts with the package's and its parents' `node_modules/.bin` in `PATH`, so `"prepack": "tsc"` works. With `bun publish -r` each package gets its own. - The `preversion`, `version` and `postversion` scripts get `$npm_old_version`, `$npm_new_version`, `$npm_command`, `$npm_package_json`, `$npm_package_name` and `$npm_package_version`. The new version is computed before `preversion` runs, as in npm, so an invalid or unchanged version fails first. - `bun create <template>` has `--ignore-scripts`, and skips the template's preinstall and postinstall tasks with `ignoreScripts = true`, `ignore-scripts=true` in `.npmrc` (both read in the folder the command is typed in and in the home folder) or `NPM_CONFIG_IGNORE_SCRIPTS`. #### `bun patch` - `bun patch` and `--commit` fetch the package again when it's installed but no longer in the cache (after `bun pm cache rm`) instead of failing with `ENOENT`, also for patched packages and git, GitHub and tarball dependencies. - `bun patch --commit` changes nothing until every check has passed. If the new patch doesn't apply to a scratch copy, or the install it runs is refused (the security scanner, `--frozen-lockfile`, a dependency that doesn't resolve), the folder, `package.json`, `bun.lock` and an older patch file stay as they were. With the global store, `--commit --lockfile-only` deleted the edited folder. - Isolated: an install with a patch that doesn't apply stops, as with the hoisted linker, and `bun add` / `bun update` write nothing. - The project's own `.gitattributes`, `.git/info/attributes` or `.git/config` could change or empty the patch. git runs without a repository. - `--commit` failed when a patched file, the project folder or the cache folder has a non-ASCII name, when git only printed a warning (more than 1,000 files deleted and 1,000 added), and for a git dependency whose repository contains symlinks. It reported "No changes detected" when git was killed. - When a file in `patchedDependencies` is missing, `bun install` and `bun patch` fail and leave `node_modules` alone. The error names `bun patch --commit <pkg>`, which writes the file again from the patched package in `node_modules`, and says how to drop the entry. - `--commit` on a package that wasn't edited and has trusted install scripts puts it back as installed and runs those scripts again. - Where the package folder can't be renamed (a lower overlayfs layer in Docker), `bun patch` lost nested packages and could leave a half-deleted folder. Nothing is deleted before the copy is in place, and an error puts everything back. - Absolute patch paths work. `--patches-dir=/abs` inside the project is written relative to it. - When several versions match, `bun patch` accepts an entry of the list exactly as it prints it. - Appending lines to a file with no trailing newline lost the last line. - A patch section that adds a binary file made an empty file. It makes none, as its warning says. - Files under a folder whose name ends in `" b"` were deleted, created or chmod-ed at the wrong path. - Applying a patch was quadratic: 40,000 changed lines in a 100,000-line file took about 7.5 s, now about 10 ms. - Swapped hunks without context that change the line count applied at the wrong place. Hunks are applied in file order. - `bun patch` dropped folders named `CMakeFiles` or `.git` from the copy. - A package's commands lost their executable bit between `bun patch` and `--commit` when files are copied rather than hard-linked. - Global store: a package being patched keeps finding its own dependencies, and `bun prune` no longer deletes the folder that `bun patch` prepared for a transitive dependency. #### Tarballs, `bun pm pack` and `bun publish` - Uncompressed `.tar` packages, and `.tgz` files served with `Content-Encoding: gzip`, failed in the streaming extractor. - A `github:` tarball whose first entry isn't its root folder was accepted or rejected depending on whether the response had a `Content-Length`. - `Bun.Archive`: `extract(dir, { glob })` and `files()` resolved with the entries read so far on a truncated or corrupted archive. They reject, like `extract(dir)`. - `bun pm pack` and `bun publish` left `bundledDependencies` out with the isolated linker, in workspaces and for linked folders. They bundle them and everything they depend on from any layout, looking no further up than the project. Sibling workspaces are packed by their own `files` and `.npmignore`, and a broken sibling that nothing needs doesn't fail the pack. - A bundled name in `dependencies` that isn't installed is an error, no longer left out. So is a ring of bundled versions that no `node_modules` can hold. - A `catalog:` dependency whose catalog entry is `workspace:^` (or another `workspace:` form) was published as written. It's replaced with the workspace's version. An entry that is itself a `catalog:` reference fails the pack. - `bun pm pack` left lockfile-named files out of a dependency of a bundled package whose name contains `node_modules`. - A file that gets shorter while it's packed was padded with NUL bytes. The pack fails. - `bun publish x.tgz` looks `bin` and `directories.bin` up in the tarball, not in the current folder. - The registry's `npm-notice` header is printed for every request of `bun publish`. - `bun publish` of `2.0.0-beta.1` without a tag made it `latest`, the version `bun add` installs. The refusal names `--tag next` and `--tag latest`, and `--dry-run` refuses too. - `bun add ./<path>.tgz` with a path longer than `PATH_MAX` fails with `ENAMETOOLONG` instead of crashing. #### Git dependencies - Checkout: files are written straight from the cached bare repository (`git ls-tree`, then `git restore --source=<sha> --worktree` with `core.symlinks=false`). The second clone and its `.git` folder are gone: about 20 % to 35 % less CPU per checkout on two repositories. An older git gets a note that 2.24 is needed. - Symbolic links: a link to a file inside the repository, also through other links and linked folders, is installed as a copy with its executable bit. Links to folders, links that leave the repository, absolute, dangling and looping links are skipped, and no longer fail the install. The rule is the same on every platform. - An ssh git URL never fell back to ssh when the https attempt was answered "not found". Bun tries every form of the URL whatever git said. - A successful install no longer prints git's complaint about the failed https attempt. A failed install shows every attempt. - Bun overrode your own ssh command in cases its hand-written git config parser missed. It asks git itself, once, before the first clone: `core.sshCommand` from any config file, plain includes, `GIT_CONFIG_*`, `GIT_SSH`, `GIT_SSH_COMMAND` and `core.askPass` all count. A value behind a conditional `includeIf` is still overruled. - `git@host:user/repo.git` installed from `bun.lock` tried a malformed https URL. It tries the same URLs as a first install. - A cached clone that git can't use (a stale lock file, a damaged ref store) failed every install. It's cloned again. Clones made by older versions keep working, also with `--offline`. - Your own `submodule.recurse = true` broke a git dependency that has a submodule. - Isolated: a second install of a shorthand or scp-like git dependency renamed its store entry. - When the `prepare` script of a git dependency fails, a note says that Bun doesn't install its `devDependencies` first. #### Migrations (npm, yarn, pnpm) - A `yarn.lock` or `pnpm-lock.yaml` that pins a GitHub dependency failed on the first install and lost the pin on the second. It installs the pinned commit and writes a `bun.lock` that loads. Migrated pnpm rows keep the full 40-character commit. - A `pnpm-lock.yaml` with a `type: git` resolution installs the pinned commit, whichever way `repo:` is written. - pnpm: a patch keyed by a version range or a bare name was lost. It becomes one `patchedDependencies` entry for each version it patches. - yarn v1: `link:` and bare path (`./dir`) entries failed with "failed to resolve". They're migrated, with their `bin`, also for workspaces and with `--frozen-lockfile`. - yarn v1: a tarball or hosted-git entry was merged with the registry package of the same name and version, so a fork was replaced by the published package. They stay apart, and an `owner/repo` entry is the archive that `yarn.lock` names. - yarn v1: a monorepo where a registry package depends on a workspace failed with "failed to resolve". - A root or workspace dependency whose `yarn.lock` entry can't be used is resolved instead of failing the install. A dependency of a dependency without a usable entry still fails. - A migrated entry takes `bin`, `os` and `cpu` from the registry's metadata only when it's that registry's tarball: the same URL, the same hash, or, when the hashes can't be compared, a URL with the registry's layout for that name and version (a mirror). - A lockfile that Bun can't migrate hid a `yarn.lock` or `pnpm-lock.yaml` next to it. Bun warns (`...; trying yarn.lock`) and tries the next one. - pnpm and npm: the project's own required peers that the old lockfile doesn't record are resolved. No migration (`bun.lockb` included) writes them as `optionalPeers` any more, so `--frozen-lockfile` accepts the `bun.lock` that `bun install` writes. `bun pm migrate` leaves such a peer out; the next `bun install` adds it. - `bun install --yarn` wrote an npm alias under the package's own name. It uses the name `package.json` gives it, as yarn v1 does, so an alias migrates back to the same package. #### Auto-install - `bun.lock` was never read without `node_modules` (only `bun.lockb` was). Bun uses the versions pinned in either, and downloads them on first import. - `import "foo"` also honors `devDependencies` and `peerDependencies`, and walks up to the first `package.json` that lists the name. - `overrides`, `resolutions`, `catalogs` and `packageExtensions` of the project's `package.json` apply. An override counts when it names a version, range, tag, `npm:` alias or `catalog:`; one that names a folder, tarball or git repository is passed over, as before. - `workspace:`, `file:` and `link:` dependencies load from their folder on disk, and tarball dependencies load through the cache. - Peer dependencies of auto-installed packages failed with "Cannot find module". They load, at the version the project's `package.json` lists if nothing has loaded that name before. Optional peers are installed when imported. - A failed tarball download (5xx, 404, truncated file) was remembered until restart. A later import tries again. - `bun --install=force` rejected an import that names a version in a project with a `package.json`. - An import waiting on a registry that accepts the connection and never answers failed after about half an hour. It fails after 30 s idle, without retries. - A cache or temp directory that can't be written (a read-only container, no `HOME`) ended the process inside `require()` / `import()`. The import throws "Cannot find package" with the reason as a note, and packages already in the cache still load. - A package that ships `bundledDependencies` can load its other dependencies, and Bun no longer downloads the bundled ones. #### `bunx` and global installs - `bunx <tool>` with a day-old copy failed when the registry answered 5xx, 429 or 408, or when the tarball of a newer version couldn't be fetched. It runs the copy it has, and asks a failing manifest once instead of retrying for 8 s. - The daily refresh never moved `pkg@<range>` or any dependency, because the tree's own lockfile pinned them. It takes what was published since, and a `tool@latest` run no longer puts it off. - `bunx pkg@3` failed in a project that has another major of `pkg` installed. With a version, tag, URL or alias, nothing from the project is used. - `bunx @scope/pkg` and `bunx -p pkg cmd` find the bin through that package's own `package.json`, in the project (no further up than its root) or in `bunx`'s tree, and run that package's file, not another package's `.bin` entry of that name (on Windows the shim is taken as is). - `bunx -p pkg cmd` installs `pkg` and runs any command, as npx does, with `pkg`'s commands first in `PATH`. - A project's own tool run through `bunx` no longer finds helper commands from an older copy that `bunx` had installed. - `bunx --no-install pkg@latest` runs the copy that is on disk. - `bunx pkg` without a version passes over a deprecated `latest`, as `bun add` does. - A `node_modules/.bin/<name>` that exists but can't be executed is `error: <path> is not executable`, exit 126, instead of a download. - A `bun install` that a package's script starts during `bunx` is an ordinary install of its own project. - `bun add -g` / `bun update -g`: when one bin can't be linked because another file has its name, the command still fails, but the package is saved, so `bun pm ls -g` lists it and `bun remove -g` removes it. - `bun remove -g second` took a bin name away from `first` for good if `second` had claimed it. A global package whose bin link is missing gets it back on the next `bun add -g`, `update -g` or `remove -g`. Nothing already in the global bin folder is replaced. #### `bun update`, `bun outdated`, `bun audit`, `bun pm *` - `bun update --interactive` rewrote every range as "first character + chosen version". Ranges such as `>=1.0.0 <2`, `1.x`, `*`, `^1 || ^2` or `latest` stay as written when they already allow the chosen version. - `bun update` with `--exact` or `install.exact` no longer pins a `peerDependencies` range, or a catalog entry that a `catalog:` peer uses. `bun update --latest` without names leaves a peer range in a `package.json` that already allows the latest version (one in a catalog is still rewritten). - `bun outdated` hid dependencies that are ahead of the `latest` tag (`next` / `canary` channels, `latest` kept on an LTS major). It lists them when `bun update` would move them. It also printed an optional peer twice. - `bun update --latest` and `bun update -i` downgraded a dependency that is ahead of `latest`, including one written as a dist-tag such as `"canary"`. No key or flag in `-i` can pick an older `latest`, and it no longer lists packages whose only other version is one. An entry in a catalog is still moved down. - `bun audit fix`: after it upgrades a package past a plugin's peer range, the "Fixed / remaining" counts and the exit code match the next `bun audit`, and the plugin's line is marked `(peer)`. `--dry-run` says the same as the real run, except when the narrow peer range is in one of the project's own `package.json` files. - `bun audit` reported "No vulnerabilities found" on an empty answer. It's `error: <registry> returned an empty audit response`, exit 1, also with `--json` and `fix` (for a scoped registry, the existing warning). `{}` still means none found. - `bun pm version <increment>` computes the same version as `npm version` (`1.0.0-rc.1` + `major` is `1.0.0`). It cleans `v2.0.0` to `2.0.0`, and rejects `2`, `x`, ranges, a bad `--preid` (also `--preid 01`, a number with a leading zero) and, with `from-git`, a tag that isn't a version. - `bun pm ls` / `bun list` support `--production`, `--omit`, `--depth=0` and package names (`bun pm ls lodash`). This covers part of #13148. - `bun pm licenses` skipped an installed package whose own version differs from `bun.lock` (a bundled copy, another spelling). It lists it under its own version. - `--dry-run` printed one line per dependency edge, skipped peers and ignored the platform. It lists what the install would put in `node_modules`, each package once (not yet narrowed by `--filter`). - `bun pm cache rm` prints "Cleared" only on success. - `bun pm cache pack` / `unpack` crashed while reporting a file name that isn't UTF-8. `bun info` and `bun create` crashed on a name longer than a path. - A flag that belongs to another `bun pm` sub-command (`bun pm ls --json`) was silently ignored. It's an error that names the sub-commands that take it. - `bun pm diff a b > changes.patch` applies with `git apply` and `patch -p1` when a version adds or removes an empty file, changes an executable bit, or has file names with spaces (they're C-quoted). Binary files and source maps keep `diff --bun`. - `bun pm diff <pkg>` compared the first lockfile entry of that name. It uses the version `<pkg>` means in the workspace you run it from, accepts an alias, and `bun pm diff <pkg> 2.0.0` starts from the installed version. - `bun pm diff` compares style sheets that contain `url()` by their canonical print too. - Commands that rewrite `package.json` turned control characters such as `\^[` into `\x1B` or `\v`, which npm and node can't parse. #### Registry requests and the manifest cache - The same commands and `bun publish` dropped the registry token on a redirect from http to https on the same host and port. They keep it, as `bun install` does. - A redirected request ignored `NO_PROXY` for the new URL. Each hop picks its proxy by its own URL, as in `fetch()`. - A registry that redirects in a circle was retried and reported as "No version matching". It's `TooManyRedirects`, without retries. - A damaged manifest cache file could crash every install. The offsets and lengths of what an install reads are checked, and a bad file is fetched again. - Bun deleted cached manifests that it couldn't read, also under `--offline`. It leaves them, and the next fetch overwrites them. - A manifest cached while the clock was ahead stayed "fresh" until that time came. It's revalidated. #### Configuration (`bunfig.toml`, `.npmrc`, flags) - An unreadable `bunfig.toml` or `.npmrc` used to be a warning. It's `error: This file cannot be read, and bun does not install packages without its settings`, before any request. Commands that neither install nor publish still warn and continue, and auto-install turns itself off. - A relative `cafile`, `cache`, `globalDir` or `globalBinDir` in `bunfig.toml` or `.npmrc` is relative to that file, wherever the command is typed. A relative `cafile` only worked from the project root. - A relative `--config`, `--cache-dir` or `--cafile` is relative to the directory the command is typed in, not the project root. - `BUN_INSTALL`, `XDG_CACHE_HOME`, `BUN_INSTALL_CACHE_DIR`, `BUN_INSTALL_GLOBAL_DIR`, `BUN_INSTALL_BIN` and `NPM_CONFIG_USERCONFIG` accept a leading `~/`. A relative value keeps its current base (the project root for the cache). - A scope whose registry URL is empty uses the default registry of the run, including one set by `BUN_CONFIG_REGISTRY`, `NPM_CONFIG_REGISTRY` or `--registry`. With credentials of its own it leaves the configured host for no other. - `--dry-run` (and `dryRun = true`) was accepted and ignored by `bun pm fetch`, `bun pm cache pack` / `unpack`, `bun patch` and `--commit`. `bun patch <pkg> --dry-run` even discarded edits. They print what they would do and leave the project alone. - `bun install --analyze` and `bun add --analyze` skipped packages that are imported only by a subpath (`lodash/debounce`). An unresolved `@/alias` import no longer aborts the run, and a file that imports the package it belongs to no longer gets that name looked up or added. #### Messages and diagnostics - Names, versions, paths, URLs and script text from packages, lockfiles, registries or the file system are printed with unprintable characters spelled out (`\x1b`, `\r`, `\^[`, bidirectional marks) in `bun pm pack`, `publish`, `audit fix`, `prune`, `pm whoami`, `run`, `create`, `pm migrate`, in error messages and in source excerpts (a NUL in an excerpt stays). `--json` is untouched. - `bun install` silently skipped parts of a patch it can't apply: a `Binary files ... differ` section, a file copy, colored output, stray text. It still installs, and warns about each one. It fails on a `GIT binary patch` section or a hunk without a file. - Patch warnings and parse errors go through the normal logger, so they show the patch file, line, column and source line. - A `patchedDependencies` key without a version was silently not applied. It gets a warning that points at the key. - The "patch no longer applies" warning also covers tarball and git dependencies, and a version that is a URL containing `@` no longer gets a false one on every install. - `bun patch` prints a note when the package has trusted install scripts: they haven't run on the fresh copy, and run again at `--commit`. - Global store: `bun patch <transitive dependency>` prints that dependents only see the changes after `--commit`. - `bun patch` and `--commit` fail with `ENAMETOOLONG` instead of crashing when a path argument or `--patches-dir` is longer than `PATH_MAX`. - Diagnostics about the project's `.npmrc` show its absolute path, as for `bunfig.toml` and `~/.npmrc`. - Git errors name the command that failed (`"git restore" for "x" failed`) instead of "git checkout". #### Windows Found by @dylan-conway in review, by hand, on CI's Windows builds, and tested again the same way on `ce845c1e2`: of 52 cases 35 go from fail to pass, 15 controls pass on both, none goes from pass to fail. - Regressions from #44968: a second isolated install crashed with a stack overflow once a link's path reached 260 characters; `bun patch` failed with `EPERM` while a program had a file of the package open (a file opened without sharing still fails, with `EBUSY` and the folder left whole); switching linkers with one file open stopped half way and left packages missing. - `XDG_CONFIG_HOME` or `USERPROFILE` without a drive letter, or on a share that doesn't exist, stopped every install with "cannot be read". An odd error counts only if the file is really there. - git is given the platform's own name for the null device (`NUL`). - Behind a `node_modules` that is a junction or a directory symlink, the isolated linker writes absolute link targets, so the links resolve. - A workspace that leaves the project through a junction counts as outside the project, as on macOS and Linux. - Links inside a `file:` folder were taken to "lead out of the project" when the command was typed in `c:\proj` for `C:\proj`, or through a junction. - `--patches-dir` takes `/`, `\` and `.\`, and always writes `/` to `package.json` and `bun.lock`. - The `preversion` / `version` / `postversion` scripts of `bun pm version` in a git repository ran with an empty environment. - A dependency's lifecycle script found the shared `node_modules\.bin` ahead of the system `PATH` when `PATH` spelled it in another letter case. - The hoisted linker installed a package through a junction at its destination. The junction is removed first. - `bun create <local template>` failed with "ENOENT: failed to copy file" when the working directory, the template or the destination has a non-ASCII character. #### Hardening - Auto-install started in a linked workspace takes the settings of the workspace root and its own, and of no folder above the link's target. - `bun install` extracts a few unusual tarball entries the way npm does, or skips them as npm does, and takes two zero blocks for the end of a package tarball (registry, URL and `file:` tarballs). It reads a tarball to its end on every extraction path. - Messages that show the URL of a git or tarball dependency, or of a registry, no longer show any of the login part of the URL, also in what git itself prints. `bun patch --commit` names new patch files without it. - A tarball URL with an invalid host is rejected with `InvalidURL` before any connection is opened. - A git dependency's `package.json` is always a regular file from its own repository. A cached checkout whose `package.json` is a link is checked out again when the dependency is resolved anew (no `bun.lock` entry, `bun update`, `bun add`). - A cached clone is used only if it's a plain bare repository without links. - The isolated linker reinstalls a package when its entry in `node_modules/.bun` doesn't really live in the project's store (macOS, Linux). - With `globalStore = true`, packages that depend on each other are shared only between projects that link them to the same versions. - Migrating a `yarn.lock` keeps the checksum of entries that have no `integrity` line. - The pnpm migration keeps an empty `onlyBuiltDependencies` as `"trustedDependencies": []` in `package.json`. - `bun pm trust` takes only its positional arguments as package names. It and `bun add --trust` leave blocked what a `trustedDependencies` entry cannot unblock for `bun install`: a tarball, git, `file:` or `link:` dependency that neither the project nor a workspace declares, and a package that another one ships inside itself. `bun pm untrusted` marks the packages that can't be trusted by name. - A folder that a package ships inside itself never runs install scripts of its own. - With a security scanner configured, auto-install is off as documented, whatever the cache state or `--install` flag. It counts when set in the global bunfig, or in a `bunfig.toml` between the working directory or the importing file and the project root. - `bun patch`, `bun patch --commit`, `bun link <pkg>` and `bun dedupe` run the configured security scanner before they install packages. - A relative `install.security.scanner` path in the global bunfig or in a `--config` file is relative to that file. File names with quotes work. - Commands typed inside a workspace that is a symlink to a folder elsewhere use the workspace root's settings and lockfile. - Auto-install takes its settings and `bun.lock` from the project of the importing file, wherever `bun` was started; a `bun.lock` in an unrelated parent folder is ignored. A `package.json` that only marks the module type (no name, workspaces or dependencies) is passed over when a real project is above it; otherwise the topmost one is the root. - `bun pm pack` never packs its own temporary files. - `bun pm diff` requests and checks downloads the way `bun install` does (the registry's integrity, and `bun.lock`'s when it's loaded). - `bun pm diff`'s terminal view checks what it folds away as "formatting only", and what it re-prints for minified files, against a full re-parse, and shows the plain diff when the check doesn't hold. The JavaScript parser gets one internal option for this (`keep_directives`); only `bun pm diff` sets it. - `bun pm diff` treats fewer files as binary and prints sizes and a sha256 for the ones that are. Its `package.json` notes compare whole values and say where the registry's metadata and the tarball disagree on a field that `bun install` uses. It escapes control characters on a terminal with `NO_COLOR` too. #### Not in this PR - `bun pm ls --json` and `--depth=N` above 0 are rejected rather than supported (#8283 stays open). `bun unlink <name>` is still not implemented. - Plain `bun update` still doesn't move a peer-only entry within its range (#13509). - A package that a tarball declares as bundled but doesn't ship is still not installed. A tarball that bundles an unpublished package still fails. An override doesn't reach a bundled copy. - An optional dependency that couldn't be resolved isn't retried by a plain `bun install`; `bun update` retries most of them. - Hoisted: the lifecycle scripts of a workspace whose name the root gives to another package don't run. - Auto-install in a project that has a `node_modules` folder still ignores `package.json` and `bun.lock` for a package that is missing on disk. #### Known to be still open Reopened because the fix was taken out of this PR again (as on `main`, unless noted): - A name in two dependency groups with different ranges: a fresh install and one from `bun.lock` can give different versions. - After `bun.lock` changes under an existing `node_modules` (a `git pull`, `rm bun.lock`), a nested copy that the old lockfile didn't place can stay and shadow the right version. Simulated pulls that ended wrong: 101 of 697 on `main`, 100 of 697 here. The `rm bun.lock` case is new with #44968. - An override named like a workspace displaces links to that workspace again (#37271). - Bin links that a package's new version no longer declares stay in `.bin` and in the global bin folder. - Ctrl-C with a `SIGINT` listener during auto-install's wait is handled only when the wait ends (30 s idle at most); before #44968, at once. - A package that is trusted already (through a `git pull`) but never ran its scripts here: `bun pm trust` doesn't run them. - A trusted `file:` folder that is edited in place isn't built again by the next install (new with #44968). - With a patch file missing, `bun patch` fails for every package. A second `bun patch <pkg>` discards uncommitted edits. - The `bin` of a tarball, GitHub or git dependency isn't linked after a `yarn.lock` or `pnpm-lock.yaml` migration: neither file records it. - A 404 below a version that couldn't be completed can make a fresh install settle on a lower version that is already in the tree; the warning names the request. - An optional dependency whose own dependency can't be resolved fails the whole install, as on `main` and in pnpm. npm leaves the optional package out and asks again on the next install. Regressions from #44968 that this PR doesn't fix: - `bun remove` empties `package.json` before it writes it, so a full disk at that moment leaves an empty file. - With the registry down, `bun install` waits 7 to 9 s through its retries without printing anything. - The install cache of 1.4 isn't reused for a registry whose URL has a path, a port or plain http, and every cached manifest is fetched again once. - A long package name from a registry other than npm's fails with `ENAMETOOLONG` from 185 characters on (204 before). - Two GitHub repositories can get the same cache folder name, and the second one is then installed from the first one's folder. - Three more are in the list above, marked "#44968". Gaps in what this PR adds: - Behind a linked `node_modules`, the hoisted linker replaces a folder or file that has the name of a dependency, as 1.4.3 and pnpm do; behind a workspace's link `main` doesn't, because it replaces the link. - `bun add p@2 --filter a` moves a shared `node_modules/p` without laying the nested copy for a workspace outside the filter, until the next plain install. - Hoisted, as on `main`: the install script of a deeply nested package can start while a folder above it is still being replaced (`install --force`, `add`, `update`), and fails because its own folder is gone. - Windows: `bun patch` fails with `EBUSY`, the folder left whole, while another process has a file of the package open without sharing deletes, or has its working directory in the package (1.4.3 worked). `--patches-dir ..\outside` fails, as before. ### How did you verify your code works? - **State of the branch.** The numbers are from CI's release builds of `cc9d7c2da` and of `main` at the merge base. The head adds one fix, to `bun add --trust`, and a test. - **Order of responses.** Random projects, each installed 24 times with a warm and 24 times with a cold manifest cache, under random response delays. Projects that got more than one `bun.lock`, `main` / branch: 6 / 0 of 950; 49 / 0 of 300 with many peers (8 + 8 runs); on the branch 0 of 1,400 with git, GitHub and tarball dependencies. - **Other commands, and faults.** 21 commands (`add`, `remove`, `update`, edits of `package.json`, ...) on random projects, cold against warm: 1,260 cases, no difference. 7,640 pairs of a command and a registry fault: the exit codes of `main`, no hang. 16,148 installs with one package answering 404: none ends with another `bun.lock` and no message. 6,168 bit flips in cached manifests: no crash. - **Wall clock of a cold install,** 450 packages, delayed responses, median of 9 runs, `main` / branch in ms: plain 1555 / 1505; three levels of peers 1603 / 1585; with one slow host 1900 / 1914; three remote tarballs that take 0.3 s 1588 / 1271, 1.5 s 1933 / 1893, 3 s 3426 / 3408. - **A script per report.** 448 repro scripts: 61 of the first 417 pass on `main`, 306 on the branch. One that passes on `main` fails here, on purpose: the error for a command typed in a linked workspace outside the project. The rest fail on both. - **The whole diff against `main`, one pass per kind of regression** (deletes, credentials, loud to quiet, works to refused, lockfile compatibility, cost): about 95,000 differential cases, 364 real lockfiles and 9,571 real package folders found 22 regressions of this branch; each is fixed or listed above. - **Changed existing tests.** The 138 whose expectations the first push touches were audited line by line: none lost, four weakened ones restored. The fixed order changed when 31 tests release a held response, but no expectation. - **Cost, user-space instructions** (millions, median of 3): 1,500 packages, 50 workspaces, bins, scripts, platform packages, overrides, a catalog, aliases; the hoisted project has peers too. | Command | hoisted, `main` | hoisted, branch | isolated, `main` | isolated, branch | | --- | --- | --- | --- | --- | | fresh, cold cache | 1501.3 | 1503.7 | 2441.0 | 2450.1 | | warm, nothing to do | 185.8 | 164.1 | 175.9 | 176.2 | | `--frozen-lockfile` | 133.5 | 122.1 | 150.3 | 150.8 | | `bun add` | 182.3 | 160.5 | 171.1 | 171.7 | | no `node_modules`, cache warm | 198.6 | 179.1 | 1291.4 | 1304.1 | | no lockfile, `--lockfile-only` | 129.7 | 128.9 | 110.8 | 111.0 | - **Above 3 %:** `bunx` of a scoped or cached tool (+0.2 M, 35 system calls); `bun pm untrusted` (+5 %, one more hoist); an isolated `bun prune` that removes something (+4 %); auto-install in a workspace without a lockfile (+13 %; double at 300 workspaces); and the hoist named in the sign-off list. A warm hoisted install makes 8,326 system calls, not 10,434. - **Upgrades over an older tree.** 400 random hoisted projects laid by `main`, taken over, then a pulled `bun.lock`: 17 end with a wrong version on `main`, 21 here (the stale copy after a pull described above). Of planted folders that Bun didn't lay, `main` loses 138, the branch none. - **On earlier builds:** semver against npm (59.9 M comparisons; 21 of 6,698 real ranges differ from npm's strict mode, none changes an install); `bun pm version` against `semver.inc` (0 of 1,530,270 differ); 1,260 tarballs extract as before; older builds of `main` leave a lockfile with the new keys unchanged. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…every file first (oven-sh#45451)
…-sh#45446) Fixes oven-sh#45049 ### Problem - With `NODE_COMPILE_CACHE` set, the cache write after the `exit` event takes about 2x longer since 1.4.3. On the issue repro (a 3.4 MB module) the post-exit time is about 300 ms on 1.4.2 and about 580 ms on 1.4.3 (linux x64). - 5f55496 added an `optimize` flag to `__bun_jsc_generate_cached_bytecode` and passed `true` from the compile cache worker (`src/jsc/NodeCompileCache.rs:919`). That runs the JSC bytecode optimizer over every module on every cold start. ### Fix - The compile cache worker passes `optimize = false`. The optimizer stays a build step of `bun build --bytecode`, where `--no-optimize-bytecode` is the opt-out. - This is correct because the other runtime producers of cached bytecode (`vm.Script` `produceCachedData`, the internal module cache) already generate unoptimized bytecode. The cache directory is keyed on the Bun version tag, so entries written by 1.4.3 are not read by a build with this change. - Warm runs do not change: release 1.4.2 and 1.4.3 both load the repro in about 300 ms. A debug build A/B on one binary (`BUN_JSC_useBytecodeOptimizer=1` to force the optimizer back on) gives a cold write of 3.3 s vs 9.6 s and warm runs within noise (44 ms delta, 100 ms spread). - Verified: `test/js/node/module/node-module-module.test.js` (new case, stock 1.4.3 fails it). Also the rest of that file. ### Background - The compile cache (`NODE_COMPILE_CACHE`, `module.enableCompileCache`) stores serialized JSC bytecode per module. Entries are written at exit on a worker thread with its own VM (`BunCompileCache`). - The bytecode optimizer is a whole-function pass in JSC that decodes the instruction stream to an IR, runs CFG and liveness analyses, and re-emits it. The optimized and unoptimized forms share one serialized format and one decoder. - Considered a named opt-in for the compile cache (an env var or `enableCompileCache` option). `BUN_JSC_useBytecodeOptimizer=1` already forces the optimizer on for any generation, so no new knob is needed. ### Downsides - Users who want optimized bytecode in the compile cache must set `BUN_JSC_useBytecodeOptimizer=1`. No load-time gain was measurable in either the release or the debug A/B above. The A/B covers short-lived runs only. A long-running process that executes cached modules for hours was not measured. - The cache entry is about 3% larger than the optimized one (7332k vs 7096k on the issue repro). - `bun build --bytecode` and its `--no-optimize-bytecode` flag are untouched. Self-reviewed: 2 concerns raised, 2 addressed (both are the body lines above). <details><summary>Notes</summary> - Repro: the `gen.cjs` from the issue, then `NODE_COMPILE_CACHE=$(mktemp -d) bun main.mjs` with a fresh directory each run. Post-exit time is the wall time minus the `performance.now()` value printed from the `exit` handler. - Release, 3 interleaved runs each: 1.4.2 post-exit 563/353/339 ms (first is a cold file system), 1.4.3 789/586/618 ms. `NODE_DISABLE_COMPILE_CACHE=1`: 8 and 9 ms. - Release warm runs (cache present), 5 interleaved pairs: 1.4.2 334/299/298/295/303 ms, 1.4.3 297/299/308/295/287 ms. - Debug build (ASAN), 2000-function module, 3 runs each, same binary: cold post-exit 3342/3230/3269 ms with `optimize=false`, 9666/9785/9430 ms with `BUN_JSC_useBytecodeOptimizer=1`. Warm runs 2344/2398/2189 ms from the unoptimized cache, 2140/2181/2220 ms from the optimized cache. - `perf`, `valgrind`, `strace` and `bloaty` are not installed in the container, so the numbers above are wall clock only. - Producers of cached bytecode: `writeOutputFilesToDisk.rs:397`, `generateChunksInParallel.rs:1126` and `:1445` (build time, `options.optimize_bytecode`), `NodeCompileCache.rs:914` (this change), `NodeVM.cpp:450` `getBytecode` (no optimize), `InternalModuleRegistry.cpp:262` (no optimize). - The test sets `BUN_JSC_reportBytecodeOptimizer=1`, which makes JSC print one `BytecodeOptimizer:` line per code block the optimizer rewrites, and asserts that the compile cache write prints none. </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/node/module/node-module-module.test.js <!-- robobun:evidence:end -->
…many files to lint a few (oven-sh#45302) ### What does this PR do? #### `bun lint` warns when a `tsconfig.json` without `include` made it load many files to lint a few ``` warn: Loaded 10,114 files in 1.4s to lint 1 file: config.ts ../../tsconfig.json has no "include", so every file below it is part of its project. To lint faster, add "include" to it, or pass --tsconfig. ``` A `tsconfig.json` with neither `include` nor `files` includes every file below it. At the top of a repository, that is the whole repository. The rules that need types load all of it for any file that falls to it, such as a `config.ts` beside `src` that the `tsconfig.json` of its package leaves out. With a copy of the sources of vscode as the rest of the repository, linting a package of two files takes 1.8 s and 1.6 GB that way, and 0.15 s and 60 MB with `--tsconfig`. Nothing said where the time went. It is printed if the project has a thousand files or more, and a hundred or more for each file that was linted in it. It goes to standard error, so `-f json` stays JSON. It is not printed where whoever runs it has already chosen the files: | | | |---|---| | `bun lint --type-aware`, `bun lint --type-aware packages/a` | warns | | `--staged`, `--changed`, standard input, `--silent` | does not | | only files are named: `bun lint --type-aware config.ts` | does not | | a file and a directory are named | warns, for what was found in the directory | | the `tsconfig.json` has `include` or `files`, also from one that it extends | does not | | most of the project is linted | does not | A run is judged once, at the end: `--fix` loads a project again for the files that it fixed, and those passes count together. With `--tsconfig` it does not suggest the flag. #### Tests for which `tsconfig.json` a file is checked with With the configuration of oxlint, a file is checked in the first project that includes it: the nearest `tsconfig.json`, then the projects it references, then the same for each `tsconfig.json` further up. A file that none includes is in one program with the others like it. `--tsconfig` is the only project that is asked: nothing further up is. The one test for the flag named the nearest `tsconfig.json`, which gives the same result whether the flag is looked at or not. #### Docs "Choosing a `tsconfig.json`" in `docs/lint/index.mdx`: `--tsconfig`, what a missing `include` means, and the warning. ### How did you verify your code works? - Seventeen rows in `lint.test.ts`, four of them at the two limits, and one test in `git.test.ts`. Each project in them declares a promise of its own, and the file names them all. `typescript/no-floating-promises` reports a name only if its declaration is in the program, so what is reported says which program it was. A second file is reported in every row, so that a run in which nothing happened does not pass. - Each row that depends on the behavior fails on a build without it: the warning on main, the rows for `--staged`, `--changed` and named files on a build that warns regardless, and the rows for the choice of the project on a build that ignores `--tsconfig` and takes the topmost `tsconfig.json`. - Without `--tsconfig`, the rows are also what oxlint 1.87.0 with tsgolint 7.0.2003 reports for these files. There the flag does not change the program of a file, so those rows are `bun lint`'s own. - `lint.test.ts`, `config-files.test.ts`, `bunfig.test.ts` and `git.test.ts`: 2,072 pass, none fail. `cargo clippy` and `cargo mordant` have nothing.
### What does this PR do?
The dev server's framework and bundler options now own their strings.
Before, they were `&'static` views into `UserOptions.arena` or into JS
strings. The second copy of every options type is deleted.
**Before.** There were two families of the same types:
- `bake_body::{Framework, FileSystemRouterType, ServerComponents, ...}`
held `&'static [u8]`. Three safe helpers made those from borrowed data:
`arena_erase`, `arena_dupe_z` and `StringRefList::track`. `from_js`,
`react`, `auto` and `bun build --app` used it.
- `bake::{Framework, ...}` held `Cow<'static, [u8]>`. The dev server
used it.
For `Bun.serve` HTML routes, `ServerConfig` copied the `Cow` family into
the arena to build the `'static` family. `server/mod.rs` then converted
back with `Cow::Borrowed`, so the dev server's copy still pointed into
the arena.
**After.** Only the `bake` family is left, and its string fields are
`Utf8Bytes<'static>`.
- `Borrowed` is only ever a literal or a path the resolver interned.
`Owned` is a computed or transcoded string. `Shared` holds a ref on an
ASCII string from the JS options, which is what `StringRefList` held,
now in the field that reads it. `UserOptions.root` is a `ZBox` and
`env_prefix` a `Box<[u8]>`.
- None of the option structs is `Clone`, so a copy of one does not
compile.
- `bun_bundler::bake_types::BuiltInModule` holds `Cow<'static, [u8]>`
instead of `Box<[u8]>`. `Borrowed` is one of Bun's embedded sources (the
react framework, the react-refresh fallback), in the options and in
every `as_bundler_view`. One from JS is owned bytes and not a ref,
because the bundler's threads read it.
- `from_js`, `react` and `auto` are `impl`s on that family. `auto` is
infallible, as `react` is.
- One `init_transpiler`, taking `&self`. There were two bodies, one per
family. It takes the source map and minify settings from
`BuildConfigSubset` in production, and ignores them in development, as
before.
- One `resolve`, in place. The production one cloned the `Framework`.
- One `resolve_path` behind `resolve_helper` and `resolve_or_null`.
**Behaviour changes**
- A framework entry point or runtime import that resolves to a module
with no usable path aborted the process (`panic: just resolved` in the
dev server, an `unwrap` in `bun build --app`). It now prints `Failed to
resolve '...' for framework (...)`, and `Bun.serve` throws or the build
exits 1. That is a module a `"browser"` map disables, and also `"fs"`
and `"node:fs"`.
- A dev server that fails to start drops `framework` and
`bundler_options`: `DevServer::init` keeps them in `options` until its
fallible steps are done. What else it has written by then still leaks,
as on main.
- `resolve_or_null` maps instead of unwrapping. The `None` cannot be
reached: `auto` uses the VM's resolver, which ignores `"browser"` maps.
**Deleted:**
- the `bake_body` copies of the types, and their duplicate `resolve`,
`resolve_helper`, `as_bundler_view`, `add_react_install_command_note`
and `clone`
- the `From` bridges and `convert_file_system_router_type`
- the three erasure helpers, and the unused `arena` parameters
- `get_optional_string`, which was `get_optional_slice` plus `track`
- `init_transpiler_with_options`'s four extra parameters and the wrapper
around it, whose `ProductionStatic` arm no caller reached
- `BuildConfigSubset.{conditions, drop, ignore_dce_annotations}`.
Nothing wrote them, on main either, and two were keyed by `&'static
[u8]`. `bake.d.ts` lists all three; they were and are ignored.
**Unchanged:**
- The arena stays: the transpilers, their `bake_types::Framework` views
and `DefineData` live in it. The drop order from oven-sh#39630 is still needed.
- The order of JS property reads in `from_js`.
- `ignoreDirs` still iterates the wrong array.
- `serverEntryPoint` is still resolved with the client's resolver.
- The dev server's plugin handle is still never destroyed: main also
takes `bundler_options` out of `UserOptions`, the only owner that
destroys it.
- `production.rs` still never drops its transpilers or their framework
views.
- `bun build --app` still unwraps the resolved path of its config file.
It cannot be `None`: the VM's resolver ignores `"browser"` maps, and the
specifier always starts with `./` or `/`. On a build of main, a config
file that a `"browser"` map names is loaded, and `fs` and `node:fs`
report `could not resolve application config file`.
- `sync_resolver_opts` still copies every built-in module's source into
the resolver's options, twice per transpiler, embedded ones included.
The resolver never reads it.
### Copies, counted from the code against main
**Fewer**
- HTML and `{dir, style}` routes: every router type string was copied
into the arena and the lists were collected again twice. They are moved,
`dir` included.
- A resolved path: the dev server copied each one. It is borrowed when
the resolver interned it.
- An embedded source: the dev server copied it at the bridge between the
families, and both modes copied it in every `as_bundler_view` (one per
transpiler, one after `resolve`, and one at the start of each dev server
bundle). None of those copies it now.
- `bun build --app`: `resolve` cloned the router types and the built-in
module map.
- `root` from the working directory: two arena copies.
**The same.** A string from a JS framework object: a ref when it is
ASCII, one transcode when it is not.
**More**
- `bun build --app`, for each of `builtInModules`: the key is boxed and
an ASCII `path` or `code` is copied once, as the dev server already did
on main. An entry point that names a `path` built-in copies that path,
also as the dev server did.
- `bun build --app`: a resolved path the resolver did not intern
(external, `data:`) is copied. Main kept a `'static` view of the
specifier.
- No bytes, only a `Vec`: the default `extensions` and `ignoreDirs`
lists in `bun build --app`, and the two lists in the router type `react`
builds and `auto` replaces for `{dir, style}` routes.
### How did you verify your code works?
Two new tests with a `clientEntryPoint` that a `"browser"` map disables:
`Bun.serve({ app })` in `test/bake/app-options.test.ts` and `bun build
--app` in `test/bake/dev/production.test.ts`. Both fail on a debug build
of main and on 1.4.3.
**On this head, a local debug + ASAN build:**
- 19 files under `test/bake` pass, with no failure: `app-options`,
`framework-router`, `deinitialization`, `serve-plugins-dev-server`,
`dev-and-prod`, `hmr-socket-protocol`, and
`dev/{bundle,html,css,plugins,production,esm,react-response,react-spa,ssg-pages-router,response-to-bake-response,request-cookies,import-meta-inline,server-sourcemap}`.
That includes the real react framework in development and in `bun build
--app`.
- `app-options` also passes with `BUN_JSC_validateExceptionChecks=1`, as
the ASAN lane runs it.
- `test/internal/source-lints`: 182 pass.
- Against a debug build of main, no difference and no ASAN report:
- every string option as an atom, a rope, a substring and a fresh
string, plus Latin-1 and UTF-16 ones and one string in four fields; the
options object dropped, two full GCs and heap churn before the first
fetch; failing starts, 15 start/stop cycles, `{dir, style}` routes, and
a server in a Worker that is terminated while it runs (65 lines of
output)
- 9 custom frameworks x 4 URLs: `extensions` with and without the dot,
non-ASCII, `"*"` and the two rejected forms; `builtInModules` with
`code`, with `path` and with a duplicate key
- the client bundle of an HTML route with react installed (the embedded
react-refresh) and with react-refresh installed, three server starts
each, by hash
- `bun build --app` with the built-in react framework over faked
dependencies: the files it writes, by hash, and stderr. It ends in
`Runtime file not found` on both.
**Behaviour, release builds from CI, 272ff43 against 4e5926d,
each with an A/A control.** The only differences are the aborts that
became errors. No report from the ASAN build of the head.
- option parsing: 39 property paths x 46 values (1,794 cases), printing
the error or the response
- the order of every property read, with logging, throwing, GC-ing and
array-mutating getters
- resolution: 4 roles x 45 specifiers
- what the options do: 48 configurations x 20 URLs, with freshly built
strings and a GC before the first fetch, comparing bodies, client
bundles and source maps
- export conditions and `import.meta.env` per graph, HTML routes with
and without react and react-refresh, 9 `[serve.static]` bunfigs, 170
`{dir, style}` routes
- lifecycle: 50 start/stop cycles, failing starts, two servers,
`reload()`, workers, exits
**Performance, the same two release builds.** User-space instructions.
The control is a byte-identical copy of the base. Not measured again
since.
| workload | head | control |
|---|---|---|
| 800 start/stop cycles, HTML routes, no react | -0.51% | +0.83% |
| the same, react installed | -0.85% | -0.48% |
| the same, react-refresh installed | -0.44% | +0.12% |
| the same, a custom framework object | -3.83% | -0.21% |
| one `server.reload()`, in those three HTML set-ups | -4.8%, -4.3%,
-4.7% | within 0.0% |
**Not covered.** `console: true` forwarding, macOS and Windows locally.
Moves mordant to a commit that fixes four lint bugs. No lint goes over the baseline.
Moves mordant to a commit with five new lints for unsafe code and binary size, and adds their 148 current findings to the baseline so only new ones fail.
… ms (oven-sh#45460) ### What does this PR do? Follow-up to oven-sh#44968 and oven-sh#45307: what the package manager prints, and how it retries failed requests. #### Messages are terse About 170 warnings, errors and notes are rewritten in the style of a command line tool. The action comes first, articles and pronouns go, the thing is named, and a short reason follows a colon. Conditions, severity and exit codes don't change. ``` skipping link vendor/lib/gone: target does not exist refusing to patch workspace package "x" lockfile is frozen. refusing to update package.json & lockfile cannot read bunfig.toml, refusing to install packages cannot prune symlink packages/a/node_modules cannot migrate package-lock.json: unsupported lockfileVersion 1, trying yarn.lock ``` The full list is at the end. #### A command that succeeds prints next to nothing These only print with `--verbose` now: - deprecated transitive dependencies (a deprecated direct dependency still gets its line) - a `latest` tag that points to an unpublished version - an unnamed dependency in somebody else's package - what a lockfile migration resolves by itself: entries that yarn.lock or package-lock.json doesn't have, unused entries, broken yarn.lock entries, folders that `npm --install-links` copied, workspaces that package-lock.json doesn't list, links that pnpm's `excludeLinksFromLockfile` left out, peers that pnpm-lock.yaml doesn't record - the note that `--frozen-lockfile` keeps a migration from being saved - the note of `bun pm fetch` about the security scanner What still prints: anything that was asked for and didn't happen, a setting that is ignored, and everything about credentials, integrity and where packages come from. #### Retries start at 50 ms Since oven-sh#44968 a failed request waited 250 ms, 500 ms, 1 s, 2 s and 4 s before its five retries. 1.4.3 sent all five within about 5 ms. - The waits are now 50, 100, 200, 400 and 800 ms (±20 %), about 1.5 s in total instead of 8 s. `BUN_CONFIG_HTTP_RETRY_BACKOFF` still sets the first wait. - Without `--verbose`, one line stands for every retry of an install: `warn: registry requests failed, retrying`. It is printed from the third retry on, or at the first retry after a timeout, where an attempt can take minutes. - With `--verbose`, every retry has its own line, as before. - A wait that the registry asks for with `Retry-After` keeps its own line. `bun install` of one package: | Registry | 1.4.3 | `main` | this PR | | --- | --- | --- | --- | | healthy | no wait | no wait | no wait | | one 503, then fine | works, 22 ms | works, 310 ms | works, 78 ms | | 503 for 100 ms | fails, 8 ms | works, 327 ms | works, 222 ms | | 503 or 429 for 1 s | fails, 7 ms | works, 1.7 s | works, 1.6 s | | 503 for 3 s | fails, 18 ms | works, 3.7 s | fails, 1.5 s | | 503 forever | fails | fails, 8.2 s | fails, 1.6 s | | nothing listening | fails, 15 ms | fails, 7.3 s, silent | fails, 1.6 s | The trade-off is the fifth row: an outage of more than about 1.5 s fails the install again. #### Also - `bun patch --commit` no longer says that another package uses a patch file when neither path can be resolved. ### How did you verify your code works? - All 114 install, migration, auto-install and patch test files (plus `test/cli/lint/init.test.ts`, which pins a config warning) on CI's release build of `9502dd99a`, the last commit that changes code: 13,499 pass. The 12 that fail need network access that the sandbox they ran in doesn't have, and fail there on `main` too. - Each reworded message was changed in the source and in the tests that pin it. The tests were then searched for parts of every old message, because many pin only a fragment. That also found checks that a message is *absent*, which would have passed without testing anything, a stream-read loop that stops at a message, and sibling code paths that print the same sentence from another string. - A test of a message that now needs `--verbose` passes the flag, including the checks that a message is absent. The default being quiet has its own checks. The stored npm migration snapshots are unchanged. - Retries: new tests check that two retries print one line without `--verbose` and two lines with it. The table above is measured on CI's build. With 30 packages against a registry that answers 503, a regular run prints the one line. - Before a warning went behind `--verbose`, its code and tests were read to see what it reports. Seven stayed visible because they report something lost: a required peer that isn't installed, a section of a patch that isn't applied, a symlinked `node_modules` where a stale copy can shadow the wanted version (two warnings), `bunx` running its cached copy right after a connection error, a pnpm named registry, a dropped entry of pnpm-workspace.yaml, and a misspelled credential option in .npmrc. ### Every reworded message `{}` stands for a name, path or number. | Before | After | | --- | --- | | `it leads to a directory` | `target is a directory` | | `a link to a directory has to be relative and stay in the folder` | `directory links must be relative and stay inside the folder` | | `it leads out of the project` | `target is outside the project` | | `it could not be created` | `failed to create link` | | `it could not be copied` | `failed to copy target` | | `text that is not a patch` | `non-patch text` | | `the registry asked to wait {}s before the next attempt` | `registry asked to wait {}s before retrying` | | `{} was written, and the install did not run` | `wrote {}, install did not run` | | `error: cannot patch {}: bun does not install it, it ships inside the tarball of a package that bundles it` | `error: refusing to patch bundled dependency "{}"` | | `error: cannot patch {}: {}@{} is not in the lockfile` | `error: cannot patch {}: {}@{} not in lockfile` | | `no npm package {}@{} is in the lockfile, and the other packages named {} have a git, tarball or folder label. Run {} <dependency>@<label> with the dependency name and the label from the lockfile instead.` | `no npm package {}@{} in lockfile. Other packages named {} have a git, tarball or folder label: run {} <dependency>@<label>` | | `edit the folder itself instead` | `edit the folder directly` | | `nothing was written: {}, package.json and the lockfile are as they were` | `no changes made to {}, package.json or the lockfile` | | `The packages that depend on {} load it from the global store. They see your changes after bun patch --commit.` | `Dependents of {} load it from the global store and will not see changes until bun patch --commit.` | | `The install scripts of {} have not run on this copy. They run again at bun patch --commit.` | `Install scripts of {} have not run on this copy. They run on bun patch --commit.` | | `{} is not in the cache, and getting it again failed` | `{} not in cache, re-download failed` | | `error: {} resolves into the global store, which other projects share; refusing to patch through it` | `error: refusing to patch {}: resolves into the shared global store` | | `run bun patch on the package first, it makes the copy to edit` | `run bun patch on the package first to create an editable copy` | | `skipped the dependencies and scripts of {} workspace{} outside the project: {}` | `skipping dependencies and scripts of {} workspace{} outside the project: {}` | | `{} has {}, but {} in package.json asks for {}` | `{} has {}, {} in package.json wants {}` | | `the lockfile is frozen, so the migration from {} was not written to {}; run` | `lockfile is frozen, migration from {} not written to {}; run` | | `the lockfile does not record an integrity for {}@{}, run` | `no integrity in lockfile for {}@{}, run` | | `other local users can add files to {}, so bun ignored the package.json in it` | `ignoring package.json in {}: directory is writable by other users` | | `workspace "{}" is a link to "{}", which is outside the project, so nothing was changed` | `workspace "{}" links to "{}" outside the project, no changes made` | | `"{}" cannot be read or parsed` | `cannot read or parse "{}"` | | `A request failed twice. Sending at most {} requests at a time` | `request failed twice, limiting to {} concurrent requests` | | `each of these needs the next one nested inside it, without end. "overrides" in package.json can keep a single version of one of them` | `each needs a nested copy of the next, forever. Pin one to a single version with "overrides" in package.json` | | `{}: symlinks cannot be created in {}, skipping the links to package binaries` | `{}: cannot create symlinks in {}, skipping bin links` | | `{}: failed to remove {}, which another linker installed` | `{}: failed to remove {} (installed by another linker)` | | `the "latest" tag of "{}" names no published version, using {}` | `"latest" tag of "{}" points to unpublished version, using {}` | | `local tarball dependencies are only allowed in the package.json files of this project` | `local tarballs are only allowed in this project's package.json files` | | `Skipping workspace "{}": its package.json has no "name", so its dependencies will not be installed` | `skipping workspace "{}": package.json has no "name"` | | `"{}@{}" is not in the cache, and only `bun install` downloads it` | `"{}@{}" not in cache, run `bun install`` | | `Bun does not install the devDependencies of a git dependency before its prepare scripts` | `devDependencies of a git dependency are not installed before its prepare scripts` | | `registry {} references {}, but that environment variable is not set; treating it as no credential` | `registry {} references unset variable {}, ignoring credential` | | `for {}@{}; its tarball will not be verified` | `for {}@{}, tarball will not be verified` | | `"patchedDependencies" is ignored in a workspace package. Move it to the root package.json` | `ignoring "patchedDependencies" in workspace package, move it to the root package.json` | | `Skipped a dependency of {}@{} that names no package (` | `skipping unnamed dependency of {}@{} (` | | `Skipped a dependency that names no package (` | `skipping unnamed dependency (` | | `the dependencies in {}/package.json changed since the lockfile was saved (run 'bun install' to update it)` | `dependencies in {}/package.json changed since lockfile was saved (run 'bun install')` | | `"{}@{}" names the folder "{}", which does not exist` | `"{}@{}": folder "{}" does not exist` | | `{} is not a known .npmrc option; ignoring this line` | `ignoring unknown .npmrc option {}` | | `{} cannot be migrated ({}); trying {}` | `cannot migrate {} ({}), trying {}` | | `{} is lockfileVersion {}, which bun cannot migrate; trying {}` | `cannot migrate {}: unsupported lockfileVersion {}, trying {}` | | `{} is lockfileVersion {}, which bun cannot migrate; resolving from package.json instead` | `cannot migrate {}: unsupported lockfileVersion {}, resolving from package.json` | | `{} is lockfileVersion {}, which bun cannot migrate` | `cannot migrate {}: unsupported lockfileVersion {}` | | `{} has no entry for "{}@{}"; bun resolves it` | `{} missing "{}@{}", resolving it` | | `no integrity, and the configured registry has none for this version` | `missing integrity, and configured registry has none for this version` | | `no integrity: downloaded from the configured registry, not from {}{}` | `missing integrity, downloading from configured registry instead of {}{}` | | `skipped npm patches for {} from {}` | `skipping npm patches for {} from {}` | | `skipped "{}@{}" from {}: unsupported version specifier` | `skipping "{}@{}" from {}: unsupported version specifier` | | `skipped "{}@{}" from package-lock.json: no entry for this dependency of` | `skipping "{}@{}" from package-lock.json: no entry for this dependency of` | | `workspace "{}" ({}) is not in {}; resolving it from package.json` | `workspace "{}" ({}) not in {}, resolving from package.json` | | `skipped "{}" from {}: transitive folder dependency "{}" is outside the project` | `skipping "{}" from {}: transitive folder dependency "{}" is outside the project` | | `skipped "{}" from {}: a folder npm copied (--install-links)` | `skipping "{}" from {}: folder copied by npm (--install-links)` | | `skipped {} {} {} not depended on by any package: {}` | `skipping {} unused {} {}: {}` | | `skipped "{}" from yarn.lock: {}` | `skipping "{}" from yarn.lock: {}` | | `skipped "{}@{}" from pnpm-lock.yaml: runtime dependencies are not supported` | `skipping "{}@{}" from pnpm-lock.yaml: runtime dependencies are not supported` | | `skipped "{}" from {}: excluded from pnpm-lock.yaml by excludeLinksFromLockfile` | `skipping "{}" from {}: excluded from pnpm-lock.yaml by excludeLinksFromLockfile` | | `"{}@portal:{}" is migrated as "file:{}"; bun installs a copy of the folder instead of linking it (make it a workspace to keep it linked)` | `migrating "{}@portal:{}" as "file:{}": folder is copied, not linked (make it a workspace to keep it linked)` | | `cache entry has a symlink that may lead out of it` | `cache entry has symlink that may point outside it` | | `the pack is cut short` | `truncated pack file` | | `the registry and the package.json of {} disagree on {}: {} vs {} (bun install goes by the registry)` | `registry and package.json of {} disagree on {}: {} vs {} (bun install uses the registry's)` | | `a prerelease needs a tag: {} would become "latest", the version that `bun add` installs` | `prerelease needs a tag: {} would become "latest"` | | `bunx is installing into the shared temp directory because no cache directory of your own is usable. Set` | `no usable private cache directory, installing into shared temp directory. Set` | | `the global store {} removed with it. Run 'bun install' again in each project that uses it.` | `global store {} removed too, run 'bun install' in each project that uses it` | | `file got shorter while it was packed:` | `file shrank while packing:` | | `cannot be trusted by name: declare it in your package.json` | `cannot trust by name: add it to your package.json dependencies` | | `lifecycle scripts are turned off (ignore-scripts), so bun pm trust cannot run any` | `lifecycle scripts are disabled (ignore-scripts), nothing to run` | | `the lockfile is frozen, and bun pm trust records the packages in package.json and the lockfile` | `lockfile is frozen. refusing to update package.json & lockfile` | | `{} The settings in {} are not applied.` | `{} ignoring settings in {}` | | `"{key}" is ignored in a project's bunfig.toml. Set it in $HOME/.bunfig.toml or in ${env_var}` | `ignoring "{key}" in project bunfig.toml, set it in $HOME/.bunfig.toml or ${env_var}` | | `{} The registry and auth settings in {} are not applied.` | `{} ignoring registry and auth settings in {}` | | `empty _auth value: this line supplies no credentials` | `empty _auth value, no credentials set` | | `Failed to link into {}: {}` | `failed to link into {}: {}` | | `error: Failed to link {}: {}` | `error: failed to link {}: {}` | | `Invalid {}: '{}'. Valid values are: *, any, {}. Use !name to negate.` | `invalid {}: '{}'. valid values: *, any, {}. use !name to negate` | | `bun is unable to write files: {}` | `cannot write files: {}` | | `bun is unable to access tempdir: {}` | `cannot access tempdir: {}` | | `{} accessing temporary directory. Please set $BUN_TMPDIR or $BUN_INSTALL` | `{} accessing temporary directory. set $BUN_TMPDIR or $BUN_INSTALL` | | `bun is unable to write files to the cache directory "{}": {}` | `cannot write files to cache directory "{}": {}` | | `{}: Failed to read temporary directory path: '{}'` | `{}: failed to read temporary directory path: '{}'` | | `{}: failed to open the global directory` | `{}: failed to open global directory` | | `{}: failed to get the full path of the global directory` | `{}: failed to get full path of global directory` | | `Invalid dependency name "{name}"` | `invalid dependency name "{name}"` | | `"{}@{}" is a git dependency, which only `bun install` loads` | `"{}@{}" is a git dependency, run `bun install`` | | `Could not find directory "{}" for linked dependency "{}"` | `directory "{}" not found for linked dependency "{}"` | | `Could not find package.json for "file:{}" dependency "{name}"` | `package.json not found for "file:{}" dependency "{name}"` | | `Could not find package.json for dependency "{name}"` | `package.json not found for dependency "{name}"` | | `to the root package.json to install that tarball for {declarer_name} as well` | `to the root package.json to install it for {declarer_name} too` | | `field in the root package.json, and bun used {} as the root, which has none` | `field in the root package.json. {} was used as root and has none` | | `{}@{} was not installed: unsupported platform (wants {})` | `skipping {}@{}: unsupported platform (wants {})` | | `Restore package.json, or write {} to it to start without dependencies` | `restore package.json, or write {} to start without dependencies` | | `a patch cannot hold a change to a binary file, and git found these in {}:` | `patches cannot contain binary file changes. found in {}:` | | `failed to copy the package to try the patch on: {}` | `failed to copy package to test the patch: {}` | | `failed to move the contents of {} back into the package folder: {}` | `failed to move {} back into package folder: {}` | | `failed to move the previous package folder back, it is at {}: {}` | `failed to restore previous package folder (now at {}): {}` | | `failed to link the dependencies of the package into {}: {}` | `failed to link package dependencies into {}: {}` | | `failed to keep the packages installed under {}, run bun install to get them back: {}` | `failed to keep packages under {}, run bun install to restore them: {}` | | `failed to move everything back into the package folder, the rest is at {}: {}` | `failed to restore package folder (rest is at {}): {}` | | `could not read the package.json of "{}@{}" declared by {}@{}: {}` | `failed to read package.json of "{}@{}" declared by {}@{}: {}` | | `failed to read the workspace's package.json files` | `failed to read workspace package.json files` | | `failed to read the workspace dependencies for --filter` | `failed to read workspace dependencies for --filter` | | `the archive holds no files` | `archive is empty` | | `cache entry has a path that cannot be packed` | `cache entry path cannot be packed` | | `file changed while it was being packed` | `file changed while packing` | | `cache entry has a name that cannot be packed` | `cache entry name cannot be packed` | | `record runs past the end of its package` | `record runs past end of package` | | `bytes after the end of the pack` | `trailing bytes after pack` | | `not the name of a cache entry` | `invalid cache entry name` | | `packages are not in order` | `packages out of order` | | `replacing the links` | `replacing links` | | `could not create the node alias for bun in {}. Scripts that run node will not find it.` | `failed to create node alias in {}, scripts that run node will not find it` | | `{}@{} is not in the catalog{}` | `{}@{} not in catalog{}` | | `Dependency name cannot be empty (in "{}")` | `empty dependency name (in "{}")` | | `Invalid package name {}` | `invalid package name {}` | | `dependency cycle cannot be laid out in node_modules: {}` | `cannot install dependency cycle into node_modules: {}` | | `, and {count} more like it` | `, and {count} more` | | `Ignoring minimumReleaseAgeExcludes entry {}:` | `ignoring minimumReleaseAgeExcludes entry {}:` | | `error: Couldn't find patch file: '{}'` | `error: patch file not found: '{}'` | | `To write it again from the patched package in node_modules, run:` | `To recreate it from the patched package in node_modules, run:` | | `To install the package without a patch, remove` | `To install without the patch, remove` | | `pnpm-lock.yaml dependency '{}' of importer '{}' has a link path that is too long` | `pnpm-lock.yaml dependency '{}' of importer '{}': link path too long` | | `pnpm-lock.yaml package '{}' has a directory path that is too long` | `pnpm-lock.yaml package '{}': directory path too long` | | `"{}" in pnpm-workspace.yaml has entries that are not strings ({}); they were not copied to package.json` | `skipping non-string entries of "{}" in pnpm-workspace.yaml ({})` | | `an argument with one of % & | < > ^ cannot be passed to a .cmd or .bat file` | `cannot pass an argument containing % & | < > ^ to a .cmd or .bat file` | | `yarn.lock patches {} packages with several files each ({}); bun applies one patch per package — merge the files of each into one` | `yarn.lock applies several patch files to {} packages ({}). bun supports one per package, merge them` | | `{f} returned an audit response that is not a list of advisories` | `invalid audit response from {f}: not a list of advisories` | | `{f} returned an empty audit response` | `empty audit response from {f}` | | `bunx could not create a cache directory that only the current user can access. Set` | `failed to create private cache directory. Set` | | `the destination {} is inside the template {}` | `destination {} is inside template {}` | | `the template files were written; run bun install in that directory once the error is fixed` | `template files were written. run bun install there once the error is fixed` | | `the template name is too long` | `template name too long` | | `git has no user.name and user.email, so the files are staged but not committed` | `git has no user.name and user.email, files staged but not committed` | | `a cycle of bundled dependencies cannot be laid out in node_modules:` | `bundled dependencies form a cycle:` | | `the catalog entry refers to another catalog` | `catalog entry refers to another catalog` | | `the package.json of workspace "{}" has no version` | `workspace "{}" has no version in package.json` | | `the package.json of workspace "{}" in the directory "{}" has no version` | `workspace "{}" in "{}" has no version in package.json` | | `refusing to {s} "{s}": it is or contains {s}` | `refusing to {s} "{s}": is or contains {s}` | | `the cache directory comes from` | `cache directory comes from` | | `Point it at a directory that holds nothing but the bun install cache.` | `point it at a directory used only for the bun install cache` | | `{} is in package.json more than once` | `{} listed more than once in package.json` | | `No package name was given and no package.json was found` | `no package name given and no package.json found` | | `invalid tag "{}": a tag cannot be a version or a version range` | `invalid tag "{}": cannot be a version or range` | | `Invalid dependency name "{}"` | `invalid dependency name "{}"` | | `Invalid package name "{}"` | `invalid package name "{}"` | | `b"Invalid package name"` | `b"invalid package name"` | | `"Failed to link {}: {}"` | `"failed to link {}: {}"` | | `skipped pnpm registry "{}" from pnpm-lock.yaml` | `skipping pnpm registry "{}" from pnpm-lock.yaml` | | `invalid `--audit-level` value: '{}'. Valid values are: low, moderate, high, critical` | `invalid `--audit-level` value: '{}'. valid values: low, moderate, high, critical` | | `skipped patch "{}" from pnpm-lock.yaml` | `skipping patch "{}" from pnpm-lock.yaml` | | `skipped peer "{}" of {}: not recorded in pnpm-lock.yaml (bun install will resolve it)` | `skipping peer "{}" of {}: not in pnpm-lock.yaml, resolving it` | | `yarn.lock has no entry for "{}@{}", a dependency of "{}"` | `yarn.lock missing "{}@{}", a dependency of "{}"` | --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…components, bun:app) (oven-sh#45461) Supersedes oven-sh#42826. Bun's core no longer carries React and React Server Components behavior. A framework adds that behavior itself. - A `"use server"` module is a build error instead of a crash. - The `--server-components` flag of `bun build` is removed. - The `bun:app` module and its `Response` class are removed, together with the dev server code that served them. The problem - A module that starts with `"use server"` crashed the dev server and `bun build --app`. Four code paths after the parser were `todo_panic!`. They came with the bake dev server in oven-sh#14025 (September 2024). Before that, the bundler turned an import of a `"use server"` file into a server reference and built the file as its own entry point. The bake rewrite did not carry that over. - `bun build --server-components` built only half of a server components build. It made a client bundler, then dropped it. Nothing in the docs or in a real project used it. - In a server component, the bundler replaced the global `Response` with the `Response` class of `bun:app`. A file that never imported `bun:app` got a different `Response`. In a production build, the output imported `Response` but used the name `import_bun_app.Response`, which was never declared. - The `Response` of `bun:app` is React glue in core. `new Response(<jsx />, init)` looked for a React element and stored `init` for the built-in React framework. `Response.render(path)` threw a response that the dev server caught to render a second route. Both needed an `AsyncLocalStorage` that the dev server passed to the framework's `render` as an extra argument, outside the documented `render(request, routeMetadata)` contract. What changed - The parse step fails a `"use server"` module with the error `"use server" is not supported yet`. The four crash sites cannot be reached. Only JavaScript loaders get the directive scan, so a text file that starts with `"use server"` is text again. This part is the change from oven-sh#42826, rebased onto main. - `--server-components` is removed from the CLI, the docs, the shell completions and the bundler test helper. - The parser does not declare an implicit `Response` in server components. - `bun:app` is removed: the native module, the `JSBakeResponse` class, its builtin and its Rust half, and `BakeAdditionsToGlobalObject`. - The dev server calls the framework's `render(request, routeMetadata)` with the two documented arguments. It no longer creates an `AsyncLocalStorage`, and it no longer catches a thrown `Response`. - The dev server code that only `Response.render()` used is removed: the `bundleNewRoute` and `newRouteParams` functions, the promise each bundle kept for them, and the one-implementation `EnsureRouteCtx` trait. - The built-in React framework no longer reads response options from the `AsyncLocalStorage`. Visible changes - `bun build --server-components` is an unknown flag. - `import ... from "bun:app"` fails to resolve. - A page that returns `new Response(<jsx />)`, or calls `Response.render()`, no longer works. A framework that wants these APIs can build them on top of `render`. - In the built-in React framework, a page that returns `Response.redirect(url)` gets the dev error page instead of a 302. The redirect only worked through the `bun:app` class. Redirects from pages come back with the framework API redesign. - With `separateSSRGraph: false`, a `"use server"` module with no exports built as a plain module. It is now a build error. Tests - `test/bake/dev/bundle.test.ts` and `test/bake/dev/production.test.ts` have the 7 tests from oven-sh#42826. oven-sh#42826 reports that they fail on canary. - `test/bake/dev/react-response.test.ts` keeps its two error tests and adds one that checks the global `Response` in a server component has no `render` method. The nine tests of `bun:app`'s `Response` are deleted with the feature. - `test/bake/dev/response-to-bake-response.test.ts` is deleted. It used the removed flag, and it asserted the broken output above. - On a local debug build, all of `bundle.test.ts` and the two `"use server"` cases in `production.test.ts` pass. The tests that install React could not install it on my machine, so CI is their first run. Not in this PR - `bun build --app` with a custom framework still panics with "Runtime file not found" (oven-sh#32143).
Moves mordant to smarter lock and struct field checks. Turns bool_cluster back on (3 findings, down from 409), adds Bun's lock types to mordant.toml, and drops 129 stale baseline entries.
…#45481) oven-sh#45461 removed the only caller, so the mordant job now reports it as unused on every PR.
…uffer (oven-sh#45454) ### What does this PR do? `Bun.indexOfLine` is declared in `BunObject.bind.ts` instead of parsing a `CallFrame` by hand, and the bindgen gets a way to take a `SharedArrayBuffer`, which `indexOfLine` searches today. ```ts export const indexOfLine = b.fn({ args: { buffer: b.bufferSource.bytes.allowShared, offset: b.f64.loose.default(0), }, returns: b.f64, }); ``` ```rust pub(crate) fn index_of_line(buffer: PossiblySharedBytes<'_>, offset: f64) -> f64 { // The cast saturates: NaN and a negative offset are 0. let offset = offset as usize; // 0x0A never appears inside a multi-byte UTF-8 sequence. match buffer .get(offset..) .and_then(|rest| rest.index_of_char(b'\n')) { Some(i) => (offset + i) as f64, None => -1.0, } } ``` #### Shared memory Another thread can write the bytes of a `SharedArrayBuffer` at any time. A `&[u8]` says that cannot happen, so `b.bufferSource` and its forms refuse a shared buffer. That left no way to port a function that accepts one. `.allowShared` is WebIDL's `[AllowShared]`: opt-in, per argument. The C++ side is marked with WebCore's `IDLAllowSharedAdaptor` and reads the buffer with `JSC::toPossiblySharedArrayBuffer`. The Rust side gets a type of its own, whether the buffer turns out to be shared or not: ```rust pub struct PossiblySharedBytes<'a> { .. } impl PossiblySharedBytes<'_> { pub fn get(self, range: RangeFrom<usize>) -> Option<Self>; pub fn index_of_char(self, needle: u8) -> Option<usize>; } ``` There is no way from it to a slice. What reads the bytes is a method, written for a byte that can differ each time it is read. `index_of_char` hands the pointer to the Highway kernel (`hn::Find`), which compares in registers and returns a lane position clamped to the length: nothing it reads is used as an index or a size. Nothing is copied. The kernel's loads are ordinary ones, not atomic. That is how JavaScriptCore reads the same memory: `Uint8Array.prototype.indexOf` is `typedArrayIndexOfImpl`, which calls `WTF::find8`, SIMD loads with no test for a shared buffer. `new Uint8Array(shared).indexOf(10, offset)` and `Bun.indexOfLine(shared, offset)` do the same thing to the same bytes. This is not hypothetical. `bun_highway::index_of_char`, which `main` calls with a `&[u8]` of the shared memory, ends with ```rust debug_assert!(found.is_none_or(|i| haystack[i] == needle)); ``` which reads the byte a second time. The new test that searches a buffer while a `Worker` writes it, and checks that the writer ran meanwhile, **panics there on a debug build of `main`**. `PossiblySharedBytes` goes to the kernel through `index_of_char_raw`, which does not. As with `b.bufferSource.bytes`, it is only for a function that cannot run JavaScript, and the generated code reads pointer and length again once every argument is converted. Two existing tests are for an `offset` whose `valueOf` transfers the buffer; that order is now what the generator emits rather than something the body has to get right. A new test grows a `SharedArrayBuffer` from `valueOf` and finds a newline in the part that was not there before. #### Also - `b.f64.loose`: `ToNumber`, WebIDL's `unrestricted double`. oven-sh#45450 adds the same lines. - `bun_jsc` depends on `bun_highway`, which `bun_core` already does. #### Behaviour Compared with `main` on 89 calls of `indexOfLine` with hostile arguments. 51 are the same, as are its `name`, `length` and property attributes. A script puts each of the other 38 in exactly one of four boxes, by whether what `main` does is a bug and whether what this PR does is. The search of a buffer that another thread is writing is not one of the 89, since its result is not fixed; it is counted apart. | | Rows | | --- | --- | | [bug → bug](#bug--bug-0) | 0 | | [bug → non-bug](#bug--non-bug-11-and-the-shared-buffer) | 11, and the shared buffer | | [non-bug → bug](#non-bug--bug-0) | 0 | | [non-bug → non-bug](#non-bug--non-bug-27) | 27 | ##### bug → bug (0) None. ##### bug → non-bug (11, and the shared buffer) | Call | `main` | Now | Rows | | --- | --- | --- | --- | | `indexOfLine(buffer, x)` where `x` is a string or an object: `"4294967302"`, `"4294967296"`, `"-4294967290"`, `"Infinity"`, `"1e300"`, `{ valueOf: () => 2 ** 32 + 6 }`, `[4294967302]` | `ToInt32`: it wraps, so these search from 6, 0, 6, 0, 0, 6, 6 | the number itself, as when it is written as a number: `-1` beyond the end, from 0 for a negative one | 7 | | `indexOfLine(buffer, Infinity)`, `2 ** 63`, `2 ** 64`, `1e300` | searches from 0 | `-1`: that is beyond the end | 4 | | `indexOfLine(shared)` while a `Worker` writes `shared` | a `&[u8]` of memory another thread writes, which is undefined behaviour in Rust; a debug build panics at `debug_assert!(found.is_none_or(\|i\| haystack[i] == needle))` | no slice is made, and nothing is read twice. The index is `-1` or within the buffer | the new test | Both rows of numbers come from `JSC__JSValue__coerceToInt64`. For a double that does not fit it does `static_cast<int64_t>`, which is undefined behaviour: x64 gives `INT64_MIN`, which was then clamped to 0. For anything that is not a number or a BigInt it does `toInt32`. ##### non-bug → bug (0) None. ##### non-bug → non-bug (27) **Stricter: a call that returned now throws (26).** These work in 1.4.3, so they are breaking changes, and deliberate ones. | Call | `main` | Now | Rows | | --- | --- | --- | --- | | `indexOfLine(x)` where `x` is not a buffer: nothing, `undefined`, `null`, a string, a number, an array, an object | `-1` | `TypeError` `ERR_INVALID_ARG_TYPE` "buffer must be an ArrayBuffer or ArrayBufferView" | 25 | | `indexOfLine(buffer, 10n)` | searches from 10 | `TypeError` "Conversion from 'BigInt' to 'number' is not allowed.", as `ToNumber` of a BigInt is everywhere else. It is `number` in `bun-types` | 1 | **A call that threw still throws, with another error (1).** | Call | `main` | Now | Rows | | --- | --- | --- | --- | | `indexOfLine("x", { valueOf() { throw .. } })` | what `valueOf` throws | the `TypeError` for the buffer, which is looked at first | 1 | #### Performance CI's release builds for linux-x64 of this branch (`538e9828b`) and of its merge base (`af5743d74`), on an AMD Ryzen AI Max+ 395. The numbers are user-mode cycles and instructions per call: the difference between N and 4N calls, each in a fresh process under `perf stat` on a reserved core, the least of 7 rounds in alternating order. "Control" is a byte-identical copy of the base's binary, measured the same way. | Call | Cycles, base | Cycles, now | Change | Control | Instructions, base | Instructions, now | | --- | --- | --- | --- | --- | --- | --- | | `indexOfLine(bytes, 32)`, 64 bytes | 39.3 | 39.3 | 0.0% | +0.7% | 235 | 214 | | `indexOfLine(bytes)`, 64 KB | 1,501 | 1,517 | +1.1% | −0.2% | 6,387 | 6,390 | | `indexOfLine(view)`, 64 KB, at each of 64 alignments in turn | 1,546 | 1,565 | +1.2% | +0.7% | 6,336 | 6,337 | | `indexOfLine(view)`, 164 bytes, at each of 64 alignments in turn | 37.2 | 39.7 | +6.7% | +1.0% | 229 | 230 | | `indexOfLine(bytes)`, 64 bytes | 30.8 | 33.6 | +9.0% | +0.1% | 189 | 192 | | `indexOfLine(bytes)`, 64 bytes of a `SharedArrayBuffer` | 31.4 | 34.9 | +11.0% | −0.7% | 197 | 192 | A call without an offset costs about 3 more cycles. `offset` is a double, and the saturating cast to `usize` is on the way to the first load whether or not an offset was passed; `main` skips its conversion when there is none. That cast is what makes `Infinity` and `2 ** 64` come out right. With an offset the cost is the same, in 21 fewer instructions. Over 64 KB the difference is 1%. The binary is the same size: 106,178,480 bytes. ### How did you verify your code works? - The calls above, each row that differs classified by a script that fails on one it does not know, on a debug build with ASAN and `BUN_JSC_validateExceptionChecks=1`, against the same build of `main`. The other 580 rows of that corpus, for functions this PR does not touch, are unchanged. - `test/js/bun/util/index-of-line.test.ts`: 22 pass. Of the 5 new tests, 3 fail on `main` and the one with the writing `Worker` panics its debug build. - `test/internal/bindgen.test.ts`, with what the generator refuses: `.allowShared.optional`, and `.allowShared` with `capability: "js"`. - `test/internal/source-lints`, `test/internal/build-codegen-declared-outputs.test.ts`, clippy on `bun_highway`, `bun_jsc` and `bun_runtime`, `tsc`, and `bun run rust:mordant` at the revision CI pins.
Bumps `WEBKIT_VERSION` from `b1e30d08cb14` to `87ee10da087b` (current oven-sh/WebKit `main`). The `autobuild-87ee10da087b63e011bd1ada64d1814a850a4395` release exists. oven-sh/WebKit changes picked up: - [JSC] PutCellButterflySlot forbids the double format for the variable it stores (oven-sh/WebKit#792) Checked locally on a debug build, whose `process.versions.webkit` is `87ee10da087b63e011bd1ada64d1814a850a4395`: `[17, 8, 1, 1]` passed as four arguments to `(v, w, x, y) => [v, w, x, y].sort((a, b) => (a + 0.5) - (b + 0.5))` comes back as `1,1,8,17` in 200,000 of 200,000 calls. Bun 1.4.3 returns a wrong array in 192,141 of them. Everything else is left to CI.
### What does this PR do?
Eighteen functions that parsed a `CallFrame` by hand are now declared in
`.bind.ts` files, and their Rust bodies receive converted arguments.
| Function | Declared as |
| --- | --- |
| `Bun.allocUnsafe(size)` | `size: b.u32.validateInteger()`, returns
`b.bufferSource` |
| `Bun.mmap(path, options)` | `path: b.string.utf8`, `{ shared, sync:
b.bool.loose, size, offset: b.f64.loose.allowBigInt }`, returns
`b.bufferSource` |
| `Bun.openInEditor(path, options)` | `path:
b.string.loose.utf8.nullable.default("")`, `{ editor, line, column:
b.string.loose.utf8.nullable }` |
| `Bun.shrink()` | no arguments |
| `Bun.sleepSync(milliseconds)` | `milliseconds: b.f64` |
| `Bun.which(command, options)` | `command:
b.string.loose.utf8.nullable.default("")`, `{ PATH, cwd:
b.string.loose.utf8.optional }`, returns `b.string.utf8.nullable` |
| `Bun.$.escape(input)` | `input: b.string.loose`, returns `b.string` |
| `Bun.semver.satisfies(version, range)`, `order(left, right)` |
`b.string.loose.utf8` twice, returns `b.bool`, `b.i32` |
| `Bun.CSRF.generate(secret, options)` | `secret:
b.string.utf8.optional`, `{ expiresIn: b.u64.validateInteger().default(a
day), sessionId: b.string.utf8.nullable, encoding:
b.string.loose.utf8.optional, algorithm: b.string.utf8.optional }` |
| `Bun.CSRF.verify(token, options)` | `token: b.string.utf8`, the same
with `secret` and `maxAge`, returns `b.bool` |
| `Bun.dns.prefetch(hostname, port)` | `hostname: b.string.utf8`, `port:
b.u16.validateInteger().nullable.default(443)` |
| `Bun.dns.getCacheStats()` | returns a dictionary of six `b.usize` |
| `Bun.unsafe.gcAggressionLevel(level)` | `level: b.f64.loose.nullable`,
returns `b.i32` |
| `Bun.unsafe.setJITPolicy(scale)` | `scale: b.f64` |
| `Bun.unsafe.memoryFootprint()`, `mimallocDump()` | no arguments,
returns `b.usize.optional`, nothing |
| `Bun.unsafe.arrayBufferToString(buffer)` | `buffer: b.any`: see "Not
in this PR" |
The Rust files lose 826 lines and gain 406; the schemas gain 207. For
example, all of `Bun.which`:
```rust
pub(crate) fn which<'a>(
heap: &JsHeap<'a>,
command: &'a [u8],
options: WhichOptions<'a>,
) -> bindgen::Result<'a, Option<JsLocal<'a, JSString>>> {
if command.len() >= MAX_PATH_BYTES {
return Err(heap.error(format_args!("bin path is too long")));
}
let vm = VirtualMachine::get();
let path = options
.path
.unwrap_or_else(|| vm.env_loader().get(b"PATH").unwrap_or(b""));
let cwd = options.cwd.unwrap_or_else(|| vm.top_level_dir());
let mut path_buf = bun_paths::path_buffer_pool::get();
bun_which::which(&mut *path_buf, path, cwd, command)
.map(|bin_path| heap.create_utf8_string(bin_path))
.transpose()
}
```
Thirteen of the eighteen take `capability: "none"` or `"heap"`, so the
compiler holds them to not running JavaScript. The other five ask for
`"js"` only because `cx.global()` is the way to the `&JSGlobalObject`
that older helpers take. `openInEditor` had a comment asking that every
coercion finish before its `RefCell` is borrowed, after a re-entrant
getter made it panic. That is now how the function is built: the
conversions are done before the body starts, and the body cannot call
back into JavaScript. `CSRF.generate` and `verify` each had their own
copy of the `sessionId` check and of the list of algorithms; they now
share them.
#### What the bindgen gained
- `b.namespace({ name: { args, returns, .. }, .. })`: a plain object
that holds functions, as WebIDL's `namespace`. It generates what
`create_host_function_object` and a lazy property callback did in Rust
for `Bun.semver`, `Bun.unsafe` and `Bun.CSRF`: the same own, writable,
enumerable, configurable properties in the same order, made when the
property of `Bun` is first read.
- `.allowBigInt`. An integer took a number or a BigInt. Now `b.u32`
takes a number, and `b.u32.allowBigInt` a BigInt in the same range too,
so a schema says what the signature does: `number`, or `number |
bigint`. `.loose` is `ToNumber` without it and `ToNumeric` with it.
`b.f64.loose.allowBigInt` is `ToNumeric`, and a BigInt becomes the
double nearest to it. The eight options that were declared with a bare
integer now say `.allowBigInt`, so they take what they took: `port` and
`fd` of a socket, and `secureOptions`, `minVersion`, `maxVersion`,
`clientRenegotiationLimit`, `clientRenegotiationWindow` and
`sessionTimeout` of `tls`.
- `nonObjectAsEmpty` now reads a function like any other object.
`nonObjectAsEmpty: "typeof"` is what it was, for `os.userInfo`, where
Node tests `typeof options !== "object"`.
- The body of a `b.fn` or of a namespace's function can return a
`b.string.utf8` result as the string itself, from
`heap.create_utf8_string(bytes)`, when its bytes are in a buffer that
does not outlive it. `which` finds its answer in a pooled path buffer,
and makes the string from there as before, without a copy in between.
- `heap.error_with_code(ErrorCode::X, ..)`, so a body that cannot run
JavaScript can still throw an error with a `code`. The number inside an
`ErrorCode` is private now: C++ indexes a table with it, unchecked.
- `heap.create_uninitialized_uint8_array(length)`.
- `?` on a `JsError` or an `AllocError` in a body that returns
`bindgen::Result`.
`bun_semver_jsc` is the third crate with bindings, so it gets a
`generated_bindgen.rs` and a `build.rs` that says where the generated
file is, as `bun_jsc` and `bun_runtime` have. `dns.getCacheStats` is the
first shipped function that returns a dictionary.
Removed with their last user: `bun_csrf::DEFAULT_EXPIRATION_MS` (the
default is in the schema), `IntegerRange::always_allow_zero`, the
`UnsafeObject` alias in `api.rs`.
`docs/project/bindgen.mdx` has what a schema or a body is written with,
and three rules the generator already enforced: an array's elements hold
no JS value, a named type is used by schemas of its own crate, and a
member cannot be named by a C++ keyword.
#### Behaviour
Compared with `main` on 2,202 calls with hostile arguments (every
primitive, boxed values, `valueOf`/`toString` that log or throw, getters
that log, inherited and null-prototype options, arrays, functions,
classes and proxies as options, a polluted `Object.prototype`, lone
surrogates, out-of-range numbers) and on the shape of each object and
function. 1,675 are the same, including the order in which members are
read and converted, and all 584 that pass numbers, BigInts and what
converts to them as those eight options. A script puts each of the other
527 in exactly one of four boxes, by whether what `main` does is a bug
and whether what this PR does is:
| | Rows |
| --- | --- |
| [bug → bug](#bug--bug-0) | 0 |
| [bug → non-bug](#bug--non-bug-76-and-two-outside-the-corpus) | 76, and
two outside the corpus |
| [non-bug → bug](#non-bug--bug-0) | 0 |
| [non-bug → non-bug](#non-bug--non-bug-451) | 451 |
One rule explains many rows: **every argument and every member is
converted before the body looks at any value.** `main` interleaved the
two.
##### bug → bug (0)
None. Three bugs were found on the way and are left exactly as they are,
so they are not among the 527. A null byte in `which`'s command or
`dns.prefetch`'s hostname ends it for the C API it goes to:
`which("sh\0x")` is `"/usr/bin/sh\0x"`, and
`dns.prefetch("localhost\0x")` looks up `localhost`. `bun-types`
declares `mmap(path: PathLike)`, and a `Buffer` or a `URL` is refused.
##### bug → non-bug (76, and two outside the corpus)
| Call | `main` | Now | Rows |
| --- | --- | --- | --- |
| `unsafe.arrayBufferToString(view)` where `view` is a detached
`Uint16Array` or `Int16Array` | the process aborts, in 1.4.3 too:
`bytemuck` refuses the unaligned empty slice | `""`, as for a detached
`Uint8Array` | not in the corpus: `main` does not survive it |
| `mmap("file\0x")` | maps `file`, in 1.4.3 too; a debug build trips an
assertion in `ZStr::as_cstr` | `TypeError` `ERR_INVALID_ARG_VALUE` "The
argument 'path' must be a string, Uint8Array, or URL without null
bytes.", from the check `Bun.file` and `node:fs` make | not in the
corpus: the debug build of `main` does not survive it |
| `mmap(path, { size: x })`, `{ offset: x }` where `x` is a string or an
object: `"4294967301"`, `"3000000000"`, `"Infinity"`, `"-4294967291"`,
`{ valueOf: () => 2 ** 32 + 5 }`, `[4294967301]` | `ToInt32`: it wraps,
so these are 5, negative, 0, 5, 5, 5 | the number itself, as when it is
written as a number. 11 of these rows returned and now throw: an offset
beyond the end is `EINVAL`, and a negative one ".. must be a
non-negative integer" | 20 |
| `mmap(path, { size: x })`, `{ offset: x }` where `x` is a BigInt that
does not fit 64 bits: `2n ** 63n`, `2n ** 64n + 5n`, `-(2n ** 64n) + 5n`
| it wraps, so these are negative, 5, 5 | the value itself: a size
beyond the end is the whole file, an offset beyond the end `EINVAL`, and
a negative one ".. must be a non-negative integer" | 6 |
| `$.escape(x)` where `x` is not a string and its string needs no
escaping: `undefined`, `null`, `true`, `false`, `NaN`, `new
String("foo")`, `{ toString }` | `x` itself, though it is declared to
return a string | the string, lone surrogates included | 11 |
| `dns.prefetch(host, 65536)`, `-1`, `2 ** 32`, `Infinity`, `±2 ** 53`,
`±1e21` | "It must be >= -9007199254740991 and <= 9007199254740991" |
"It must be >= 0 && <= 65535", "It must be an integer" | 8 |
| `shrink`, `unsafe.gcAggressionLevel`, `mimallocDump`,
`memoryFootprint`, `CSRF.generate`: `length`; `dns.prefetch`: `length` |
1, with no required argument; 2, with one | 0; 1 | 6 |
| `unsafe.gcAggressionLevel(x)` where `x` is a string or an object:
`"4294967297"`, `"4294967296"`, `"Infinity"`, `{ valueOf: () => 2 ** 32
+ 2 }` | `ToInt32`: levels 1, 0, 0, 2 | ignored, as the same number is |
6 |
| `mmap(path, { offset: NaN })`, `{ size: NaN }`, `{ size: Infinity }`,
`{ size: 2 ** 64 }`, `{ offset: 2 ** 64 }` | ".. must be a non-negative
integer" | as `{ offset: {} }`, `{ size: {} }`, `{ size: 2 ** 53 }` and
an offset beyond the end already were: the whole file, or `EINVAL` from
`mmap` | 5 |
| `allocUnsafe(1.9)`, `2 ** 32`, `2 ** 53`, `Infinity` | "Expected a
positive number", which they are | "It must be an integer", "It must be
>= 0 && <= 4294967295" | 4 |
| `CSRF.generate(undefined)`, `generate(undefined, options)` | "Secret
is required", so there was no way to pass options and use the default
secret, which `secret?: string` allows and `verify(t, { secret:
undefined })` does | the default secret, as `generate()` | 3 |
| `CSRF.generate(s)`, `verify(s)` where `s` is an instance of a subclass
of `String` whose `toString` returns another string | emptiness is
tested on the object and the bytes come from `toString`, so `""` from
`toString` signs with an empty secret | both are the converted string:
"Secret must be a non-empty string" | 3 |
| `openInEditor(undefined)`, `openInEditor(null)` | opens a file called
`undefined`, `null` | "No file path specified", as `openInEditor()` | 3
|
| `allocUnsafe(-0)` | "Expected a positive number" | 0 bytes | 1 |
Three of these come from `JSC__JSValue__coerceToInt64`. For a double
that does not fit it does `static_cast<int64_t>`, which is undefined
behaviour: x64 gives `INT64_MIN`, which read as negative. For a BigInt
it does `toBigInt64`, which wraps. For anything else that is not a
number it does `toInt32`. The bodies now take a double and cast in Rust,
where `NaN` is 0 and the rest saturates.
`mmap` takes a BigInt that fits as it did, for example
`fs.statSync(path, { bigint: true }).size`, and `bun-types` now says so:
`size?: number | bigint`, `offset?: number | bigint`. So does
`indexOfLine`'s `offset`, which is `b.f64.loose.allowBigInt` too.
##### non-bug → bug (0)
None.
##### non-bug → non-bug (451)
**Stricter: a call that returned now throws (10).** These work in 1.4.3,
so they are breaking changes, and deliberate ones.
| Call | `main` | Now | Rows |
| --- | --- | --- | --- |
| `unsafe.gcAggressionLevel(1n)`, `0n`, `2n`, `10n` | sets the level; 10
is not one | `TypeError` "Conversion from 'BigInt' to 'number' is not
allowed.", as `ToNumber` of a BigInt is everywhere else | 4 |
| `CSRF.generate(s, { expiresIn: NaN })`, `CSRF.verify(t, { maxAge: NaN
})`, `dns.prefetch(host, NaN)` | the default: 24 hours, 443 |
`RangeError` `ERR_OUT_OF_RANGE` "It must be an integer. Received NaN",
as Node's `validateInteger`, which they now are | 3 |
| `which("", { PATH: Symbol() })`, `which(null, { cwd: Symbol() })`,
`which(undefined, { get PATH() { throw .. } })` | `null`, without
looking at the options | what the conversion throws | 3 |
`csrf.test.ts` had "NaN is treated as an absent option", from the fix
that stopped `NaN` from becoming `0`, which means that a token never
expires. Refusing `NaN` keeps that property; the test now says so.
**More lenient: a call that threw now returns (45).** An argument that
is left out is `undefined`.
| Call | `main` | Now | Rows |
| --- | --- | --- | --- |
| `semver.satisfies()`, `satisfies(v)` | "Expected two arguments" | what
`satisfies(v, undefined)` returns, on `main` too: `false` where `v` is
not a version, `true` for `satisfies("1.0.0")`, since "a `range` with no
parseable part behaves like `*`" (`docs/runtime/semver.mdx`). Or what
converting `v` throws | 27 |
| `indexOfLine(buffer, 6n)`, and any other BigInt or what converts to
one | `TypeError` "Conversion from 'BigInt' to 'number' is not
allowed.", since oven-sh#45454, which is in no release | searches from there,
as 1.4.3 does. What wraps in 1.4.3 does not: `2n ** 64n + 6n` is beyond
the end, so `-1`, where 1.4.3 searches from 6 | 11 |
| `mmap(path, { size: Object(5n) })`, `{ size: { valueOf: () => 5n } }`,
and the same for `offset` | `TypeError` "Conversion from 'BigInt' to
'number' is not allowed." | 5, as `5n` | 4 |
| `which()` | "which: expected 1 argument, got 0" | `null`, as
`which(undefined)` | 1 |
| `semver.order("日本")` | "Expected two arguments" | `0`, as `order("日本",
undefined)`, on `main` too: an argument that is not ASCII is answered
before the other is parsed | 1 |
| `$.escape()` | "shell escape expected at least 1 argument" |
`"undefined"`, as `$.escape(undefined)` | 1 |
**A call returns something else (6).**
| Call | `main` | Now | Rows |
| --- | --- | --- | --- |
| `openInEditor(path, { editor: x })` where `x` is not `""` and converts
to it (`[]`, `new String("")`), after another editor was chosen | looks
for an editor again | keeps the editor, as for `""` | 3 |
| `which(null, options)`, `which("", options)` | `null`, without reading
`options` | `null`, after reading `options`: its getters can tell | 2 |
| `unsafe.gcAggressionLevel({ valueOf() { gcAggressionLevel(2); return 1
} })` | returns the level before `valueOf` ran | the level after | 1 |
**A call that threw still throws, and another of two errors wins or more
getters run (13).** Conversions come first.
| Call | `main` | Now |
| --- | --- | --- |
| `which(tooLong, { PATH: thrower })`, `mmap(tooLong, { size: thrower
})`, `mmap(path, { size: -1, offset: thrower })` | "bin path is too
long", "Path too long", "size must be a non-negative integer" | what
`thrower` throws |
| `CSRF.generate("", { expiresIn: "x" })`, `verify("", { secret: 1 })`,
`generate(s, { encoding: "utf8", algorithm: 5 })` | "Secret must be a
non-empty string", "Token must be ..", "Invalid format: .." | the error
of the member that does not convert |
| `CSRF.verify(t, { sessionId: "", maxAge: -1 })` | `TypeError`
`ERR_INVALID_ARG_TYPE` "sessionId must be a non-empty string" |
`RangeError` `ERR_OUT_OF_RANGE` for `maxAge` |
| `CSRF.generate("", options)`, `generate(s, { sessionId: "", .. })`,
`verify("", options)`, `verify(t, { secret: "", .. })`, `mmap(path, {
size: -1, offset })` | the same error, after reading none or some of the
members | the same error, after reading all of them |
**A call that threw still throws, with the bindgen's error (377).** The
class and `code` are the same except where noted.
| Call | `main` | Now | Rows |
| --- | --- | --- | --- |
| `CSRF.generate(null)`, `generate(1)`, `verify()`, `verify(null)`,
`verify(1)` | "Secret is required", "Secret must be a non-empty string",
"Missing required token parameter", "Token is required", "Token must be
a non-empty string" | "secret must be a string", "token must be a
string". An empty one is still ".. must be a non-empty string" | 237,
with the next two |
| `expiresIn`, `maxAge`: not a number, `1.5`, out of range | 'The
"expiresIn" property must be of type number. Received string',
`TypeError` '.. must be of type integer', 'It must be >= 0 and <=
9007199254740991.' | Node's: 'The "CSRFGenerateOptions.expiresIn"
property must be of type number. Received type string (\'100\')',
`RangeError` 'It must be an integer.', 'It must be >= 0 && <=
9007199254740991.' The range is the same | |
| `sessionId`, `secret`, `algorithm`: not a string | 'The "sessionId"
property must be of type string, got number' |
"CSRFGenerateOptions.sessionId must be a string" | |
| `mmap(1)`, `mmap(path, 1)` | "Expected a path", "Expected options to
be an object" | "path must be a string", "MMapOptions must be an object"
| 36 |
| `semver.order()`, `order(v)` | "Expected two arguments" | "Invalid
SemVer: undefined", or what converting the argument throws | 27 |
| `allocUnsafe("1")`, `allocUnsafe(10n)`, `allocUnsafe(-1)` |
`TypeError` `ERR_INVALID_ARG_TYPE` "Expected a positive number" |
Node's: 'The "size" argument must be of type number. Received type
string (\'1\')'; `RangeError` `ERR_OUT_OF_RANGE` 'It must be >= 0 && <=
4294967295. Received -1' | 21 |
| `dns.prefetch()`, `prefetch(1)`; `port`: not a number, `1.5` |
`ERR_MISSING_ARGS` "Not enough arguments to 'prefetch'. Expected 1, got
0."; 'The "port" property must be of type number. Received ..',
`TypeError` '.. must be of type integer' | "hostname must be a string";
Node's: 'The "port" argument must be of type number. Received ..',
`RangeError` `ERR_OUT_OF_RANGE` 'It must be an integer.' | 20 |
| `sleepSync()`, `sleepSync("1")` | `ERR_MISSING_ARGS` "Not enough
arguments to 'sleepSync'. Expected 1, got 0.", "Expected milliseconds to
be a number for 'sleepSync'." | `ERR_INVALID_ARG_TYPE` "milliseconds
must be a number" | 18 |
| `unsafe.setJITPolicy("1")` | 'The "scale" argument must be of type
number. Received ..' | "scale must be a number" | 18 |
**What a Linux run does not show**, from reading. On Windows `main`
throws "mmapFile is not supported on Windows" before it looks at
anything; now the arguments are converted first, so `Bun.mmap(1)` is
"path must be a string" there too. `main` reads `sync` on Linux only;
now its getter runs everywhere, and the value is used on Linux only. The
counters `csrf_generate` and `csrf_verify` count the calls whose
arguments convert.
`allowArray` and `nonObjectAsEmpty` in the schemas are there so that
options which were accepted or ignored still are.
#### Performance
CI's release builds for linux-x64 of this branch (`47a12c44c`) and of
its merge base (`af5743d74`), on an AMD Ryzen AI Max+ 395. The numbers
are user-mode cycles and instructions per call: the difference between N
and 4N calls, each in a fresh process under `perf stat` on a reserved
core, the least of 7 rounds in alternating order. "Control" is a
byte-identical copy of the base's binary, measured the same way. It
differs from the base by up to 2.8% in cycles, so a change of that size
with no change in instructions is noise. A checksum of what the calls
return is the same in all binaries.
| Call | Cycles, base | Cycles, now | Change | Control | Instructions,
base | Instructions, now |
| --- | --- | --- | --- | --- | --- | --- |
| `dns.getCacheStats()` | 1,119 | 218 | −80.5% | 0.0% | 3,905 | 1,210 |
| `which("missing", { PATH, cwd })` | 919 | 358 | −61.0% | 0.0% | 2,758
| 1,447 |
| `which("sh", { PATH, cwd })` | 914 | 396 | −56.6% | −1.0% | 2,646 |
1,419 |
| `CSRF.verify(token, { secret })` | 2,352 | 1,500 | −36.2% | 0.0% |
7,800 | 5,581 |
| `mmap(path, { shared, size, offset })` | 1,889 | 1,314 | −30.5% |
−0.3% | 5,718 | 3,973 |
| `unsafe.gcAggressionLevel()` | 20.5 | 15.0 | −26.8% | +1.1% | 98 | 82
|
| `CSRF.verify(token, options)`, five options | 4,465 | 3,292 | −26.3% |
−0.1% | 20,310 | 17,769 |
| `semver.satisfies("1.2.3", "1.2.3")` | 364 | 271 | −25.5% | +0.6% |
1,669 | 1,527 |
| `semver.satisfies("1.2.3", "^1.0.0")` | 392 | 298 | −24.2% | +0.3% |
1,789 | 1,662 |
| `semver.order("1.2.3", "1.10.0")` | 215 | 177 | −17.7% | −0.4% | 1,368
| 1,248 |
| `$.escape("hello world")` | 269 | 224 | −16.9% | −0.4% | 1,111 | 1,122
|
| `allocUnsafe(16)` | 43.3 | 36.6 | −15.6% | −1.2% | 232 | 220 |
| `CSRF.generate("secret", options)`, four options | 6,301 | 5,390 |
−14.4% | 0.0% | 23,132 | 21,035 |
| `which("sh")` | 442 | 403 | −8.8% | +1.0% | 1,372 | 1,375 |
| `$.escape("hello")` | 117 | 112 | −3.9% | −0.4% | 233 | 309 |
| `CSRF.generate("secret")` | 3,059 | 2,988 | −2.3% | 0.0% | 5,983 |
5,984 |
| `mmap(path)` | 1,218 | 1,229 | +0.9% | −0.9% | 3,629 | 3,665 |
| `allocUnsafe(4096)` | 287 | 291 | +1.5% | +2.8% | 1,145 | 1,136 |
| `sleepSync(0)` | 84.0 | 86.0 | +2.4% | 0.0% | 179 | 182 |
| `unsafe.arrayBufferToString(sixteenBytes)` | 104 | 106 | +2.5% | 0.0%
| 394 | 408 |
| `$.escape("日本 語")` | 316 | 330 | +4.6% | +1.0% | 1,474 | 1,514 |
- What has options gains the most. For each member `main` calls into
C++, where an `Identifier` is made from the bytes of the name. The
generated code reads them all in C++, with the identifiers the VM holds
for the names in schemas. `dns.getCacheStats` is the same the other way
round: six `put`s by name on `main`, one call now.
- `$.escape("hello")`, which needs no escaping, runs 76 more
instructions in fewer cycles: it returns a new string cell for the same
characters where `main` returned its argument.
- `$.escape("日本 語")` is the one that is slower, by 14 cycles. The result
is made with `String::clone_utf8`, which validates and measures the
UTF-8 before it converts it; `main` converted it in one call.
- `sleepSync(0)` and `unsafe.arrayBufferToString` have almost nothing to
convert, and run 3 and 14 more instructions, 2 and 3 cycles.
- `CSRF.verify(token, { secret })` costs one of two amounts, fixed for
the life of a process, in every binary: about one process in seven runs
1,755 fewer instructions per call. Its row is the median of 40 rounds,
which is the common amount. The rare one goes from 2,067 cycles to
1,227.
The binary is 4,096 bytes larger: 106,182,576 against 106,178,480.
#### Not in this PR
- What takes "a string, a buffer or a `Blob`, as bytes": `Bun.hash.*`,
`gzipSync` and its siblings, `zstdCompressSync`, `TOML.parse` and its
siblings, `randomUUIDv5`. `b.bufferSource.bytes` cannot be an
alternative of a union, and they all accept a `SharedArrayBuffer` today,
whose bytes `.allowShared` lends but has no way to copy out.
`unsafe.arrayBufferToString` is one of them: it moved with its
namespace, takes `b.any` and checks the value as before.
- `Bun.resolve` turns an error from converting its arguments into a
rejected promise, and shares its parsing with `resolveSync`. There is no
promise-returning function yet.
- `Bun.randomUUIDv7` takes a `Date` and `Bun.registerMacro` a function.
Neither is a type of the schema yet.
- The `length` of a function in a static property table (`Function 1` in
`BunObject.cpp`) is written there, beside the schema's. They agree for
every function here.
### How did you verify your code works?
- The 2,202 calls above, each row that differs classified by a script
that fails on one it does not know, on a debug build with ASAN and
`BUN_JSC_validateExceptionChecks=1`, against the same build of `main`.
Two runs of `main` agree with each other.
- Tests in each function's own file, for what changed and for what has
to stay: `unsafe.test.js`, `jit-policy.test.ts`, `sleepSync.test.ts`,
`mmap.test.js`, `which.test.ts`, `open-in-editor-gc.test.ts`,
`bunshell.test.ts`, `csrf.test.ts`, `dns-prefetch.test.ts`,
`semver.test.ts`. All pass here, as do `os.test.js`, `socket.test.ts`,
`tcp-server.test.ts`, `bun-serve-ssl.test.ts`, `index-of-line.test.ts`,
`test/internal/bindgen.test.ts` and the types test. (Locally one
differential test of `semver.test.ts` fails here and on `main`, against
a newer `node-semver` than the one `test/package.json` pins.)
- `test/internal/source-lints`,
`test/internal/build-codegen-declared-outputs.test.ts`, clippy on the
workspace, `tsc`, `bun run rust:mordant` at the revision CI pins, and
`cargo check -p bun_semver_jsc` without `BUN_CODEGEN_DIR`.
…odules (oven-sh#45477) ### What does this PR do? `bun install` works again in a project that pnpm installed and whose `node_modules` is a symlink. pnpm writes the commands in `node_modules/.bin` as shell script files. Since oven-sh#45307, a file or directory that has a command's name is kept when `node_modules` is a symlink, because what is behind the link need not be bun's. So every command of such a project failed the install: ``` error: failed to link dep: EEXIST ``` - A **file** at a command's name is replaced by the link again, as in 1.4.3. - A **directory** there is still kept and still fails the install. No package manager puts one there, and removing it would delete a tree that may lie outside the project. - The global bin directory is unchanged: it holds programs that aren't bun's. - Windows is unchanged: bun writes `<name>.exe` and `<name>.bunx` there, and pnpm writes neither. There is no extra system call: `unlink` fails on a directory, which tells the two cases apart. The file is only replaced when the link failed because the name is taken. After any other failure, the file stays and the original error is reported. ### How did you verify your code works? - The test of this case in `symlink-path-traversal.test.ts` is split in two, each for both linkers and for the `node_modules` of the project and of a workspace. A directory in the way is kept and fails the install, as before. A file gives way to the link, and every other file behind the symlink is untouched, including another file in `.bin`. - On a build of `main`, the 4 file cases fail with the error above and the 4 directory cases pass. - On CI's release build of `9ece36989`, all 8 cases pass. - All 113 install, migration, auto-install and patch test files on that build: the only failures are 12 tests that need network access that the sandbox they ran in doesn't have, and that fail there on `main` too. 6 tests of `bun-pm-diff.test.ts` hit their 5 s timeout on the loaded machine; run alone, that file passes 75 of 75 on this build and on `main`'s. - The `mordant` check fails on `JSPromiseStrong::from_value` in `src/jsc/JSPromise.rs`, which this PR doesn't touch: 4b1681a on `main` removed its only caller, and the check judges the PR merged with `main`.
…tar (oven-sh#45588) ### What does this PR do? Fixes `test/cli/install/bun-install-windows-rename-retry.test.ts`, which is red on the Windows 11 aarch64 lane on main and on most branches: the two "reinstalled project-local store entry" tests find 299 files in the package instead of 300, on every retry. The test is at fault, not `bun install`. Its `beforeAll` packed the fixture with the system's `tar -czf ... .quiet()`. On the failing machines `tar.exe` (bsdtar 3.8.8) prints `package/files/N.txt: Can't add archive to itself`, leaves that file out and still exits with 0, so the tarball has 299 files. That check in libarchive is wrong on Windows. It compares a 16-bit hash of the archive's file id (`getino` in `archive_windows.c`) with each file's whole NTFS record number (`bhfi_ino` in `archive_read_disk_windows.c`), so an unrelated file can be taken for the archive. The current Windows aarch64 image has two ranges of free records that are 2 * 65536 apart, which makes the two numbers meet. The measurements are in oven-sh#45579. The fixture is now packed in memory with `Bun.Archive`, as other tests in `test/cli/install` do. The temp directory and the `afterAll` that only served `tar` are gone. Bun's own packing never sets that check. ### How did you verify your code works? The tests only run on Windows, so CI's Windows lanes are the check. Locally (Linux) the file loads with its 8 tests skipped, and the same `Bun.Archive` call gives a gzip tarball in which both Bun and GNU tar find all 303 entries, 300 of them under `package/files/`.
…#45489) This PR moves the Bake code into folders by job. It changes no behavior. It is the first step before a redesign of the Bake API, so that the code is easier to read first. The problem - `DevServer.rs` was 6,200 lines. It held the request handlers, rendering, bundling, failure reporting, file watching and the server-side HMR FFI in one file. - The config code was split between `mod.rs` and `bake_body.rs` for no reason other than history. `bake_body.rs` also held router and HMR-runtime code. - `FrameworkRouter.rs` (2,000 lines) held the router, the pattern encoding, the Next.js naming rules and the JS test class. The new layout of `src/runtime/bake/` - `config/`: the one config type that both `bun index.html` and an `app` export produce. `framework.rs`, `bundler.rs`, `transpiler.rs`, and `defaults.rs` for `Framework::auto` and the built-in React preset. - `router/`: the file system router. `pattern.rs`, `style.rs`, `tiny_log.rs`, `js.rs`. - `production/`: `bun build --app`, from `production.rs`. `ffi.rs`, `entry_points.rs`, `per_thread.rs`. (A folder named `build` is ignored by `src/.gitignore`.) - `dev_server/`: `DevServer.rs` and the old `dev_server/mod.rs`, one file per job: `lifecycle.rs`, `http.rs`, `ensure_route.rs`, `render.rs`, `bundle.rs`, `finalize.rs`, `failures.rs`, `trace.rs`, `deferred_request.rs`, `router_glue.rs`, `protocol.rs`, `server_hmr_ffi.rs`, `hot_reload_event.rs`, `watcher_atomics.rs`, `directory_watch_store.rs`, `bundler_vtable.rs`, `watch.rs`. - `hmr_runtime.rs`: the embedded HMR runtime code. How to read it Each commit moves one area and builds on its own: router, config, production, `dev_server/mod.rs`, then `DevServer.rs`. Use `git diff -M -C --color-moved=dimmed-zebra` per commit. For every commit I compared the old and new files line by line, ignoring blank lines, `use` lines and visibility keywords. The only other lines that differ are: - module declarations and re-exports; - paths that pointed at the old modules (`framework_router::` is now `router::`, `bake_body::` is now `bake::`, the inline `mod c` is now `server_hmr_ffi`); - `pub(super)` or `pub(in crate::bake)` on items that a sibling file now uses. Each one keeps the visibility the item had before the move; - shim re-exports and comments that only existed to join the old split files. Small changes that are not a pure move - `ASSET_PREFIX` and `CLIENT_PREFIX` were defined twice with the same values. One copy is left. - `bake::jsc` no longer re-exports all of `crate::jsc`. Nothing used it after the merge. - `#![allow(unexpected_cfgs)]` from `bake_body.rs` moves to the two files that use `runtime_embed_file!`. Paths outside `src/runtime/bake` that this updates - `src/codegen/generate-js2native.ts`: `FrameworkRouter.rs` now maps to `runtime/bake/router/js.rs`. - `src/codegen/bake-codegen.ts` and `scripts/glob-sources.ts`: the protocol enums are read from `dev_server/protocol.rs`. - `mordant-baseline.toml` and `test/internal/source-lints/vm-thread-door.inventory.json`: three keys move to the new file names. Tests On a local debug build, these pass: `hmr-socket-protocol`, `deinitialization`, `framework-router`, `app-options`, and the dev server suites `hot`, `html`, `css`, `bundle`, `esm`, `sourcemap`, `plugins`, `vfile`, `import-meta-inline`, `incremental-graph-edge-deletion`, and the production tests that do not need React. The tests that install React could not install it on my machine, so CI is their first run. Open bake PRs that edit `DevServer.rs`, `bake_body.rs`, `production.rs` or `FrameworkRouter.rs` will conflict with this one. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…-sh#45605) This PR fixes three bugs in the Bake config and router. Review bots found them on oven-sh#45489. They were in the code before that PR. The bugs 1. A rejected `app` config leaked the bundler plugin. `Framework::from_js` creates the plugin when the framework has `plugins`. If a later step failed (for example `root: 123`), `UserOptions::from_js` returned the error, and nothing unprotected the plugin's JS object. Only the `Drop` of `UserOptions` freed it, and that value was never built. 2. `fileSystemRouterTypes[].ignoreDirs` was ignored. The parser walked the outer `fileSystemRouterTypes` array instead of the `ignoreDirs` array. When a framework set `ignoreDirs`, the router got one `"[object Object]"` entry per router type in its place, so it scanned every directory, including `.git` and `node_modules`. 3. A route with more than 64 params crashed the dev server. The router wrote the error message but set no cursor, so `TinyLog::print` sliced the path from index `u32::MAX`. The fixes 1. `SplitBundlerOptions::destroy_plugin` frees the plugin. The `Drop` of `UserOptions` calls it, and `UserOptions::from_js` also calls it when parsing fails after the plugin exists. 2. The parser reads `ignoreDirs` from its own array. The error message no longer says that `"*"` is allowed, because it is not. 3. The error highlights the full route path. Tests `test/bake/app-options.test.ts` has one test for each bug. All three fail on Bun 1.4.3: - the plugin test counts 10 leaked `BundlerPlugin` objects instead of 0; - the `ignoreDirs` test reports the invalid route in the ignored directory; - the 64-params test crashes with `range start index 4294967295 out of range for slice of length 389`. On a local debug build, `app-options`, `framework-router`, `dev/bundle` and `dev/html` pass.
… has been read (oven-sh#45593) ### What does this PR do? `Bun.YAML.parse` names the property for a key with `ToString`. A YAML key can be a sequence or a mapping (`? [a, b]`), and `ToString` of an array or an object is JavaScript that a program can replace: `Array.prototype.toString`, `join`, `Object.prototype.toString`, `Symbol.toPrimitive`, `Symbol.toStringTag`. It ran halfway through turning the tree into values. The tree points into the input. When the input is a typed array or an `ArrayBuffer`, that code can write to the buffer, or transfer it and let it be collected, and the rest of the values are then read from memory that has changed or been freed. With a replaced `Array.prototype.toString` that frees it, a document of 201 keys comes back with 4. 1.4.3 does the same. `ToString` now runs once the tree is gone. While the tree is turned into values, a property whose key is a sequence or a mapping is set aside, with the properties that follow it in the same mapping so that they stay in the order they are written in. When the parse has returned, and with it the borrow of the input, the tree and the arena, those keys are named and the properties defined. What names a key, and what a program sees of it, is as before. The values are rooted by a `MarkedArgumentBuffer` around the whole call, through the closure form `stringify` uses, so `ParserCtx::run` and the two fields that carried its argument and its result are gone. #### Behaviour Compared with `main` on 2,833 calls: the 2,375 of the five parsers' corpus, and 458 for this change. Those are 26 keys (17 that are sequences or mappings, with aliases to themselves and to each other, and 9 scalars) in four positions as a string, a buffer and a `Blob`; keys nested 10, 1,000 and 100,000 deep; eight replaced methods that return a name, throw, overwrite the input, or free it; documents with several such keys, where the order of the properties and of the calls is recorded; and a replaced method that calls `YAML.parse` again. 2,811 rows are the same, among them every row where the method returns a name or throws, and the order of the properties and of the calls. | | Rows | | --- | --- | | bug → bug | 0 | | bug → non-bug | 22 | | non-bug → bug | 0 | | non-bug → non-bug | 0 | | Call | `main` | Now | Rows | | --- | --- | --- | --- | | `YAML.parse(buffer)` with such a key, and a replaced method that overwrites `buffer` | 3 keys of 41, named by what was written | the document | 13 | | the same, and the method transfers `buffer`, collects, and allocates again | 3 keys of 41, or by luck the document: two runs of `main` disagree on 6 of these rows, and on no other | the document | 9 | A script checks that in each of the 22 the method is called on both, and that the result now is what it is when the method only returns a name. ### How did you verify your code works? - `yaml.test.ts`: with an `Array.prototype.toString` that overwrites the input, it is called once, the key is named what it returns, and every value before and after is whole. It fails on `main` and on 1.4.3. Also: the properties of mappings with such keys, duplicates among them, are in the order they are written in, and each of 17 keys is named as `String()` names the same node parsed on its own. - These also pass with `BUN_JSC_slowPathAllocsBetweenGCs=3` and zombie mode, and with `BUN_JSC_collectContinuously=1`. - `yaml.test.ts`, `yaml-test-suite.test.ts`, `yaml-block-scalar-matrix.test.ts` and `resolve/yaml/yaml.test.js` pass, on a debug build with ASAN and `BUN_JSC_validateExceptionChecks=1`.
oven-sh#45616) ### What does this PR do? Fixes a regression from oven-sh#45307: `bun install --frozen-lockfile` refused the `bun.lock` that `bun pm migrate` had just written from a yarn (berry) lockfile, when `package.json` has a resolution scoped to a range. ```json { "dependencies": { "x": "^1.1.0", "p": "^1.0.0" }, "resolutions": { "x@npm:^1.0.0": "1.0.0" } } ``` Here `p` depends on `x@^1.0.0`. yarn applies such a key only to a dependency declared with exactly that range, so `p` gets `x@1.0.0`, the root's `^1.1.0` isn't touched, and yarn.lock has `x@1.1.0` for it. The migration takes both over. Since oven-sh#45307 every install checks the lockfile against the rules in `overrides` or `resolutions`. That check goes by whether the two ranges overlap, which is npm's and pnpm's rule for `overrides`. `^1.1.0` overlaps with `^1.0.0`, so: ``` error: lockfile had changes, but lockfile is frozen note: bun.lock has x@1.1.0, resolutions in package.json wants 1.0.0 ``` A plain `bun install` moved the root's `x` to 1.0.0, below the range it declares. For rules that come from `resolutions`, the check now also accepts a row that is right by yarn's reading. It looks the rule up with the same code as the resolver, but with the key's range and the declared range compared as text (`npm:^1.0.0` and `^1.0.0` are the same). The row has to fit that rule, or its own range if yarn applies no rule to it. So both lockfiles pass: one migrated from yarn.lock, and one that Bun resolved itself. The same goes for two keys that overlap, such as `x@npm:^1.0.0` and `x@npm:^1.0.1`. Bun picks the key whose text sorts first, yarn the key with the declared range, and `main` refuses that migrated lockfile too. So does a dependency declared as `*` next to a ranged key, when `latest` is a prerelease. When a frozen install does fail, the note names the version that yarn's rule wants. A rule whose value is `catalog:` is compared by the version in the catalog, as for `overrides`. Not changed: - `overrides`: the check is as strict as before. - What a fresh install resolves. Without a lockfile, Bun still applies a ranged key in either field to every range that overlaps with it, as 1.4.3 does. That differs from yarn for `resolutions`, and changing it would be a change in behavior of its own. The new code is on a path that only runs when some row looks out of line, and it reads `package.json` from the cache. ### How did you verify your code works? - A new test in `yarn-berry.test.ts` migrates the project above, checks that `bun.lock` has both versions, that a frozen install passes, and that a plain install leaves `bun.lock` as it is. On a build of `main` it fails with the error above. - Three more tests put a version in `bun.lock` that the key for the row's own range forbids, and expect a frozen install to fail with the whole message: next to a second key that overlaps, and for a dependency declared as `npm:^1.0.0` and as `^1.0.0`. The first also checks that the migrated lockfile passes, which fails on `main`. - One test migrates a `*` dependency whose `latest` is a prerelease. It passes before oven-sh#45307 and fails on `main`. - One test runs the same project with the rule in `overrides` and in `resolutions`. In `overrides` a frozen install still fails and a plain install moves the version. In `resolutions` the frozen install passes and the version stays. It fails if the check stops telling the two fields apart. - A script with 9 shapes of a range-scoped resolution (direct, transitive, both, next to another major, in a workspace, with and without `npm:`, another range of the same major as a direct and as a transitive dependency). Before oven-sh#45307 all pass. On `main`, the two with another range of the same major fail. - On CI's release build of `5405a15ff`, the last commit that changes code: the 7 new tests pass and all 9 shapes pass. All 113 install, migration, auto-install and patch test files: 13,411 pass. The 12 that fail need network access that the sandbox they ran in doesn't have, and fail there on `main` too. Two differences from yarn are left as they are on `main`, because each needs a key that has no effect in yarn: a key with a ranged parent (`parent@npm:^1.0.0/x`), which yarn never matches, and a key written after another key that matches the same dependency, which yarn never reaches, because it takes the first match in the file.
…n native transports (consolidates the open WebSocket PRs) (oven-sh#45223) ### What does this PR do? This PR replaces the open WebSocket pull requests (listed in the table at the end) with one change. It covers the global `WebSocket` client, `Bun.serve({ websocket })`, the built-in `ws` package, the parts of `node:http` that `ws` depends on, undici's WebSocket exports, and the types, docs and tests for all of them. I treated every open PR as a bug report. For each one I reproduced the bug on main, ran the same script on Node 26.3 (undici) and on npm `ws`, checked the RFCs, and then wrote the fix the way the code should have been structured in the first place. Often that meant deleting a mechanism that caused several bugs at once. Many fixes below had no PR. They came out of those comparisons and are marked "(no PR)". Size: native and builtin source outside `ws` is +5,893 / -5,374. `ws.js` (1,744 lines) becomes `ws.ts` (3,399 lines, about 450 of them types). Tests are +22,739 / -1,786. How I decided close calls: - `require("ws")` behaves like npm `ws` on Node. - The global `WebSocket` behaves like the WHATWG spec and Node's global. Bun's extras stay. - `Bun.serve` follows RFC 6455 / 7692 on the wire and does what its docs and types say. ### Behavior changes to look at first These are the changes I expect users to notice. They're listed so that any of them can be vetoed. **Global `WebSocket`** 1. **Real closing handshake.** Before, `close` fired about 0.2 ms after the `close()` call with our own code and `wasClean: true`, no matter what happened next. Now `close()` sends the Close frame, sets `readyState` to CLOSING and keeps the process alive. Once both Close frames are exchanged, the client closes its side (FIN, close_notify), and `close` fires **when the connection closes**, as in Node (undici and npm `ws`) and browsers. The event has the **server's** code and reason (1005 if its frame was empty) and `wasClean: true`. - If the server echoes the Close frame but never closes the connection, the client drops it at the timeout. The close is still clean and still has the server's code. - If the connection ends before the server's Close frame, or that frame doesn't arrive within `BUN_CONFIG_WS_CLOSE_TIMEOUT`, the result is `1006`, not clean. The timeout is 30 s, in practice 32 to 36 s because uSockets sweeps every 4 s. - Cost: a script that ends in `ws.close()` exits one round trip later. If the server never answers, the process stays alive until the timeout. `terminate()` and `process.exit()` are the ways out. npm `ws` also waits 30 s; undici waits forever. - Data queued before `close()` still goes out first. The timeout starts over whenever more of it can be written, so a slow upload isn't cut off (100 MB at 2 MB/s arrives whole, also through a proxy), and a peer that stops reading is dropped one timeout after its last read. Progress means the kernel reports the socket writable: on loopback with the default 30 s, a reader at 20 KB/s counted as stalled and one at 50 KB/s didn't. main waits forever. - Two tests from #44618 pinned the old model on purpose. I rewrote them. 2. **`close()` without a code sends an empty Close frame**, and both ends report 1005, as in browsers, Node and npm `ws`. A `Bun.serve` `close(ws, code)` handler now sees 1005 when a Bun client calls `close()`. When the server closes first, the client echoes what it received: 1001 stays 1001, empty stays empty. 3. **One failure path.** Whenever the client fails the connection (bad handshake, TLS, proxy, protocol error, message too big, invalid UTF-8, bad close code), it fires `error`, then `close` with **1006**, reason `""`, not clean. Before, `close` had 1002 / 1007 / 1009 / 1011 / 1015 with an English reason, sometimes `wasClean: true`, and often no `error` fired. The text is now in `ErrorEvent.message`, and `ErrorEvent.error.code` is new (`ECONNREFUSED`, `ENOTFOUND`, `DEPTH_ZERO_SELF_SIGNED_CERT`, `ERR_TLS_CERT_ALTNAME_INVALID`, `WS_ERR_INVALID_UTF8`, ...). If the failure happens after open, the client sends a Close frame with 1002 / 1007 / 1009 first. Losing the TCP connection after open isn't a failure: only `close` 1006 fires, with reason `""` (was "Connection ended"). 4. **The 101 response is validated strictly.** The connection fails if the server picks an extension or permessage-deflate parameter that wasn't offered or is invalid (RFC 6455 9.1, RFC 7692 7.1). That includes a server that echoes `client_max_window_bits` without a value. A response head over `--max-http-header-size` + 1 KB now fails no matter how TCP splits it into reads. Before, a head of about 17 KB or more got through if it arrived in one read. 5. **Header values are sent one byte per code unit**, like `fetch` and Node: "café" is `63 61 66 e9` (was UTF-8). `Host` / `Proxy-Connection` in `proxy.headers` replace Bun's own line in the CONNECT request. `fetch` shares that writer, so this changes `fetch` too. 6. `terminate()`, a failed connection and the close timeout now close with a FIN instead of a RST. The kernel still sends a RST if data is unread. The client answers a peer's FIN with a FIN. 7. Messages that arrive together with the 101 are delivered after `open`'s microtasks, as in browsers and undici. npm `ws` has the old order, so `ws` on Bun now differs from it there, in the harmless direction. 8. Smaller changes: - An invalid `binaryType` is ignored (used to throw). - `ws.url` is `ws:` / `wss:` when the input was `http:` / `https:`. - `MessageEvent.origin` is the origin, not the URL with its password. - `bufferedAmount` no longer drops to 0 at close. - `send(Bun.file())` throws (used to send an empty frame). - A ping or pong that nothing listens for is dropped. It used to be queued for a later listener, so a flood of them grew memory to gigabytes. Automatic pongs are unaffected. - Interim 1xx responses count toward the response head's size limit (`--max-http-header-size`). An endless stream of `100 Continue` used to freeze the event loop. **`Bun.serve({ websocket })`** 9. **`websocket.error` is called as `(error, ws)`** and is now typed. It used to be called as `(error)`, so existing handlers keep working and new ones can tell which socket failed. The docs said `(ws, error)`, which never matched the implementation. They now say `(error, ws)`. 10. **`server.stop(true)` closes each WebSocket with 1001 "Server closed"**, including through `using server`. The client and the `close` handler used to see 1006. `stop()` is unchanged. 11. A long `ws.close(code, reason)` reason is cut at 123 bytes on a character boundary. It used to be cut mid-character, so the client saw 1007. Every numeric code is still accepted and sent as before. 12. **`server.upgrade()`**: - reads the handshake from `req.headers` only; - returns `false` once a `Response` has been accepted for the request; - throws for a `Request` that another server received; - ignores `Upgrade`, `Connection`, `Sec-WebSocket-Accept / -Key / -Version / -Extensions`, `Content-Length` and `Transfer-Encoding` in `options.headers`; - treats an empty `Sec-WebSocket-Protocol` in `options.headers` as declining the subprotocol. - Related: an HTTP/1.1 Upgrade or CONNECT request without `Host` gets a 400 (node:http is unchanged). 13. **The configured compressor is actually used.** Every `compress: "dedicated"` / `"4KB"`...`"256KB"` setting ran as `"3KB"`. Each now uses the window its name says, which costs memory. Per connection after traffic: `"dedicated"` 149 → 282 KB, `"128KB"` → 218, `"64KB"` → 186, `"16KB"` → 154. Off, `true` and `"shared"` stay at 2 KB. `"3KB"` is what every setting used to run as. The libdeflate compress fast path is removed (because of the Safari bug described below; numbers are under Performance). `x-webkit-deflate-frame` (Safari 14 and older) is no longer accepted, so those clients run uncompressed. A dedicated compressor accepts and echoes an offer of `server_max_window_bits=8`, as npm `ws` does. The shared compressor (the default) is always 15 bits, so it declines a smaller window, and so does any offer that also has `server_no_context_takeover`. Those connections run uncompressed. Released Bun answered it with 9, which the RFC doesn't allow. (zlib deflates with 9 bits, and that output always inflates with an 8-bit window.) 14. When the server fails a connection, the client now gets a Close frame with 1002 / 1007 / 1009 and the reason. The `close` handler still gets `1006`, but its reason is never empty now. The socket then stays open until the peer's FIN arrives, or for 4 to 16 s. 15. `idleTimeout: 0` arms no timer, so there are no automatic pings either. Those connections used to die at about 256 s. 16. **`maxPayloadLength` no longer applies to Ping, Pong and Close frames.** They have their own 125-byte limit. 17. The error messages of `ServerWebSocket` methods are now consistent with each other. A closed socket still ignores the call, whatever the arguments are. 18. A throw after `server.upgrade()` is no longer swallowed. It goes to `error()` or stderr, the server keeps serving, a surrounding `bun test` fails, and the process exits with code 1 when it ends naturally, same as for any printed `fetch` error. This holds for a sync `fetch`, and for an async one that was really suspended (a timer, I/O) before it called `upgrade()`. If an async `fetch` upgrades before its first real suspension and then rejects, that is still an unhandled rejection and ends the process, as on main. 19. After `server.reload()`, the new `websocket` options also apply to sockets that are already open. The handlers already did. **`ws`** 20. `'error'` gets an `Error` (was an `ErrorEvent`) and is **fatal without a listener**, as in Node. A failed connection attempt, or on a server a hostile frame, ends a process that has no `'error'` listener. 21. `'close'` is `(code, reason: Buffer)` (was a string plus a third argument). It fires when the connection closes, or at `closeTimeout`. Failures are 1006: `'error'` fires immediately and `'close'` when the connection closes, as in npm. (The global `WebSocket` fires both immediately, as undici does.) `close()` without a code gives 1005. `'unexpected-response'` is not followed by `'close'`. 22. A text message is a `Buffer` for every `binaryType`. 23. `ws` servers have no idle timeout, no automatic pings and no backpressure limit. They used to inherit 120 s, a ping at 104 s and 16 MiB from `Bun.serve`. - Automatic Pongs are the one thing still capped at 16 MiB, because the peer controls how many there are. They're skipped while that much is buffered for the socket (RFC 6455 5.5.3; npm queues them all). The cap applies on every transport and also to the clients (`ws` and the global `WebSocket`), which had no cap on main. - The default `maxPayload` is 100 MiB (was 16). - uWS no longer preallocates the whole length a frame header announces before the bytes arrive. It preallocates at most 16 MiB. This also applies to `Bun.serve` with a large `maxPayloadLength`. 24. `WebSocketServer({ perMessageDeflate })` actually negotiates permessage-deflate, and `ws.extensions` is a string. The uWS workaround for Safari 15.0-15.3 no longer applies to `ws` on node:http (npm `ws` has none). `Bun.serve` keeps it. 25. **With an `HttpsProxyAgent`, each hop is verified with its own TLS options**, as on Node. The agent's options (`rejectUnauthorized`, `ca`, ...) apply only to the connection to the proxy, and the WebSocket's options only to the target. Before, both sets were merged and used for both hops, so `new HttpsProxyAgent(url, { rejectUnauthorized: false })`, meant for a self-signed proxy, also turned off certificate checks for the **target**. Code that relied on that has to pass the option to `new WebSocket(...)` too. - All 42 combinations I measured on Node with the real `https-proxy-agent` (14 sets of options, for http→wss, https→ws and https→wss) now give the same result. - The native client gets a new option for this, `proxy: { url, headers, tls }`, also on the global `WebSocket`. With `proxy.tls` (even `{}`), the proxy hop uses only `proxy.tls` and the target only `tls`. Without it, `tls` applies to both hops as before. `serverName` and `checkServerIdentity` are never shared: `tls`'s are for the target and `proxy.tls`'s for the proxy. Proxy agents stay on the native client. - `proxy.tls: true` means `{}`, falsy values mean "not set", and any other non-object throws. `pfx` works in it. - An agent that has neither `connectOpts` nor `proxyRequestOptions` gets no `proxy.tls`, so it behaves as on main. With credentials in an agent's proxy URL, they win over a `Proxy-Authorization` header, as in `https-proxy-agent`. - `'error'` gets the value `checkServerIdentity` returned, as in Node. It used to get a generic TLS error. If the proxy can't be reached, `'error'` names the proxy, not the target. 26. Smaller changes: - A client stays CONNECTING when an `'upgrade'` listener throws, as in npm. It used to emit `'open'`. - A client no longer fails on a `Sec-WebSocket-Version` response header, a repeated `permessage-deflate` entry, or window bits written like `015` or `0xf`, all of which npm `ws` accepts. - `send()` while CONNECTING throws. - `ws.url` is `undefined` on server sockets. - `on*` / `addEventListener` are built on the emitter, as in npm: they use ws's own event classes (not `instanceof Event`), `addEventListener("upgrade")` does nothing, and the `signal` option is ignored. - A server socket is `instanceof WebSocket`. - After `'open'`, only protocol errors are `'error'` events. A reset, a broken TLS record or a failed renegotiation is just `'close'` 1006, as in npm and on main. **Differences from npm `ws` that I kept** (veto any): - An invalid close code throws and leaves the socket OPEN. That matches ws 8.22.0. 8.18.3, the version in `test/node_modules`, leaves it CLOSING forever. - A header whose value is an array is sent as one line joined with ", ". npm sends one line per element. Both join `cookie` with "; ". - `mask: false` / `generateMask` are ignored, `binaryType: "fragments"` gives one fragment, and the `Sender` / `Receiver` classes throw. - After a native upgrade, `req.socket.pause()` doesn't stop reads (`ws.pause()` does), and `req.socket` emits no `'end'`. - `terminate()`: on a server, `'close'` fires before the callbacks of sends made after `terminate()`. On a natively connected client, data that is already buffered is delivered during `terminate()`, not after it returns. `terminate()` in a `'message'` listener drops the rest of the packet being parsed. - A `'message'` listener that throws doesn't stop later messages (npm's receiver stalls). `addEventListener("ping" | "pong")` works. - A Close reason that isn't valid UTF-8 is sent with U+FFFD (npm sends the raw bytes). If that makes it longer than 123 bytes, `close()` throws `RangeError` and the socket stays OPEN. - After the body of a non-101 response, the client closes the connection (Node keeps it for its agent). - `send()` of a file-backed `Blob` throws (npm reads it asynchronously). - Pings aren't answered while more than 16 MiB is buffered for the socket (npm queues a Pong for each, with no limit). - A frame that breaks two rules at once (say RSV2 set and unmasked) is rejected with Close 1002 like in npm, but `'error'` may name the other broken rule. Frames that break one rule get npm's exact error. Released Bun emits no `'error'` and sends no Close frame for any of them. - On a paused, natively connected client, bytes that arrive after `pause()` stay in the kernel's buffer, so `terminate()` and the close timer drop them. Node has already read them and delivers them. - A paused native socket (client or server) doesn't read, so it doesn't see the peer's FIN until `resume()` or `terminate()`. That includes a `createWebSocketStream()` whose consumer has stalled. Node keeps reading up to 64 KiB while paused, so it sees a FIN that follows little or no data. - `req.destroy()`, `req.abort()` or an aborted `signal` inside an `'upgrade'` listener of a natively connected client fails the handshake: `'error'`, then `'close'` 1006. Node emits `'open'`, then `'close'` 1006. I tried Node's order and took it out again: it leaked the connection when a listener threw. - `req.setTimeout()` does nothing on a natively connected client, so an `'unexpected-response'` listener can't change or turn off `handshakeTimeout`, which stays armed until the body ends. Until the response head is complete, `handshakeTimeout` is a fixed deadline that interim responses don't restart. - On a server, send callbacks that are still waiting when the closing handshake finishes run when the connection closes, right before `'close'`, not at the moment of the flush: with no error if the kernel took everything, else with `ERR_STREAM_DESTROYED`. `readyState` is already CLOSED there. On TLS, a drain can report success while uSockets still holds the last record. - If a send callback throws, the ones queued behind it still run, each once. Node drops some of them, depending on how the writes were batched. - `ws.send()` followed by `ws._socket.destroy()` in a `'message'` or `'ping'` listener of a server drops that message. `terminate()` and `close()` deliver it. - Through a proxy, the request passed to `finishRequest` emits no `'socket'` event before `end()`. The native client connects with the final headers, so there is no socket yet. **Not taken**: - #35031 (WHATWG-only close codes): breaks `close(1001)` / `close(1011)`, which work today, with no gain for portable code. - #41712 (env proxy by default): a product decision, stays open. - #36749 (serialize async `message` handlers). - #39255 (keep the ping deadline across sends): TCP already notices a dead peer, and it could drop a slow one that is alive. - #44122, #20533, #37559: refactors. - #35730, #36561: speed-ups of tests that CI skips. - #31205: can't be reproduced. Also not here: - `closeOnBackpressureLimit` over TLS still takes about 12 s to close. It's an old bug, and the fix changes EOF routing in the TLS engine. - `ws.remoteAddress` can still be `undefined` when it's first read inside `close()`. Filling it in eagerly costs a `getpeername` per connection. The docs say to read it while the socket is open; the value is kept after that. - Throws inside `websocket.*` handlers are routed the way they are today. - The `ws` client option `secureContext` is honored on direct connections but still ignored when connecting through a proxy (same as before this PR). ### Changes #### Global `WebSocket` (client) The connected client now tracks two flags (our Close frame was sent, the peer's Close frame was received) and has one exit, `finish()`. The C++ side has one `finish()`, one constructor path and one `sendFrame()`. That replaces six dispatch sites, three copies of the failure block, a 130-line switch and 14 `send` / `ping` / `pong` overloads. Crashes, hangs, lost data: - `send(); send(); close()` lost data. On `ws://` the socket was shut down in both directions right after the Close frame, so anything the server sent next triggered a RST that also destroyed unsent bytes. A plain echo server got 10 of 32 small messages (2 of 32 at 64 KiB) while the client reported `1000`, clean, and 50 ordinary closes caused about 50 RSTs. Now nothing is shut down after the Close frame, and the FIN is sent once both Close frames are exchanged: 32 of 32 in each case, including when the script just ends. (#45196) - `close()` with unsent data queued and a peer that stopped reading never finished, and `terminate()` was a no-op in CLOSING, so the process hung forever. The timeout is now armed when the Close frame is queued. `terminate()` works in every state except CLOSED and reports a close code that was already received. (#36112; the timeout: no PR) - A server's Close frame that arrived under send backpressure left `readyState` OPEN, and later `send()`s were queued behind our own Close echo. (#36112) - A send queue whose tail landed on the end of its ring buffer never flushed the wrapped part (`bufferedAmount` stuck at 7). (#44821) Growing such a queue blocked `send()` for 2.9 / 11.4 / 43.6 s at 16 / 32 / 64 MB. `LinearFifo::realign` is now one `rotate_left`. (no PR in this set; #44929 touches the same function) - `wss://` through a proxy: Close and Pong frames overtook queued messages (`BIN x185, CLOSE, BIN x71`), so 71 of 256 messages were lost behind a "clean" close. (#44822) - A masked Ping, Close or continuation frame from a server was parsed with the mask key as payload, and the connection continued out of sync. Any masked frame now fails the connection. (no PR) - `permessage-deflate; client_max_window_bits=8` was accepted, and then the first compressed message failed with `1011 "Compression not implemented yet"`. (#38632) - `send()` / `ping()` / `pong()` of `Bun.file()` or an S3 file silently sent an empty frame. They now throw `ServerWebSocket`'s error, in every state. (#39082) - A detached buffer hit `ASSERTION FAILED: m_ptr` on debug builds. It's now an empty payload, as in WebIDL, `ServerWebSocket`, `Blob` and `Response`. (#35029, without its TypeError) - `new WebSocket(url, ["ÿ".repeat(357913942)])` aborted the process. (#42216, reduced to the one call site that expands its input) - Deleted `EncodedSlice::utf16_global`, a safe fn that marked a borrowed slice as "C++ frees this". Text frames now reach C++ as an owned string. Fixes #31968. (#31982) - `pause(); close()` kept the fd open (on TLS the server's too) until the timeout. Closing now resumes reading. `ws` is the exception: there the socket stays paused, as in npm. (#42974) Events and state: - Every failed connection fires `error`, then `close` 1006. Subprotocol mismatches used to give `close 1002 ... wasClean: true` and no `error`. (#45122) Failures after open do the same and send a Close frame first. Before, 22 kinds of bad frame reset the TCP connection and reported 1002 / 1007 / 1009 / 1011 without `error`. RSV1 without the extension was `1011`, "clean". An illegal close code from the server was `1002`, "clean". Corrupt deflate data was 1002 and is now 1007. (no PR) - Errors have `error.code`, and the message includes the detail: `Expected 101 status code: 503`, `TLS handshake failed: self signed certificate`, `Proxy connection failed: 403 Forbidden`. (no PR) - A connect failure detected synchronously (a missing unix socket) left `readyState` at 3 right after the constructor and put the whole sentence, URL included, in `close.reason`. It now looks like an asynchronous failure and reports `ENOENT`. (no PR) - The automatic Pong is sent before the `ping` event fires and never after our Close frame. Before, `close()` in a `ping` listener on `wss://` sent an empty Pong after the Close. (no PR) - Events run in the `AsyncLocalStorage` context of the `new WebSocket()` call, as in Node. Sockets created inside a `Bun.ModuleGraph` get their store too. The snapshot leaves out the graph's own frames, so an open socket doesn't keep its graph alive. (#33066) - Frames that arrive with the 101 wait for `resume()` if the socket was paused in `open` or earlier (#43018), and are delivered after `open`'s microtasks (#43696). This takes one field, with no native microtask and no special cases for nested event loops. - `isPaused` is false, and `pause()` / `resume()` return false, unless the socket is CONNECTING or OPEN. (#42978) - After the connection closes, `bufferedAmount` keeps the bytes that were never sent, and then grows only by the payload bytes of later `send()` calls. It used to count framing and pings and skip Blobs. This is what #31760 (already closed) asked for. (#31761, as one snapshot in the single exit, with no FFI override) - A length of 0 encoded in 16 or 64 bits is accepted, like every other non-minimal length (undici and npm `ws` accept them too). A length of 2^53 or more has its own error. (no PR) Constructor and arguments: - Options are read through the prototype chain, stopping at `Object.prototype`. Before, inherited keys and getters were ignored, while `Object.prototype.perMessageDeflate = false` turned compression off everywhere. (#41674) - The documented `protocols: "chat"` threw "Value is not a sequence", and `protocol: "chat"` was dead code. (#41705) A second argument such as `null` or `5` is converted per WebIDL (was `TypeError: Type error`). `['a', 'A']` is a duplicate. (no PR) - One invalid `proxy.headers` entry dropped all of them, including `Proxy-Authorization`. It now throws. (#37638) - `ws://:token@host` sent no credentials, and `user:p%E9ss` sent the wrong password. (#44964) An explicit `Proxy-Authorization` header beats the proxy URL's credentials, matching `fetch` and the target (non-proxy) path. #44964 pinned the opposite. (no PR) - The URL's credentials are kept out of `ErrorEvent.message` and the `SyntaxError`s. (#41698, using `URL::removeCredentials()`; its `fetch` half landed in #42692) - `ping(undefined)` sent "undefined". (#39071) `ping(null)` sends no payload either, (no PR) Opening handshake: - A proxy's 407 was reported as "Proxy connection failed" whenever it arrived in one read, and `HTTP/1.1 200` without a reason phrase was rejected. (#43445) Any 2xx opens the tunnel, as in `fetch`. (#43511) - A 101 with more than 128 header fields failed with "Invalid response". `server.upgrade(req, { headers })` with 125 headers triggers it. (#43243, bounded by the byte limit instead of a new constant) The byte limit gave different results for different TCP segmentation. (#41573, WebSocket half) - `100 Continue` / `103 Early Hints` before the 101, or before the proxy's 200, are skipped. (no PR) - Bytes read while a handshake listener spins the event loop were parsed as a new HTTP head. (#43763, the buffer simplification only) - Header values are sent as latin-1. (#41482) `close()` / `terminate()` while CONNECTING close TCP with a FIN. (no PR) - Found by fuzzing (no PR): the two floods in item 8, and a missing exception check where a binary message is queued because nothing listens yet. - A response of fewer than 9 bytes that can't be HTTP (`foo\r\n`) left the client waiting for the 120 s handshake timeout. It now fails immediately with "Invalid response". A RST instead of a response reports "Connection ended: read ECONNRESET". (no PR) #### `Bun.serve` WebSocket server Crashes and memory safety: - **`server.upgrade(req)` returned `true` after a streamed `Response` had started.** It wrote the handshake headers into the chunked body and then segfaulted the release build (heap-use-after-free at the next `ws.send()` under ASAN). A request can no longer be upgraded once a `Response` has been accepted for it. (no PR) - `ws.subscribe(new String(...))` with a `toPrimitive` that closes the socket caused a heap-use-after-free at the next `publish()`, and `send` / `ping` reported bytes sent on a dead socket. All arguments are now converted first, followed by one state check. Nine methods became four bodies, and `server.publish()` shares their payload helpers. (#39385) - Upgrading a socket that had already been ended (a `ws` `'headers'` listener that calls `req.socket.end()`) was a SEGV on debug and a hang on release. `HttpResponse::upgrade()` now rejects it up front, and `us_socket_adopt` returns NULL instead of the same pointer. The check runs again after the 101 is written, because a head longer than the cork buffer is a real TLS write, which can shut down the socket. `upgrade()` then returns `false`. (#37447, only this part; #44618 fixed its two other paths) - `ws.send(sharedMemory, true)` while another thread writes to that memory overran libdeflate's output buffer. That code path is deleted, and #42318's two race tests pass under ASAN. (#42318 stays open for `Bun.gzipSync` and friends) - `server.reload()` leaked a uWS context per route (12 KiB per reload with 12 routes). There's now one context per owner. A reload refreshes settings and never replaces a live context's closures. (#41680) Lost data, hangs: - A new TEXT / BINARY frame inside a fragmented message was appended to it, and both were delivered as one message. Fixes #33010. (#33011) - `ws.close()` while over `backpressureLimit` lost accepted frames and the Close frame (the peer got 62 of 79, then a bare FIN). `ws.close()` inside `ws.cork()`, `drain`, or `open` after an async upgrade sent its FIN 16 s late. `end()` is now the only writer of a Close frame and owns the uncork and the FIN. (#39843, plus TLS variants of its tests) - `terminate()`, a throwing `open`, `stop(true)` and process exit dropped data that `send()` / `publish()` had accepted in the same tick. (#43711, a third of it; no `flush` flag) A throwing `open` now always lets the 101 be sent first. - The reply to a valid frame was lost when a later frame in the same read failed the connection (Autobahn 3.2, 4.1.4, 5.15). (falls out of #35874) - `idleTimeout: 65536`, `maxPayloadLength: 2 ** 32 + 16` and `backpressureLimit: 2 ** 32 + 1` wrapped to 0, 16 and 1. `idleTimeout` above 960 now always throws, and the other two are clamped. Fixes #13918. (#36998; docs from #40097) - `idleTimeout: 0` meant about 256 s (`0 - 4` in an unsigned short). `static_assert`s now guard the arithmetic. Part of #26554. (#34222) Protocol and API: - When the server fails a connection, it sends a Close frame with 1002 / 1007 / 1009 and the reason. That includes an invalid Close payload, which used to be echoed as an empty Close. Fixes #22723. (#35874, but the handler keeps 1006, which is what npm `ws` and browsers report) - `stop(true)` sends 1001. Part of #25722. (#34961, the `stop(true)` half, in `TemplatedApp::close()`) - permessage-deflate: - The negotiator is new and does a plain split and compare. The old tokenizer compared the *sum of the characters*, so `deflate-permessage` matched. `server_no_context_takeover` or `server_max_window_bits` is honored and echoed, invalid offers are declined, and fallback offers are tried. (#41629) - `compress: false` actually turns compression off, and `decompress: "16KB"` no longer sets the *compressor*. `{ compress: on, decompress: off }` keeps working. (#41637, without its throw) - Inbound BFINAL=1 blocks are accepted. (#33592) - Corrupt data closes with 1007. A message that inflates past the limit closes with 1009. - Each dedicated compressor and each dedicated decompressor uses 4 KB less. - `perMessageDeflate: true` sent messages that end in a final DEFLATE block, which makes Safari close with 1002 (WebKit bug 326651). All outbound messages now end in a sync flush. Fixes #25673. I checked this on the wire; I couldn't run Safari. (no PR) - A long `ws.close()` reason is truncated to 123 bytes on a character boundary. It used to be cut mid-character, so the client saw 1007. (#41665) - `upgrade({ headers })` duplicated `Sec-WebSocket-Accept`, `Upgrade` and `Connection`, and removed a field from the caller's `Headers`. (#41875) `headers: {}` threw, and a subprotocol couldn't be declined. Fixes #18243. (no PR) The result depended on an unrelated `req.headers` read or `await`. (#41962) Another server's `Request` cross-wired handlers and topics. (#41811, `upgrade()` only) - The 101 for an upgrade made after an `await` was sent in 25 writes (33 TLS records). It's now one write. (no PR) - An Upgrade / CONNECT request without `Host` gets a 400. It used to reach `fetch` with a relative URL. (#41670) - `upgrade(); throw`, and a rejection from a handler that upgraded after an `await`, were silently dropped. (#45000, but reported through `error()` / the `fetch` reporter instead of ending the process) `async: upgrade(); throw` is still an `unhandledRejection`. - `websocket.error(error, ws)`, typed. Fixes #11634. (no PR in this set; #41728 touches the types) - New: `ws.pause()`, `ws.resume()`, `ws.isPaused`. The idle timer is suspended while paused. Fixes #32483. (instead of #36749) - `drain` only fires after a `send()` that returned `-1`. A TLS socket used to get one right after the upgrade with nothing buffered, and each `pause()` / `resume()` would have caused one. (no PR) - `ws.cork(cb)` runs `cb` on a closed socket. `server.publish(t, d, undefined)` compresses like `publish(t, d)`. (#39253) `websocket: {}`, the first example in the docs, no longer throws "expects a message handler". (no PR) #### `ws` `src/js/thirdparty/ws.js` (two classes that forwarded lazily to native objects) is replaced by `ws.ts`: one `class WebSocket extends EventEmitter` ported from ws 8.22.0 (`websocket.js`, `websocket-server.js`, `event-target.js`, `stream.js`, `extension.js`, `subprotocol.js`, `validation.js`, and the negotiation half of `permessage-deflate.js`). Framing, masking, UTF-8 validation and deflate stay native, so there is no JS `Sender` / `Receiver`. All nine JS-only PRs together moved the old shim from 18 to 24 of my 212 client cases. The state machine had to live in one place. There are three transports: 1. **Client**: the native client connects. In this mode it calls one shared handler table (`this` = the `ws` object) instead of dispatching DOM events, so there are no listener objects per socket and no `MessageEvent` per frame. 2. **Server**: upstream's code builds the 101 in JS, native code writes those bytes, and uWS adopts the node:http connection into a context that has this `WebSocketServer`'s options. `_socket` is the real node:http socket, and its `'close'` is what emits the `ws` `'close'`. There are no queues and no ticks in JS. 3. **Any JS stream** (the stream transport), both roles: the native client engine reads from and writes to a JS stream. It's the slower path (numbers under Performance), and only these cases use it: - Server: any `Duplex`, `server.emit("connection", socket)`, http2 `allowHTTP1`, fastify's `injectWS`, a manual `setSocket()`, and a node:http connection on which frames were read before `handleUpgrade()` ran. - Client: a custom `agent` (SOCKS agents), `createConnection`, `lookup`, `finishRequest`. These go through the real `http.request()`, which is upstream's own path. `family`, `localAddress` and `localPort` take that path too. Crashes, lost or corrupted data: - **One request from any client ended the process** with an uncaught `TypeError` when the connection wasn't native: `http2.createSecureServer({ allowHTTP1: true })` (`vite --https`), `server.emit("connection", socket)` (socket.io's sticky sessions), or any `Duplex`. They now get a 101 over the stream transport. (#44460: took its native binding and `can_upgrade()`, replaced its 500; #28152's Proxy case is a test) - `await events.once(ws, "open")` on a port that refuses connections killed the process, and `once("message")` + `on("message")` delivered each message twice. (#37972; there are no per-socket closures at all now) - Server: a message after `ws.onmessage = null` was an uncaught `TypeError`, `addEventListener("message")` got a Buffer for text, and `onclose` got a number. Fixes #36060. (#44111) - Server: 64 x 1 MiB arrived as 83 frames, 62 of them out of order, and `close()` behind a backlog delivered 4 of 24. The second send queue in JS is deleted. Fixes #44971, #21376. (#44520, #39802) - `send(Buffer.from([0xff, 0x68]), { binary: false })` arrived as `ef bf bd 68`. `binary` didn't decide the frame type for ArrayBuffer and Blob. `{ fin: false }` sent three whole messages. (#44535: its native entries, plus `fin` and `compress`) - `send()` after `close()` acknowledged a dropped message with `cb(null)`. (#39840) Callbacks now mean "written": they wait if the peer isn't reading, and they fail when the socket dies. (no PR) - `finishRequest` with a late `req.end()` sent two handshakes, and `'open'` never fired. A request that was never ended connected anyway. (#44123, with no deferred native socket) - `maxPayload` was ignored in both directions: a limit of 100 let 101 bytes through, the default rejected 20 MiB, and socket.io's `maxHttpBufferSize` did nothing. It's now a native limit per `WebSocketServer`, checked at the frame header. Fixes #8261. (#36650) Client `maxPayload` works too. (no PR) - A custom `agent` was silently dropped, so a SOCKS proxy the user asked for was bypassed by a direct connection. The agent is now used. The tests run three kinds of SOCKS agent against a SOCKS5 server and check that a dead proxy isn't bypassed. Fixes #23208. (no PR) - Frames that arrive before a *deferred* `handleUpgrade()` (after an `await`, or a `verifyClient` that answers late) were lost: `["later"]` where Node gives `["early", "later"]`. Such a connection now uses the stream transport, which starts with `head` and whatever the socket has buffered. (#43176, without its replay on `nextTick`) - Server `close(999)`, `close(1004)` and `close(-1)` were sent on the wire. (#42011) Features and parity: - The closing handshake, in both roles (#39802; client: no PR): - After `close()` the socket is CLOSING, and messages that arrive before the peer's Close frame are still delivered. - `'close'` reports the peer's code (`4444 "theirs"`, was our own `4000 "mine"`), or 1006 if the peer only sends a FIN. - A listener added after `close()` or `terminate()` fires. - `terminate()` after `close()` is 1006. - `close()` leaves a paused socket paused. - The `closeTimeout` option works. - uWS got a mode for this: write the Close frame, keep reading, and report when the peer's Close, the FIN or `socket.destroy()` arrives. `Bun.serve` doesn't use it. - `createWebSocketStream()` is upstream's `stream.js`, with backpressure. Fixes #4568. (#35459) - `options.WebSocket` is honored, server sockets are `instanceof WebSocket`, and patches to `WebSocket.prototype` apply to them. Fixes #13317. (no PR) - `ws._socket` is the node:http socket on servers. On clients it's a small object with `remoteAddress` / `remotePort` / `remoteFamily`, `ref()`, `unref()` (engine.io-client's `autoUnref`) and `destroy()`. (no PR) - Server `pause()` / `resume()` / `isPaused`. Fixes #44453. (#44554; the tests fail against a `pause()` that only sets a flag) - Options that were accepted but ignored now work. Client: `handshakeTimeout`, `origin`, `auth`, `followRedirects` / `maxRedirects` / `'redirect'`, `protocolVersion`, `autoPong`, `skipUTF8Validation`. Server: `autoPong`, `skipUTF8Validation`, `perMessageDeflate`. `handshakeTimeout` and `closeTimeout` are no longer capped by the native 120 s / 30 s timers, because no native timer runs for a `ws` client. `protocolVersion: 8` is actually sent, with `Sec-WebSocket-Origin`. (no PR) - Headers that `'headers'` listeners push are included in the 101 (engine.io's cookies and `headers` event). `handleProtocols: () => false` no longer selects a protocol (graphql-ws). (no PR) - Errors match npm's: `Unexpected server response: 403` for every kind of listener (#36375) and with or without a listener (#45125, minus its two regressions); `connect ECONNREFUSED ...` with `code`; `WS_ERR_*` for protocol errors on both sides, with Close 1002 / 1007 / 1009 on the wire. (no PR) - `req.destroy()` / `req.abort()` work in `'unexpected-response'` and `finishRequest`. (#37973) `ping(null | false)` sends an empty frame. (#39093) Text is a Buffer for every `binaryType`, built natively. (#44255, #44553, #44112) `terminate.call(undefined)` throws. (#34873) - `send(null)`, `send([1, 2])` and `send({})` do what npm does (empty binary, bytes, `ERR_INVALID_ARG_TYPE`). `ping` / `pong` callbacks run later, with `null`. Accessors are enumerable, and the ready-state constants are read-only. `["Foo", "foo"]` are two subprotocols (the global `WebSocket` treats them as duplicates). (no PR) - Bun's extras carry over: `proxy`, `tls`, `HttpsProxyAgent`-style agents (connected natively), `ws+unix:`, `wss+unix:`. - `perMessageDeflate: { serverNoContextTakeover, clientMaxWindowBits, ... }` goes into the client's offer (it was always `permessage-deflate; client_max_window_bits`), and the server's response is validated against what was offered. (no PR) - A RST instead of a response is `read ECONNRESET` with `syscall: "read"`. A FIN is `socket hang up`. (no PR) - The body of a non-101 response in `'unexpected-response'`: `res` now gets the whole body, decoded (Content-Length, chunked with `fetch`'s decoder, or until EOF), even when it arrives after the head. The event itself isn't delayed. Before, a chunked body arrived raw and a body that came after the head was lost, so a gateway's JSON error didn't parse. The body has backpressure: a `res` that isn't read pauses the native read. A complete body sets `res.complete`. A truncated one ends as in Node: `'aborted'`, `'error'` `ECONNRESET`, no `'end'`. (no PR) - A 101 with no value in `Upgrade`, or without the `upgrade` option in `Connection`, is a final response with no body, as it is for node:http's parser: `'unexpected-response'`, then `'end'` on the response immediately. Without a listener it's `'error'` "Unexpected server response: 101". The global `WebSocket` fails such a 101 as before. (no PR) - Found by fuzzing (no PR): - A `handleUpgrade()` that runs a few ms late (an `await`, an async `verifyClient`) lost bytes, or the peer's FIN, that node:http had read but not yet handed to JS. Lost bytes failed a valid client in about 1% of such connections. A lost FIN left the socket open for good, which a remote peer could provoke (main too). Both now take the stream transport. - `req.socket.remoteAddress`, `servername`, `authorized` and `authorizationError` can still be read in `'close'`, whether the upgrade came from an `'upgrade'` or a `'request'` listener. - A send callback that throws no longer skips `'close'`, a stream's `'finish'`, or the callbacks behind it. On the stream transport each runs exactly once, although node:net on Bun drops some write callbacks and repeats others. - `handshakeTimeout` also covers the body of an `'unexpected-response'`. A refused upgrade behind a pipelined request answers on the right response. `createWebSocketStream()` finishes when the closing handshake is done. - `tls` / `proxy.tls`: inherited, non-enumerable and getter options are honored, also next to a `pfx`, and `Object.prototype` is ignored. #### node:http - `socket.bytesWritten` of an `'upgrade'` / `'connect'` socket was 0. It now counts raw writes (queued ones too) and the 101 that `ws` sent, and it keeps its value after close. `bytesRead` counts the tunnel's bytes. engine.io closes every upgraded socket on its `http.Server` whose `bytesWritten` is 0 after 1 s, which is the cause of "vite / SvelteKit dev + socket.io reloads every second". Fixes #23747. (#45030: its sum and its test lines, plus corked, backpressured and closed sockets) Not counted: WebSocket frames after a native upgrade, and HTTP heads in `bytesRead`. - A node:http server no longer registers a `Bun.serve`-style `websocket` route. The `NodeHTTPResponse` arm of `server.upgrade()`, `NodeHTTPResponse::upgrade()` and `UpgradeCTX` are deleted (375 lines). - Pinned by tests: `socket.destroy()` and `socket.setTimeout(0)` after an upgrade; `http.Server#close()` leaves `ws` connections open. #### undici - Added `require("undici").ping(ws, body?)`, and `undici:websocket:ping` / `:pong` on `node:diagnostics_channel`, with a `Buffer` payload for every `binaryType`. Without them, Slack Socket Mode / Bolt reconnected forever. Fixes #37110. (#37111, with two shared listeners instead of two closures per socket, undici's order of argument checks, and `"blob"` handled) - `require("undici").WebSocket` is now a subclass of the global one instead of the same object. Only its sockets publish to those channels. The `open` / `close` / `socket_error` channels stay silent. #### Types and docs - `error` / `onerror` are `ErrorEvent`. Fixes #36329. (#42673, #39069) Added `ping` / `pong` events, `binaryType: "blob"` and `Blob` payloads. (#39069) `headers` accepts `Headers` and pairs, and with lib.dom loaded, `Bun.WebSocket`'s `message` event has `.data` again. (no PR) The fixtures assert on `Bun.WebSocket`, so the list of expected lib.dom diagnostics didn't change. - `serve.d.ts`: `error?(error, ws)`, `pause` / `resume` / `isPaused`, `message?` is optional, `cork<R>` (its `T` shadowed the data type), option ranges and what 0 means, `close()` codes, compress defaults, and four examples that called the wrong method. - `websockets.mdx`: every example runs and type-checks as pasted (the first one threw). New "Closing connections" and "Errors" sections. One Origin check for cookie-authenticated upgrades, which allows a missing `Origin` so that non-browser clients work (#36155). Also compress defaults (#39253), backpressure (#41679, WebSocket half), `pause()` (#42976, the paragraph but not its JSDoc) and casing (#42443). The hand-copied Reference block had drifted on nine members and is replaced by links, which makes #40879 moot. - `ws.ts` is fully typed (`tsc` on `src/js`: 0 errors, also with `noImplicitAny`). Adding the types left the generated module byte-identical. #### Tests - #39370 took `websocket-server.test.ts` on a debug build from 40 s with 4 timeouts to 13 s and green (the file has grown since). - The writev fault test never reached `us_socket_write2`. (#38174) - `websocket-pause.test.ts` streams 32 MiB instead of 256 MiB and fails immediately when a send is dropped. (#41109, the constant and about 15 lines) - `ws-syscall-fault.test.ts` went from 7.6 s to 3.2 s on debug. (#37115, without the asserts that froze four differences from npm) - `setSocketOptions()` of `bun:internal-for-testing` (`SO_SNDBUF` / `SO_RCVBUF`) now takes any socket Bun owns: a `ws` or global `WebSocket`, a `ServerWebSocket`, TLS sockets, node:net and node:http sockets. It also works on Windows, where it did nothing. With a 64 KiB send buffer a sender hits backpressure after about 200 KB instead of 4.2 MB, so the backpressure tests send 1 MiB instead of 12 to 64 MiB. - No test sits through a real close timeout (uSockets sweeps every 4 s, so "1 s" is 4 to 8 s). Those were removed, and the numbers are under "Fuzzing before merge". - Autobahn fan-out (#35421): not run here, it needs docker. - `hostname: "127.0.0.1"` where a test connects to 127.0.0.1. (#41331, plus `websocket-proxy.test.ts`) I haven't swept the other files that have the same pattern. - Two stale `todo`s are now real tests, and so are `ws`'s five early-frame `todo`s (#43176). The empty `perMessageDeflate` todo and an empty `skip` are removed. - New in `test/js/first_party/ws`: the `ws-*-compat-*.test.ts` files, `ws-stream-transport.test.ts`, a rewritten `ws-upgrade-events.test.ts` (18 files) and one raw-peer helper. Every block runs twice: on the built-in, and on `node_modules/ws` loaded by path. No sleeps. ### Performance **`bench/websocket-server`** (the chat room in this repo, 32 clients in 4 client processes as its README suggests, server on 4 reserved cores, 6 interleaved rounds, this PR at `6564a70`): Throughput against main: `Bun.serve` pub/sub -1.1% (main against a copy of itself: +1.4%), `ws` -4.6% (copy: -4.1%). That's inside the noise, so I also counted instructions per delivered message. `Bun.serve`: 758 on main, 775 here. `ws`: 12,477 on main, 12,628 here (95% of the time is in the kernel's TCP stack). **Receiving compressed messages in a client** (global `WebSocket`, server without context takeover, which is `Bun.serve`'s default; user-space instructions per message, this PR at `6605b96`): | JSON message | main | this PR | |---|---|---| | 512 B | 35,958 | 16,375 | | 2 KB | 57,948 | 26,022 | | 16 KB | 224,533 | 101,941 | | 64 KB | 718,398 | 333,248 | The client tried libdeflate first, which can't inflate an unfinished stream, so it decoded every message, failed at the end, and zlib decoded it again. It now appends a final empty block, as uWS does on the server, and only uses libdeflate without context takeover. **My own scripts:** Both binaries are CI's own release builds: this PR at `d51fcce`, and `9510e74`, the main commit it's based on. 8 physical cores were reserved: 4 for the side being measured and 4 for the other end, which is always main's binary. Runs are interleaved, and each number is the median of 7 rounds. "main again" is a byte-identical copy of main's binary, so the difference between the first two columns is the noise. `ws` client (echo server: main's `Bun.serve`): | shape | main | main again | this PR | |---|---|---|---| | ping-pong 64 B, round trips/s | 43,469 | 43,838 | 46,403 | | pipelined echo 64 B, msg/s | 261,627 | 263,397 | 299,713 | | echo 16 KB, msg/s | 98,230 | 94,591 | 97,603 | | compressed echo 2 KB, msg/s | 106,904 | 106,657 | 107,253 | | idle connection, KB | 9.2 | 9.1 | 10.4 | `ws` server (load generator: main's native `WebSocket`): | shape | main | main again | this PR | |---|---|---|---| | ping-pong 64 B, round trips/s | 43,011 | 43,784 | 45,172 | | pipelined echo 64 B, msg/s | 271,840 | 276,301 | 273,717 | | echo 16 KB, msg/s | 81,858 | 83,697 | 82,962 | | `send()` to 100 clients in a loop, 128 B text, deliveries/s | 345,531 | 352,509 | 381,826 | | idle connection, KB | 7.9 | 7.7 | 7.4 | The loop row is `for (const c of wss.clients) c.send(data, { binary: false })`, not pub/sub. It measures per-`send()` overhead. The old shim decoded the `Buffer` to a string on every such call; the new code passes the bytes and the opcode straight through. Compressed echo on the server can't be compared, because the old shim never negotiated permessage-deflate. The global `WebSocket` client and `Bun.serve` are level with main on the same three shapes (ping-pong, pipelined echo, echo 16 KB): every difference is under 1.6%, and main differs from its own copy by up to 2.4%. `Bun.serve`, `ws.send(x, true)` echo with the libdeflate compress path deleted (older numbers: local release builds, a busy machine): the shared compressor is 10% to 2.5x faster than main from 64 B to 16 KB. The dedicated one is 7% slower at 64 B and 1.3x to 5x faster from 512 B to 16 KB. Bytes on the wire for 64 B to 4 KB slices of `package.json`: the shared compressor is within 4 bytes of main, and the dedicated one is 11 to 26% smaller from 512 B up. Supporting the server role in the client engine costs a natively connected client one predictable branch per received frame and two per sent frame. The global `WebSocket` table above measures that. Every `JSWebSocket` cell grew by 16 bytes (owner and handler table). Each upgrade client is 4 KB smaller (a header array was removed). Binary size: +64 to +128 KB per target (CI's table). ### How did you verify your code works? **Oracles.** Node 26.3.0 (its global `WebSocket` is undici 8.3); npm `ws` 8.18.3 (the repo's test dependency), on Node and, loaded by path, on Bun; the source and tests of `ws` 8.22.0; RFC 6455, RFC 7692 and the WHATWG standard. Where Node and a spec disagree, I say so above. **Against Node + npm `ws`** (release build): | what | main | this PR | |---|---|---| | `ws`'s own test suite, 271 tests, in a sandbox without network | 94 | 234 | | 212 client cases: one process each, raw `node:net` peer, output compared with Node's byte for byte | 18 | 179 | | 159 server cases, same method | 67 | 150 | | new permanent suite, built-in rows that fail (measured at 1,233 rows; it has grown since) | 800 of the 1,082 that finish | 0 | - The remaining failures in `ws`'s own suite reach into the JS `Sender` / `Receiver` / `net.Socket` of a natively connected client (`ws._receiver`, `ws._socket.on`, `generateMask`, file-backed Blobs), which don't exist here. - The remaining client cases differ in the wording of engine errors (V8 / JSC), ws 8.22 versus the 8.18 that produced the recorded output, the order of request headers, and the kept differences listed above. **Each new or changed test was first run on main's release build**, and I read the failure. It has to fail for the stated reason, not by timing out. Some new tests pass on main: they pin existing behavior to guard the rewrites. **Debug + ASAN build**, with `BUN_JSC_validateExceptionChecks=1` as on CI's ASAN lane: every file under `test/js/web/websocket`, `test/js/bun/websocket` and `test/js/first_party/{ws,undici}`, plus `node-http`, `node-https`, `node-http-connect`, `node-http-with-ws`, `node-tls-connect`, `bun-server`, `request-smuggling`, fastify-websocket and `bake/deinitialization`. 80 files and 6,001 tests, all pass. - Wire logs from a raw peer and the client on one clock: `send` x 32 + `close` delivers 32 of 32 (main: 10, and 2 at 64 KiB); 256 of 256 through the tunnel with the Close last; Pong before Close on `wss://`. **CI**: every platform built, and the new `ws` suite passed on macOS and Windows. **Fuzzing before merge.** Up to seven agents, one area each, ran seeded generators in a jail on CI's release builds of this PR, main, released Bun and Node, and on debug + ASAN. After each round I fixed what they found, another agent turned each repro into a test that fails on the old binary, and the hunt ran again on the new one. | | round 1 | round 2 | round 3 | round 4 | |---|---|---|---|---| | cases | over 1M | 400k | 300k | 200k | | crashes, sanitizer reports | 0 | 0 | 1 use-after-free | 0 | | regressions from main | 6 | 5 | 0 | 0 | | bugs in the fixes of the round before | | 6 | 5 | 2, low | - Proxy trust: over 350,000 combinations of options, certificates and routes against a model. Nothing opened that must be rejected, and no credential or client certificate reached the wrong peer, except a `pfx` next to inherited or hidden options (fixed). The same model flags main in 90 of 800 cases. - The use-after-free came from one of my fixes for round 1: it ran the `ws` drain handler after uWS's close handler, when GC may already have freed the native socket. It needed a script that clears private symbols. That code is gone, and uWS again calls nothing after the close handler. - Close timeout, measured on CI's release build on ws, wss, a unix socket and through http, https and authenticated proxies: `send(100 MB); close()` at 2 MB/s delivered 82.5 MB and closed 1006 before the fix, 100 MB and a clean 1000 after it. A peer that stops reading at once, halfway or 1 KB before the end is dropped one timeout after its last read. One that reads everything but never answers is dropped one timeout after the flush. - Late `handleUpgrade()`: 9 of 1,250 connections lost bytes before the fix, 0 of 1,500 after it, then 0 of about 50,000 generated cases of valid traffic (the generator fails 119 of 8,000 on the build before). **`Bun.serve` against main**, for "keeps accepting what released Bun accepts": the same 8,700 server calls, 1,492 client calls, 26 handshakes and 19 close codes on both. What only this PR refuses: an HTTP/1.1 upgrade without `Host`, a repeated `Sec-WebSocket-Key`, `idleTimeout` of 65,536 or more, and `upgrade()` with another server's request. Five more were found this way and taken out. At scale (50,000 connections, a 20-minute soak with 438,000 connections, fan-out, dead consumers, reconnect storms, `stop()` / `reload()` with 20,000 open) only item 13's memory and item 10's close code differ from main. **Two security scans** of the whole diff found nothing above low severity except two denial-of-service bugs in `ws`. All four findings were reproduced and fixed with tests: - Unbounded automatic Pongs (described above). The cap is on all three transports, because a peer can choose the transport by sending a frame before a deferred upgrade. - A redirect `Location` (`/#`, `ws+unix::/x`) threw out of the native handshake callback and ended the process. - Bun's `tls: { ca, checkServerIdentity, ... }` option was dropped whenever the client connects through `http.request()` (`finishRequest`, `lookup`, `family`, ...), so a certificate pin was lost (1.4.3 enforces it). - `tls`, `proxy` and `agent` could be set through a polluted `Object.prototype` (1.4.3 has one such case too). **A review of the whole branch** found 15 issues, fixed with tests. The worst: a use-after-free of the server when `handleUpgrade()` gets a request from a server that was already freed, a stream wrapping the socket that `handleUpgrade()` bypassed, and the ignored client options `signal` and `secureContext`. **Lints**: `cargo clippy --workspace`, `cargo check --workspace --all-targets`, `rust:check-all` (every target), `bun lint`, the three `tsc` projects, clang-format, rustfmt, prettier, `test/internal/source-lints`. **Types**: `bun-types.test.ts` passes 22 of 22. With main's `.d.ts` and the new fixtures, 10 of 22 fail. **Not run**: - #44633's script on Windows. It's worth trying, since the 101 and the Close now leave in one write. Some backpressure tests are skipped on Windows, as they were in the original PRs. - Autobahn (needs docker), Safari, WebView with a real Chrome. - "No ping and no idle close on a `ws` server" has no automated test, because it needs 135 s. I checked it by hand, with the three options injected into main. ### The pull requests Thanks to the outside contributors whose reports and patches are in here: @C0casio45 (#28152), @hughescr (#34222), @steipete (#44553, #44554, #45030), @davidhevn (#42673), @deepshekhardas (#40879, #42443), @ousamabenyounes (#36749), @alex-mextner (#31205); and to @1cedsoda, whose #44061 / #44165 stay open. "ext" = from an outside contributor. taken = as it was, or nearly. reworked = same bug, different fix. superseded = falls out of a larger change here. | PR | author | verdict | why | |---|---|---|---| | #20533 | Jarred-Sumner | not taken | clang already emits `bswap` / `rol`, so the assembly is identical | | #24914 | Jarred-Sumner | obsolete | fixed by #25475; in BoringSSL the line marks every TLS socket low priority | | #28152 | C0casio45 (ext) | obsolete | next-ws awaits before `handleUpgrade()`; #39642 fixed that. Proxy case kept as a test | | #31205 | alex-mextner (ext) | not taken | its cases compile on main; I widened `headers`, which is probably what the author hit | | #31761 | robobun | reworked | one snapshot in the single exit; counts payload bytes only | | #31982 | robobun | taken | | | #33011 | robobun | taken | plus BINARY inside TEXT | | #33066 | robobun | reworked | one `dispatchEvent` override, not 20 renamed call sites | | #33592 | robobun | taken | | | #34222 | hughescr (ext) | reworked | the one-line fix + `static_assert`s; no test binding, no 20 s tests that pass without the fix | | #34873 | robobun | superseded | by the port | | #34961 | robobun | reworked | `stop(true)` half only; `stop()` stays graceful as documented | | #35029 | robobun | reworked | `span()` kept; detached = empty, no TypeError | | #35031 | robobun | not taken | breaks `close(1001)` / `close(1011)` | | #35421 | robobun | taken | | | #35459 | robobun | superseded | upstream's `stream.js`, with the backpressure the PR dropped | | #35730 | robobun | not taken | 0.8 s on a test that CI skips | | #35874 | robobun | reworked | handler keeps 1006; 1007 vs 1009 for inflate errors | | #36112 | robobun | taken | plus the missing timeout | | #36155 | robobun | reworked | one check and a short warning; a missing `Origin` is allowed | | #36375 | robobun | superseded | upstream's `abortHandshake`, every kind of listener | | #36561 | robobun | not taken | same as #35730 | | #36650 | robobun | reworked | native limit per `WebSocketServer`; `'close'` is 1006, not 1009 | | #36749 | ousamabenyounes (ext) | not taken | frames from one read still overlap; deadlocks request / response over one socket. `ws.pause()` instead | | #36808 | robobun | obsolete | replaced by #37275 | | #36998 | robobun | taken | | | #37111 | robobun | reworked | shared listeners, `"blob"`, undici's check order | | #37115 | robobun | reworked | `describe.concurrent` only | | #37439 | robobun | stays open | WebSocket half landed in #40624; the rest is `fetch` | | #37447 | robobun | reworked | the early rejection only; #44618 fixed two of its three paths | | #37463 | robobun | obsolete | landed in #43557; test kept | | #37559 | robobun | not taken | refactor of code that #40018 rewrote | | #37638 | robobun | taken | | | #37972 | cirospaciari | superseded | one handler table, so no listeners to install twice | | #37973 | cirospaciari | superseded | one request class; npm's rows | | #38174 | robobun | taken | | | #38177 | robobun | obsolete | fixed by #40566 | | #38632 | robobun | taken | | | #39069 | robobun | reworked | `.d.ts` taken; fixture on `Bun.WebSocket` | | #39071 | robobun | taken | plus `null` | | #39082 | robobun | reworked | the server's error class, one FFI call | | #39093 | robobun | superseded | upstream's line | | #39253 | robobun | taken | plus `publish(t, d, undefined)` | | #39255 | robobun | not taken | a `send()` cancels the ping deadline, as released | | #39370 | robobun | taken | | | #39385 | Jarred-Sumner | taken | ported by hand; `sendBinary`, `close`, ping / pong too | | #39608 | robobun | stays open | not WebSocket; its two `ErrorEvent` lines are here | | #39802 | robobun | reworked | no JS queue, no tick; native "wait for the peer's Close" | | #39840 | robobun | superseded | upstream's lines | | #39843 | robobun | taken | plus TLS tests | | #40097 | robobun | superseded | by #36998; docs lines kept | | #40879 | deepshekhardas (ext) | obsolete | the block it fixes is deleted | | #41109 | robobun | reworked | the constant + 15 lines | | #41331 | robobun | taken | | | #41482 | robobun | reworked | Rust only, smaller | | #41573 | robobun | reworked | WebSocket half; `fetch` half dropped | | #41629 | robobun | reworked | new negotiator, fallback offers, no x-webkit | | #41637 | robobun | reworked | no throw for a config that works today | | #41665 | robobun | taken | | | #41670 | robobun | taken | | | #41674 | robobun | taken | | | #41679 | robobun | reworked | three sentences; `routing.mdx` half not taken | | #41680 | robobun | reworked | never replaces a live context's closures | | #41698 | robobun | reworked | `removeCredentials()`; `fetch` half landed in #42692 | | #41705 | robobun | taken | inside #41674's code | | #41712 | robobun | stays open | product decision | | #41728 | robobun | stays open | not only WebSocket; overlaps in `websockets.mdx` and `error` | | #41811 | robobun | reworked | `upgrade(Request)` only | | #41875 | robobun | taken | plus `headers: {}` and declining a protocol | | #41962 | robobun | taken | | | #42011 | robobun | superseded | upstream's checks, both roles | | #42216 | robobun | reworked | 5 lines, no budgets | | #42318 | robobun | stays open | the libdeflate patch is a vendoring decision; the WebSocket path is deleted, its two tests are kept | | #42443 | deepshekhardas (ext) | taken | | | #42673 | davidhevn (ext) | taken | its two lines; a different fixture, because the lib.dom run fails with the PR's own | | #42974 | robobun | reworked | `resume()` on close; the rest follows from `readyState` | | #42976 | robobun | taken | trimmed | | #42978 | robobun | taken | derived from `readyState` | | #43018 | robobun | reworked | with #43696: one field instead of four | | #43176 | robobun | reworked | these connections use the stream transport; as written, the PR would deliver `head` twice | | #43243 | robobun | reworked | bounded by the byte limit | | #43445 | robobun | taken | | | #43511 | robobun | taken | | | #43614 | robobun | stays open | goes with #43441 | | #43696 | robobun | reworked | the deletion only; no nested-event-loop cases | | #43711 | robobun | reworked | `terminate()` and exit flush; no flag | | #43763 | robobun | reworked | "always buffer"; no `handshake_dispatched` | | #44061 | 1cedsoda (ext) | stays open | RFC 8441 is a separate feature; needs a rebase | | #44111 | robobun | superseded | upstream's `event-target.js`, both roles | | #44112 | robobun | reworked | second native handler set, no per-event flag | | #44122 | robobun | not taken | used its options struct and test table; no `validate()` / `dial()` split | | #44123 | robobun | reworked | real `http.request()`; no deferred native socket | | #44165 | 1cedsoda (ext) | stays open | built on #44061 | | #44255 | robobun | superseded | native Buffer for text | | #44460 | robobun | reworked | binding + `can_upgrade()`; 101 instead of 500 | | #44520 | robobun | reworked | no backpressure limit, JS queue deleted | | #44535 | robobun | reworked | native entries + `fin`, `compress`; no lint | | #44553 | steipete (ext) | superseded | same result, built natively; test kept | | #44554 | steipete (ext) | reworked | public `ServerWebSocket.pause()`; tests that fail against a no-op `pause()`; no 120 s kill | | #44821 | robobun | taken | plus `realign` | | #44822 | robobun | reworked | three small changes instead of five mechanisms | | #44964 | robobun | taken | explicit `Proxy-Authorization` wins | | #45000 | robobun | reworked | reports the error and keeps serving | | #45030 | steipete (ext) | reworked | its sum and tests; plus cork, backpressure, close, `bytesRead`, the 101 | | #45122 | robobun | taken | extended to failures after open, and 1006 | | #45125 | robobun | superseded | falls out of `emit("error")`; two regressions avoided | | #45196 | robobun | reworked | moves the `close` event; no second parser | Issues that can also be closed: - #22870: the client option works; the reporter set it on the server. - #18869: the code in its trace was rewritten by #39642 / #39435, and it has no s…
…the fetches have responded (oven-sh#45600) ### What does this PR do? Fixes the flaky "worker exit with streaming-request-body fetches whose response.body was touched" in `test/js/web/workers/worker-terminate-lifetime.test.ts`. It fails with `worker exited 3` on the macOS x64 lane: in the 328 recent builds I read, it failed every retry in 64 and passed on a retry in 35 more. Each of the 6 workers starts 8 fetches against one server and exited on a timer, 150 to 202 ms after it started, with code 3 if no response had arrived by then. On a busy machine none has. The timer now starts when all 8 fetches of the worker have their response and have touched `response.body`, which is the state the test is about. Its length is unchanged, so at exit the streams have run at least as long as before, and all 8 fetches are in that state instead of at least one. A fetch that fails is now printed and fails the test; it was swallowed before. `test/flaky-tests.txt` keeps the file, with the reason that is left: "a worker ends while a stream is still being piped into a FileSink" crashed with a segmentation fault on a libuv-worker thread in 4 builds on Windows. That is a bug in Bun, not in the test, and this PR does not touch it. ### How did you verify your code works? With the Linux x64 binary that CI built for main (f8bc171), under `systemd-run --user --scope -p CPUQuota=N`, 10 runs each: | CPU quota | main | this PR | |---|---|---| | 25% | 9 pass, 1 `worker exited 3` | 10 pass | | 12% | 10 `worker exited 3` | 10 pass | | 6% | not run | 10 pass | The whole file passes without a quota. Not checked: that the test still fails on a runtime without the fix it was written for (oven-sh#38457), which needs an ASAN build of that commit.
…t.ts (oven-sh#45598) ### What does this PR do? Fixes a race in `test/js/bun/net/socket.test.ts`. "close() error after the peer resets the connection > tcp > the accepted socket reports the reset as read ECONNRESET" fails now and then on the macOS 14 x64 lane, main included: `close()` gets no error. It is named in the error annotations of 25 of the 328 recent builds I read. The test wrote to a paused child process and killed it at once: ```ts accepted.write("left unread in the peer's receive buffer"); peer.kill("SIGKILL"); ``` The kernel resets a connection on close only if unread data is already in the receive buffer. Otherwise it sends a FIN, which is a clean end. On macOS loopback delivery is asynchronous, so a busy machine can kill the peer before the bytes arrive, and nothing in JavaScript can tell whether they have. Measured on macOS: 800 of 800 runs report `ECONNRESET` on an idle machine, 40 of 40 report no error under loopback load, and 40 of 40 report `ECONNRESET` under the same load with a 100 ms wait before the kill. `terminate()` resets without that condition. The test next to it already used it, for tcp and tls, and is in none of those annotations. Both are now one test body with the roles swapped: the accepted socket reports the connected socket's `terminate()`, and the other way round. The child process and its temp directory are gone. ### How did you verify your code works? With the Linux x64 binary that CI built for main (f8bc171): the 4 tests pass, 40 of 40 runs. With `end()` in place of `terminate()` they fail with `reported: false, code: undefined`. macOS and Windows are left to CI.
…ven-sh#45635) ### What does this PR do? Fixes the flaky "load the same file 10000 times" in `test/js/bun/resolve/load-same-js-file-a-lot.test.ts`. It timed out on the Windows 2019 x64 lane in 7 of the last 27 builds of main, each time in the shard's parallel batch, and passed when it was run again alone. CI runs every test with `--timeout=90000`. This test passed a timeout of its own, 5000 ms on a release build, so it had an eighteenth of the time every other test gets. The second test of the file does the same number of imports without a timeout of its own and is in none of those failures. The test now has no timeout of its own. That timeout was also what gave a debug build 20 s. `bun bd test <file>` passes no `--timeout`, so there the default of 5 s applies, and 2000 imports do not fit in it. A debug build now does 200 imports. Release builds still do 10000 and ASAN builds 2000. ### How did you verify your code works? **Release.** With the Linux x64 binary that CI built for main (f8bc171), `bun test <file> --timeout=90000` under `systemd-run --user --scope -p CPUQuota=6%`, 3 runs each: | | result | |---|---| | main | 3 of 3: `this test timed out after 5000ms`, the second test passes | | this PR | 3 of 3: both tests pass | Without a quota both tests pass. Not run on Windows locally; CI covers that lane. **Debug.** With a debug build, `bun-debug test <file>` and no `--timeout`: | | first test | second test | |---|---|---| | main | passes 5 of 5 | passes 4 of 5, times out once | | this PR | passes 6 of 6 | passes 6 of 6 | 6 copies side by side at `CPUQuota=200%` also all pass.
…est.ts without their order (oven-sh#45640) ### What does this PR do? Fixes the flaky "overrides" test in `test/cli/run/run-autoinstall.test.ts`. It failed on the macOS 14 x64 lane in 3 of the last 27 builds of main and passed on the retry. All three times the registry had served the two expected tarballs, in the other order: ``` [ - "/one-range-dep/-/one-range-dep-1.0.0.tgz", "/no-deps/-/no-deps-1.0.1.tgz", + "/one-range-dep/-/one-range-dep-1.0.0.tgz", ] ``` `registry.tarballs()` lists requests in the order they reach the test's server. In this test the two downloads are in flight at the same time, on separate connections, so that order is not Bun's to decide. The test is about which version of `no-deps` the override selects. The list is now sorted before it is compared, as in the two other `registry.tarballs()` assertions of this file that expect more than one tarball. "bun.lock, for a dependency of a package" expects the same two tarballs in order, through `registry.requests`. It stays as it is: there the second download starts only once the first has finished (see below). ### How did you verify your code works? With the Linux x64 binary that CI built for main (f8bc171): the "overrides" and "resolutions" tests pass, and the whole file passes (303 pass, 1 skip). With the expected `no-deps` version changed to 1.1.0 both tests fail, so the assertion still tells a wrong version. Are the two downloads concurrent? With a debug build I had the test's server hold the response for the `one-range-dep` tarball until the `no-deps` tarball was requested, or for 3 s. 3 runs per test: | test | while `one-range-dep` was held | |---|---| | overrides | `no-deps` was requested, 3 of 3 | | resolutions | `no-deps` was requested, 3 of 3 | | bun.lock, for a dependency of a package | nothing was requested in 3 s, 3 of 3 | I could not reproduce the swapped order itself on Linux: 480 runs of the two tests on main, 24 copies side by side at `CPUQuota=100%`, all passed.
… own cache (oven-sh#45644) ### What does this PR do? Fixes the flaky "large single package with many dependencies" test in `test/cli/install/migration/pnpm-comprehensive.test.ts`. In 2 of the last 27 builds of main it failed in the parallel batch of the macOS 14 x64 lane and passed alone, both times with this one difference from the snapshot: ``` - "fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, ... + "fsevents": ["fsevents@2.3.3", "", { "os": "darwin", "hasInstallScript": true }, ... ``` pnpm-lock.yaml does not record `hasInstallScript`, so after a migration it is taken from the package's manifest (`fetch_necessary_package_metadata_after_yarn_or_pnpm_migration`). That lookup also reads the manifest cache on disk, expired entries included. `startNpmjs()` stands in for `https://registry.npmjs.org/` and answers 404 for `fsevents`. Its users did not set `BUN_INSTALL_CACHE_DIR`, so they used the one of the runner. The runner makes one per `bun` process it starts: a file run alone has it to itself, but all files of the parallel batch share one. A manifest that another file cached for that registry URL was then used here. `startNpmjs()` now has a cache directory of its own, passed in `env` and removed by `stop()`. That covers its three users: `pnpm-comprehensive`, `pnpm-migration-complete` and `pnpm-lock-migration`. ### How did you verify your code works? With the Linux x64 binary that CI built for main (f8bc171). I filled a cache directory with an `fsevents` manifest that has `hasInstallScript: true`, from a local server behind the same kind of tunnel, and ran the tests with `BUN_INSTALL_CACHE_DIR` set to it: | outer cache | main | this PR | |---|---|---| | empty | the test passes | the three files pass (26 tests) | | has the `fsevents` manifest | the test fails with exactly the difference above | the three files pass (26 tests) | | manifests the three files leave in it | 12 | 0 | I did not find out which file of the batch caches the `fsevents` manifest in CI.
…strings.test.ts (oven-sh#45639) ### What does this PR do? Fixes the flaky "a compiled executable runs what is in it" tests in `test/js/node/disallow-code-generation-from-strings.test.ts`. In 6 of the last 27 builds of main one of the four variants ran into the 90 s timeout on a Linux lane, a different variant each time. The program under test spins until the DFG has compiled a small function: ```js function hot(a, b) { return a + b; } while (!numberOfDFGCompiles(hot)) sum = hot(sum, 1) | 0; ``` `numberOfDFGCompiles(hot)` counts an optimized replacement of `hot`'s own code block (`CodeBlock::numberOfDFGCompiles`). Two things race to be compiled: `hot`, and the function around the loop. If the outer function wins, `hot` is inlined into it and is never called again, so it never reaches its own threshold and the loop never ends. On a machine with CPU to spare `hot` wins. The process then spins on one core, and outlives the test that timed out. `noInline(hot)` from `bun:jsc` takes `hot` out of the inliner's candidates, so it keeps being called and gets compiled. This is the only loop on `numberOfDFGCompiles` in `test/`. The hang needs neither `--compile` nor `=strict`: the loop alone in a plain script does it. ### How did you verify your code works? With the Linux x64 binary that CI built for main (f8bc171), copies run side by side in a cgroup with a CPU quota, main and this PR in alternating rounds: | | main | this PR | |---|---|---| | the four tests, 12 copies at `CPUQuota=200%` | 10 of 144 runs fail, each with one test timed out | 0 of 144 | | the loop alone, 40 copies at `CPUQuota=100%` | 5 of 800 hang | 0 of 800 | What a hung process shows: - `perf`: 90% of the samples are in `numberOfDFGCompiles` and what it calls, the rest in JIT code. - `BUN_JSC_reportDFGCompileTimes=1`: the only function compiled is the anonymous outer one. In every run that finishes, with or without the fix, `hot` is compiled.
…rator-stream.test.ts (oven-sh#45688) ### What does this PR do? Fixes the flaky "Streaming body via > async generator function" in `test/js/bun/http/async-iterator-stream.test.ts`, and "[Symbol.asyncIterator]" next to it, which has the same flaw. The first failed in the parallel batch of the macOS 14 x64 lane in build 126506 of main (`Expected length: 2`, `Received length: 1`) and passed alone. Both bodies yield, `await Bun.sleep(30)`, and yield again, and the tests expect the client to read two chunks. That holds only if the client reads the first chunk within those 30 ms. On a busy machine both are there by the time it reads, as one chunk. The generators now wait until the client has read a chunk before they yield the next one. There are two chunks whatever the machine is doing, and a body that was held back until the generator is done would now hang the test. The two other `Bun.sleep(30)` in the file are left alone: their tests only compare the whole text. ### How did you verify your code works? Linux x64, the binary CI built for main (0ffddcd), the whole file, 24 copies side by side under `CPUQuota=100%`, main and this PR in alternating rounds: | | main | this PR | |---|---|---| | runs that failed | 231 of 288 | 0 of 288 | | runs in which "async generator function" failed | 217 | 0 | | runs in which "[Symbol.asyncIterator]" failed | 93 | 0 | No other test of the file failed in either. Without a limit this PR's file passes 30 of 30. I have not run it on macOS.
…t.js (oven-sh#45686) ### What does this PR do? Fixes the flaky "node:dgram close() inside 'message' handler stops remaining batch datagrams" in `test/js/node/dgram/node-dgram.test.js`. It failed in the parallel batch of the macOS 14 x64 lane in builds 125737 and 126172 of main, both times with `"trace": []`, and passed alone. After its 16 sends the fixture waited 8 turns of the event loop and printed what it had seen. That is enough where a loopback datagram is in the receiver's buffer by the time `send` returns, as on Linux. An empty trace means that none had arrived after 8 turns. The fixture now waits for the callback of `close()`, and then for one more turn. The poll that dispatches datagram 0 is the one that would dispatch the rest of its batch, and it is over by then. It also starts all 16 sends before the loop gets a turn, where it awaited each one. That puts more of them into the first batch when delivery is late (second table). ### How did you verify your code works? All on Linux x64 with release binaries. I have not run it on macOS. A late loopback is imitated in a network namespace with `tc qdisc add dev lo root netem delay <d>`. **The failure.** The test on Bun 1.4.3, 10 runs per cell. Failing runs, all with `"trace": []`: | delay | none | 1 ms | 5 ms | 30 ms | 200 ms | |---|---|---|---|---|---| | main | 0 | 10 | 10 | 10 | 10 | | this PR | 0 | 0 | 0 | 0 | 0 | **The bug it was written for.** Bun 1.3.14 is the last release without oven-sh#33669. Runs of the fixture on it that show `message:1` or more after the close, the two send loops in alternating rounds of 40: | delay | none | 1 ms | 5 ms | 30 ms | |---|---|---|---|---| | each send awaited | 40 of 40 | 13 of 200 | 9 of 200 | 179 of 200 | | all sends started first (this PR) | 40 of 40 | 32 of 200 | 46 of 200 | 186 of 200 | So the test finds the bug every time where the burst is one batch. Where delivery is late it often has a batch of one datagram to look at, and passes without having tested anything. These rates move: a round of 40 at 5 ms gave between 2 and 15 with this PR's loop, and one earlier round gave 40 of 40 at 1 ms and at 5 ms with either loop. On a fixed Bun (1.4.3) this PR's fixture prints the expected trace in 40 of 40 runs at each of the four delays. With a delay, main's fixture fails on both releases with `[]`. **The extra turn is needed.** Without the `setImmediate` after `closed.promise`, the fixture prints `message:0, closeEvent, closeCallback` on 1.3.14 in 20 of 20 runs, which is what a fixed Bun prints. **A lost datagram 0** ends in the test's timeout, as on main: with `netem loss 100%` both versions of the test report `this test timed out after 5000ms`. The whole file passes on Bun 1.4.3 (4 pass).
### What does this PR do?
`test/cli/install/bunx.test.ts` was red on Windows 2019 x64 in build
127229 of main: "-p installs the package for a command that is none of
its bins, from $PATH" failed in all 4 attempts with `error: Package
mypkg does not provide a binary named caller`.
The test puts a directory in front of `PATH` with `{ ...box.env, PATH
}`. On Windows `bunEnv` spreads `process.env`, whose key is `Path`, so
that object holds one variable under two spellings. `Bun.spawn` passes
both on, and which of the two the child ends up with changes with the
other variables. When `Path` wins, the directory the test added is not
there.
`bunEnv` now has the key `PATH` on Windows, so `{ ...bunEnv, PATH }`
replaces it. `test/preload.ts` copies `bunEnv` over `process.env` in
every test process, so the key is `PATH` there too. In `bunx.test.ts`
that also matters to the four tests that plant a decoy in `PATH` that
must not run: they pass without testing anything when `Path` wins.
`mergeWindowEnvs` in the harness exists for this, but a test has to
think of it. It stays for the envs that are not `bunEnv`, and its
comment no longer says that the first spelling wins.
What worked around the `Path` of `bunEnv` is gone: the deletions in
`bun-create.test.ts` and `bun-pm.test.ts`, the extra `Path` keys in
`lint/git.test.ts` and `bun-install-git-deps.test.ts`, the lookup of the
spelling in `bun-install-git-deps.test.ts`, and the `??
process.env.PATH` fallbacks in `bunx.test.ts`. The filters that drop
every spelling of `PATH` from an env (`bun-patch.test.ts`,
`bun-run.test.ts`, `bun-install-lifecycle-scripts.test.ts`) are still
right and stay.
`Path` is what a Windows user has, and so far every spawn with `env:
bunEnv` used it. Nothing else passes it, so `spawn-path.test.ts` gets a
test for it: a program that is only in a directory named by `Path` is
found, and a Bun child reads the value as `process.env.PATH`.
### How did you verify your code works?
**The cause**, measured on the Windows lanes of CI, from a throwaway
branch with main's harness. Inside that test, `bun x -p mypkg caller`
ran with the test's own env plus 0 to 40 more variables, and `cmd /d /c
set PATH` with the same env told which spelling comes later in the
child's block.
| 41 runs each | 2019 x64 | 11 aarch64 |
|---|---|---|
| as on main, `PATH` later: runs `caller` | 34 | 35 |
| as on main, `Path` later: `does not provide a binary named caller` | 7
| 6 |
| as on main, any other combination | 0 | 0 |
| `Path` deleted from the env, one key: runs `caller` | 41 | 41 |
`box.env` spelled the key `Path` on both lanes. Which variable was
different in build 127229 I do not know.
**Windows.** Build 127787 passed all 16 jobs of the two Windows lanes.
`spawn-path.test.ts` reports `1 pass, 4 skip, 0 fail` on both: the other
four tests of the file are skipped on Windows, so that one is the new
test. On Linux x64, where the new test is skipped and the change to
`bunEnv` is a no-op, `bunx.test.ts`, `bun-create.test.ts`,
`lint/git.test.ts`, `spawn-path.test.ts` and
`bun-install-git-deps.test.ts` pass with a release binary of main.
…ent.cpp (oven-sh#45764) ### Problem - 29 lines in 6 files have no user. - `source_file`, `compare_types` and `symbol_to_string` in the `queries!` list of `src/lint/types/mod.rs` have no caller. The list generates the only call of the `Services` functions of the same names in `bun_sema`. The trait is `pub`, so rustc does not check it. - `BunProcess.cpp` declares five `using X = JSC::X;` aliases that it never names. `DirEntType` in `NodeDirent.cpp` has three enumerators that nothing names. ### Fix - Delete the three list lines with their `Services` functions (21 lines), the five aliases and the three enumerators. - Correct because a word search over the sources, the tests and the generated code finds no user, and the workspace compiles for 12 targets. - Verified: `bun bd`, `bun run rust:check-all`, `cargo check --workspace --all-targets`, 8 test files (Notes). - Self-reviewed: 41 concerns raised, 33 addressed. Rejected: a `pub(crate)` narrowing of `Queries` (`resolve_name` needs a caller first), and a hold until earlier sweeps merge. ### Background - `queries!` lists each question that a lint handle (`Type`, `TsSymbol`) can ask the type checker. One line generates a `Queries` method and a forwarding call to `Services` in `bun_sema`. - `BunProcess.cpp` compiles as its own unit with `using namespace JSC;`, so the aliases change no name lookup. - A deletion of unreferenced code has one possible place, so no other design was weighed. ### Downsides - A later lint handle that needs one of the three queries must restore it. - None found for users. Checked: the heads of all 5,559 open pull requests, 12 targets, the tests in Notes. Behaviour change: none <details><summary>Notes</summary> **Removed** - `src/lint/types/mod.rs`: the `queries!` lines `source_file`, `compare_types` and `symbol_to_string`. No call of these names has a `Queries` receiver. The calls that exist go to other functions: `Checker::compare_types` and `Checker::symbol_to_string` in `bun_sema`, a private `compare_types` in `prefer_optional_chain.rs`, and the parser's `source_file`. With `Queries` made crate-private in a scratch edit, rustc reports `resolve_name` as the only query left that is never used. - `src/sema/check/services.rs`, `services/types.rs`, `services/symbols.rs`: `Services::source_file`, `Services::compare_types`, `Services::symbol_to_string`. The generated forwarding call was the only user. What they call (`Files::by_path`, `Checker::compare_types`, `Checker::symbol_to_string`, `Checker::prop_to_string`) keeps other callers. - `src/jsc/bindings/BunProcess.cpp`: the aliases `Exception`, `JSModuleLoader`, `JSModuleRecord`, `SourceOrigin`, `JSNonFinalObject`. None of the names appears in the rest of the file. The same block in `Path.cpp` and `ZigGlobalObject.cpp` is left to oven-sh#43378 and oven-sh#42431. - `src/jsc/bindings/NodeDirent.cpp`: the enumerators `Whiteout`, `Door` and `EventPort` of `DirEntType`, all equal to `Unknown`. No C or C++ file names them. **Open pull requests** - Compared at 2026-10-17 16:40 UTC on main 017b243 with the heads of all 5,559 open pull requests, drafts and every author included. - 156 of them touch a file of this diff (151 touch `BunProcess.cpp`). A trial merge of each with main and with this branch (`git merge-tree`) shows that the branch adds a conflict to none. - None deletes one of these lines. None adds a use of a removed name: the added lines were searched for the three query names, the five alias names in `BunProcess.cpp`, and the three enumerators. **Found dead in this run, and left alone** - Already gone on main: three lines in `JSSQLStatement.cpp` and `NodeSqlite.cpp` (oven-sh#45304). - Another open pull request deletes or rewrites them: 13 commented-out lines in uSockets, uWS, `JSEventTargetCustom.cpp` and `src/node-fallbacks/url.js` (oven-sh#45709, and oven-sh#43730 for `url.js`), two typedefs in `InspectorHTTPServerAgent.cpp` (oven-sh#45725), the `BUN_WEBKIT_VERSION` fallback in `BunProcess.cpp` (oven-sh#41729), a commented-out line in `src/js/internal/html.ts` (oven-sh#45682). - An open pull request uses them: `Event::currentTargetIsInShadowTree()` and the `MediaTime` overload of `JSConverter<IDLUnrestrictedDouble>` (oven-sh#45415). - Held by an earlier sweep for a reason that still holds: the QUIC setters `scid_len` and `delay_onclose` (oven-sh#43976, and oven-sh#40220 changes the line above them). - `resolve_name`, the fourth query with no caller. tsgolint's `prefer-readonly` calls `ResolveName` and Bun's port does not, which gives a wrong autofix for a global enum. The fix is handed off and needs the query. After it, `Queries` can become `pub(crate)`, and rustc then guards the list. - Mirrors of typescript-eslint and tsgolint: the write-only fields `ThenCall::on_fulfilled`, `FinallyCall::on_finally`, `BaseTypeOfClassMember::base_type` and `heritage_token`, `TargetElement::is_shorthand`, with `TsNode::token` (as oven-sh#45402 held them). Also 33 functions from macro lists in `src/lint/types/` that nothing calls, and `File::expr_tags_in_tree`. - Debugging aids: 25 hand-written `Debug` impls in `bun_lint`, 5 in `bun_format`, and `Display` for `Difference`, `FilePoll` and `Part`. - Handed off: the plugin `target` value that nothing reads (`BundlerPlugin::target`, three more members, a parameter of `JSBundlerPlugin__create`, and the Rust enum). Its removal changes an FFI signature. - Needs a decision: `PerformanceResourceTiming` cannot be constructed, so `ResourceTiming.cpp`, `NetworkLoadMetrics.cpp` and `ServerTiming.h` are reachable only from its getters. **How the candidates were found** - Ten text searches, one for each area: `src/js`, the C++ bindings (three parts), uSockets and uWS, the build scripts, the side packages, the three new crates, and commented-out code. - A relink of the debug build with `-Wl,--gc-sections`, with each dropped function mapped to its source line by `llvm-nm -l`: 517 C++ functions and 607 Rust function lines. Almost all are Windows or macOS code, release-only code, or are deleted by an open pull request. - For `bun_sema`: `pub` narrowed to `pub(crate)` on each item that no other crate names, in a scratch tree. rustc reported nothing new. - For 39 trait impls that the linker drops: delete all, compile for linux, windows and darwin and with `--all-targets`, put back each impl that an error names. What stayed deleted is a debugging aid. **Tests, debug build of this branch** (with `--timeout 600000`, because the lint and check tests need more than the default with a debug build) - `test/cli/lint/conformance.test.ts`: 3 pass. - `test/cli/lint/rules.test.ts`: 16 pass. - `test/cli/lint/native-rules.test.ts`: 166 pass. - `test/cli/lint/bun-rules.test.ts`: 100 pass. - `test/cli/check/check.test.ts`: 599 pass, 42 skip. - `test/js/node/process/process.test.js`: 188 pass, 5 skip. - `test/js/node/fs/dir.test.ts`: 23 pass. - `test/regression/issue/24129.test.ts`: 2 pass. </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 0 · the description declares no behaviour change, so there is no failing test to prove; the existing suite in CI is the check <!-- robobun:evidence:end -->
This PR adds `bun bake`. It is the first step of the new Bake design (https://github.com/alii/bake-rfc). `bun bake` serves the app in the working directory with Bun's dev server: - Every `.html` file in `app/pages/` is a page. Its path is its URL: `app/pages/docs/index.html` is `/docs`, and `app/pages/docs/install.html` is `/docs/install`. The `Bun.serve` route is percent-encoded, so `app/pages/über.html` is served at `/%C3%BCber`. - `app/server.ts` is optional. Its default export can have `routes`, `fetch` and `websocket`, as in `Bun.serve()`. Bake starts the server. The pages and the routes are `Bun.serve()` routes. `fetch` gets each request that no page and no route matched. `bun bake --check`, `--watch`, `--hot` and `--no-clear-screen` mean the same as before `bake`. `bun bake --check` (or `bun --check bake`) type checks the project first, as `bun --check` does for a package.json script, and does not start the server when there are type errors. With `bun --check --watch bake`, it type checks again after each change and starts the server when there are no errors. As for `bun --check`, `--hot` with `--check` runs as `--watch`, in both spellings. `bun bake` always runs the dev server, also with `NODE_ENV=production`. The production build is a later PR (`bun bake build`). `bun bake` is behind the Bake feature flag. It is on in canary and debug builds. In release builds, set `BUN_FEATURE_FLAG_EXPERIMENTAL_BAKE=1`. How it works - `bun bake` goes through the normal script lookup. When the project has no `bake` script, no `bake` file and no `bake` binary, the CLI starts the VM with the built-in module `src/js/internal/bake.ts` as the program. Before this PR, that lookup printed `Script not found "bake"`. So no project that runs today changes behavior. - `bun ./index.html` already starts the VM with a built-in module (`internal/html.ts`). The VM flag `main_is_html_entrypoint: bool` becomes the enum `MainEntryPoint { File, Html, Bake }`. `Bun__loadHTMLEntryPoint` becomes `Bun__loadBuiltinEntryPoint`, with a named enum on both sides of the FFI. The C++ file is renamed to `BuiltinEntryPoint.cpp`. - `bun bake` has no entry module, so `vm.main` becomes `Option<RawSlice<u8>>`, `None` for `bun bake`. `set_main`, `reload_entry_point` and `load_entry_point` take `Option<&[u8]>`, and `RunCommand::boot_main` takes `BootMain::File { path, loader }` or `BootMain::Bake`. Every reader of `vm.main` handles `None`: `process.argv` is `[bun, ...arguments]`, as for `bun -e`; `Bun.main` is `""`; `import.meta.main` is `false` in `app/server.ts`; the checks that compare a module with the entry find no match. An earlier version used an empty path for this, and on Windows the `Bun.main` getter opened `""`, which is the working directory. - In `reload_entry_point`, a built-in entry point is chosen before the other ways to start: the patched `Module.runMain` applies only to a user file, and a built-in entry point also runs when transpiling is off. Before this PR, both of these ran the main path in place of `internal/html.ts`. - A built-in entry point sets `has_loaded` before it runs. That flag means "the main module has started"; it was only set when a loaded module's path equals `vm.main`. Without it, a file of an unknown type that `app/server.ts` imports (`./icon.png`) was parsed as a script, and imports were never transpiled off-thread. - The first commit moves the server startup of `internal/html.ts` into `internal/serve_cli.ts` with no change: the flags, the port retry, the startup message and the keyboard shortcuts. `bun bake` uses the same code, so both commands behave the same. - `--check` in Bake mode type checks the project (`check_project_before`), because there is no entry module to start from. Under `--watch`, `Run::start` does the same check before each run (`Unchecked::Project`), and registers each file it reads with the watcher. - The rule that turns `--hot` into `--watch` for `--check`, and the Windows watcher-manager setup, move into `cli::watch_instead_of_hot_for_check` and `cli::set_up_watch_restarts`. The argument parser and the trailing `bun bake --check` both use them. - Changes to the shared code, which `bun ./index.html` gets too: `--port 3000` works as well as `--port=3000` (before, `bun ./index.html` ignored the first form); `--host` takes IPv6 addresses (`::1`, `[::1]:3000`); the `o` shortcut no longer stops the server when no browser opener is installed, and runs `cmd /c start` on Windows; under `--hot` the shortcut and signal listeners are added once, not once per reload. What `bun bake` rejects, with an error - A route in `app/server.ts` that has the URL of a page. - Two pages with the same URL, for example `docs.html` and `docs/index.html`. - An option in the default export other than `routes`, `fetch` and `websocket`. - A default export that is not an object. - A page with `[`, `:` or `*` in its path. - No pages and no `app/server.ts`. - An unknown argument, or a flag with no value. Not in this PR - `bake.config.ts` and plugin objects. Until then, `[serve.static] plugins` in `bunfig.toml` applies, because the pages are HTML imports. - `.tsx` pages, `bun bake build`, page data functions and ISR. These are the next PRs. Tests - `test/cli/bake/bake.test.ts` (new): pages, nested `index.html`, routes and `fetch` in order; `--port 0 --host localhost`; `--check` in both spellings with a type error, and with none; a `bake` script that takes precedence; each error above; `--help`. All fail on Bun 1.4.3 except the test that a `bake` script still runs, which is also true on 1.4.3. - `bun --check --watch bake` and `bun --hot bake --check` were checked by hand: with a type error the server does not start; after the fix, it starts. - Known limit: under `--watch`, adding `app/pages/` to a project that had nothing to serve does not restart the process, because the watcher restarts only on changes to files it loaded. `bun --watch missing.ts` behaves the same. - `test/js/bun/http/bun-serve-html-entry.test.ts`, `test/regression/issue/23569.test.ts`, `test/regression/issue/29298.test.ts` and `test/bake/app-options.test.ts` pass, for the `bun ./index.html` path. Docs are in a separate PR, to merge when Bake is on in a release (`ali/bake-docs`). --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…of cron.test.ts (oven-sh#45809) ### What does this PR do? Fixes the flaky "cron end-to-end (macOS) > force-triggered job receives correct controller properties" in `test/js/bun/cron/cron.test.ts`. It failed on the macOS 14 aarch64 lane in build 127287 of main and passed on the retry: ``` 1176 | while (!existsSync(markerPath)) await Bun.sleep(10); 1177 | 1178 | const output = JSON.parse(readFileSync(markerPath, "utf8")); ^ SyntaxError: JSON Parse error: Unexpected EOF ``` The job that launchd starts writes the marker with `writeFileSync`, and the test parses the file as soon as it exists. `writeFileSync` creates the file and then writes to it, so for a moment it exists and is empty. The job now writes to `<marker>.tmp` and renames that to the marker, so the marker appears with its content. ### How did you verify your code works? The test needs `launchctl`, so I have not run it myself. On CI's macOS aarch64 lane, in build 127827, the file has 88 pass, 0 fail and 29 skip; the blocks for Linux and Windows hold 29 tests, so the macOS ones ran. **The race**, on Linux x64 with the binary CI built for main (47c8687): a child process writes the same JSON, and the parent reads and parses the file as soon as `existsSync` sees it, polling without a pause to widen the window. | the child | reads that failed | |---|---| | `writeFileSync` to the marker, as on main | 1101 of 1500, all `JSON Parse error: Unexpected EOF` | | `writeFileSync` to `.tmp`, then `renameSync` | 0 of 1500 | **The job file**, as the test now writes it, run with `bun run --cron-title=t --cron-period="0 0 * * *"` on Linux: exit code 0, the marker holds the four fields and no `.tmp` is left. The rest of the file on Linux: 54 pass, 63 skip.
…RESTART (oven-sh#45811) ### What does this PR do? Fixes the two flaky "epoll EINTR retry accounts for elapsed time > ...: setTimeout fires on time under signal storm" tests in `test/js/web/timers/setTimeout.test.js`. One or the other failed and passed on a retry on the Debian 13 x64 ASAN lane in builds of 27 different branches between 127085 and 127706, always like this: ``` expect(filteredStderr).toBe(""); + "==29314==WARNING: ptrace appears to be blocked (is seccomp enabled?). LeakSanitizer may hang. + ==29314==Child exited with signal 42." ``` The fixture preloads a library that delivers `SIGALRM` every 20 ms to a handler installed with `sa_flags = 0`, so that `epoll_pwait` keeps returning `EINTR`. The storm is still on when the process exits, which is when LeakSanitizer runs. Before it stops the world it forks a child to see whether `ptrace` is allowed ([`TestPTrace`](https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_stoptheworld_linux_libcdep.cpp)): ```cpp int wstatus; internal_waitpid(pid, &wstatus, 0); // Handle SCMP_ACT_KILL if (WIFSIGNALED(wstatus)) { ``` The result of `waitpid` is not checked. When a `SIGALRM` interrupts it, `wstatus` is never written, and the warning is about whatever was on the stack. Nothing sends signal 42. The handler now has `SA_RESTART`. The kernel then restarts `waitpid`, and it never restarts `epoll_wait` and its variants ([signal(7)](https://man7.org/linux/man-pages/man7/signal.7.html)), which is what the test is about. ### How did you verify your code works? Linux x64. **The failure**, with the ASAN binary CI built for 5b468a6 and the `ASAN_OPTIONS`, `LSAN_OPTIONS` and `BUN_DESTRUCT_VM_ON_EXIT` that `scripts/runner.node.ts` sets: | | main | this PR | |---|---|---| | the two tests, runs in which one or both failed | 25 of 25 | 0 of 25 | | the fixture alone, runs with the warning, `epoll_pwait2` | 14 of 60 | 0 of 60 | | the fixture alone, runs with the warning, `epoll_pwait` fallback | 5 of 60 | 0 of 60 | **`epoll_pwait2` is interrupted as before.** `strace` of the fixture on a release binary of main (47c8687): 10 of its 12 calls return `EINTR` with `sa_flags = 0`, and 10 of 12 with `SA_RESTART`. **The bug it was written for.** Bun 1.3.14 does not have oven-sh#34780. When the fixture's 200 ms timer fires, 20 runs per cell (the test wants 200 to 500 ms): | | `sa_flags = 0` | `SA_RESTART` | |---|---|---| | 1.3.14 | 993 to 995 ms | 993 to 995 ms | | main, `epoll_pwait2` | 200 ms | 200 to 201 ms | | main, `epoll_pwait` fallback | 200 to 201 ms | 200 to 201 ms | The whole file passes on the release binary of main (37 pass). **In CI**, build 127831: the file passed on the Debian 13 x64 ASAN lane at the first attempt (37 pass, 0 fail).
…e worker's dispose() has returned (oven-sh#45813) ### What does this PR do? Fixes a flaky cell of `test/js/bun/module-graph/module-graph-workers.test.ts`: "B: a worker hosts the graph > the graph has a child process running, in a node Worker: the worker is woken to dispose the graph, then terminate()". The file failed and passed on a retry in builds of 29 different branches between 126996 and 127701, 25 of them on the Debian 13 x64 ASAN lane. The 8 I read are all this cell, and all end the same way: after 30 s the fixture's watchdog prints `"stillRunning": true`, with `"worker": "exited"` as the last thing done. The next thing the fixture does is wait for the graph's child process to be gone, which it does if the graph was disposed: ```js result.childGone = !main.disposed() || (await main.pidGone(...)); ``` `disposed()` reads a flag that the worker sets before it disposes, because it comes with the counters that later ones are compared to: ```js dispose: () => { snapshot(); graph.dispose(); }, ``` In this cell the main thread wakes the worker and calls `terminate()` in the same turn. When that stops the worker between the two calls, the flag is set, nothing has killed the child, and the fixture waits for it until the watchdog. The worker now sets a second flag once `dispose()` has returned, and the wait for the child goes by that one. The order that disposes an inner graph took the snapshot and disposed by itself; it now goes through the same `control.dispose`, so it sets the flag too. ### How did you verify your code works? Linux x64, that cell alone, 16 at a time. Release is the binary CI built for main (47c8687), ASAN the one for 5b468a6. Runs that failed: | | main | this PR | |---|---|---| | release | 41 of 3000 | 0 of 3000 | | ASAN | 10 of 400 | 0 of 400 | | release, with a 5 ms busy loop between `snapshot()` and `graph.dispose()` | 258 of 1500 | 0 of 3000 | | release, with a 5 ms busy loop after `graph.dispose()` | 14 of 1500 | 0 of 3000 | The third row is the window made wider. The fourth gives `terminate()` 5 ms to land right after a `dispose()` that returned: the child was gone every time, so this is the fixture's flag and not `dispose()`. In `JSModuleGraph::dispose` the context is stopped, which ends the child, before the one step that can run JavaScript. (The two rows with a busy loop are of the first commit.) **The wait still happens.** The two flags in 2000 runs of the cell: | snapshot taken | `dispose()` returned | runs | |---|---|---| | no | no | 1708 | | yes | no | 136 | | yes | yes | 156 | **A `dispose()` that kills nothing still fails.** The 27 cells with a child process, with a `dispose()` call taken out of the fixture, three runs each. Cells that failed: | taken out | main | this PR | |---|---|---| | the graph's | 13, 12, 12 | 12, 12, 12 | | the inner graph's | 1 | 1, 1, 1 | The whole file passes on the release binary (479 pass).
…olation.test.ts (oven-sh#45821) ### What does this PR do? Fixes the way "a server whose HTML routes were still being built is released with it" in `test/js/bun/module-graph/module-graph-isolation.test.ts` fails on Windows. The file failed and passed on a retry on the Windows 2019 x64 lane in builds of 8 branches between 127026 and 127507. Of the 4 I read, 3 are this test, each time after 30 s: ``` - "exitCode": 0, - "stdout": "collected", + "exitCode": 137, + "stdout": "", ``` The fixture asks a server for six HTML routes, and disposes the graph once it sees a request that is waiting for its build: ```js await until(() => server.pendingRequests >= 1); ``` `until` looks once per turn of the event loop. oven-sh#45816 ran a copy of the fixture that reports where it is after 10 s, and it was there on both Windows lanes of build 127847: ``` STUCK: use 3: waiting for a pending request, pendingRequests 0, polls 3534993, sockets {"open":6,"data":6,"close":0,"error":0,"rejected":0} STUCK: use 2: waiting for a pending request, pendingRequests 0, polls 6509681, sockets {"open":6,"data":6,"close":0,"error":0,"rejected":0} ``` All six requests had been answered. They arrived and their builds finished between two looks, so the count went from 0 to 6 and back unseen, and it stays 0. The wait now also ends when every request has an answer. A server that was not disposed during a build shows nothing, so it does not count towards the three, and the fixture says so on stderr. This is not the `kept 1` that the same test gives on the macOS x64 lane. ### How did you verify your code works? **Windows 11 x64**, 24 cores, with a binary CI built from main's code (f1fc570). That test alone, main's file and this PR's in alternating rounds, 48 at a time for 1500 runs of each and 96 at a time for 6000: | | runs | exit 137, no output | servers whose builds were all done unseen | |---|---|---|---| | main | 7500 | 6 | | | this PR | 7500 | 0 | 4 | The four are runs that printed the new line, made another server, and passed. **Linux x64**, with the binary CI built for main (47c8687). To miss the builds on purpose, the third server's wait starts 200 ms late. The two tests that run this fixture: | | main | this PR | |---|---|---| | "a server whose HTML routes were still being built is released with it" | exit 137, no output | pass | | "and the plugins those builds borrowed from it stay the server's" | exit 137, no output | pass | With every server's wait 200 ms late, the fixture is killed after 30 s as before, with "every build was done before a turn saw one" 148 times in the output. As it is: the two tests pass 300 of 300 times on that binary and 60 of 60 on the ASAN one (5b468a6), and the whole file passes (413 pass). **A server that is kept still fails.** With every server the fixture makes pushed to a global array, both tests fail with `kept 3`. In CI: the file passed at the first attempt on both lanes in build 127869 (the first commit).
…est of react-compiler.test.ts (oven-sh#45808) ### What does this PR do? Fixes the flaky "react-compiler time does not grow with the cube of a chain of functions" in `test/bundler/transpiler/react-compiler.test.ts`. It failed in the parallel batch of a Windows lane in two builds of main, 125476 (11 aarch64) and 125686 (2019 x64), and passed alone. Both times: ``` Expected: < 8 Received: 8 ``` The test divides the CPU time of compiling a chain of 2n functions by that of n. On Windows `proc.resourceUsage().cpuTime` comes from `GetProcessTimes`, which counts in ticks of 1/64 s (15625 µs). That is the kernel's accounting and has nothing to do with the resolution of timers, which oven-sh#34834 made fine. The smaller compile takes a tick or two, so the quotient there is one of two small whole numbers. Exactly 8 is 8 ticks over 1. In a release build on Windows the two files now hold eight such components instead of one. The work per file is eight times as much and the ratio is the same, so a tick more or less no longer decides it. The other platforms, and a debug build on Windows, compile what they did. ### How did you verify your code works? **Windows 11 x64**, with the binary CI built for this branch (f1fc570, which is main's code). The CPU time of 78 child processes (30 that spin for 0 to 140 ms, 48 compiles): all 234 values of `user`, `system` and `total` are multiples of 15625 µs. A process that spins for 7 ms reads as 15625, one that does nothing as 0 or 15625. The test's two compiles, 100 rounds per row: | components | n | 2n | ratio | not `< 8` | |---|---|---|---|---| | 1, as on main | 1 to 6 ticks | 3 to 9 ticks | 0.67 to 9.00 | 1 of 100 | | 4 | 2 to 6 ticks | 12 to 18 ticks | 2.17 to 7.50 | 0 of 100 | | 8, this PR | 4 to 12 ticks | 22 to 31 ticks | 2.00 to 6.00 | 0 of 100 | In 12 earlier rounds with one component, n once read as 0 ticks, which makes the ratio `Infinity`. **In CI**, build 127826: the file passed on both Windows lanes with eight components. It took 39.33 s on 2019 x64 and 46.21 s on 11 aarch64, against 37.54 s twice and 43.61 to 44.43 s in two builds of the same hour without this change. **Linux x64**, release binaries, 12 runs per row, for what the compiler fix is worth with eight components: | | components | n | 2n | ratio | |---|---|---|---|---| | main (47c8687) | 1 | 16.9 to 23.3 ms | 52.8 to 75.9 ms | 2.3 to 3.8 | | main (47c8687) | 8 | 80.4 to 97.6 ms | 359.7 to 431.2 ms | 4.1 to 4.6 | | Bun 1.4.3, which does not have the compiler fix | 1 | 349.5 to 371.3 ms | 3754.8 to 4729.5 ms | 10.6 to 13.4 | | Bun 1.4.3 | 8 | 2753.6 to 2906.5 ms | 30235.3 to 37826.6 ms | 10.8 to 13.3 | The test alone, with main's binary: 10 of 10 pass as it is, and 10 of 10 with eight components forced on Linux. On 1.4.3 it fails 3 of 3 (10.4 to 10.8). With eight components a Bun without the fix would need over 30 s for the larger file, so on Windows the test would then fail by its timeout.
…nd `strategy` in `gzipSync` and `deflateSync`, and pass dictionaries to Rust by reference (oven-sh#45794) ### What does this PR do? 1. Adds a test: `packages/bun-types` promises nothing that a schema of the bindgen breaks. 2. Fixes what it found. For `Bun.gzipSync` and `Bun.deflateSync` that is at run time: `memLevel` and `strategy` did nothing. 3. Fixes how the output of those two grows, which those options make reachable. 4. Hands a dictionary of the bindgen to its Rust body by reference. The bindgen already writes `Bindgen.d.ts` from the schemas. The new case in `test/integration/bun-types/bun-types.test.ts` generates it and compares it with the published types through the TypeScript compiler API, which that file already uses: - **Arguments.** Every call the published types allow has arguments the schema takes. An argument they leave out is `undefined`. Where the schema has overloads, one of them has to take the call. - **Results and properties.** Each one the published types name is one the schema gives. They may be narrower: `Uint8Array` where the schema says a buffer, `-1 | 0 | 1` where it says a number. Not by leaving out `null` or `undefined`, which the schema says only of a result that can be that. `void` promises nothing. - **Members.** A dictionary or a class of the schema lists all it has, so no published object has a member it lacks: an option that nothing reads, a member of a result that nothing gives, a method that is not there. - **Names.** Every export of the schema is published. - **Classes.** The constructor, the static members and the prototype, each in the same way. That is 66 exports of 15 of the 22 schema files. Of the other 7, JavaScript wraps 1 (`node:os`), only tests reach 3, 1 has `module._stat`, which is not published, and 2 have only types. It compares types, so it does not see what a body does with a member that its dictionary has. `windowBits` of `Bun.gzipSync` and `Bun.deflateSync` is converted and not used, and so are `level`, `memLevel` and `strategy` of `Bun.gunzipSync` and `Bun.inflateSync`. The schema says so of each. All four are as on `main`, and this PR leaves them. The test runs in `.github/workflows/bun-types.yml`, which now also runs when a `.bind.ts` file or `src/codegen` changes. `PUBLISHED_AS` in the test says what the published types call the exports of each schema file, one line per file, or `null`. A schema file that is not listed fails the test, and so does a listed one that `Bindgen.d.ts` does not have, so none goes unchecked. #### The generated types say what an argument takes They said what it is meant to be given, so `Bun.semver.satisfies({ toString() {} }, …)` looked refused when it is not. | Schema | Was | Now | | --- | --- | --- | | `b.string.loose`, `b.string.loose.utf8`, `.owned` | `string` | `any` | | `b.bool.loose` | `boolean` | `any` | | `b.i32.loose`, `.wrap`, `.enforceRange()`, and the other integers | `number` | `any` | | `b.f64.loose`, `.allowBigInt` | `number`, `number \| bigint` | `any` | | `b.null` in a union, which takes `undefined` too | `null` | `null \| undefined` | `.validateInteger()` refuses what is not a number, so it stays `number`. This changes only `Bindgen.d.ts`, in 63 lines. #### A dictionary is handed to its Rust body by reference The thunk made the struct and handed it over by value. The structs were `Copy`, and clippy refuses a `Copy` type over 64 bytes by value, so a body whose options grew past that needed `#[allow(clippy::large_types_passed_by_value)]`, and `docs/project/bindgen.mdx` said so. Three had it: `CSRF.verify`, `inflateSync` and `gunzipSync`. The struct now stays in the frame of the thunk and the body takes `options: &ZlibCompressionOptions<'a>`. A generated dictionary is not `Copy`, since nothing copies it, and neither is a union that holds one. So no lint about the size of a `Copy` type has a say at either end: `BracesOptions` is 2 bytes, and clippy refuses a `Copy` type of 8 bytes or less by reference. No `allow` of either lint is added, and the three are gone. That is for an argument that is a dictionary, with `.default()` or without, Inside a union, as for `Glob#scan` and `password.hash`, inside `.optional` or in an array it is held by value. Against the generator of `main` (c814409) on the same schemas: 15 calls pass `&options`, 17 types lose `#[derive(Clone, Copy)]` (16 dictionaries and the union of `Glob#scan`), and nothing else differs. That is in the 2 generated Rust files; the other 47 generated files are byte for byte the same. The 15 bodies change their signature, and so do the two helpers they hand the options to. Nothing else changes but what follows from that: `inflate_sync` and `gunzip_sync` pass `options` on where they passed `&options`, and one `match` in the test bindings borrows a nested dictionary that it moved out. #### What it found Each is the same in 1.4.3 as on `main`. | Published | At run time | Now published | | --- | --- | --- | | `Bun.mmap(path: PathLike)` | a `Buffer`, an `ArrayBuffer` and a `URL` throw | `path: string` | | No `byteLength` on an instance of `MD4`, `MD5`, `SHA1`, `SHA224`, `SHA256`, `SHA384`, `SHA512`, `SHA512_256` | `new Bun.MD5().byteLength` is 16 | `readonly byteLength: number` | | `Bun.hash(data, seed?): number \| bigint` | always a `bigint` | `bigint` | | `memLevel` and `strategy` of `Bun.gzipSync` and `Bun.deflateSync` | the same bytes whatever they are | as it was: they now work | #### `memLevel` and `strategy` Published since 2022 and described in `docs/runtime/utils.mdx`. The wrapper around zlib had both fields and gave them to `deflateInit2_`; `gzip_or_deflate_sync` never filled them in. The schema's dictionary now has the two members and the body passes them on. Each is an integer in its range, `b.i32.validateInteger(1, 9)` and `(0, 4)`: anything else is refused by name and never turned into another value, so `{ strategy: "Z_RLE" }` is not the default strategy. | `Bun.gzipSync(data, …)`, 29880 bytes | `main` | Now | | --- | --- | --- | | no options | 19183 bytes | 19183 bytes | | `{ strategy: 2 }` | 19183 | 19191 | | `{ strategy: 3 }` | 19183 | 19192 | | `{ memLevel: 1 }` | 19183 | 24039 | | `{ memLevel: 9 }` | 19183 | 19409 | | `{ level: 9 }` | 20016 | 20016 | | `{ level: 9, strategy: 1 }` | 20016 | 19194 | libdeflate has neither, so with `library: "libdeflate"` they change no byte. The JSDoc of `strategy`, and its copy in `docs/runtime/utils.mdx`, named five constants and not where they are. It now gives the number of each, says they are `constants` of `node:zlib`, and says that any other value throws. The type stays `number`: `@types/node` has `Z_RLE: number`, which `0 | 1 | 2 | 3 | 4` would refuse. #### The output can be longer than `deflateBound()` The output starts with the room `deflateBound()` asks for: in zlib-ng the input, an eighth of it, 3 bytes and the wrapper, whatever `memLevel` and `strategy` are. `{ memLevel: 9, strategy: 4 }` can need more. `Z_FIXED` has no tree of its own, and a block of 32767 symbols that the window has slid past cannot be stored, so bytes from 144 up cost 9 bits each and every block 10 more. `ZlibCompressorArrayList::read_all` on `main` has a branch for that, and it is wrong in three ways: - The length of the `Vec` is still 0 there. Reserving 4096 more does nothing, zlib is handed the start of the buffer again, and the length is then set past the capacity. Nothing reaches it on `main`, where both options are the defaults, but for the next. - The room is given to zlib as `capacity as uInt`, the low 32 bits of it. From an input of about 3.82 GB the bound is over 4 GiB, so zlib gets a small part of the buffer and that branch has to carry on. - The place in the output is read from zlib's `total_out`, a `c_ulong`, which is 32 bits on Windows. It now does what `step` in that file does. Each pass hands zlib the spare capacity of the `Vec`, at most `u32::MAX` of it, and adds what zlib used of it to the length. The length is the only record of the place, and it grows when there is no spare capacity. `src/zlib/lib.rs` is 34 lines shorter. | `Bun.deflateSync(input, { memLevel: 9, strategy: 4, level: 2 })`, 168099840 bytes | | | --- | --- | | `deflateBound()` | 189112323 | | `node:zlib`, which grows its output | 189112331 | | The loop of `main` with the two options passed on, on a debug build | `panic: Vec::set_len requires that new_len <= capacity()` | | Now | 189112331, and it decompresses to the input | An input has to be that long. The first block is always stored, which leaves 4092 bytes of the bound to spare, and each block after it uses up less than one: 5120 blocks of 32768 bytes are the fewest that pass it. ### Behaviour `Bun.gzipSync` and `Bun.deflateSync`, each counted once per row. 1.4.3 does what the `main` column says in every row. | | Now a bug | Now not a bug | | --- | --- | --- | | **A bug on `main`** | 0 | 12 | | **Not a bug on `main`** | 0 | 6 | ##### bug -> bug None. ##### bug -> non-bug (12) The third, fourth and fifth rows are breaking changes, and deliberate ones: a call that returned now throws. | Call | `main` | Now | | --- | --- | --- | | `{ strategy: 1 }` to `4` | the bytes of the default strategy | the bytes of that strategy. `1` differs from the default at `level` 7 to 9 | | `{ memLevel: 1 }` to `9` | the bytes of `memLevel: 8` | the bytes of that level | | `{ memLevel: 0 }`, `10`, `-1`, `2 ** 32 + 1`, `1.9`, `NaN`, `Infinity` | returns | `RangeError` `ERR_OUT_OF_RANGE`: The value of "ZlibCompressionOptions.memLevel" is out of range. It must be >= 1 && <= 9. Received 10. For the last three: It must be an integer. Received 1.9 | | `{ strategy: 5 }`, `-1`, `1.9`, `NaN`, `Infinity` | returns | the same for `strategy`: It must be >= 0 && <= 4, or an integer | | Either one as `null`, `true`, `"9"`, `1n`, `[2]`, `{ valueOf: () => 2 }`, `Symbol()` | returns | `TypeError` `ERR_INVALID_ARG_TYPE`: The "ZlibCompressionOptions.memLevel" property must be of type number. Received null | | 3900000000 bytes, `{ level: 0 }` | passes 20 GB of memory, where it was killed | returns, and decompresses to the input | ##### non-bug -> bug None. ##### non-bug -> non-bug (6) | | Call | `main` | Now | | --- | --- | --- | --- | | Stricter | `{ library: "libdeflate", memLevel: 10 }`, or `null`, or `Symbol()` | returns | the errors above: the dictionary converts every member, whichever the library | | Reads more | any options object | reads `windowBits`, `library`, `level` | then `memLevel`, `strategy`, so a getter of either runs, and does what one of `level` does: what it throws is thrown, and an input it detaches is empty | | Other error | `{ library: "nope", memLevel: {} }`, `{ level: 99, memLevel: 99 }` | the error about `library`, or `level` | the error about `memLevel`: every member is converted before the body looks at one | ### How did you verify your code works? The types test passes, and fails without each of the three changes to `bun.d.ts`. Each of these breaks, made one at a time, fails it with the message shown: | Break | Message | | --- | --- | | Schema: `semver.satisfies` becomes strict | `SemverObject.bind.ts: satisfies(version: StringLike, range: StringLike): boolean: version takes string, not StringLike` | | Published: `which()` returns `number` | `BunObject.bind.ts: which() is string \| null, not number` | | Published: `which()` returns `string`, without `null` | `BunObject.bind.ts: which() is string \| null, not string` | | Published: `sleepSync` takes a string too | `BunObject.bind.ts: sleepSync(ms: string \| number): void: milliseconds takes number, not string \| number` | | Published: the argument of `sleepSync` becomes optional | `BunObject.bind.ts: sleepSync(ms?: number \| undefined): void: milliseconds takes number, not number \| undefined` | | Published: `Glob#match` takes a number too | `Glob.bind.ts: new Glob().match(str: string \| number): boolean: str takes string, not string \| number` | | Published: `CSRF.generate`'s `expiresIn` takes a string too | `csrf_jsc.bind.ts: generate(…): options takes CSRFGenerateOptions \| null \| undefined, not CSRFGenerateOptions \| undefined` | | Published: `WhichOptions` gains `all?: boolean` | `BunObject.bind.ts: which(command: string, options?: WhichOptions \| undefined): string \| null: options has no all` | | Schema: the dictionary of `gzipSync` loses `strategy` | `BunObject.bind.ts: gzipSync(…): Uint8Array<...>: options has no strategy`, and the same for `deflateSync` | | Published: `dns.getCacheStats()` gains `evictions: number` | `dns.bind.ts: getCacheStats() has no evictions` | | Published: `CryptoHasher` gains `reset()` and `static fromState()` | `CryptoHasher.bind.ts: CryptoHasher has no fromState`, `new CryptoHasher() has no reset` | | Published: `unsafe.memoryFootprint()` returns `number`, without `undefined` | `UnsafeObject.bind.ts: memoryFootprint() is number \| undefined, not number` | | Published: `indexOfLine` is gone | `BunObject.bind.ts: indexOfLine is not published` | | A new schema file that is not listed | the list of files differs | | Generator: `Bindgen.d.ts` names the files otherwise | `TypeError`, where nothing would have been compared | | Generator: a type is spelled with a name that does not resolve | `Cannot find name 'Strng'. Did you mean 'String'?` | Making the first argument of `which` optional in the published types does not fail it, rightly: the schema has it optional, and `Bun.which()` is `null` on `main`. **By reference.** `cargo clippy` passes on `bun_runtime` and `bun_jsc`. On a debug build, the tests of all 15 functions pass: `test/internal/bindgen.test.ts` (117), `zlib.test.js` (457), `zstd.test.ts` (100), `csrf.test.ts` (40), `which.test.ts` (8), `mmap.test.js` (26), `brace.test.ts` (47), `os.test.js` (59), `open-in-editor-gc.test.ts` (4). **`memLevel` and `strategy`.** 39 new tests in `test/js/node/zlib/zlib.test.js`. For each option the bytes of the 29880 are those `node:zlib` gives for it, are not those without it, and decompress to the input. 35 of them fail on 1.4.3 and on `main`; the other 4 say that libdeflate ignores both and that the defaults are 8 and 0. The file passes on a debug build: 457 pass, 2 skip. **The output longer than the bound.** One of the 39 is the call in the table above. It asserts that the output is longer than the bound, so it fails if it stops reaching the branch. Without the step that grows the `Vec` it is the only one of the 39 that fails, with `buffer error`. The same work through `node:zlib` takes 1.6 s and 564 MB on the release binary of `main`. It has a timeout of its own only on a debug build, as `leak.test.ts` beside it does. **The input over 3.82 GB.** By hand, since it takes a 3.9 GB input and as much output: both functions, each under a limit of 20 GB of memory, on 1.4.3, on `main` and on a debug build of this branch. The first two are killed at the limit, and the third returns 3900297555 bytes (`deflateSync`) that decompress to the input. 3700000000 bytes are right on `main` and on this branch. 4294967295 bytes, whose output is too long for an `ArrayBuffer`, throw `RangeError: Out of memory`. There is no test of it, and nothing of it was run on Windows. **The same bytes.** 960 calls with neither option, on `main` and on a debug build of this branch: both functions, no options and each `level` from -1 to 9, 20 lengths from 0 to 5000011 around the sizes of blocks, of bytes that compress and of bytes that do not. The length, the length of the `ArrayBuffer` and the CRC of each are the same. **The cost of a call.** Two more members are read from an options object. Instructions in user mode for each call, on 16 bytes so that the call itself shows, from the release binaries of CI for `main` (c814409) and for this branch (86e3fe9, which has the loop of `main` with the length set before it grows; the dictionary and the thunk are those of the head). The same binary measured twice differs by at most 20. | | `main` | Now | | | --- | --- | --- | --- | | `gzipSync(data)` | 19875 | 19876 | +1 | | `gzipSync(data, {})` | 20149 | 20291 | +142 | | `gzipSync(data, { level: 1 })` | 9566 | 9722 | +156 | | `gzipSync(data, { library: "libdeflate" })` | 38340 | 38479 | +139 | | `deflateSync(data)` | 19540 | 19559 | +19 | | `deflateSync(data, {})` | 19822 | 19969 | +147 | | `deflateSync(data, { level: 1 })` | 9257 | 9424 | +167 | | `deflateSync(data, { library: "libdeflate" })` | 38246 | 38401 | +155 | **The matrix.** 792 calls of the four functions: 37 values for each of `memLevel`, `strategy`, `level` and `windowBits`, and every `level` with every `strategy`, through a `Proxy` that records what is read, on 1.4.3, on `main` (c814409, the release binary of its CI build) and on a debug build of this branch. 310 are identical, among them every call of `gunzipSync` and `inflateSync`. A script puts each of the other 482 in exactly one row above and fails on one it does not know.
…oven-sh#45711) ### What does this PR do? Follow-up fixes to oven-sh#45304, and three older bugs that were found on the way. Each is something that an ordinary program can run into. #### Regressions of oven-sh#45304 **A pool that the server refuses more connections dialed again for every caller that had to wait.** Since oven-sh#45304 a pool that lost connections grows back: a caller that has to wait makes the pool dial, unless a dial in flight covers it. When `max` is above the number of connections the server has left (several processes whose `max` adds up to more than `max_connections`), those dials are refused, nothing remembered that, and every waiting caller cost another refused login: a forked backend and a FATAL line in the server's log each. The client paid too: it got through 40 % of the queries. | A real PostgreSQL with 7 free slots, `max: 20`, a 20 ms query every 2 ms for 5 s | before oven-sh#45304 | after oven-sh#45304 | |---|---|---| | refused logins | 13 | about 1,750 | | queries issued | about 2,310 | about 920 | With no connection ready (none, all reserved, only dials in flight) the pool still dials until `max`, and reads no clock and no memory there. Only the growth beside a ready connection is limited now: back to the most connections the pool has had open since a dial last failed, and by one more no sooner than 20 ms after that failed dial. That pause doubles with each such try, up to 1 second; the refusals of a pool's start do not count. A login ends the pause, and if somebody still waits the pool asks for one more connection at once. So a caller that finds nothing ready is dialed for as before oven-sh#45304, and the clock can only add a dial. On a release build: 17 to 22 refused logins and about 2,390 queries in that set-up. What oven-sh#45304 gained stays. With `max: 10` under a load that needs all 10: | | before oven-sh#45304 | after oven-sh#45304 | now | |---|---|---|---| | 9 of 10 connections cut, one still usable | 1 connection for ever, median latency 16 to 21 s | 10 again in about 30 ms | 10 again in about 30 ms | | all cut, the server lets the first login in and tells nine "starting up" for 300 ms | 1 connection for ever | 10 in 304 ms, 1,125 dials | 10 in 334 ms, 31 dials | | four `reserve()` calls queued behind one busy connection, a lost one is replaced, nobody new arrives | 0 of 4 served | 4 of 4 | 4 of 4 | | 500 `reserve()` calls at once on a cold pool of `max: 100`, room for 1 at the start and for all 20 ms later | 25.1 s on 1 connection | 25.1 s on 1 connection | 1.0 s on 49 connections | An earlier commit of this pull request (646e259) damped the storm with a pause that skipped dials, which could leave a query unanswered although the server had room. A release build and three CI lanes showed it. That mechanism is gone. Another (6f7bdef) let a login start the pool again when its callers were `reserve()` or `begin()`: 40 to 120 dials where 20 were due. The commit behind it limits a login to one more connection. **A burst of `reserve()` or `begin()` cost the square of its size.** When the oldest waiter was a reservation that could not be served, every call walked the whole queue in search of a plain query to serve. | User CPU per caller, release builds, `max: 5`, a burst of | 1,000 | 4,000 | 16,000 | 64,000 | |---|---|---|---|---| | `reserve()`, before oven-sh#45304 | 38 µs | 32 | 27 | 20 | | `reserve()`, after oven-sh#45304 | 41 µs | 42 | 82 | 222 | | `begin()`, before oven-sh#45304 | 65 µs | 61 | 51 | 44 | | `begin()`, after oven-sh#45304 | 113 µs | 77 | 111 | 219 | A second measurement, on another machine's release builds, with the fix (the median of 3 interleaved rounds): | | 1,000 | 4,000 | 16,000 | 64,000 | |---|---|---|---|---| | `reserve()`, before oven-sh#45304 | 32 µs | 20 | 16 | 12 | | `reserve()`, after oven-sh#45304 | 44 µs | 28 | 55 | 136 | | `reserve()`, now | 39 µs | 16 | 18 | 18 | | `begin()`, before oven-sh#45304 | 41 µs | 30 | 30 | 27 | | `begin()`, after oven-sh#45304 | 50 µs | 29 | 57 | 164 | | `begin()`, now | 34 µs | 27 | 27 | 29 | Behind as many waiting reservations as the pool has connections nobody can be served, so the walk stops there: 11 steps per caller at any size, where it was as many as there were callers. The order of service does not change. Plain queries were not affected (9 to 26 µs after oven-sh#45304, 16 to 23 before). **A query that is only started ended the process when it was refused before it had a connection.** A query that is started with `.execute()` and that nobody handles does not end the process when it fails. oven-sh#45304 kept that only for a query that had reached a connection. Three refusals that are new since oven-sh#45304 were reported as unhandled rejections where the same program used to be silent: - `close({ timeout: 0 })` right behind the query, which now closes at once. A last `` sql`insert ...`.execute() `` before `await sql.close({ timeout: 0 })` made the process exit with code 1. - A statement on a released reservation, or on an ended transaction or savepoint, which is now refused instead of running on a connection that the handle no longer owns. - `query.cancel()` before `query.execute()`. They are silent again. What was reported before still is: a refused or timed out dial, a refused login, a pool that is already closed, a `cancel()` while the query waits for a connection, and whatever a `then()` or `catch()` handler throws, and a query that is still waiting for a connection when the time of `close({ timeout })` runs out. Only `close({ timeout: 0 })`, where the caller says to drop what is queued, is silent. **The "Unsupported protocol" error could repeat part of the connection string.** It names what stands in front of `://` only if that has the form of a scheme. #### Older bugs **`bun:sqlite`: a Worker that was terminated while it wrote left a full-text index without the rows of its table.** With `INSERT`, `UPDATE` or `DELETE ... RETURNING`, SQLite makes all the changes before it returns the first row. A `worker.terminate()` that fell into that step was raised behind it, and the call returned without ending the statement; the end of the thread then rolled back what the index held and committed the rest. No callback is involved, and every way of running the statement was hit: `all()`, `get()`, `values()`, `raw()`, `iterate()` in 1.4.3 too, `run()` since oven-sh#45304; `db.query()`, `Bun.SQL` with the SQLite adapter, FTS5, FTS4 and FTS3. | a Worker in a loop of one-row `insert ... returning id`, terminated a few milliseconds later; files with a row that `MATCH` does not find | 1.4.3 | `main` | now | |---|---|---|---| | `st.get()`, a table whose triggers feed an external-content FTS5 index | 43 of 180 | 46 of 180 | 0 | | the same through kysely (`insertInto(...).returning('id')`) | 53 of 180 | 42 of 180 | 0 | | the same through `Bun.SQL` | 15 of 60 | 14 of 60 | 0 | | one statement of 2,000 rows through kysely | 49 of 120 | 39 of 120 | 0 | Nobody is told: the Worker's exit code is as usual, and for an index that triggers feed `PRAGMA integrity_check` says "ok". Without `RETURNING`, inside a transaction, or through `db.run(sql)` and `db.exec(sql)` it did not happen; `INSERT INTO ft(ft) VALUES('rebuild')` mends a file that has it. A timeout of `node:vm` in the main thread took the same way: the call threw `ERR_SCRIPT_EXECUTION_TIMEOUT`, the write stayed on its row, and every other connection was told "database is locked", for a read as for a write, until something ended the statement. A run now ends with the call that makes it, however that call is left: a destructor resets the statement, as in `node:sqlite`, in place of the resets that were written out before some of the returns. Outside a transaction that reset is the commit, so the rows of a call that threw are in the file, whole, as in `node:sqlite` and as with a row that `all()` cannot convert. Inside `BEGIN` the transaction stays the user's. `iterate()` streams as before, so a write that is read row by row stands on a row between two `next()` calls on purpose. A Worker that is terminated there, or an iterator that is dropped half-read, is as on `main`. The loop that collects the rows of `all()`, `values()` and `raw()` defines the elements of its array where it used `push()`, so that no script (an indexed setter on `Array.prototype`) can finalize the statement in the middle of a run; the check for that goes. The array is contiguous from the start, which makes a call about 110 instructions cheaper (-0.55 % at 10 rows, +0.02 to +0.08 % at 1,000). **`sql.begin()` rejected a transaction that was committed, and `sql.beginDistributed()` one that stayed prepared,** when the server's answer and its goodbye were read together. A server that shuts down, fails over or has the backend terminated right behind `COMMIT` sends its answer and `FATAL 57P01` (MySQL: `ERR 1053`) back to back. When the client read both in one go, the goodbye rejected the transaction before the answer had been looked at. | A real PostgreSQL, the backend terminated right behind the answer | `begin()` | rows in the table | `beginDistributed()` | in `pg_prepared_xacts` | |---|---|---|---|---| | every version so far | rejected | 1 | rejected | 1 | | now | resolved | 1 | resolved | 1 | A caller that tried again wrote twice; a prepared transaction kept its locks, and the caller had been told that it failed. It takes no stalled client: in 679 of 1,933 such terminations an ordinary client read both in one go. This also covers the array form and `begin()` on a reserved connection. In the same situation, a `COMMIT` that is answered with an error or with the tag `ROLLBACK`, and a callback that threw, now reject with that error and not with the goodbye, as they do when the goodbye comes a moment later. A connection that is lost while `COMMIT` is unanswered still rejects: nobody can know the outcome then, and the docs say so now. **MySQL, MariaDB: a server that refuses the connection in place of its greeting was reported as an unsupported protocol version.** | | before | now | |---|---|---| | the server is at `max_connections` | `ERR_MYSQL_UNSUPPORTED_PROTOCOL_VERSION`, no `errno`, "Server is using an unsupported protocol version" | `ERR_MYSQL_SERVER_ERROR`, `errno` 1040, "Too many connections" | | no account for this host | the same | `errno` 1130, "Host '...' is not allowed to connect to this MySQL server" | It rejects what waits at once, like a failed login; it is not tried again. **An error for a connection URL that cannot be parsed no longer repeats the URL.** It is the error of `new URL()` in everything but `input`, which is `scheme://[REDACTED]`: anything behind the scheme can be something else than its position suggests. The same for the hint at `{ adapter: "sqlite" }`, which no longer has a `cause`, and for the error about a null byte in a startup parameter, which no longer prints the value. **A key on `Object.prototype` is no longer taken for an option, a URL parameter or a variable.** `Bun.SQL` merged its arguments into a plain object, looped over the URL's parameters and `options.connection` with `for ... in`, and read the parsed options by name for every connection. `Bun.env` inherits from `Object.prototype` as well. So a key that something had put there (a dependency that merges untrusted JSON, or one enumerable helper) counted as an option, a URL parameter, a startup parameter or an environment variable: `tls: false`, another `hostname`, a `path` that the pool then dialed with the user's password, a `DATABASE_URL` that sent a client configured with `PGHOST`, `PGUSER` and `PGPASSWORD` to another host with that user and password, or `unrecognized configuration parameter "helper"` from the server for every `postgres://` pool. The objects that `Bun.SQL` makes for itself have no prototype now; what is read from an object of the caller's, with what that inherits (the options, `options.connection`), skips a name only if `Object.prototype` has it and nothing below it in the chain does, so what a class or `Object.create()` hands over still counts; a variable counts only if the environment has it; and every key that `sql.options` has only when it is set (`path`, `transform`, `onconnect`, `onclose`, the three timeouts) is read as an own property wherever the client acts on it (only the comparison of two clients' settings under `--hot` still reads by name, and sees the same on both sides): `transform.undefined`, for one, is what is bound for a parameter that is `undefined`. `sql.options` is the object that it was. With a clean prototype one thing changes: the pool reads the own enumerable properties of `sql.options`, so a later write to it that is not one (`Object.defineProperty` without `enumerable`, `Object.setPrototypeOf`) no longer counts. Still looked up as before: the keys inside a `transform` that the caller passes. `Object.prototype` is the one of the realm that `Bun.SQL` is from: an options object that was made in a `node:vm` context inherits from another one, which is not looked at. Only for a name that `Object.prototype` has, an options object that is a `Proxy`, or has one in its chain, is also asked through `getOwnPropertyDescriptor` and `getPrototypeOf`: what those traps do then reaches the caller, and a `Proxy` that answers through `get` alone is not asked for that name. ### How did you verify your code works? - The pool: 18 tests in `sql-pool.test.ts` against scripted servers, and the tables above against a real PostgreSQL. - A query that is only started: 28 ways of being rejected, in up to three forms, on PostgreSQL, MySQL, MariaDB and SQLite: 283 cells, one process per cell, exit code checked. Cells that were silent before oven-sh#45304 and are reported: 37 on `main`, 0 now. - `bun:sqlite`: 12 tests raise an exception behind a step: ten by a timeout of `node:vm` that doubles from 1 ms until the write has run, so that one of them falls into the step, two by the termination of a Worker; each compares the rows that the index finds with the rows that are stored. 59 of 60 runs fail on a release build of `main`. On AddressSanitizer and release builds, with the same `JSSQLStatement.cpp` but for `iterate()` of a write: 1,152 cells of a Worker terminated while it writes (230 bad on `main`, 0), 660 ends of endless loops of one-row inserts (0 left a lock or a row that `MATCH` misses; `main`: 12 to 22 of every 60), kysely, lucia, typeorm and `Bun.SQL` over 123 runs (no other line differs from `main`). - `Object.prototype`: one test per adapter parses the same arguments and runs a query before and after 25 keys are put there: with a URL, with everything from `PG*` or `MYSQL_*` variables, and with `adapter`, `username` and `readonly` from `Object.create()`, and listens on the unix socket whose path is one of them; it fails on `main`, says that the socket was dialed if only the pool's read is put back, binds the value from the prototype if only the reads of `transform` are, and takes the URL from the prototype if only the reads of the environment are. A server that closes every connection at once has to be dialed a third time before the query gives up.
…sh#45851) ### What does this PR do? Moves 58 small natives of `node:*` and `Bun` to schemas of the bindgen (`docs/project/bindgen.mdx`). Each was a host function that took a `CallFrame`, counted and type-checked its arguments by hand, and made its result by hand. `src/js` reached it with `$newRustFunction("file.rs", "name", length)`. Each now has a line in a `.bind.ts` file that says what it takes and gives, and a Rust body with typed parameters. `src/js` reaches it with `$bindgen("file.bind.ts", "name")`, which is typed from the schema. ```ts export const crc32 = b.fn({ args: { data: b.bytes({ from: ["string", "buffer"], refuse: ["stringObject"] }), value: b.u32.validateInteger().default(0), }, returns: b.u32, }); ``` ```rust pub(crate) fn crc32(data: &[u8], value: u32) -> u32 { bun_zlib::crc32_bytes(value, data) } ``` | | `main` (d0831bc) | Now | | --- | --- | --- | | `$newRustFunction` and `$rust` in `src/js`, outside `internal-for-testing.ts` | 91 | 29 | | Schema files | 22 | 38: 15 for these, 1 for tests | | Rust, in 24 files | | 571 lines added, 1289 removed | | Schemas | | 297 lines added | | Schema | Functions | | --- | --- | | `node_util_binding.bind.ts` | `internalErrorName`, `internalErrorEntries`, `etimedoutErrorCode`, `enobufsErrorCode`, `ecanceledErrorCode`, `uvTranslateSysError`, `extractedSplitNewLinesFastPathStringsOnly`, `normalizeEncoding`, `assertEncodingValid`, `parseEnv` | | `udp_socket.bind.ts` | `dgramIsFdAdopted`, `dgramNewSocketFd`, `dgramAdoptFd`, `dgramBindFd`, `dgramGetSockNameFd`, `dgramGuessHandleType`, `dgramListenFd`, `dgramCloseFd`, `connect`, `disconnect`, `bufferSize`, `getFd` | | `node_http_binding.bind.ts` | `getMaxHTTPHeaderSize`, `setMaxHTTPHeaderSize`, `getInsecureHTTPParser`, `getBunServerOpenCount`, `upgradeNodeHTTPResponseToWebSocket`, `sendWebSocketFrame` | | `node_cluster_binding.bind.ts` | `onInternalMessageChild`, `onInternalMessagePrimary`, `setRef`, `channelIgnoreOneDisconnectEventListener`, `clusterValidateFd`, `clusterCloseHandle` | | `socket_body.bind.ts` | `upgradeTLSDeferred`, `releaseHeldFlight`, `checkServerIdentityInHandshake`, `firstFlightBeforeFin`, `isNamedPipeSocket`, `getBufferedAmount` | | `node_net_binding.bind.ts` | `getDefaultAutoSelectFamily`, `setDefaultAutoSelectFamily`, `getDefaultAutoSelectFamilyAttemptTimeout`, `setDefaultAutoSelectFamilyAttemptTimeout`, `newDetachedSocket` | | `node_assert_binding.bind.ts` | `myersDiff`, `printSimpleMyersDiff`, `printMyersDiff` | | `Listener.bind.ts` | `addServerName`, `setSecureContext` | | `jest.bind.ts` | `fileGeneration`, `nodeTestMarkResult` | | One each | `crc32`, `getUseSystemCA`, `getRuntimeDefaultResultOrderOption`, `hotReloadGeneration`, `monotonicNowMs`, `Bun.registerMacro` | The four natives of `UDPSocket` read the socket from `this`, so `node:dgram` called them with `.$call(socket, …)`. The socket is now their first argument. `generate-js2native.ts` loses the 8 entries of `rustIdentifierPaths` that nothing names any more. #### The generator **`b.classRef(name)`**: an instance of a class of a `.classes.ts` file, as a `&'a T` of what it wraps. It is what `JSValue::as_class_ref` gives, with the lifetime of the call. `T` is `crate::generated_classes::<name>`, the type the class's own thunks use, so the schema cannot name one class and another struct. 16 of the 58 take a `TCPSocket`, a `TLSSocket`, a `Listener`, a `UDPSocket`, a `SecureContext`, a `NodeHTTPResponse`, a `ServerWebSocket`, a `Subprocess` or one of the four servers. The argument's slot in the frame is what holds the instance, so it is only for an argument, alone, `.nullable` or in a union: in a dictionary or an array it is a build error. Two `b.classRef` of one class are one kind of value to a union and to overloads. A native that did nothing for a value that is not an instance says so in its schema: ```ts /** The `_handle` of a `net.Socket`: `new net.Socket({ handle })` keeps any object. */ export const SocketHandle = b.union({ tcp: TCPSocket, tls: TLSSocket, ignored: b.any }); ``` **`expected` of a function**: what Node's own source expects of an argument, as it spells it. ```ts export const crc32 = b.fn({ args: { data: b.bytes({ from: ["string", "buffer"], refuse: ["stringObject"] }), value: b.u32.validateInteger().default(0) }, expected: { data: ["Buffer", "TypedArray", "DataView", "string"] }, returns: b.u32, }); ``` A value the argument's type refuses then throws Node's `ERR_INVALID_ARG_TYPE(name, expected, actual)`, with Node's words. What the type takes is the same, and a call that succeeds does nothing more: the names are read where the error is made. `util.parseEnv`, `zlib.crc32` and `net.setDefaultAutoSelectFamily` are exported as they are, with no JavaScript in front to validate, and use it. **`refuse: ["stringObject"]` of `b.bytes`**: `"string"` is then only a string. `zlib.crc32(new String("abc"))` throws, here and in Node, and `b.bytes` takes a `String` object otherwise. **A `b.string` argument is a `&bun_core::String`.** The thunk handed over the string by value, and clippy's `needless_pass_by_value` refuses that in a body that only reads it. `braces` had an `#[allow]` for it, and 9 arguments of this PR would have needed one. The string now stays in the frame of the thunk, as a dictionary does, and a body that keeps it clones it, which counts a reference and copies nothing: `Bun.$.escape` does, for an input that needs no escaping. No `#[allow(clippy::needless_pass_by_value)]` is left in `src/runtime`. **A `b.array` result of more than 8 elements is written into the array.** It was staged in a `MarkedArgumentBuffer`, which allocates past 8. The array is made with the length the iterator promises (`size_hint`) and grows if there is more. `util.inspect` of a string of many lines gets its lines this way, and `main` wrote them into an array of the right length. #### What is not moved These are the 29 that are left. | Native | Why | | --- | --- | | `BlockList`, `SocketAddress`, `NativeZlib`, `NativeBrotli`, `NativeZstd`, `SecureContext`'s `getConstructor`, `H2FrameParserConstructor` | each gives the constructor of a class, not a function | | `createBinding` of `node_fs_binding`, `mysql` and `postgres`, `createNodeQuicBinding`, `createNodeCryptoBindingZig` | each is a whole binding | | `doConnect`, `jsUpgradeDuplexToTLS`, `Resolver.newResolver` | they ask the `CallFrame` who the caller is | | `getBunServerAllClosedPromise` | needs a `&mut` of the server | | `clusterRawBind`, `sendHelperPrimary`, `parseArgs`, `Bun__FFI__cc`, `emitHandleIPCMessage` | not small, or every argument is an object the body reads or writes itself | Of `Bun`: `color`, `file` and `write` take many shapes of input and are not small. `resolve` gives a promise that rejects for an argument it refuses, which the bindgen has no result for, and `resolveSync` shares its parsing. `serve`, `spawn`, `spawnSync`, `listen`, `connect`, `udpSocket` and `build` take large option bags. The natives that only `internal-for-testing.ts` reaches are as they were. ### Behaviour A call that returned on `main` returns the same, but for `http.maxHeaderSize = value`. Node has no setter for `http.maxHeaderSize`: the assignment is ignored, or throws in strict mode. Ours stays, and now takes what Node's `maxHeaderSize` option of `http.createServer()` takes, a number that is a safe integer, through `.validateInteger()`. It refuses 0, as on `main`. `main` converted the value to an integer, and wrapped what did not fit. What it throws is what the option throws, word for word, but for ">= 1" where the option says ">= 0", and for `undefined`, which to the option is an option that was not given. 2456 calls, made on CI's release build of `main` (d0831bc) and on a debug build of this branch. 65 values go to each argument that JavaScript can reach: through the public API where there is one, through a replaced `_handle`, a `lookup` that gives any value as the address, or `--expose-internals`. 1897 are the same and 559 differ. A script sorts the 559 and stops at one it has no rule for. A case is one change of behaviour, or of one message, on one API. | | Cases | | --- | --- | | bug → bug | 0 | | bug → non-bug | 9 | | non-bug → bug | 0 | | non-bug → non-bug | 26 | ##### bug → bug None. ##### bug → non-bug | Call | `main` | Now | | --- | --- | --- | | `zlib.crc32("hello", undefined)` | throws `ERR_INVALID_ARG_TYPE` | 907060870, as `zlib.crc32("hello")`. In Node `value = 0` is a default parameter | | `http.maxHeaderSize = "5000000000"`, or an object whose `valueOf` gives that | 705032704, the low 32 bits. `"-4294950912"` was 16384 | throws `ERR_INVALID_ARG_TYPE` | | `http.maxHeaderSize = 2n ** 64n + 16384n` | 16384, the low 64 bits. `-(2n ** 64n) + 5n` was 5 | throws `ERR_INVALID_ARG_TYPE` | | `socket.connect(port)` of `node:dgram` with a port the engine holds as a double, as `new Float64Array([port])[0]` is | read as an int32: the socket is not connected | connected | | `socket.connect("41234")` of `node:dgram`, or `" 41234 "` or `"0xa112"` | the callback gets `ERR_INVALID_ARG_TYPE`, Expected "port" to be an integer | connected, as in Node, whose `validatePort` returns `port \| 0`. Ours returned the string. `socket.send()` took one already | | `net.setDefaultAutoSelectFamily()` | `Error`, no code, "missing argument" | `TypeError`, `ERR_INVALID_ARG_TYPE`, as in Node | | `net.setDefaultAutoSelectFamilyAttemptTimeout()` | `Error`, no code, "missing argument" | `TypeError`, `ERR_INVALID_ARG_TYPE`, as in Node | | `util.getSystemErrorName(-4294967297)`, or any number below -2<sup>31</sup> | "Unknown system error -2147483648" | "Unknown system error -4294967297", as in Node | | An `ERR_OUT_OF_RANGE` made in C++ for `-0`: `util.getSystemErrorName(-0)`, `crypto.generatePrimeSync(-0)`, `zlib.createGzip({ chunkSize: -0 })` | "Received 0" | "Received -0", as in Node | The last is in `appendOutOfRangeReceived` of `ErrorCode.cpp`. `.validateInteger()` reports through it, and the validator in Rust that `net.setDefaultAutoSelectFamilyAttemptTimeout(-0)` used on `main` had the sign. ##### non-bug → bug None. ##### non-bug → non-bug The first two returned on `main` and throw: the setter is as strict as the option. The other 24 threw and throw, and the class and the code are the same but where the table says otherwise. | Call | `main` | Now | | --- | --- | --- | | `http.maxHeaderSize = "16384"`, `1n`, `true`, `[1]`, `new Number(5)` or `{ valueOf: () => 7 }` | converted: 16384, 1, 1, 1, 5, 7 | `TypeError`, `ERR_INVALID_ARG_TYPE`, The "maxHeaderSize" argument must be of type number. Received … | | `http.maxHeaderSize = 1.5` or `2 ** 53` | 1, 9007199254740992 | `RangeError`, `ERR_OUT_OF_RANGE` | | `http.maxHeaderSize = undefined`, `null` or `"a"` | The "maxHeaderSize" argument must be of type non-negative integer. Received undefined | The "maxHeaderSize" argument must be of type number. Received undefined | | `http.maxHeaderSize = 0`, `-1` or `2 ** 63` | `TypeError`, `ERR_INVALID_ARG_TYPE`, … must be of type non-negative integer. Received type number (0) | `RangeError`, `ERR_OUT_OF_RANGE`, … It must be >= 1 && <= 9007199254740991. Received 0 | | `http.maxHeaderSize = -1.5`, `NaN` or `Infinity` | `TypeError`, `ERR_INVALID_ARG_TYPE`, … must be of type non-negative integer. Received type number (-1.5) | `RangeError`, `ERR_OUT_OF_RANGE`, … It must be an integer. Received -1.5 | | `http.maxHeaderSize = Symbol("s")` | no code, Cannot convert a symbol to a number | `ERR_INVALID_ARG_TYPE`, … must be of type number. Received type symbol (Symbol(s)) | | `http.maxHeaderSize =` an object whose `valueOf` throws | what `valueOf` threw | `TypeError`, `ERR_INVALID_ARG_TYPE`. `valueOf` is not called | | `http.maxHeaderSize =` an object whose `toString` throws | what `toString` threw | `TypeError`, `ERR_INVALID_ARG_TYPE`. `toString` is not called | | `http.maxHeaderSize =` a typed array whose buffer is detached | no code, Underlying ArrayBuffer has been detached from the view or out-of-bounds | `ERR_INVALID_ARG_TYPE`, … Received an instance of Uint8Array | | `http.maxHeaderSize = { valueOf: () => 5n }` | no code, Conversion from 'BigInt' to 'number' is not allowed. | `ERR_INVALID_ARG_TYPE`, … Received an instance of Object | | `util.parseEnv(1)` | The "content" property must be of type string, got number | The "content" argument must be of type string. Received type number (1), as in Node | | `net.setDefaultAutoSelectFamily(1)` | autoSelectFamilyDefault | The "value" argument must be of type boolean. Received type number (1), as in Node | | `net.setDefaultAutoSelectFamilyAttemptTimeout("a")` | The "value" property must be of type number, got string | The "value" argument must be of type number. Received type string ('a'), as in Node | | `net.setDefaultAutoSelectFamilyAttemptTimeout(2 ** 40)` | … Received 1099511627776 | … Received 1_099_511_627_776, as in Node | | `zlib.crc32(1)` | The "data" property must be an instance of Buffer, TypedArray, DataView, or ArrayBuffer. Received number | The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (1), as in Node | | `zlib.crc32("hello", -1)` | … It must be >= 0 and <= 4294967295. Received -1 | … It must be >= 0 && <= 4294967295. Received -1, as in Node | | `zlib.crc32("hello", Infinity)` | … It must be >= 0 and <= 4294967295. Received inf | … It must be an integer. Received Infinity, as in Node | | `zlib.crc32("hello", "a=\ud800")` | … Received type string ('a=�') | … Received type string ('a=\ud800') | | `Bun.registerMacro("a", fn)` | Internal error registering macros: invalid args | id must be a number | | `Bun.registerMacro(7)` | `TypeError`, `ERR_INVALID_ARG_TYPE`, Internal error registering macros: invalid args | `Error`, Macro must be a function, as `Bun.registerMacro(7, undefined)` on `main` | | `addContext` of a `tls.Server` whose `_handle` was replaced | `Error`, Expected a Listener instance | `TypeError`, `ERR_INVALID_ARG_TYPE`, listener must be an instance of Listener | | `myersDiff()` | `ERR_MISSING_ARGS`, Not enough arguments to 'myersDiff'. Expected 2, got 0. | `ERR_INVALID_ARG_TYPE`, actual must be a string | | `myersDiff(1, "b")` | The "actual" argument must be of type string. Received type number (1) | actual must be a string | | `myersDiff("a")` | `ERR_MISSING_ARGS`, Not enough arguments to 'myersDiff'. Expected 2, got 1. | `ERR_INVALID_ARG_TYPE`, expected must be a string | | `myersDiff("a", 1)` | The "expected" argument must be of type string. Received type number (1) | expected must be a string | | `printSimpleMyersDiff(1, "b")` | The "actual" argument must be of type string. Received type number (1) | actual must be a string | The last five are of `internal/assert/myers_diff`, which only `--expose-internals` reaches with such values. #### Cost The first `util.getSystemErrorMap()` of a process makes an object, `{ code, name }`, for each entry of libuv's error table, where `main` made one flat array of codes and names. The map it fills has 85 codes on Linux, and is cached. ### How did you verify your code works? - A test for each of the nine fixes, for Node's words from `util.parseEnv`, `zlib.crc32` and `net.setDefaultAutoSelectFamily`, and for what `http.maxHeaderSize = value` takes and throws next to what the option throws, in `zlib.test.js`, `node-http-maxHeaderSize.test.ts`, `node-net.test.ts`, `node-dgram.test.js` and `util.test.js`. Each fails on the release build of `main` and passes here. - `test/internal/bindgen.test.ts` has `b.classRef` through `class_ref_test.bind.ts`, which like `bindgen_test.bind.ts` is only in a build with assertions: an instance, what is not one (the prototype, an object that inherits from an instance, an instance of another class), `.nullable`, both kinds of union, overloads that name the class twice, and a collection while the body holds the `&`. It also has a borrowed `b.string` with and without a default, `refuse: ["stringObject"]`, `expected` next to an argument without it, and the new build errors. - `test/integration/bun-types/bun-types.test.ts` lists the new schema files. None is published: JavaScript wraps them, or they are not for users. `Bun.registerMacro` and `getRuntimeDefaultResultOrderOption` are in files of their own for that, `BunObjectInternals.bind.ts` and `dns_internals.bind.ts`, since every export of `BunObject.bind.ts` and `dns.bind.ts` is compared with the published types. - One script of real flows prints the same bytes as Node 26.7.0: TLS with SNI, `addContext` and `setSecureContext`, TLS over a connected `net.Socket`, a connected `dgram` socket with its buffer sizes and `disconnect`, `http.Server#close`, the `net` defaults, `crc32`, `parseEnv`, `getSystemErrorName`, `realpathSync` with encodings, the diff in an `assert` message. - The same as the release build of `main`: a `cluster` worker with an HTTP server and a UDP socket, a `ws` server with `perMessageDeflate`, `t.skip()` and `t.todo()` of `node:test` under `bun test`, a macro, and `tls.getCACertificates("default")` with and without `--use-system-ca`. - Locally on a debug build: the test files above, `udp_socket.test.ts`, `node-tls-server.test.ts`, `node-tls-context.test.ts`, `ws-server-compat-transports.test.ts`, `brace.test.ts`, `assert.test.cjs`, and 98 of Node's own tests for `dgram`, `net`, `zlib`, `util`, `tls`, `http` and `dns` in `test/js/node/test/parallel`. 96 pass. `test-uv-errmap.js` and `test-uv-errno.js` fail on this machine only, whose kernel headers define `EFTYPE`; both pass on CI's build of `main`, and this PR touches neither table. - `cargo clippy`, `test/internal/source-lints`, the three `tsc` projects of `src`, and `cargo check` for `x86_64-pc-windows-msvc` and `aarch64-apple-darwin`. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
### What does this PR do? Follow-up fixes to oven-sh#45223, all in the WebSocket client (the global `WebSocket` and `ws`). 38 lines of product code. #### 1. Through a proxy, a healthy connection died with 1006 when something was sent during a TLS renegotiation A `wss://` connection through a proxy runs its TLS session in the tunnel's own TLS engine instead of in the socket. That engine refuses a write where a socket just takes 0 bytes. The client treated a refused write as the end of the connection and tore it down from inside `send()`. `wss://` through an `http:` or `https:` proxy, TLS 1.2 origin that renegotiates: a `send()`, or anything else that writes a frame, between the origin's request and the end of the new handshake closed the connection with 1006, and `ws.send(data, callback)` called back with `null` for a message that was never sent. A direct connection was fine. Old, 1.4.3 has it too. The frame is now queued, like it is for a socket (`bufferedAmount` shows it), and goes out when the renegotiation ends. The `ws` callback runs after the write. As on a direct connection, messages that arrive together with the end of the handshake are delivered before the write is retried. The same line covers the other case in which the engine refuses a write: a read failed (a fatal alert, a corrupt record) and what was decrypted before the failure is still being handed over. A listener that wrote at that moment lost every message that had already arrived. They are delivered now, then `close` 1006, as on a direct connection. #### 2. `ws`: `send()`, `ping()` and `pong()` reported success for a frame that was dropped When the connection of a natively connected client ends, its `close` event is queued behind the current task. Until then `readyState` is still `OPEN`, and a frame sent in that window was dropped without a word: the callback got `null`. A timer that is due in the loop turn in which the peer's FIN is read hits that window. So does this: ```js ws._socket.destroy(); ws.send("x", error => console.log(error)); // null, now ERR_STREAM_DESTROYED as on Node ``` The callback now gets `ERR_STREAM_DESTROYED`, right before `close`. The server side of `ws`, and clients over a custom stream, already did this. Old. #### 3. `ws` ignored the headers of an `hpagent` agent ```js new WebSocket(url, { agent: new HttpProxyAgent({ proxy: "http://proxy:8080", proxyRequestOptions: { headers: { "Proxy-Authorization": "Basic ..." } }, }), }); ``` `proxyRequestOptions.headers` never reached the CONNECT request, so a proxy that wants that header answered 407. Only `proxyHeaders`, the name `https-proxy-agent` uses, was read. As in hpagent, credentials in the proxy URL replace `Proxy-Authorization`. Old. #### 4. Regression from oven-sh#45223: a paused client lost the messages that came with the 101 when a TLS read failed Direct `wss://`, certificate verification on, `pause()` in `open`, messages in the same read as the 101, then a fatal alert or a corrupt record before `resume()`: 0 messages, where main before oven-sh#45223 and Node + `ws` deliver all of them. On a direct connection a failed read is reported to the client as a failed handshake, and that path dropped what was still waiting for `resume()`. The two ordinary close paths parse it first, and now this one does too. One line. #### Also - Two comments said that a body declared by an upgrade request is never parsed as frames. That only holds for the part of the body in the same read as the request. The comments now say so. No behavior change. - `tlsRecords()` and the "hold N TLS records, forward them in one write" logic moved from one test into `proxy-test-utils.ts`, and that test uses the shared helper. - One missing `ws.data` type in `websocket-custom-headers.test.ts`. ### How did you verify your code works? - 38 new tests (14 in `websocket-proxy.test.ts`, 18 in `ws-proxy.test.ts`, 6 in `ws-client-compat-send.test.ts`). 24 of them fail on a release build of main. The other 14 pass there too: they are the reference rows, the same cases on a direct connection and on npm `ws`. - The timing in those tests is pinned, not left to the network: what stands in front of the origin forwards whole TLS records and puts the request to renegotiate, the end of the handshake, or the corrupt record in the same write as the message next to it, and the tests assert which record types that write carried. The renegotiation tests also assert `bufferedAmount` right after the `send()`, so they cannot pass by sending before the renegotiation started. Every client in them verifies the certificate. - The real `hpagent` 1.2.0 against a recording proxy, Bun next to Node: the same `Proxy-Authorization` and extra headers. - The WebSocket test folders (74 files, 4,732 tests) on a debug + ASAN build with `BUN_JSC_validateExceptionChecks=1`: no failure.
…45710) ### What does this PR do? Follow-up fixes for `bun install` and related commands after oven-sh#44968, oven-sh#45307, oven-sh#45460, oven-sh#45477 and oven-sh#45616. Every fix has a test that fails without it, except the performance fix, which is measured. #### Regressions on `main` (not in any release) - **`bun ci` no longer fails when `package.json` changed but the installed packages didn't.** Since oven-sh#44968, `--frozen-lockfile` failed on any difference between `package.json` and `bun.lock`: changing `~11.2.0` to `^11.2.0`, moving a dependency to `devDependencies`, or adding a dependency that another workspace already has. 1.4.3 allowed all of these, because the same packages get installed. 2 of the 51 projects in this repo that have a `bun.lock` would have failed CI after upgrading. It still fails when an edit adds a package, removes one, or changes a version. - **`bun pm ls` works without `node_modules` again.** Since oven-sh#44968 it exited 1 with "node_modules not found, nothing to list". It now lists what's in `bun.lock` and exits 0, like 1.4.3. When `node_modules` exists, it still lists what's on disk. - **Removed `warn: registry requests failed, retrying`** (added in oven-sh#45460). A retry that succeeds doesn't need a message, and one that fails ends in an error that explains why. `--verbose` still prints every retry. Two messages stay: the same warning after a timeout, where the install has been silent for minutes, and the wait time when a registry sends `Retry-After`. - **Stale nested packages could survive an upgrade from 1.4.3 forever** (hoisted linker, since oven-sh#45307). When an install skipped some dependencies (`--production`, `--omit`, `--filter`, or an optional dependency for another platform), 1.4.3 nested packages at paths that `bun.lock` has no entry for, like `n/node_modules/b`. oven-sh#45307 cleans those up, but it skipped the cleanup in five cases: - the project has an install script and scripts are disabled, e.g. `"prepare": "husky"` with `bun install --production --ignore-scripts` - `n` isn't in the cache, e.g. a CI or Docker layer that keeps `node_modules` but not the cache - `n` is trusted and has lifecycle scripts - `n` is a git, `github:` or `file:` dependency - `n` is inside the `node_modules` of a workspace that has an install script In those cases `n` kept loading `b@2.0.0` after `bun.lock` had moved to `b@2.1.0`. See "Behavior change to review" below. - **The isolated linker could link a bin from the wrong package.** When two direct dependencies have the same bin (`typescript` and `typescript7`, or `esbuild` and `esbuild-wasm`), the first one by name is supposed to win. If another package that sorts earlier had the first one as a peer dependency, like `msw` has `typescript`, `node_modules/.bin/tsc` pointed at the second one. - **Renaming a workspace deleted the packages in its `node_modules`** (hoisted linker). After changing a workspace's `"name"`, `bun install` removed `packages/two/node_modules/y`, so the workspace loaded the hoisted version until the next install. The cleanup now identifies a workspace by its folder instead of its name. - **`bun --filter="*" publish` published every workspace.** `bun --filter "*" publish` runs each workspace's `publish` script, so the `=` form doing something completely different was a trap. `--filter` and `--recursive` now only apply to `bun publish` when they come after `publish`. Otherwise it stops with an error that shows both spellings. - **`bunx` failed when the install cache was under a path that isn't valid UTF-8**, like a home directory named `/home/jos\xE9`. It installed the package and then reported "could not determine executable to run for package". #### Bug fixes - **A peer dependency could resolve to an unrelated package's out-of-range version.** `elysia` has a peer dependency on `@sinclair/typebox@>=0.34.0 <1`. In a project that also uses jest, `@jest/schemas` depends on `@sinclair/typebox@^0.27.8`. If that was the only version installed, elysia got it and failed to load. 1.4.3 printed a warning, and since oven-sh#45307 nothing is printed. Installing elysia first and jest second gave each its own version, so the same `package.json` could produce two different lockfiles. Now, when no installed version satisfies a peer range, Bun doesn't fall back to a version that's only there as a private dependency of unrelated packages. It resolves the peer from the registry instead. Everything else works as before: - `react@17` in the project, a workspace, or any package above the library is still used by a library that wants `^18`. - A version that satisfies some other package's peer range stays shared. `@octokit/rest` and its plugins share one `@octokit/core`, and `type-graphql` and its helpers share one `graphql`. Hoisting a second copy above it would split them up. - Two peer dependencies that nobody provides still share one version. Limitations: - The fix doesn't apply if any other package in the project has a peer range that the old version satisfies. - A lockfile that already has the wrong version keeps it until the package with the peer dependency changes version (`bun update`, `bun add`) or is removed and re-added. - Optional peer dependencies still aren't resolved. They use the nearest copy in `node_modules`. With the hoisted linker, elysia's dependency `exact-mirror` still loads the old typebox if that's the hoisted one, which requires `@jest/schemas` itself to be in `package.json`. - Bun now requests a manifest that it didn't need before. If that request fails, `bun add` exits 1 and leaves `package.json` and `bun.lock` untouched, but the new package's folder is already in `node_modules`. - **`bun update --latest` narrowed a peer range that already allowed the latest version.** A library with `"solid-js": "catalog:"` in `devDependencies` and `"solid-js": ">=1.9.0"` in `peerDependencies` had the peer range rewritten to `^1.9.15`. The docs say such a range is left alone. The check was reading the `devDependencies` entry from the lockfile, which is a catalog reference, not a range. - **`bun update --latest` downgraded catalog entries that were ahead of `latest`**, whether a pin (`3.0.1`), a range (`^3.0.1`) or a dist-tag (`canary`, `next`), in the default catalog and named ones. oven-sh#45307 only fixed this for entries in `dependencies`. - **`bun run --filter <pkg> deploy` kept going after `predeploy` failed.** It ran `deploy` and `postdeploy` anyway. The exit code was right, but the deploy had already happened. A failing script now skips the rest of its package's `pre`/main/`post` chain, like `bun run` without `--filter`, and the skipped scripts are shown as `Skipped`. Other packages still run. - **`bun install --use-system-ca` accepted the flag and ignored it.** Only `NODE_USE_SYSTEM_CA=1` worked. The flag now works for every package manager command, before or after the command name, and in `BUN_OPTIONS`. `--use-openssl-ca` and `--use-bundled-ca` are handled too, and passing more than one is an error, same as `bun run`. - **A new workspace wasn't linked if the same name and version was already installed** (hoisted linker). After copying `dep@1.0.0` into `packages/dep` and depending on it with `"workspace:*"`, `bun install` reported "no changes" and `node_modules/dep` stayed the registry copy. - **`bun prune` deleted bundled dependencies that `bun.lock` doesn't mark as bundled** (hoisted linker, since 1.4.0). This repo's `test/bun.lock` is one example: `bun prune` removed all 34 folders in `node_modules/aws-cdk-lib/node_modules`. It now also checks `bundleDependencies` in the installed package's `package.json`. - **`bun pm cache rm` and `bun pm cache prune` printed "Cleared" without deleting anything** when `~/.bunfig.toml` sets the cache directory. They now read that file. A project's `.env`, `bunfig.toml` and `.npmrc` still can't choose what gets deleted, and neither can `cache` in `~/.npmrc`, because that's npm's cache too. - **`bun pm fetch` downloaded nothing in a monorepo's Docker layer.** With `COPY package.json bun.lock ./` and `RUN bun pm fetch`, workspaces matched by a glob aren't on disk yet, so they were dropped along with their dependencies: "No packages to fetch", exit 0, and the later `bun install --offline` failed. It now keeps them, like `--frozen-lockfile` does. - **`bun pm fetch` looked up a missing workspace in the registry** when the root depends on it as `"lib": "*"` or `"^1.0.0"`. It either failed with a 404 or cached a public package with the same name. It now stops with the same error as a frozen install. - **`packageExtensions` stopped working after migrating from pnpm.** Editing or removing an extension had no effect, and `--frozen-lockfile` didn't notice. `bun.lock` records which dependencies came from an extension, but `pnpm-lock.yaml` doesn't distinguish them. The migration already fetches every manifest, so it now records any extension dependency that the manifest doesn't declare. - **`node-gyp` in a lifecycle script re-ran itself hundreds of times when `TMPDIR` wasn't valid UTF-8.** Bun's `node-gyp` shim runs `bun x node-gyp`, which skips the shim by comparing an environment variable against `PATH`. One of the two was lossily formatted, so they never matched. - **Output from a failed lifecycle script was garbled if it wasn't UTF-8.** Each invalid byte became U+FFFD, which made Latin-1, Shift-JIS or GBK compiler errors unreadable. The bytes are now printed as-is, like in 1.3.14. - **The `--frozen-lockfile` error now says which dependency changed**, for example `dependencies in package.json changed since bun.lock was saved: basic-1 was added`, `a-dep was removed in packages/app`, or `no-deps changed from ^1.0.0 to ^2.0.0`. - **Docs:** fixed two wrong statements on the auto-install page. The `bunfig.toml` closest to the importing file wins, not the one where you started `bun`. And auto-install is disabled by a `node_modules` above the importing file, not above the working directory. #### Performance - **Hoisted installs were quadratic in the number of nested `node_modules` folders**, even when there was nothing to do (1.4.3 too). Every time a folder finished, Bun scanned every other folder for installs that were waiting on it. It now keeps a count of waiting installs and skips the scan when it's zero. User-mode instructions for a no-op `bun install`, on CI's release builds: | Project | Before | After | |---|---|---| | 2,000 packages, no nested folders | 77.2 M | 77.2 M | | 500 packages, each with a nested `node_modules` | 53.5 M | 29.8 M | | 2,000 packages, each with a nested `node_modules` | 491.6 M | 114.9 M | #### Behavior change to review The stale-nested-packages fix changes which folders an install deletes. As before, a folder is only considered if 1.4.3's layout for this install would have put a package at that exact path, the current layout doesn't, and the package that owns the `node_modules` is still installed in the same place. What changed is the decision after that: | Case | `main` | This PR | |---|---|---| | A script could have created the folder (an install script that won't run again, or a trusted owner) | kept | removed | | The owner is a registry package that isn't in the cache | kept | removed | | The owner is a git dependency or `file:` folder | kept | removed, because Bun never installs their `node_modules` | | The owner is a `github:` dependency | kept | kept only if the cached archive has the folder | | The owner's cached tarball has the folder, or the owner is a tarball dependency | kept | kept | | A parent package has bundled dependencies | kept | kept | I don't know of any script that creates a folder named after a dependency at exactly the path where the old layout nested that dependency. Two tests from oven-sh#45307 that pinned "such a folder is kept" are removed. What this could get wrong: a script that does exactly that, or a tarball that ships `node_modules/b` without listing `b` in `bundledDependencies` while also depending on `b`, in a project with no cache. The owner would then load the `b` that `bun.lock` resolved for it, which is within its declared range. ### How did you verify your code works? **Tests.** New or updated tests in `hoist`, `bun-install-registry`, `bun-lock`, `frozen-lockfile-outdated`, `frozen-lockfile-pruned`, `frozen-lockfile-member-pins`, `bun-pm`, `bun-pm-fetch`, `bun-prune`, `bun-publish`, `bun-install-retry`, `bun-install-lifecycle-scripts`, `bun-update-lockfile-sync`, `bun-update-transitive`, `bun-workspaces`, `bun-add-filter`, `bunx`, `isolated-install`, `pnpm-lock-v9` and `filter-workspace`. The 23 `--frozen-lockfile` tests for edits that don't change any package also pass on 1.4.3's install code. **Peer dependency fix, with the real packages.** Every install order, both linkers: | Scenario | 1.4.3 and `main` | This PR | |---|---|---| | `elysia` with `@jest/schemas` or `jest-extended`: loads and serves a request | 4 of 18 | 18 of 18 | | A project with only `ajv-keywords@3.5.2` and `ajv-formats@3.0.1`: `ajv-keywords` works | 1 of 5 | 5 of 5 | | 15 monorepo layouts that differ in which package declares `react`, checked with a real render: lockfiles with two copies of `react` | 0 of 30 | 0 of 30 | | `type-graphql` without `graphql` in `package.json`, then `bun add` a package that wants an older `graphql`: `buildSchema` works | yes | yes | **Peer dependency fix, on the 51 projects in this repo that have a `bun.lock`,** compared with `main`: - A fresh resolve produces an identical lockfile in 47 of the 48 projects that resolve. The other one is `test/`, where elysia now gets a matching typebox. - `install`, `ci`, `add`, `remove` and `update` on the checked-in lockfiles produce identical lockfiles in 239 of 239 runs. - In 84 `bun add` runs chosen so the new package's peer range doesn't match what's installed, 67 produce identical lockfiles. In the other 17, the new package gets a version within its range. - In all of these, no peer dependency that was within range on `main` moved to a different version, and no regular dependency resolved differently. - 1.4.3 and `main` can read the lockfiles this PR writes for the two elysia projects without modifying them, and elysia works. **Upgrade from 1.4.3.** 996 random projects nested up to 7 levels deep, installed with 1.4.3's install code, then updated and reinstalled over the old tree. Number of projects where a package ends up loading a version outside its declared range: | Project | 1.4.3 (299 projects) | `main` | This PR | |---|---|---|---| | Has a `prepare` script, installed with `--ignore-scripts` | 0 | 50 | 0 | | No scripts | 0 | 0 | 0 | | Has a `prepare` script that runs | 0 | 0 | 0 | | No scripts, a quarter of the packages bundle a dependency | 0 | 5 | 5 | | `prepare` script, `--ignore-scripts`, and bundled dependencies | not run | 49 | 5 | The remaining 5 are inside packages with bundled dependencies, which the cleanup skips entirely, same as `main`. **CI.** Linux (including ASAN), Windows and macOS arm64 passed on `610f3ca95`. The commits after it merge `main` and reword comments, test names and three messages. The macOS x64 test jobs never got a machine. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
This PR adds build plugins to `bun bake`, from a `bake.config.ts` file. It is the second step of the new Bake design (https://github.com/alii/bake-rfc), after oven-sh#45682. ```ts // bake.config.ts import { defineConfig } from "bun:bake"; import tailwind from "bun-plugin-tailwind"; export default defineConfig({ plugins: [tailwind], }); ``` - The plugins are Bun plugins (`{ name, setup(build) }`). The dev server runs them on the pages and the files that the pages import, the same as the plugins in `[serve.static] plugins` of `bunfig.toml`. - The default export can also be a function. It gets `{ mode: "development" }`. - `bun:bake` is a new built-in module. It has one function, `defineConfig`, which returns its argument. It resolves only where `bun bake` is (the Bake feature flag). - The plugins of `bunfig.toml` still apply, before the plugins of `bake.config.ts`. How it works - `internal/bake.ts` loads `bake.config.ts` before it starts the server, and checks it. Then it gives the plugin objects and their names to the VM (`$newRustFunction("bake/config/plugins.rs", "setConfigPlugins")`). The VM keeps them in `bake_config_plugins` (a JS array) and `bake_config_plugin_names`. - The server already keeps the names of the `bunfig.toml` plugins on the VM, and loads them when a route needs plugins. Now it also starts that load when only `bake.config.ts` has plugins, and `loadAndResolvePluginsForServe` takes plugin objects as well as module names. - The dev server's Tailwind check (which bundles the Tailwind CSS again after a change) also looks at the names of the `bake.config.ts` plugins. What `bun bake` rejects, with an error - A default export that is not an object or a function. - An option other than `plugins`. - A plugin that is not an object or a function, has no `name`, or has no `setup()`. As in `Bun.build()`, a function with `name` and `setup` is a plugin. `bun bake` now always reloads `app/server.ts` in place on a change, as `bun --hot` does, so it has no `--watch` and `--hot`: - The CLI sets `HotReload::Hot` for `bun bake`. `bun bake --watch` is an unknown argument, and `bun --watch bake` is an error, also with `--check`. The CLI records `--watch` in `ctx.debug.watch_flag`, because `HotReload::Watch` alone does not tell it from `--hot` with `--check`. `bun bake --check` still runs as `--watch` (it starts again after a change, to type check again), as `bun --check` does with `--hot`. - An error that a change to `app/server.ts` or `bake.config.ts` can fix no longer exits: the process shows it and reloads after the fix, as under `bun --hot`. An error in the arguments, in the names of the pages, or "nothing to serve" still exits with code 1, because no change to a loaded file can fix it. - A bad `--port` or `--host` must not leave the process waiting for a change: the shared flag parser (`internal/serve_cli.ts`) takes a port only as a whole number from 0 to 65535, and a `Bun.serve()` failure at the start exits 1 when `--port` or `--host` is set. `bun ./index.html` gets the same port check: `--port=abc` is an error there too. - A mistake that `bun bake` finds in the arguments or the app's files (a `BakeError`) prints as one line, `error: <message>`, with no stack trace into `internal/bake.ts`. Then the process exits 1, or waits for a change when one can fix it. "Nothing to serve" now says what to add. An error that the app's own code throws keeps its stack trace. - `Bun__loadBuiltinEntryPoint` marks the promise of the built-in entry point as handled. The VM reports its rejection; before, the rejection tracker reported it a second time. `bun ./index.html` gets this too. - The plugins load once. When `bake.config.ts` changes, `bun bake` prints `"bake.config.ts" changed. Restart "bun bake" to use the new plugins.` Also in this PR: `bun bake` reads the `host` option of the `app/server.ts` export when it has no `hostname`, as `Bun.serve()` does (a review comment on oven-sh#45682). Not in this PR - Vite-style plugins (`transform`, `load` and other hooks). `bun bake` takes Bun plugins. - A restart of the process when `bake.config.ts` changes, so the new plugins apply on their own. Today `bun bake` asks for a restart. A follow-up PR. - The types of `bun:bake` and the docs are in the docs PR (oven-sh#45751), to merge when Bake is on in a release. Until then, `bun --check` on a project with `bake.config.ts` reports that `bun:bake` has no types. Tests - `test/cli/bake/bake.test.ts`: a `bake.config.ts` plugin (in a `defineConfig` function, with `mode`) and a `bunfig.toml` plugin both change the CSS of a page; a plugin function sees the project as `build.config.root` when there is no bunfig.toml; each error above; the restart note; a reload of `app/server.ts` on the same server, with no flag; `--watch` is refused. - `test/bake/serve-plugins-dev-server.test.ts`, `test/bake/dev/css.test.ts`, `test/bake/hmr-socket-protocol.test.ts` and the `Bun.serve` HTML tests pass, for the `bunfig.toml` plugins.
This PR adds `.tsx` and `.jsx` pages to `bun bake`, and frameworks in `bake.config.ts`. It is the next step of the new Bake design (https://github.com/alii/bake-rfc), after oven-sh#45902. ```tsx // app/pages/blog/[slug].tsx export default function Post({ params }) { return <h1>{params.slug}</h1>; } ``` - A framework decides how files become routes and how a route becomes HTML. `bun bake` has a default framework: the `.tsx` and `.jsx` pages of `app/pages/`, in Next.js pages style (`index.tsx`, `[slug].tsx`, `_layout.tsx` for layouts; files and folders that start with `_` are not pages). - The default framework uses the project's own `react` and `react-dom`, not a copy in Bun. It streams the HTML from the server, then hydrates it in the browser. Fast Refresh keeps the state of a page when you edit it. - Before it starts, `bun bake` checks the project's React for the default framework. It exits with a one-line error when React is not installed, when the version is not 18 or 19, or when `react` and `react-dom` are different versions. Each error gives the `bun add` command that fixes it. - A plugin in `bake.config.ts` can bring its own framework instead. Its `framework` is the same object as the `framework` of the `app` option of `Bun.serve()`, read by the same parser: its routers (`fileSystemRouterTypes`: folder, style, entry points, extensions, layouts), `serverComponents`, `reactFastRefresh`, `builtInModules` and `plugins`. ```ts export default { plugins: [{ name: "my-framework", framework: { fileSystemRouterTypes: [{ root: "src/routes", style: "nextjs-pages", serverEntryPoint: "./framework/server.ts", clientEntryPoint: "./framework/client.ts", clientPages: true, }], }, }], }; ``` A plugin with a `framework` needs no `setup()`. Two plugins with a framework are an error. `.html` pages in `app/pages/` and `app/server.ts` work with every framework. - A router type has a new option, `clientPages`. With it, the pages and layouts also go into the browser build, and the client entry point loads those of the current route from `route` of `bun:bake/client`, to hydrate the server's HTML. It cannot be used with `serverComponents`, where the browser gets the client components instead. The default framework sets it. How it works - `internal/bake.ts` checks the `framework` of a plugin when it loads `bake.config.ts`: it runs the parser of `Bun.serve()` on it (`checkFramework`), and resolves its entry points from the project root. A mistake is then a one-line error of `bake.config.ts`, and `bun bake` waits for a fix. - `internal/bake.ts` gives the framework to its own `Bun.serve()` through a private function (`setPages`), not a new public option. `ServerConfig` parses the plugin's object with `Framework::from_js`, or makes the default with `Framework::bake_pages`. - With `clientPages`, the dev server adds the page and its layouts to the client graph, traces them into the route's browser bundle, and writes their IDs into `config.route` of the bundle. The browser runtime gives them to `bun:bake/client` as `route.page()` and `route.layouts()`. - A change to a page changes both its server copy and its browser copy. The browser runtime used to see only "server code changed" and dropped the JavaScript part of the update. When `config.route` is set, it now applies that part too, so Fast Refresh updates the page in place. - In the dev server, `bun:bake/client` and `bun:bake/server` now resolve for every framework. Before, they resolved only with server components. The HMR runtime always provides them. Fixes in the same code - Two pages with the same URL pattern (`[a].tsx` and `[b].tsx`) panicked with "TODO: propagate aliased route error". The router now reports its usual "Multiple pages matching the same route pattern is ambiguous" error, with both files, and keeps serving. - A route of the server (`app/server.ts` routes, `.html` pages) with the same path as a framework page hid the page with no message. The dev server now prints `error: The server has a route "/about", which hides the page app/pages/about.ts`, and keeps serving. It checks when the server sets its routes (at the start and on a reload of `app/server.ts`) and when the router adds a page; the test covers the start. - A request for the browser bundle of a route whose router has no client entry point panicked (`assertion failed: self.current_chunk_len > 0`). It now gets a 404, and `meta.modules` is empty for such a route. Not in this PR - `bun bake build` (a production build). - A custom routing style: `style` is one of the three Next.js styles. The function form in the types was never implemented. - The docs of frameworks in `bake.config.ts` and of `route`. They go in the docs PR (oven-sh#45751). Tests - `test/cli/bake/bake.test.ts`: the default framework with a layout and a param, next to an `.html` page and `app/server.ts` routes; the three React errors; a framework plugin with its own folder (`src/routes`) and extension, no React and no client entry point; `clientPages` puts only the current route's page in its browser bundle; two pages with the same pattern; a page that an `app/server.ts` route hides; two frameworks; a framework that the parser rejects; a framework entry point that cannot be found; `clientPages` with `serverComponents`. - `test/bake/dev/react-spa.test.ts`: in the test browser, a page of the default framework hydrates (a click changes the state), then an edit to the page keeps the state (`clicks 2`, not a reload). - All of the new tests fail on the released Bun. React 18 (the test suite's) and React 19 both pass the hydration test locally. - In `test/bake/`, 31 tests fail locally: the `production`, `server-sourcemap`, `react-response`, `ssg-pages-router` and `request-cookies` tests install the experimental React of the server components framework first, and that install fails on my machine's package mirror. The rest pass. CI runs them all.
Merge oven-sh/bun at ad4089b while retaining the fork's compatibility, fd tracking, cork, source-map and source-sharing fixes. Reconcile binding migrations and Bake entry points, preserve structured file-URL referrer decoding, and adopt native WebSocket pause/resume after the original TCP/TLS regressions pass on both parents and the candidate. The published WebKit manifest remains unchanged pending the matched engine archive. Qualification uses the replay of WebKit 87ee10da with the fork patches retained.
|
Hi! I'm the It looks like you correctly set up a CI job that uses the autofix.ci GitHub Action, but the autofix.ci GitHub App has not been installed for this repository. This means that autofix.ci unfortunately does not have the permissions to fix this pull request. If you are the repository owner, please install the app and then restart the CI workflow! 😃 |
autofix.ci
|
Dependency limit exceeded — report not shown. This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report. Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard. Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account. |
steipete
marked this pull request as ready for review
October 19, 2026 04:33
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs oven-sh/bun main at
ad4089bf49d9f4c3ab7c8d95a58f3e0f0f186054into fork main7bd4fdff3d4035bcea50718673c663f5474bae89, preserving both parents and upstream history. This imports 58 commits sincef2405c382d43b89d91a2e7c90d9c5f53505a1e71, including the lint/format fixes for oven-sh#45479 and oven-sh#45480 and adaptation to WebKit87ee10da.The resolution retains Worker descriptor tracking, corked HTTP write ownership, sync-aware CI selection, source maps, VM source sharing, CA flag handling, and the other fork runtime contracts. Qualification repaired the built-in Bake entry's interaction with loop-start tracking, iterator-aware Headers conversion in the new upgrade parser, and native WebSocket close scheduling after HTTP handoff. The imported auto-install fixture now reserves the runtime-only flag for runtime commands.
This PR must land as a merge commit.
Published engine
The manifest is a whole-file copy from the immutable WebKit release autobuild-a2976f02e69fde8ccc45dc8fb346173f9d63f8a1. Downloaded manifest SHA-256:
50aee9887e8bcd083e13aa625fbd30a5e42f2e8312e843df111261b4f54d3ac3, byte-identical to the verified release copy.WebKit qualification used Bun
cabfa8138d20ca57ccfb0a9309103ebb55900125. Subsequent drift consists of the four-line lint cleanup at346dbe95ab307b840c935e477cf8a5918efbfde5, the manifest pin ate2144a73893c6dee76ce98ee31d875deee16cb7a, and the two macOS-native realpath expectations atcc31b986a39a9f9c9b1f520f4082725d0e123ca5described below. The cleanup caches a Bake property read and removes two unused bindings.Pinned-candidate proof
The Linux x64 release build used the normal prebuilt WebKit path with archive verification, without local engine overrides. The binary reports the pinned WebKit commit. Build time: 192 seconds.
bun lint --no-scriptis accepted on clean OpenClaw main at8a23c56a83384d40e89c3dd8d92f7cee5e520c10. It matches oxlint 1.87.0 exactly: 4,469 findings (4,281 errors and 188 warnings) across 49,370 files and 258 rules, with zero extra or missing file/rule/location tuples, severities and all labeled spans at either thread setting. Existing diagnostics produce exit 1 in both tools.Single-run wall times for the JSON finding comparison:
The exact plain
bun lint --no-scriptinvocation took 2.498 seconds.bun fmt --check --no-scriptis accepted and matches oxfmt 0.72.0: zero files differ across 52,208 files, confirmed with both tools'--list-differentoutput. Check times were 0.559 seconds for Bun and 12.529 seconds for oxfmt. The nativefmtspelling is used because this project defines aformatpackage script. Source and package configuration remained unchanged.Upstream-change qualification
Before the published pin, the matched engine replay passed the native parity set, fork regressions, TLS/duplex, HTTPS, HTTP and HTTP/2 suites. Owner-exit stress had 0/50 failures. Five paired cork-benchmark rounds per size averaged −0.107% at 5 B and +2.482% at 64 KiB, inside the 3% regression gate. Both lint repros pass.
Every one of the 1,120 selected files ran, including all 308 surviving directly changed tests. The final full sweep at
cabfa8138d20ca57ccfb0a9309103ebb55900125records 1,116 pass / 4 environment failures, with both dependency steps passing. Every observed failure has three-way comparison evidence. The four final failures concern missing S3 credentials, an unassigned custom IPv4 address, a file-mode fixture mismatch, and system Git's Username/Password prompt difference. No unresolved merge bug remains; no environment failure or intermittent test was skipped.The original and intermediate sweeps were 1,095 pass / 25 fail and 1,111 pass / 9 fail. Their failures remain recorded; the final sweep passes the diagnosed SQL saturation, CSS timing, proxy memory and parallel-runner cases. The lint cleanup was rebuilt and passed 80 Bake tests, 155 WebSocket/Undici tests, JavaScript lint, formatting and all five CI TypeScript configurations.
The narrowly marked inherited differences remain: a POSIX trailing directory backslash in
path.relative(), malformed or unreadable entry-adjacent package configuration failing before user code, and npmwsover TLS losing queued frames when terminated inside a large-write callback. Corresponding Node expectations and follow-up evidence are retained.Independent Codex review at P2 covers the resolution, subsequent repairs, published pin and realpath expectation change. No accepted actionable finding remains.
macOS test results
The native expectations for
file/andfile/..now follow macOSrealpath(3), as used by Node's native variants. Node 24.22.0 and 26.12.0 each match all 175 matrix cells with zero differences. Only those two native expectation cells depend on the platform; Linux still expectsENOTDIR. The fork's descriptor-free realpath implementation remains unchanged, preserving process-owned POSIX locks.The SQLite
a timeout of node:vmfailure remains unresolved and not reproduced. The old engine passed in the pre-pin CI run; the new engine failed thevalues()anditerate()cases in the post-pin CI run. On a macOS 27 arm64 machine, using the identical CI artifact binaries and the full SQLite file with the CI runner and runtime settings, both engines passed 12/12 runs versus 12/12, and 6/6 versus 6/6 under CPU load. Each run reported 657 pass, 5 skip and 0 fail. That local comparison does not clear the original macOS 15 CI failure. No SQLite test, timeout or engine change was made.Current CI
All 11 required checks pass at
cc31b986a39a9f9c9b1f520f4082725d0e123ca5, including native macOS and Linux and all Rust checks. No CI rerun was requested. The earlier SQLite timeout remains documented above as unresolved and not reproduced.