Skip to content

Commit 7bde3ba

Browse files
Rebuild gem native extensions in agent-mode apply (#1996) (#1998)
* Rebuild gem native extensions in agent-mode apply (#1996) Agent-mode apply rewrote a gem's ext/**/*.c in place and reported it patched, while Ruby kept loading the extension compiled at install time. When a gem record changes a native-extension build input, apply now writes the patched sources and rebuilds the extension with RubyGems' own builder (the step `gem pristine --extensions` runs), using the Ruby that owns the install. The old build is moved aside first (ExtConfBuilder never overwrites files already in the extension dir) and backed up once inside the gem dir; a marker records the source hashes it was built from and every artifact's hash, including the lib/ copies Ruby may load. - A failed build puts the sources and the previous build back: native_extension_build_failed. - A copy that cannot be rebuilt (precompiled platform gem, no specifications entry, no matching Ruby) is refused with native_extensions_unsupported and nothing is written. - Sources an earlier release patched without a rebuild are rebuilt (native_extension_rebuilt warning). - vex / apply --check attest only when the marker matches (native_extension_not_rebuilt otherwise); hosted copies are unaffected. - rollback restores the backed-up original build, or rebuilds from the restored sources without one (native_extension_restore_failed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Harden gem native rebuild; add --no-gem-native-rebuild Review fixes: - rollback restores only the gem's own extensions/<plat>/<abi>/<gem> dir under its home; a backup or marker naming any other path (or a symlink) is refused before anything is deleted. - In a gem that declares no extensions, only C sources under ext/ are native inputs (templates and fixtures elsewhere stay plain files); a root-level build script makes only top-level files inputs. - A build whose record cannot be written is undone like a failed one. - The Ruby must build into an installed extension dir; a gem without one (RubyGems < 2.2) is refused. - Lib copies are restored by stage + rename, never rewritten in place under a process that has them mapped. - A build timeout kills the whole process group (Unix). - Probe output tolerates stray stdout lines; the script requires rubygems itself (Ruby 1.8 with RUBYOPT cleared) and names a RubyGems older than 2.2. - Share gemspec_declares_extensions and copy_tree_blocking. --no-gem-native-rebuild (SOCKET_NO_GEM_NATIVE_REBUILD) is for a caller that compiles the extension itself, such as the patches evaluation harness: sources are written without a rebuild, with a native_extension_not_rebuilt warning, and vex does not attest them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bound gem builds in utils::process; guard reads; dry-run rebuild - CI (child_deadline_guard): the Ruby probe/build no longer keeps its own kill_on_drop deadline. New utils::process::build_output_within is the bounded spawn for builds: stdout and stderr captured, the child in its own process group (Unix) killed whole at the deadline. - Bugbot: the lib-copy snapshot, the previous marker, backup.json and the backed-up lib copies are read through the regular-file guard, so a FIFO in the gem dir fails fast instead of hanging apply or rollback. - Bugbot: a dry run over sources that are already patched but whose extension was not built from them warns native_extension_rebuilt ("apply would rebuild it") instead of reading as plain already_patched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 531f591 commit 7bde3ba

16 files changed

Lines changed: 2683 additions & 20 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1271,8 +1271,10 @@ jobs:
12711271
# Ruby <= 3.3 and 1.17-2.1 need Ruby <= 3.1 (`untaint`). 2.7.2
12721272
# (no boundary of its own) is in e2e-full.
12731273
# Hosted and vendored share each era's leg (one Ruby + bundler
1274-
# setup; the run step loops over `suite`).
1275-
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build, ruby: '3.4', bundler: '4.0.21'}
1274+
# setup; the run step loops over `suite`), as does the agent-mode
1275+
# C-extension rebuild capstone (#1996), which needs the same Ruby
1276+
# plus the runner's C compiler.
1277+
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build e2e_gem_native_ext_build, ruby: '3.4', bundler: '4.0.21'}
12761278
# The live-API smoke suites (e2e_npm, e2e_pypi, e2e_gem,
12771279
# e2e_scan) are intentionally NOT in the PR matrix — their
12781280
# `#[ignore]`-gated tests hit the real public proxy at
@@ -2008,7 +2010,7 @@ jobs:
20082010
# composer 1.10 keeps the PHP 8.2 it has always run on here
20092011
# (composer-compatibility.yml runs 1.10.28 on 8.1).
20102012
- {os: ubuntu-latest, suite: e2e_vendor_composer_build e2e_redirect_composer_build, composer: '1', php: '8.2'}
2011-
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build, ruby: '3.1', bundler: '1.17.3'}
2013+
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build e2e_gem_native_ext_build, ruby: '3.1', bundler: '1.17.3'}
20122014
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build, ruby: '3.1', bundler: '2.1.4'}
20132015
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build, ruby: '3.1', bundler: '2.2.33'}
20142016
- {os: ubuntu-latest, suite: e2e_redirect_gem_build e2e_vendor_gem_build, ruby: '3.3', bundler: '2.5.23'}

‎crates/socket-patch-cli/CLI_CONTRACT.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -705,6 +705,49 @@ are patched by `socket-patch apply` like every other ecosystem:
705705
or a bare `--global-prefix` cache with no known `go install` directory) the same warning code says
706706
so, and VEX is not withheld (#422).
707707

708+
### Ruby gems with native extensions in agent mode
709+
710+
Ruby loads the extension RubyGems compiled at install time
711+
(`extensions/<platform>/<abi>/<gem>-<version>/`, and its copy under the gem's `lib/`), never
712+
the `ext/` sources beside it. When a gem record changes a native-extension build input, `apply`
713+
writes the patched sources and then rebuilds the extension from them with RubyGems' own builder
714+
(the step `gem pristine --extensions` runs), using the Ruby that owns the install: the one in
715+
`<prefix>/bin` of a `<prefix>/lib/ruby/gems/<abi>` home, else the `ruby` on `PATH` run from
716+
`--cwd`. A build input is a changed `.c`, `.h`, `.cc`, `.cpp`, `.cxx`, `.hh`, `.hpp`, `.hxx`,
717+
`.m`, `.y` or `.rl` file, an `extconf.rb`, a `depend` under `ext/`, or any file under the
718+
directory of a build script the installed gemspec's `extensions` names (a script at the gem root:
719+
the files beside it) (#1996). In a gem that declares no extensions, only such a source under
720+
`ext/` is one (where a precompiled platform gem keeps the sources of the binary it ships), and
721+
never a grammar file (`.y`, `.rl`), whose generated output the gem packs. The Ruby must build into
722+
the gem's installed `extensions/<platform>/<abi>/` dir; a gem without one (installed by a RubyGems
723+
older than 2.2) is refused. The rebuild runs the gem's own build scripts, as `gem install` did.
724+
725+
The first rebuild of a copy backs up the install-time build (extension directory and `lib/`
726+
copies) to `.socket-patch-native-orig/` in the gem directory, and every rebuild records
727+
`.socket-patch-native-build.json` there: the source hashes it was built from and the hash of every
728+
artifact it produced. A record whose sources are patched and whose marker matches is
729+
`already_patched`; sources an earlier release patched without a rebuild are rebuilt, with a
730+
`native_extension_rebuilt` warning. `rollback` (and `remove`) puts the backed-up build back with
731+
the sources; without a backup it rebuilds from the restored sources. Only the gem's own
732+
extension dir under its home is ever replaced, whatever path the backup or marker names.
733+
734+
`--no-gem-native-rebuild` (env `SOCKET_NO_GEM_NATIVE_REBUILD`) is for a caller that compiles the
735+
extension itself (the patches evaluation harness): the patched sources are written as before #1996,
736+
with no Ruby run and no backup or marker, and the run succeeds with a `native_extension_not_rebuilt`
737+
warning. `vex` and `apply --check` then omit the patch until a rebuild by `apply` records it.
738+
739+
| Code | Run result | Meaning |
740+
|---|---|---|
741+
| `native_extensions_unsupported` | refused, exit 1 | The record changes build inputs that cannot be rebuilt in place: the copy is not a RubyGems install with a `specifications/` entry (a Bundler git checkout), the gem declares no extensions (a precompiled platform gem), or no Ruby whose extension directory for the gem is the installed one was found. Nothing is written; use `scan --mode hosted`, where RubyGems compiles the patched gem. |
742+
| `native_extension_build_failed` | refused, exit 1 | The extension did not build from the patched sources (no compiler or Ruby headers, a build error). The previous build and the original sources are put back; the error carries the build output's tail. |
743+
| `native_extension_rebuilt` | warning | The sources were already patched (by a release before #1996) and the extension was rebuilt from them. |
744+
| `native_extension_not_rebuilt` | warning | `--no-gem-native-rebuild`: the sources were written without a rebuild; Ruby loads the old build until the caller compiles it. |
745+
| `native_extension_restore_failed` | rollback: that copy fails | The sources were restored, but the original compiled extension could not be put back. Run `gem pristine <gem> --extensions`. |
746+
747+
`vex` and `apply --check` attest such a copy only while its marker matches the record's patched
748+
sources and every artifact it lists; otherwise the purl fails with `native_extension_not_rebuilt`.
749+
Hosted copies are judged on their files alone, since RubyGems compiled them from the patched gem.
750+
708751
### sbt / Mill / scala-cli caches in agent mode
709752

710753
A JVM project root (any `JVM_PROJECT_MARKERS` file, `build.sbt` / `build.mill` / `build.sc` /

‎crates/socket-patch-cli/src/args.rs‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -350,6 +350,21 @@ pub struct GlobalArgs {
350350
value_parser = parse_bool_flag,
351351
)]
352352
pub no_vlt_install_cleanup: bool,
353+
354+
/// Agent mode (`apply`, `scan`/`get --mode agent`): write a gem patch
355+
/// that changes native-extension sources WITHOUT rebuilding the
356+
/// extension, for a caller that compiles it itself. Ruby keeps loading
357+
/// the old build until then, so `vex` and `apply --check` do not attest
358+
/// those patches (`native_extension_not_rebuilt`). Other subcommands
359+
/// accept it silently.
360+
#[arg(
361+
help_heading = GLOBAL_OPTIONS,
362+
long = "no-gem-native-rebuild",
363+
env = "SOCKET_NO_GEM_NATIVE_REBUILD",
364+
default_value_t = false,
365+
value_parser = parse_bool_flag,
366+
)]
367+
pub no_gem_native_rebuild: bool,
353368
}
354369

355370
impl GlobalArgs {
@@ -641,6 +656,7 @@ pub const GLOBAL_ARG_ENV_VARS: &[&str] = &[
641656
"SOCKET_NO_TRUST_LOCKFILE_CONFIG",
642657
"SOCKET_NO_NPM_ALLOW_REMOTE_CONFIG",
643658
"SOCKET_NO_VLT_INSTALL_CLEANUP",
659+
"SOCKET_NO_GEM_NATIVE_REBUILD",
644660
];
645661

646662
/// Every env var a subcommand-local flag binds (one per `env = "..."`
@@ -709,6 +725,7 @@ impl Default for GlobalArgs {
709725
no_trust_lockfile_config: false,
710726
no_npm_allow_remote_config: false,
711727
no_vlt_install_cleanup: false,
728+
no_gem_native_rebuild: false,
712729
}
713730
}
714731
}

‎crates/socket-patch-cli/src/commands/apply.rs‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ use socket_patch_core::patch::apply::{
1212
apply_package_patch, verify_file_patch, ApplyResult, MismatchPolicy, PatchSources, VerifyStatus,
1313
};
1414
use socket_patch_core::patch::apply_lock::LockGuard;
15+
use socket_patch_core::patch::gem_ext;
1516
use socket_patch_core::patch::redirect::golang_local::{
1617
apply_go_redirect, reconcile_go_redirects, verify_go_redirect_state,
1718
};
@@ -717,6 +718,10 @@ fn describe_check_failure(reason: &str) -> &'static str {
717718
"hash_mismatch" => "an installed copy matches neither the original nor the patched bytes",
718719
"file_not_found" => "a patched file is missing from an installed copy",
719720
"no_files" => "the patch record lists no files to verify",
721+
"native_extension_not_rebuilt" => {
722+
"the patched native-extension sources were not compiled into the extension \
723+
Ruby loads"
724+
}
720725
"no_matching_variant" => {
721726
"an installed copy matches none of the manifest's release variants \
722727
(no matching variant found)"
@@ -2703,15 +2708,42 @@ async fn apply_patches_inner(
27032708
}
27042709

27052710
attempted = true;
2706-
let result = apply_package_patch(
2707-
variant_purl,
2708-
pkg_path,
2709-
&patch.files,
2710-
&sources,
2711-
args.common.dry_run,
2712-
policy,
2713-
)
2714-
.await;
2711+
// A gem patch to native-extension sources rebuilds the
2712+
// extension, or is refused with nothing written (#1996).
2713+
let result = if Ecosystem::from_purl(purl) == Some(Ecosystem::Gem) {
2714+
let applied = gem_ext::apply_gem_package_patch(
2715+
variant_purl,
2716+
pkg_path,
2717+
&patch.files,
2718+
&sources,
2719+
args.common.dry_run,
2720+
policy,
2721+
&gem_ext::RubyEnv {
2722+
cwd: args.common.cwd.clone(),
2723+
rebuild: !args.common.no_gem_native_rebuild,
2724+
},
2725+
)
2726+
.await;
2727+
// A best-effort fallback-home copy's refusal is
2728+
// its per-copy skip below, not a run warning.
2729+
if let Some((code, detail)) = applied
2730+
.warning
2731+
.filter(|_| applied.result.success || !best_effort)
2732+
{
2733+
run_warnings.push(RunWarning { code, detail });
2734+
}
2735+
applied.result
2736+
} else {
2737+
apply_package_patch(
2738+
variant_purl,
2739+
pkg_path,
2740+
&patch.files,
2741+
&sources,
2742+
args.common.dry_run,
2743+
policy,
2744+
)
2745+
.await
2746+
};
27152747

27162748
warn_mismatch_overwrites(&result, &args.common);
27172749
// A variant that reached apply is the installed distribution

‎crates/socket-patch-cli/src/commands/rollback.rs‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3276,14 +3276,32 @@ pub(crate) async fn rollback_patches_inner(
32763276
match try_rollback_local_go(purl, pkg_path, patch, common).await {
32773277
Some(r) => r,
32783278
None => {
3279-
rollback_package_patch(
3279+
let mut result = rollback_package_patch(
32803280
purl,
32813281
pkg_path,
32823282
&patch.files,
32833283
&blobs_path,
32843284
common.dry_run,
32853285
)
3286-
.await
3286+
.await;
3287+
// A gem whose native extension `apply` rebuilt gets its
3288+
// original compiled extension back with its sources
3289+
// (#1996).
3290+
if result.success
3291+
&& !common.dry_run
3292+
&& Ecosystem::from_purl(purl) == Some(Ecosystem::Gem)
3293+
{
3294+
use socket_patch_core::patch::gem_ext;
3295+
let env = gem_ext::RubyEnv {
3296+
cwd: common.cwd.clone(),
3297+
rebuild: !common.no_gem_native_rebuild,
3298+
};
3299+
if let Err(e) = gem_ext::restore_original_build(pkg_path, &env).await {
3300+
result.success = false;
3301+
result.error = Some(format!("{}: {e}", gem_ext::RESTORE_FAILED_CODE));
3302+
}
3303+
}
3304+
result
32873305
}
32883306
}
32893307
};

‎crates/socket-patch-cli/src/commands/vex.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2091,6 +2091,10 @@ fn omission_phrase(reason: &str) -> &'static str {
20912091
against"
20922092
}
20932093
"stale_install" => "the installed copy is not patched",
2094+
"native_extension_not_rebuilt" => {
2095+
"the patched native-extension sources were not compiled into the extension Ruby \
2096+
loads; run socket-patch apply to rebuild it"
2097+
}
20942098
DEPENDENCY_NOT_INSTALLED => {
20952099
"the patched package.json declares a dependency that is not installed beside \
20962100
the package"

‎crates/socket-patch-cli/tests/cli_global_args.rs‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,9 @@ fn global_flag_cases() -> Vec<(&'static str, Option<&'static str>, fn(&GlobalArg
126126
("--no-vlt-install-cleanup", None, |c| {
127127
assert!(c.no_vlt_install_cleanup)
128128
}),
129+
("--no-gem-native-rebuild", None, |c| {
130+
assert!(c.no_gem_native_rebuild)
131+
}),
129132
("--lock-timeout", Some("30"), |c| {
130133
assert_eq!(c.lock_timeout, Some(30))
131134
}),
@@ -233,6 +236,7 @@ fn global_flag_cases_cover_every_global_field() {
233236
no_trust_lockfile_config: _,
234237
no_npm_allow_remote_config: _,
235238
no_vlt_install_cleanup: _,
239+
no_gem_native_rebuild: _,
236240
strict: _,
237241
allow_patch_conflicts: _,
238242
vendor_source: _,
@@ -241,11 +245,11 @@ fn global_flag_cases_cover_every_global_field() {
241245
maven_config: _,
242246
} = common;
243247

244-
// 27 fields ↔ 27 long-flag cases. Bump both this count and add a case when
248+
// 28 fields ↔ 28 long-flag cases. Bump both this count and add a case when
245249
// the destructure above forces you to add a field.
246250
assert_eq!(
247251
global_flag_cases().len(),
248-
27,
252+
28,
249253
"every GlobalArgs field needs a long-flag case in global_flag_cases()",
250254
);
251255
}
@@ -592,7 +596,7 @@ fn bool_env_vars_reject_zero_and_falsey() {
592596
#[serial_test::serial]
593597
fn empty_bool_env_var_resolves_to_false_not_crash() {
594598
// (env var, accessor) for every boolean global.
595-
let bool_vars: [(&str, fn(&GlobalArgs) -> bool); 13] = [
599+
let bool_vars: [(&str, fn(&GlobalArgs) -> bool); 14] = [
596600
("SOCKET_OFFLINE", |c| c.offline),
597601
("SOCKET_STRICT", |c| c.strict),
598602
("SOCKET_GLOBAL", |c| c.global),
@@ -612,6 +616,7 @@ fn empty_bool_env_var_resolves_to_false_not_crash() {
612616
("SOCKET_NO_VLT_INSTALL_CLEANUP", |c| {
613617
c.no_vlt_install_cleanup
614618
}),
619+
("SOCKET_NO_GEM_NATIVE_REBUILD", |c| c.no_gem_native_rebuild),
615620
];
616621

617622
for (var, accessor) in bool_vars {

‎crates/socket-patch-cli/tests/common/clap_parse.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ pub fn global_defaults() -> Globals {
7979
no_trust_lockfile_config: false,
8080
no_npm_allow_remote_config: false,
8181
no_vlt_install_cleanup: false,
82+
no_gem_native_rebuild: false,
8283
})
8384
}
8485

0 commit comments

Comments
 (0)