From 800503c156fe7c2fac6b4664749f85f2b39865f2 Mon Sep 17 00:00:00 2001 From: Steven Pritchard Date: Thu, 6 Aug 2026 12:57:33 -0500 Subject: [PATCH 1/3] Fall back to the vendored PEM CA bundle for the system trust store include_system_store requests from the server (report processors posting to external HTTPS endpoints, and any Puppet.runtime[:http] caller passing the option) stopped trusting publicly-signed certificates in the 9.0.0 betas, failing with PKIX path building errors. This is what broke acceptance/suites/tests/https/client_may_use_external_cert_chains.rb on all platforms in acceptance run 31098859281. Root cause: Puppet::Server::Config.load_puppet_and_system_ssl_context builds the system trust by merging a vendored Java keystore from /opt/puppetlabs/puppet/ssl/puppet-cacerts, and puppet-runtime deliberately stopped shipping that keystore for the 9.x agent (puppet-runtime@066fd48, on the reasonable suspicion it was an unused PE-era artifact -- OpenVox does not package Java). The server was its one consumer; with the file absent the context silently degrades to puppet-CA-only trust with a single warning. Fall back to the PEM CA bundle the runtime still ships at /opt/puppetlabs/puppet/ssl/cert.pem, loaded through the same SSLUtils reader already used for the ssl_trust_store setting. The keystore is still preferred when present, so 8.x-era agents are unaffected. Trust content is identical either way (both forms are generated from the same Mozilla bundle). Verified against openvox-server 9.0.0~beta4 with the 9.0.0~beta2 agent on a local beaker rig: the isolated reproduction (puppetserver ruby fetching https://voxpupuli.org with include_system_store: true) fails with PKIX before and returns 200 after, and the full client_may_use_external_cert_chains.rb acceptance test passes with the patched jar. Longer term this pairs with openvox#554 (stop bundling CA certificates): when that lands, this method should grow a further fallback to the system trust anchors. This change keeps 9.0 compatible with the trust behavior 8.x shipped. Signed-off-by: Steven Pritchard Co-authored-by: Claude Fable 5 --- spec/puppet-server-lib/puppet/jvm/config_spec.rb | 14 +++++++++++++- src/ruby/puppetserver-lib/puppet/server/config.rb | 11 ++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/spec/puppet-server-lib/puppet/jvm/config_spec.rb b/spec/puppet-server-lib/puppet/jvm/config_spec.rb index 9fd05e842..a03a2844d 100644 --- a/spec/puppet-server-lib/puppet/jvm/config_spec.rb +++ b/spec/puppet-server-lib/puppet/jvm/config_spec.rb @@ -42,11 +42,23 @@ end end + it "falls back to the vendored PEM CA bundle when the keystore is absent" do + stub_const('Puppet::Server::Config::PUPPET_KEYSTORE_LOCATION', + 'spec/fixtures/does-not-exist') + stub_const('Puppet::Server::Config::PUPPET_CA_BUNDLE_LOCATION', + 'spec/fixtures/ca-cert.pem') + + expect(Puppet).not_to receive(:warning) + + ssl_context = Puppet::Server::Config.puppet_and_system_ssl_context + expect(ssl_context).to be_a_kind_of(Java::JavaxNetSsl::SSLContext) + end + it "warns if :ssl_trust_store is set but not readable" do Puppet[:ssl_trust_store] = "spec/fixtures/foo.pem" allow(File).to receive(:exist?).and_return(false) - expect(Puppet).to receive(:warning).with(/Could not find OpenVox-vendored keystore/) + expect(Puppet).to receive(:warning).with(/Could not find an OpenVox-vendored trust store/) expect(Puppet).to receive(:warning).with(/The 'ssl_trust_store' setting does not refer to a file/) Puppet::Server::Config.puppet_and_system_ssl_context diff --git a/src/ruby/puppetserver-lib/puppet/server/config.rb b/src/ruby/puppetserver-lib/puppet/server/config.rb index a9c714178..2a1711f6b 100644 --- a/src/ruby/puppetserver-lib/puppet/server/config.rb +++ b/src/ruby/puppetserver-lib/puppet/server/config.rb @@ -19,6 +19,7 @@ class Puppet::Server::Config PUPPET_KEYSTORE_LOCATION = '/opt/puppetlabs/puppet/ssl/puppet-cacerts' + PUPPET_CA_BUNDLE_LOCATION = '/opt/puppetlabs/puppet/ssl/cert.pem' CERT_REGEX = /.*-----BEGIN CERTIFICATE-----.*/ def self.initialize_puppet_server(puppet_server_config) @@ -82,8 +83,16 @@ def self.load_puppet_and_system_ssl_context truststore = stores['truststore'] if File.exist?(PUPPET_KEYSTORE_LOCATION) associate_entries(truststore, PUPPET_KEYSTORE_LOCATION) + elsif File.exist?(PUPPET_CA_BUNDLE_LOCATION) + # puppet-runtime stopped shipping the Java keystore form of the + # vendored CA bundle (puppet-runtime@066fd48); load the PEM bundle + # it still ships instead. + SSLUtils.associateCertsFromReader( + truststore, + 'openvox_vendored_ca_bundle', + FileReader.new(PUPPET_CA_BUNDLE_LOCATION)) else - Puppet.warning("Could not find OpenVox-vendored keystore at '#{PUPPET_KEYSTORE_LOCATION}'") + Puppet.warning("Could not find an OpenVox-vendored trust store at '#{PUPPET_KEYSTORE_LOCATION}' or '#{PUPPET_CA_BUNDLE_LOCATION}'") end if additional_store_location = Puppet[:ssl_trust_store] From 841111bd9ab748ec24480c98ad8b9a41fbd4bdfe Mon Sep 17 00:00:00 2001 From: Steven Pritchard Date: Thu, 6 Aug 2026 17:19:51 +0000 Subject: [PATCH 2/3] Fix the report authorization test's is_pe? misfire The report/notme step asserted that the primary may submit reports on behalf of other nodes when master.is_pe? is true. This suite's beaker options do not set a host type, and beaker's default type is 'pe', so is_pe? returned true on every FOSS host and selected the PE assertion -- which the shipped allow: "$1" report rule correctly rejects with 403. This is the step where default_rules.rb actually failed in acceptance run 31098859281, before the filebucket assertions were ever reached. OpenVox has no PE edition, so drop the PE arm and assert the denial unconditionally, with a comment warning against reintroducing is_pe? guards while the suite's host type is unset. Verified on a local beaker rig against openvox-server 9.0.0~beta4: the unmodified test fails at report/notme exactly as in the pipeline; with this commit plus the filebucket fix already on this branch, default_rules.rb passes end to end (including the new agent-cert filebucket denial and primary-cert 404 assertions). Co-Authored-By: Claude Fable 5 Signed-off-by: Steven Pritchard --- .../suites/tests/authorization/default_rules.rb | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/acceptance/suites/tests/authorization/default_rules.rb b/acceptance/suites/tests/authorization/default_rules.rb index eb2366a4e..368759e49 100644 --- a/acceptance/suites/tests/authorization/default_rules.rb +++ b/acceptance/suites/tests/authorization/default_rules.rb @@ -98,15 +98,14 @@ def report_query(node) assert_allowed(stdout) end - # In PE, the master (specifically the orchestrator) - # is allowed to make report submissions on behalf of - # other nodes + # Nodes may submit only their own reports (allow: "$1"); submitting on + # behalf of other nodes was a PE orchestrator capability that OpenVox + # does not have. Note: this must not be guarded on master.is_pe? -- + # this suite's beaker options do not set a host type, and beaker's + # default type is 'pe', which made is_pe? return true on FOSS hosts + # and select the wrong assertion. curl_authenticated(report_query('notme')) do |stdout| - if master.is_pe? - assert_allowed(stdout) - else - assert_denied(stdout, /\/puppet\/v3\/report\/notme \(method :put\)/) - end + assert_denied(stdout, /\/puppet\/v3\/report\/notme \(method :put\)/) end curl_unauthenticated(report_query(masterfqdn)) do |stdout| From 7de7ac36607ba10252d3a0ba66b291cd0bae42b5 Mon Sep 17 00:00:00 2001 From: Steven Pritchard Date: Thu, 6 Aug 2026 15:18:39 +0000 Subject: [PATCH 3/3] Fix the filebucket read authorization test The filebucket check in default_rules.rb asserted that an authenticated request is refused a read, but curl_authenticated uses the primary's own certificate, and that certificate carries the pp_cli_auth extension so that the primary's CLI tooling can reach administrative endpoints: 1.3.6.1.4.1.34380.1.3.39: ..true The auth.conf rule gates reads on exactly that extension, so the request is allowed by design and the assertion could never pass. It failed on every platform. Assert instead that the primary's certificate is allowed through to the endpoint, and generate an agent certificate, which does not carry the extension, to cover the case the rule is meant to refuse. Also check that the agent certificate may still issue 'head', so that the store path stays open to agents while reads are restricted. Verified against openvox-server 9.0.0~beta4 on el9: primary cert HEAD -> 404 agent cert HEAD -> 404 primary cert GET -> 404 agent cert GET -> 403 no client cert GET -> 403 Co-authored-by: Claude Signed-off-by: Steven Pritchard --- .../tests/authorization/default_rules.rb | 53 ++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/acceptance/suites/tests/authorization/default_rules.rb b/acceptance/suites/tests/authorization/default_rules.rb index 368759e49..c851c1059 100644 --- a/acceptance/suites/tests/authorization/default_rules.rb +++ b/acceptance/suites/tests/authorization/default_rules.rb @@ -34,6 +34,22 @@ def curl_unauthenticated(path) on(master, curl) end +# Issue a request with a certificate other than the primary's own. The +# primary's certificate carries the pp_cli_auth extension so that its CLI +# tooling can reach administrative endpoints, which makes it unsuitable for +# testing rules that gate on that extension. An ordinary agent certificate +# does not carry it. +def curl_with_cert(certname, path, &block) + curl = 'curl ' + curl += "--cert $(puppet config print ssldir)/certs/#{certname}.pem " + curl += "--key $(puppet config print ssldir)/private_keys/#{certname}.pem " + curl += '--cacert $(puppet config print localcacert) ' + curl += "--write-out '\\nSTATUSCODE=%{http_code}\\n' " + curl += "https://#{master}:8140#{path}" + result = on(master, curl) + block.call(result.stdout) +end + def assert_allowed(stdout, expected_statuscode = 200) refute_match(/Forbidden request/, stdout) assert_match(/STATUSCODE=#{expected_statuscode}/, stdout) @@ -52,6 +68,27 @@ def report_query(node) curl += '\",\"metrics\":{},\"logs\":[],\"resource_statuses\":{}}"' end +# An ordinary agent certificate, used to test rules that are gated on the +# pp_cli_auth extension. Generated once the server is up, and cleaned up in +# the teardown below. +agent_certname = 'filebucket-read-test-agent' + +# 'puppetserver ca generate' refuses to run while any key file for the certname +# is still on disk, so clear them before generating as well as afterwards. That +# keeps the test repeatable if an earlier run was interrupted before teardown. +def revoke_and_remove_cert(certname) + on(master, "puppetserver ca clean --certname #{certname}", + :accept_all_exit_codes => true) + on(master, "rm -f $(puppet config print ssldir)/certs/#{certname}.pem " \ + "$(puppet config print ssldir)/private_keys/#{certname}.pem " \ + "$(puppet config print ssldir)/public_keys/#{certname}.pem", + :accept_all_exit_codes => true) +end + +teardown do + revoke_and_remove_cert(agent_certname) +end + with_puppet_running_on(master, {}) do masterfqdn = on(master, '/opt/puppetlabs/bin/facter fqdn').stdout.chomp @@ -142,6 +179,9 @@ def report_query(node) sum = 'a' * 64 bucket_path = "/puppet/v3/file_bucket_file/sha256/#{sum}?environment=production" + revoke_and_remove_cert(agent_certname) + on(master, "puppetserver ca generate --certname=#{agent_certname}") + # Agents are allowed 'head' so they can test whether content is already # stored before uploading it. We'd actually need to store a file in the # filebucket in order to get back a 200, but we know that a 404 means we got @@ -150,9 +190,20 @@ def report_query(node) assert_allowed(stdout, 404) end + curl_with_cert(agent_certname, "#{bucket_path} --head") do |stdout| + assert_allowed(stdout, 404) + end + # Reading content back out is restricted to certificates carrying the - # pp_cli_auth extension, which an ordinary agent certificate does not have. + # pp_cli_auth extension. The primary's own certificate carries it, so the + # request reaches the endpoint and is answered with a 404. curl_authenticated(bucket_path) do |stdout| + assert_allowed(stdout, 404) + end + + # An ordinary agent certificate does not carry pp_cli_auth, so reads are + # refused even though the same certificate may store content. + curl_with_cert(agent_certname, bucket_path) do |stdout| assert_denied(stdout, /\/puppet\/v3\/file_bucket_file\/sha256\/#{sum} \(method :get\)/) end