mtpublisher: Store the mirror's checkpoint line and subtree signatures by beautifulentropy · Pull Request #9097 · letsencrypt/boulder

beautifulentropy · 2026-10-14T17:43:17Z

mtpublisher: Split .Publish() into .PublishCheckpoint(), storing the mirror's checkpoint cosignature line from add-entries, and .CosignSubtrees(), storing the mirror's signature on each subtree that lacks one, both are run at each tick.
mtpublisher: Fetch the mirror's checkpoint from its tlog-mirror monitoring prefix and prove each unsigned subtree against it for sign-subtree.
mtpublisher/mtpublishertest: Add .CosignSubtree() to TestMirror.
cmd/boulder-mtpublisher: Add the required monitoringPrefix configuration.
trees/cosignature: Add Cosigner.CosignSubtree() and Verifier.VerifySubtree() over any subtree, and return the Line, Timestamp, and Signature of a checkpoint cosignature from FilterByVerify() as a CheckpointCosignature struct.
mtca: Sign the checkpoint's subtree with Cosigner.CosignSubtree().
treedb: Add .GetUnsignedSubtrees() and .AddSubtreeMirrorSignature(), and drop subtreeID1 and subtreeID2 from the checkpoints table.
sa/db: Index checkpointSubtrees by mtcLogID and mirrorID, and grant the mtpublisher user UPDATE.

Note: tlog-witness v1.1.0 was updated to respond with a bare (base64 encoded) subtree cosignature. Sunlight v0.9.0 responds to sign-subtree requests with a note signature line. We'll update our client to expect that once Sunlight ships an update.

Closes #9037
Closes #8991

github-actions · 2026-10-14T17:43:27Z

@beautifulentropy, this PR appears to contain configuration and/or SQL schema changes. Please ensure that a corresponding deployment ticket has been filed with the new values.

…tures-to-subtrees

jsha

Looking great, glad to see this making its way out. I think the most important decision to make here is related to batching of work and how we recover from getting behind. I talk about it in line comments below, but overall I think: small batches, short intervals, last-in-first-out (i.e. stack semantics), and mirrors should continue to get entries pushed regardless of whether subtrees are getting signed successfully.

jsha · 2026-10-19T18:51:01Z

+// with origin.
+//
+// https://c2sp.org/tlog-cosignature#subtree-cosignatures
+func (v *Verifier) VerifySubtree(origin string, start, end int64, subtreeHash tlog.Hash, signature []byte) bool {


Suggested change

func (v *Verifier) VerifySubtree(origin string, start, end int64, subtreeHash tlog.Hash, signature []byte) bool {

func (v *Verifier) VerifySubtreeSignature(origin string, start, end int64, subtreeHash tlog.Hash, signature []byte) bool {

To emphasize that we're verifying a signature, not an inclusion proof.

jsha · 2026-10-19T19:12:15Z

+-- mtpublisher: follows the latestCheckpoint pointer to a checkpoint awaiting
+-- a cosignature and writes the mirror's cosignatures on it and its subtrees.


Probably best to just remove this comment altogether; we're likely to miss updating it as code elsewhere changes, and I think we're unlikely to remember to go and update the user grants SQL.

jsha · 2026-10-20T18:53:50Z

 	return model.ID, nil
 }

+// GetUnsignedSubtrees returns the log's subtrees without a mirror signature, oldest first.


I think we want this to be newest-first, with a LIMIT just in case the list gets really big.

Reasoning: as written, this acts kind like a queue. In #6042 we decided stack-like behavior fails nicer. If we get overwhelmed, we handled the freshest stuff first since older stuff may already be forgotten by the client that wanted it.

In the happy case, there will only be a tiny handful of subtrees with mirrorID IS NULL so it won't matter, but it's good to fail gracefully when we fail.

jsha · 2026-10-20T18:55:59Z

-// log's latest checkpoint.
+// ErrSubtreeNotMirrored reports that the mirror's checkpoint does not include
+// the subtree yet.
+var ErrSubtreeNotMirrored = errors.New("the mirror has not reached the subtree")


Suggested change

var ErrSubtreeNotMirrored = errors.New("the mirror has not reached the subtree")

var ErrSubtreeNotMirrored = errors.New("the mirror's latest checkpoint does not include the subtree")

jsha · 2026-10-20T18:57:58Z


-// Start attempts to publish the latest checkpoint at each interval until ctx is
+// CosignSubtrees stores the mirror's signature on each subtree of the log that
+// lacks one, oldest first, until it reaches one the mirror's checkpoint does


Suggested change

// lacks one, oldest first, until it reaches one the mirror's checkpoint does

// lacks one, newest first, until it reaches one the mirror's checkpoint does

(if you take my suggestion below about selecting from the DB newest-first)

jsha · 2026-10-20T19:00:25Z

+		err = subtree.Valid()
+		if err != nil {
+			return fmt.Errorf("validating subtree %d: %s", subtree.ID, err)
+		}


If we somehow write an invalid subtree to the DB, the mtpublisher will forever get stuck on it. Given that such a subtree can't possibly ever succeed in getting a signature, I think this case should intentionally write an invalid mirrorID to the table so we don't keep trying.

jsha · 2026-10-20T19:12:33Z

+		err := p.PublishCheckpoint(ctx)
+		if err != nil {
+			p.log.Errf("Publishing checkpoint failed: %s", err)
+		}
+		err = p.CosignSubtrees(ctx)


I think it makes sense for PublishCheckpoint and CosignSubtrees to have independent loops, in separate goroutines (with wg.Do() so this function can block on them finishing).

Reasoning: if for any reason CosignSubtrees is slow or gets stuck in a loop, we don't want that to get in the way of publishing our latest tree state.

Relatedly: we should decide how much work CosignSubtrees does per call. Or another way to put it: how big a batch of work does it get from GetUnsignedSubtrees? I think the answer should be "very small, like 2 items, and the interval should be quite small." If we chose big batches instead, the information from GetUnsignedSubtrees would be stale when we're working on the tail end of the batch. Specifically, there could already be a mirror signature available for a given subtree but this mtpublisher instance wouldn't necessarily know that.

jsha · 2026-10-20T19:21:19Z

+	separator := bytes.Index(body, []byte("\n\n"))
+	if separator < 0 {
+		return nil, nil, errors.New("mirror checkpoint has no signature lines")
 	}
-	zeroTimestampMirrorCosignature, err := m.verifier.FilterByVerify(noteText, subtreeCosignatureLines)
+	noteText := body[:separator+1]
+	signatureLines := body[separator+2:]


I think we can replace this parsing code with checkpoint.Open(). If that's not possible we should have a comment explaining.

jsha · 2026-10-20T19:27:28Z

+	if mirrored.Tree.N == end && mirrored.Tree.Hash != subtreeHash {
+		return nil, fmt.Errorf("mirror checkpoint at size %d has root hash %x, the log's is %x", end, mirrored.Tree.Hash, subtreeHash)
+	}


I think we should replace this check with a call to subtree.VerifyConsistency immediately after subtree.ConsistencyProof. That verifies the same property, but for all cases instead of the special case that the subtree lines up with the current tree size.

beautifulentropy added 3 commits October 14, 2026 12:59

treedb: Add subtree queries and drop the checkpoint subtree IDs

c037c53

trees/cosignature: Cosign and verify subtrees

28050cb

mtpublisher: Store the mirror's checkpoint line and subtree signatures

0e7d637

beautifulentropy requested a review from a team as a code owner October 14, 2026 17:43

beautifulentropy requested a review from aarongable October 14, 2026 17:43

beautifulentropy requested review from jsha and removed request for aarongable October 14, 2026 17:43

beautifulentropy added 2 commits October 14, 2026 13:59

test: Update TestIssuanceMTC

891538e

Merge remote-tracking branch 'origin/main' into publisher-write-signa…

4c94c2e

…tures-to-subtrees

jsha reviewed Oct 20, 2026

View reviewed changes

Navigation Menu

Repository navigation

mtpublisher: Store the mirror's checkpoint line and subtree signatures - #9097

mtpublisher: Store the mirror's checkpoint line and subtree signatures#9097
beautifulentropy wants to merge 5 commits into
mainfrom
publisher-write-signatures-to-subtrees

beautifulentropy commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

jsha left a comment

Uh oh!

jsha Oct 19, 2026

Uh oh!

jsha Oct 19, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

jsha Oct 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	func (v *Verifier) VerifySubtree(origin string, start, end int64, subtreeHash tlog.Hash, signature []byte) bool {
	func (v *Verifier) VerifySubtreeSignature(origin string, start, end int64, subtreeHash tlog.Hash, signature []byte) bool {

		-- mtpublisher: follows the latestCheckpoint pointer to a checkpoint awaiting
		-- a cosignature and writes the mirror's cosignatures on it and its subtrees.

	var ErrSubtreeNotMirrored = errors.New("the mirror has not reached the subtree")
	var ErrSubtreeNotMirrored = errors.New("the mirror's latest checkpoint does not include the subtree")

	// lacks one, oldest first, until it reaches one the mirror's checkpoint does
	// lacks one, newest first, until it reaches one the mirror's checkpoint does

Conversation

beautifulentropy commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

jsha left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

beautifulentropy commented Oct 14, 2026 •

edited

Loading