[pjhartlieb]

What changed

dev now requires a pull request with checks and no bypass (#1013). Two parts of the SignatureBot didn't fit that:

• signature-history.yml pushed a hash to dev after every merged signature PR. That push is refused now, and opening a pull request instead doesn't work: one opened with the workflow token starts no workflows, so its required checks never report. It would also add a second PR to every signature merge.
• read-sources.yml opened the signature PRs with DNSBOT_TOKEN, a personal token, because that is what makes their CI start.

This PR:

• Deletes signature-history.yml. Instead, each signature PR records its own history: the bot adds the hash of the content it proposes to baddns/signatures/history/<signature>.sha256, in the same PR. The line merges with the PR, so nothing runs after a merge: no push, no extra PR. It records what the bot proposed at that time, so editing the signature, in the PR or later, never brings the original back. One file per signature, so concurrent bot PRs never conflict. signature_history.txt is still read. history/ is backfilled from the bot's own commit of every merged SignatureBot PR.
• Opens signature PRs as the bls-signaturebot App. PRs it opens start CI like before. Its token lasts an hour and reaches baddns only; the key is in the signature-bot environment, which only stable (where the schedule runs) can use, so pull request workflows can't read it. The workflow token drops to contents: read.
• signature-test.yml posts its result comment with the workflow token (it already has pull-requests: write), and tests only the signature .yml, not the history files. No workflow uses DNSBOT_TOKEN after this.

One PR per signature, reviewed and merged by a person, as today. The schedule runs stable's copy of read-sources.yml, so this takes effect at the next promotion; DNSBOT_TOKEN is removed after that.

Validation

On ci/signaturebot (7031117):

$ actionlint 1.7.12
(no output)
$ zizmor 1.30.1 --offline .
No findings to report. Good job! (4 ignored, 33 suppressed)

The seen check, taken verbatim from read-sources.yml, against the content of each SignatureBot PR as the bot proposed it (its own commit, before any edit):

with history/:
  merged, bot-proposed content                 seen 141   not seen   0   (1 unreadable: #34)
  merged, edited after the bot proposed it     seen  87   not seen   0
  closed unmerged                              seen 295   not seen  83   (matches only where the same content merged later, or signature_history.txt)
  random content                               not seen
negative control, history/ removed:
  merged, bot-proposed content                 seen  51   not seen  90
  merged, edited after the bot proposed it     seen  26   not seen  61

Two bot branches from the same base: merging one, then merging dev into the other, does not conflict.

Read Sources runs take 1 to 2 minutes (last 30 runs), well inside the App token's hour.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (719eb83) to head (b90b140).

Additional details and impacted files

@@           Coverage Diff           @@
##              dev    #1014   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          28       28           
  Lines        2437     2437           
=======================================
  Hits         2414     2414           
  Misses         23       23           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pjhartlieb]

Round 2, for the point-in-time concern: the earlier replay used refs/pull/N/head, after edits, so it never tested the bot's original content. The bot now records the hash of what it proposed in its own PR (history/<signature>.sha256). Validation is in the description. To reproduce:

python3 cases.py <clone> > cases.tsv
bash verify_history.sh <clone at 7031117> cases.tsv
python3 backfill_history.py <clone> --check   # backfill complete: exit 0

cases.py

#!/usr/bin/env python3
"""Emit verification cases as TSV: state  pr  name  sha256(bot-proposed content)  edited(0/1).
state = merged | closed. 'edited' = the file on origin/dev differs from what the bot proposed.
Usage: cases.py <baddns clone>"""
import hashlib, json, pathlib, subprocess, sys

REPO = "blacklanternsecurity/baddns"
TITLE = "[SignatureBot] Add or update signature "
QUERY = """query($q:String!,$endCursor:String){search(query:$q,type:ISSUE,first:50,after:$endCursor){
 pageInfo{hasNextPage endCursor} nodes{... on PullRequest{number title merged
 commits(first:100){nodes{commit{oid message}}}}}}}"""

clone = pathlib.Path(sys.argv[1]).resolve()


def git(*a):
    r = subprocess.run(["git", *a], cwd=clone, capture_output=True)
    return r.stdout if r.returncode == 0 else None


for state in ("merged", "closed"):
    q = f'repo:{REPO} is:pr is:{"merged" if state == "merged" else "unmerged is:closed"} in:title "{TITLE.strip()}"'
    out = subprocess.run(["gh", "api", "graphql", "--paginate", "-f", f"q={q}", "-f", f"query={QUERY}",
                          "--jq", ".data.search.nodes[]"], check=True, capture_output=True).stdout
    for l in out.decode().splitlines():
        pr = json.loads(l)
        if not pr["title"].startswith(TITLE):
            continue
        name = pr["title"][len(TITLE):].strip()
        bot = [c["commit"]["oid"] for c in pr["commits"]["nodes"] if c["commit"]["message"].startswith(TITLE + name)]
        if not bot:
            continue
        git("fetch", "-q", "origin", f"pull/{pr['number']}/head")
        data = git("show", f"{bot[0]}:baddns/signatures/{name}")
        if data is None:
            print(f"#{pr['number']} {name}: unreadable", file=sys.stderr)
            continue
        cur = git("show", f"origin/dev:baddns/signatures/{name}")
        edited = int(cur is not None and cur != data)
        print(state, pr["number"], name, hashlib.sha256(data).hexdigest(), edited, sep="\t")

verify_history.sh

#!/usr/bin/env bash
# Verify the SignatureBot point-in-time record, using the seen check copied from read-sources.yml.
# Usage: verify_history.sh <baddns clone at the PR head> <cases.tsv from cases.py>
set -euo pipefail
clone=$1; cases=$(cd "$(dirname "$2")" && pwd)/$(basename "$2")
cd "$clone"

# The workflow's own definitions, extracted verbatim, not retyped.
eval "$(sed -n '/^        HISTORY_DIR=/,/^        }/p' .github/workflows/read-sources.yml | sed 's/^        //')"
type seen_in_history | sed 's/^/  /'
prev_hashes=$(cat baddns/signatures/signature_history.txt)

seen() {  # $1 file  $2 hash -- the two checks read-sources.yml makes, in its order
  if echo "$prev_hashes" | grep -q "$2"; then return 0; fi
  seen_in_history "$1" "$2"
}

report() {  # $1 label  $2 awk filter on the cases
  local s=0 n=0
  while IFS=$'\t' read -r _ _ name hash _; do
    if seen "$name" "$hash"; then s=$((s+1)); else n=$((n+1)); fi
  done < <(awk -F'\t' "$2" "$cases")
  printf '%-48s seen %3d   not seen %3d\n' "$1" "$s" "$n"
}

echo "== with history/ (this PR)"
report "merged, bot-proposed content"            '$1=="merged"'
report "merged, edited after the bot proposed it" '$1=="merged" && $5==1'
report "closed unmerged"                          '$1=="closed"'
rand=$(head -c 64 /dev/urandom | sha256sum | cut -d' ' -f1)
seen dnsreaper_aha.yml "$rand" && echo "random content: SEEN (FAIL)" || echo "random content: not seen"

echo "== negative control: history/ removed (signature_history.txt only)"
HISTORY_DIR=$(mktemp -d)
report "merged, bot-proposed content"            '$1=="merged"'
report "merged, edited after the bot proposed it" '$1=="merged" && $5==1'

backfill_history.py

#!/usr/bin/env python3
"""One-time backfill of baddns/signatures/history/<signature>.sha256.

For every merged [SignatureBot] PR, hash the signature as the bot proposed it (every commit the bot
itself made on the PR branch, before any human edit) and record it. Rerunnable: existing lines are kept,
duplicates are not added. Usage: backfill_history.py <baddns clone> [--check]
--check writes nothing and exits 1 if any bot-proposed hash would be missing.
"""
import hashlib, json, pathlib, subprocess, sys

REPO = "blacklanternsecurity/baddns"
TITLE = "[SignatureBot] Add or update signature "
QUERY = """query($q:String!,$endCursor:String){search(query:$q,type:ISSUE,first:50,after:$endCursor){
 pageInfo{hasNextPage endCursor} nodes{... on PullRequest{number title
 commits(first:100){nodes{commit{oid message}}}}}}}"""


def run(*a, cwd=None):
    return subprocess.run(a, cwd=cwd, check=True, capture_output=True).stdout


def main():
    clone = pathlib.Path(sys.argv[1]).resolve()
    check = "--check" in sys.argv
    q = f'repo:{REPO} is:pr is:merged in:title "{TITLE.strip()}"'
    out = run("gh", "api", "graphql", "--paginate", "-f", f"q={q}", "-f", f"query={QUERY}",
              "--jq", ".data.search.nodes[]")
    prs = [json.loads(l) for l in out.decode().splitlines() if l.strip()]
    hist = clone / "baddns/signatures/history"
    missing, added, total = [], 0, 0
    for pr in sorted(prs, key=lambda p: p["number"]):
        if not pr["title"].startswith(TITLE):
            continue
        name = pr["title"][len(TITLE):].strip()
        bot = [c["commit"]["oid"] for c in pr["commits"]["nodes"]
               if c["commit"]["message"].startswith(TITLE + name)]
        if not bot:
            print(f"#{pr['number']} {name}: no bot commit found", file=sys.stderr)
            continue
        subprocess.run(["git", "fetch", "-q", "origin", f"pull/{pr['number']}/head"], cwd=clone)
        for oid in bot:
            try:
                data = run("git", "show", f"{oid}:baddns/signatures/{name}", cwd=clone)
            except subprocess.CalledProcessError:
                print(f"#{pr['number']} {name}: unreadable at {oid[:9]}", file=sys.stderr)
                continue
            total += 1
            h = hashlib.sha256(data).hexdigest()
            f = hist / f"{name}.sha256"
            have = f.read_text() if f.exists() else ""
            if h in have:
                continue
            missing.append((pr["number"], name, h))
            if not check:
                hist.mkdir(parents=True, exist_ok=True)
                with f.open("a") as fh:
                    fh.write(f"{h} #{name}\n")
                added += 1
    print(f"bot-proposed versions: {total}; missing before run: {len(missing)}; added: {added}")
    if check and missing:
        for m in missing:
            print("missing", *m)
        sys.exit(1)


if __name__ == "__main__":
    main()

[pjhartlieb]

Agreed. SignatureBot PRs now run the full test suite: b90b140 drops the skip from tests.yml. A person approves every bot PR, so it goes through the same checks as any other PR into dev. signature-test still runs on them as well.