Summary
The five amber controls return WARN on an artifact that a USENIX Security '23 AEC independently reproduced, not because it lacks a reproducibility story, but because that story lives in the README and an external image rather than in files the controls look for.
git clone https://github.com/AutomatedAnalysisOf/AEADProtocols.git
cd AEADProtocols && git checkout V1 # cad164e
darnit audit --show-all --framework amber .
Total: 5 | Pass: 0 | Fail: 0 | Warn: 5 | N/A: 0 | Error: 0 | Pending LLM: 0
RE-01.01 .. RE-03.01: WARN - Could not automatically verify
No handler errors.
The artifact:
35 MB: Models/, README.md, tamarin-prover.zip. The README gives a pinned container and run command:
docker pull aeads/tamarin
docker run -it -v $PWD:/opt/case-studies aeads/tamarin bash
and vendors the prover source for manual builds. Badges: Available, Functional, Results Reproduced.
RE-01.02 (BuildEnvDeclared) is the clearest case, the build environment is declared, just not as a Dockerfile. And the WARN is indistinguishable from one on a repo with no evidence at all, which is the same complaint as #427.
Suggestions:
- Distinguish no candidate evidence found from candidates found but unevaluable, reporting what was searched for would be a start.
- Consider whether a documented
docker pull <image> counts as a declared build environment, even at suggestive authority
From darnitdevorg/amber-corpus (#534) - artifacts with externally established expected outcomes, so a verdict can be judged right or wrong. Entry: entries/usenixsec2023-aeadprotocols.yaml. Run on the #532 branch.
Summary
The five
ambercontrols return WARN on an artifact that a USENIX Security '23 AEC independently reproduced, not because it lacks a reproducibility story, but because that story lives in the README and an external image rather than in files the controls look for.git clone https://github.com/AutomatedAnalysisOf/AEADProtocols.git
cd AEADProtocols && git checkout V1 # cad164e
darnit audit --show-all --framework amber .
Total: 5 | Pass: 0 | Fail: 0 | Warn: 5 | N/A: 0 | Error: 0 | Pending LLM: 0
RE-01.01 .. RE-03.01: WARN - Could not automatically verify
No handler errors.
The artifact:
35 MB:
Models/,README.md,tamarin-prover.zip. The README gives a pinned container and run command:docker pull aeads/tamarin
docker run -it -v $PWD:/opt/case-studies aeads/tamarin bash
and vendors the prover source for manual builds. Badges: Available, Functional, Results Reproduced.
RE-01.02 (BuildEnvDeclared) is the clearest case, the build environment is declared, just not as a
Dockerfile. And the WARN is indistinguishable from one on a repo with no evidence at all, which is the same complaint as #427.Suggestions:
docker pull <image>counts as a declared build environment, even at suggestive authorityFrom
darnitdevorg/amber-corpus(#534) - artifacts with externally established expected outcomes, so a verdict can be judged right or wrong. Entry:entries/usenixsec2023-aeadprotocols.yaml. Run on the #532 branch.