Guidelines for writing a Poc (proof-of-concept) for a vulnerability report by gilles-peskine-arm · Pull Request #214 · Mbed-TLS/mbedtls-docs

gilles-peskine-arm · 2026-07-24T13:54:23Z

Written based on recent experience.

I tend to write with humans in mind, but this document is also intended for LLM. Advice on writing for LLM would be appreciated.

…eport Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>

bjwtaylor

So, I think this is a good addition as it puts in writing things that are bothering the security team. One thing that I think would be useful is to make it more LLM friendly. So that when the LLM scans it, it will produce a more accurate understanding of what we want in terms of POC's. With this in mind it might be worth making the document more procedural, as LLM's will be more likely to interpret this accurately. So it might be worth adding:

A mandatory rules section summarising what we require, something like this:

Mandatory checklist

Before submitting a PoC, verify all of the following:

The victim uses only documented public APIs.
The victim obeys every documented API precondition.
The victim does not access private structure fields or define
MBEDTLS_ALLOW_PRIVATE_ACCESS.
The attacker controls only inputs or actions available across the stated
trust boundary.
Every API, program, option and file path exists in the exact revision being
tested.
The PoC reproduces from a clean checkout using the supplied commands.
The report distinguishes observed behavior from inferred or potential impact.
Instrumentation is used only to detect a bug reached through otherwise valid
execution; it must not manufacture the invalid state.

It also might be worth adding a preferred package structure, so for my bug-triage skill I currently use something similar to the following:

poc/
├── README.md
├── build_and_run.sh
└── victim.c

I also usually compress the files into an archive to make it easier to handle, do we want to recommend/mandate this?

It would also be useful to require the Readme to list the following:

affected repository, revision and configuration;
attacker capabilities and controlled input;
expected versus observed behavior;
exact build and execution commands;
expected output and exit status;
reproduction rate;
instrumentation used;
whether the result demonstrates a crash, memory-safety violation, secret disclosure, incorrect output, or only a suspected primitive.

I've also added a few comments that tighten up the specification to try and constrain what they do, let me know what you think though.

When we have something final it also might be worth testing it with my POC skill with a known bug and seeing if it produces something we like. If it doesn't we can then constrain it further until we get something better.

bjwtaylor · 2026-07-27T10:04:20Z

+
+### PoC using SSL test programs
+
+For TLS bugs, depending on the way the vulnerability can be triggered, it may be more convenient to use the test programs `programs/ssl/ssl_client2` and `programs/ss/ssl_server2`. For bugs involving non-nominal DTLS traffic, `programs/test/udp_proxy` can be useful. These programs are normally invoked via `tests/ssl-opt.sh`. A test case in `tests/ssl-opt.sh` is a convenient way to demonstrate a TLS bug. Our tooling is set up to either run `ssl_client2` against `ssl_server2`, or run one of them against an OpenSSL or GnuTLS peer.


Typo in path programs/ss/ssl_server2

bjwtaylor · 2026-07-27T10:05:24Z

+
+## Requirements on a PoC
+
+This section discusses requirements on a PoC. We have found that submissions based on LLM (large language models, “AI”) often violates these requirements, and that leads to unfounded claims. Please pay careful attention to these requirements in any LLM-assisted report.


I think it should be often violate

bjwtaylor · 2026-07-27T10:05:59Z

+
+This section discusses the preferred structure of a PoC. These are guidelines, not requirements. You do not have to follow them, but they make our life easier.
+
+Note that LLM tend to be fairly good at picking up the structure of our test code, including finding poorly documented auxiliary functions. However, without proper guidance, LLM tends to use internal functions. Internal functions should not be used in victim code, since their behavior is not part of the library's contract.


Should it be "LLM's tend"

bjwtaylor · 2026-07-27T10:06:17Z

+
+* add a new test case for an existing function;
+* or add an assertion to the existing code;
+* or write a new test fnuction.


should it be "Test function"

bjwtaylor · 2026-07-27T10:06:38Z

+
+Note that if the bug causes memory corruption, you should not try to detect it in the code: instead, rely on compile-time or run-time instrumentation such as AddressSanitizer (ASan) or Valgrind.
+
+Note that `ssl_client2` and `ssl_server2` have access to internal functions that can create malformed traffic or put the SSL context in unexpected states. When demonstrating a vulnerability, the attaker side may use whatever is convenient, but the victim side must not rely on calls to internal functions. If the vulnerability affects a client or server in the default runtime configuration, you may use `programs/ssl/ssl_client1` or `programs/ssl/ssl_server` as the victim.


attacker not attaker

bjwtaylor · 2026-07-27T10:32:40Z

+
+If you write a new test function, it is helpful if you can provide a variant of the test that passes, to help locate the cause of the bug.
+
+Note that unit tests have easy access to internal interfaces of individual library modules. When demonstrating a vulnerability, make sure that the victim code only uses public interfaces.


Maybe something like this:
Unit tests may use internal interfaces to locate and reproduce a defect. However, a test that reaches the defect only through an internal interface does not establish that the defect is reachable through the public API. To demonstrate a vulnerability, separately show that correctly used public interfaces can reach the same defective state.

I mostly followed your suggestion, although for the last sentence, I don't think we should encourage having two ways to reproduce the problem. We just want the reproducer that only uses public interfaces (for the victim code — if there's attacker code, it can use internal interfaces).

bjwtaylor · 2026-07-27T10:34:35Z

+
+### PoC using SSL test programs
+
+For TLS bugs, depending on the way the vulnerability can be triggered, it may be more convenient to use the test programs `programs/ssl/ssl_client2` and `programs/ss/ssl_server2`. For bugs involving non-nominal DTLS traffic, `programs/test/udp_proxy` can be useful. These programs are normally invoked via `tests/ssl-opt.sh`. A test case in `tests/ssl-opt.sh` is a convenient way to demonstrate a TLS bug. Our tooling is set up to either run `ssl_client2` against `ssl_server2`, or run one of them against an OpenSSL or GnuTLS peer.


You should add a phrase that specifies revision checking of the programs, something like:
Before submitting the PoC, verify that every referenced program, command-line option, test script and file path exists in the exact source revision being tested.

That doesn't seem related to SSL? We obviously want the PoC to actually work in the claimed version, but I didn't insist on that because it's not a mistake that we've encountered often.

So, yes this isn't specifically related to ssl. It's a more general comment about use of programs. One way in which LLM's can hallucinate is by mixing up versions. So by adding this it will force them to check they haven't done this. Though if we haven't seen this before is could be surplus to requirement, though it might be a good belt a braces approach anyway?

bjwtaylor · 2026-07-27T10:35:39Z

+
+For TLS bugs, depending on the way the vulnerability can be triggered, it may be more convenient to use the test programs `programs/ssl/ssl_client2` and `programs/ss/ssl_server2`. For bugs involving non-nominal DTLS traffic, `programs/test/udp_proxy` can be useful. These programs are normally invoked via `tests/ssl-opt.sh`. A test case in `tests/ssl-opt.sh` is a convenient way to demonstrate a TLS bug. Our tooling is set up to either run `ssl_client2` against `ssl_server2`, or run one of them against an OpenSSL or GnuTLS peer.
+
+Note that if the bug causes memory corruption, you should not try to detect it in the code: instead, rely on compile-time or run-time instrumentation such as AddressSanitizer (ASan) or Valgrind.


You should probably add the following clarification to this:
If the bug causes memory corruption, use instrumentation such as AddressSanitizer or Valgrind to detect it. The victim must reach the defective operation through otherwise valid execution; instrumentation, manual memory poisoning, debugger changes or forced memory layouts must not be used to manufacture the invalid state.

Reaching a problem state through out-of-band memory corruption is not a common mistake in PoC, so I don't think we should dwell on it.

bjwtaylor · 2026-07-27T10:36:56Z

+
+Note that if the bug causes memory corruption, you should not try to detect it in the code: instead, rely on compile-time or run-time instrumentation such as AddressSanitizer (ASan) or Valgrind.
+
+Note that `ssl_client2` and `ssl_server2` have access to internal functions that can create malformed traffic or put the SSL context in unexpected states. When demonstrating a vulnerability, the attaker side may use whatever is convenient, but the victim side must not rely on calls to internal functions. If the vulnerability affects a client or server in the default runtime configuration, you may use `programs/ssl/ssl_client1` or `programs/ssl/ssl_server` as the victim.


Maybe this would be more explicit:
The attacker side may use internal functions to construct malformed traffic. The victim side must not use internal functions to reach the defect. A PoC that manipulates the victim through internal interfaces may help diagnose a bug, but it does not demonstrate public reachability or security impact.

bjwtaylor · 2026-07-27T10:40:52Z

+
+If the vulnerability is a memory corruption (buffer overflow, use-after-free, double free, etc.), then the PoC only needs to reach the stage where the memory is corrupted. It is not useful to demonstrate concrete effects beyond that.
+
+Our preferred tool for analyzing memory corruption is AddressSanitizer (ASan). If a program causes ASan to crash from correct use of the library, that is a good PoC.


It might be worth adding a reproducability and minimization recommendation similar to:
First establish stable reproduction from a clean checkout. Then minimize the PoC, rerunning it after every reduction to ensure that it still demonstrates the same defect under the same preconditions. Record the tested revision, configuration, commands, expected output, observed output and reproduction rate.

I agree in principle, but this hasn't historically been a problem, and I don't want to add too much content that isn't directly useful.

I have added a request for reproduction instructions though.

Reproduction rate is almost always 1 in practice — we hardly ever get reports about race conditions. So I don't want to open that can of worms.

bjwtaylor

On additional response and what did you think to the checklist here #214 (review)

gilles-peskine-arm requested a review from bjwtaylor July 24, 2026 13:54

gilles-peskine-arm added this to Non-roadmap pull requests Jul 24, 2026

gilles-peskine-arm added needs-review Every commit must be lightly reviewed by a team member needs-reviewer priority-medium size-xs Estimated task size: extra small (a few hours at most) labels Jul 24, 2026

github-project-automation Bot moved this to In Development in Non-roadmap pull requests Jul 24, 2026

Guidelines for writing a Poc (proof-of-concept) for a vulnerability r…

27d2c60

…eport Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>

gilles-peskine-arm force-pushed the poc-guidelines-create branch from 6e36440 to 27d2c60 Compare July 24, 2026 14:57

bjwtaylor reviewed Jul 27, 2026

View reviewed changes

gilles-peskine-arm added 3 commits July 29, 2026 18:15

Fix typos

e10b481

Miscellaneous minor improvements

590c04e

Request reproduction instructions

9a2f8fa

gilles-peskine-arm requested a review from bjwtaylor July 29, 2026 17:01

bjwtaylor reviewed Jul 30, 2026

View reviewed changes


		### PoC using SSL test programs

		For TLS bugs, depending on the way the vulnerability can be triggered, it may be more convenient to use the test programs `programs/ssl/ssl_client2` and `programs/ss/ssl_server2`. For bugs involving non-nominal DTLS traffic, `programs/test/udp_proxy` can be useful. These programs are normally invoked via `tests/ssl-opt.sh`. A test case in `tests/ssl-opt.sh` is a convenient way to demonstrate a TLS bug. Our tooling is set up to either run `ssl_client2` against `ssl_server2`, or run one of them against an OpenSSL or GnuTLS peer.


		## Requirements on a PoC

		This section discusses requirements on a PoC. We have found that submissions based on LLM (large language models, “AI”) often violates these requirements, and that leads to unfounded claims. Please pay careful attention to these requirements in any LLM-assisted report.


		This section discusses the preferred structure of a PoC. These are guidelines, not requirements. You do not have to follow them, but they make our life easier.

		Note that LLM tend to be fairly good at picking up the structure of our test code, including finding poorly documented auxiliary functions. However, without proper guidance, LLM tends to use internal functions. Internal functions should not be used in victim code, since their behavior is not part of the library's contract.


		Note that if the bug causes memory corruption, you should not try to detect it in the code: instead, rely on compile-time or run-time instrumentation such as AddressSanitizer (ASan) or Valgrind.

		Note that `ssl_client2` and `ssl_server2` have access to internal functions that can create malformed traffic or put the SSL context in unexpected states. When demonstrating a vulnerability, the attaker side may use whatever is convenient, but the victim side must not rely on calls to internal functions. If the vulnerability affects a client or server in the default runtime configuration, you may use `programs/ssl/ssl_client1` or `programs/ssl/ssl_server` as the victim.


		If you write a new test function, it is helpful if you can provide a variant of the test that passes, to help locate the cause of the bug.

		Note that unit tests have easy access to internal interfaces of individual library modules. When demonstrating a vulnerability, make sure that the victim code only uses public interfaces.


		If the vulnerability is a memory corruption (buffer overflow, use-after-free, double free, etc.), then the PoC only needs to reach the stage where the memory is corrupted. It is not useful to demonstrate concrete effects beyond that.

		Our preferred tool for analyzing memory corruption is AddressSanitizer (ASan). If a program causes ASan to crash from correct use of the library, that is a good PoC.

Uh oh!

Conversation

gilles-peskine-arm commented Jul 24, 2026

Uh oh!

bjwtaylor left a comment

Choose a reason for hiding this comment

Mandatory checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjwtaylor left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants