Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
166 lines (122 loc) · 6.12 KB

CONTRIBUTING.md

File metadata and controls

166 lines (122 loc) · 6.12 KB
Raw

Contributing a test case

This repository is a measuring instrument. Every file under vulns/ exists so an automated scanner can be scored against it. That purpose sets the rules below — a submission that does not meet them is not useful as a measurement and will not be merged.

By opening a pull request you confirm that your submission is intended for scanner-efficacy research and defensive tooling validation, and that it is inert by construction.

The five safety rules

These are not stylistic preferences. They are what keeps this repository a testbed rather than a liability, and scripts/validate-test-case.sh enforces all of them mechanically.

1. Nothing executes

Every payload must be unreachable. Use the idiom natural to the language:

Language Guard
JavaScript if (false) { … }
Python if False:
Go if neverRun { … } plus a //go:build ignore tag
Java if (NEVER_RUN) { … }
Ruby if false

A reader must be able to see in one glance that the code cannot run. Data files (.env, .json, .txt) carry no guard because they carry no code; they are kept inert by rules 2–4 instead.

2. Nothing reaches the network

Every host literal must be unresolvable. Use a domain reserved by RFC 2606 or RFC 6761 — .invalid, .example, .test, .localhost, or example.com. .invalid is preferred: it can never be registered by anyone, ever.

The single exception is naming the real API a credential belongs to (api.github.com in a GitHub-token fixture), because the fixture is not realistic otherwise. Those call sites are still dead code, and the allow-list lives in scripts/validate-test-case.sh. Do not extend it without saying why in the pull request.

3. Nothing destroys anything

No disk writes, no deletion, no process spawning, no resource exhaustion, no mining, no worm or self-replication logic. If the guard were removed by accident, the worst outcome must still be nothing.

4. No live credentials

A secrets test case needs a credential of the right shape, never a working one. Acceptable sources:

  • A value the vendor publishes as an example (AWS's AKIAIOSFODNN7EXAMPLE).
  • A structurally valid placeholder with an obviously synthetic body — a repeating filler no issuer would ever mint.
  • Key material generated for the sole purpose of being committed here, which therefore secures nothing.

Never a redacted real secret, and never a value you found somewhere. If you have to think about whether to rotate it, it does not belong here.

5. Nothing installs

Dependency manifests live under vulns/supply-chain/ and never at the repository root. Every dependency pins a version that exists on no registry, and the sibling .npmrc points at an unresolvable host. See vulns/supply-chain/README.md.

Required metadata

Every test case declares its own metadata in a header comment. The catalog is generated from these tags, so they are the source of truth — there is no separate list to keep in sync.

/**
 * @id js-xss-dom-innerhtml          // unique, kebab-case, <lang>-<vuln>
 * @test-case Reflected XSS via innerHTML
 * @cwe CWE-79
 * @severity critical|high|medium|low|info
 * @language javascript
 * @expected-detection true
 * @description What the flaw is and why it works.
 * @safe-guard Exactly what makes this inert.
 * @detection-target What a scanner must reason about to catch it.
 */

JSON files cannot carry comments, so they use "_id", "_cwe", … keys instead.

Required markers

Coverage is scored line by line, so mark the lines:

  • VULNERABLE: on every line a scanner is expected to flag.
  • SAFE: on every line a scanner must not flag.

Please include a safe counterpart. A test case that only contains broken code measures recall and nothing else. The SAFE: lines are the control group: each one is a correct implementation of the same pattern, so a scanner that flags them is pattern-matching on syntax rather than analysing data flow. That distinction is most of this repository's value.

el.innerHTML = req.query.name;   // VULNERABLE: CWE-79 sink
el.textContent = req.query.name; // SAFE: textContent does not parse HTML

Findings are attributed to a marker within ±2 lines. Files with no VULNERABLE: marker fall back to file-level scoring.

Obfuscation

Rule: payloads are readable and annotated. Reviewers cannot approve what they cannot read.

There is one narrow exception, because obfuscation-detection rules cannot be tested against un-obfuscated code. A test case may use an obfuscation technique if it also states in plain text exactly what the obfuscated form means — see vulns/supply-chain/obfuscated-eval-utility.js, where the encoded blob is inert and its decoded plaintext is written out in a comment directly above it.

The technique may be hidden from a scanner. The meaning may never be hidden from a reviewer. A submission that obfuscates without annotating will be closed.

Submitting

  1. Fork and branch: feat/add-<vuln>-<language>.

  2. Add your file under vulns/<language>/.

  3. Run the linter and regenerate the catalog:

    bash scripts/validate-test-case.sh
python3 scripts/generate-catalog.py

  4. Commit the regenerated vulns/VULNERABILITY_CATALOG.json and docs/VULNERABILITY_CATALOG.md.

  5. Open the PR and fill in the template, including the safety statement.

CI runs the same linter, then scans your case and reports whether ThreatCrush detected it. A missed detection is a good submission, not a failed one — it is the most useful kind, because it marks a real gap in the scanner. Say so in the PR rather than reworking the case until it gets caught.

What will be rejected

  • Working exploits, live payloads, or anything targeting a real system.
  • Credentials that authenticate to anything.
  • Obfuscation without a plaintext explanation.
  • Techniques whose purpose is evading detection in someone else's environment rather than measuring detection in this one.
  • Anything you are not comfortable having permanently in public git history.