Skip to content

docs(discovery): refresh project documentation #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pengyuzhang wants to merge 1 commit into
base: main
Choose a base branch
from
+192 −25
Open

docs(discovery): refresh project documentation #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions Discovery/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# ADR Discovery

ADR Discovery inventories AI tooling on an endpoint, including AI binaries,
desktop applications, agents, MCP servers, skills, plugins, and related
programmable surfaces. It records the evidence behind each finding and reports
coverage gaps so an incomplete scan is not mistaken for a clean machine.

The collector runs on Python 3.11+ and has no runtime dependencies outside the
standard library.

## Quick start

Run a scan from this directory with [`uv`](https://docs.astral.sh/uv/):

```sh
uv run adr-discovery --dry-run --json
```

To see exactly what the collector would emit without writing a snapshot:

```sh
uv run adr-discovery --dry-run --explain
```

Use `--output-dir` to save scan output:

```sh
uv run adr-discovery --output-dir ./output
```

Run `uv run adr-discovery --help` for all options.

## Common workflows

Scan a fixture tree instead of the live endpoint:

```sh
uv run adr-discovery --root /path/to/fixture --dry-run --json
```

Compare the current scan with an earlier snapshot:

```sh
uv run adr-discovery --diff /path/to/snapshot.json --dry-run --json
```

Apply a tenant policy or last-used telemetry during a scan:

```sh
uv run adr-discovery \
--policy /path/to/policy.json \
--telemetry /path/to/telemetry.json \
--dry-run --json
```

Discovery does not execute binaries that it finds. Machine access is bounded,
and risky text is redacted at collection time.

## Exit codes

| Code | Meaning |
| ---: | --- |
| `0` | The scan completed with full available coverage. |
| `2` | The scan completed, but one or more surfaces were unavailable, unreadable, or truncated. A snapshot is still produced. |
| Other | The command could not complete because of an error or invalid input. |

## Development

Install the development dependencies and run the checks:

```sh
uv sync --extra dev
uv run pytest -q
uv run ruff check .
```

The test suite has two layers:

- `adr_discovery/tests_unit/` verifies modules, contracts, security boundaries,
and the composed pipeline.
- `tests/` is the black-box endpoint harness used to compare known installations
with the collector's reported snapshot.

## Project layout

```text
adr_discovery/
catalog/ known-tool catalog and validation
contracts/ shared records, evidence, and snapshot types
coverage/ coverage accounting
enumerator/ endpoint source enumeration
extractor/ declaration and surface extraction
identifier/ evidence-based identification
judge/ risk, sanction, and finding evaluation
redact/ sensitive-data redaction
reporter/ snapshots and deltas
resolver/ observation merging and confidence
world/ bounded, platform-specific machine access
cli.py command-line interface
pipeline.py pipeline composition
tests/ black-box endpoint test harness
```

For the architecture, data model, security boundaries, current limitations,
and implementation details, see the [package documentation](adr_discovery/README.md).
For end-to-end validation and VM provisioning, see the
[test harness documentation](tests/README.md).

## License

Apache-2.0.
28 changes: 19 additions & 9 deletions Discovery/adr_discovery/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The endpoint collector is implemented in this repository. Its seven-stage pipeli

| | |
| --- | --- |
| **Design of record** | `adr-discovery-design.html` — the full argument, module contracts and complete target corpus |
| **Design overview** | The architecture, contracts, target corpus and constraints documented below |
| **Implementation** | This package: M1–M7, the cross-cutting catalog/redaction/coverage concerns, and the `adr-discovery` CLI |
| **Verification** | The fast module and pipeline suite under `tests_unit/`, plus the black-box endpoint harness in [../tests/README.md](../tests/README.md) |
| **Current maturity** | Core architecture complete; production source coverage and the three-OS golden-endpoint corpus remain incomplete |
Expand Down Expand Up @@ -116,7 +116,7 @@ One directory per module. Directory names are the module names above verbatim, s

```
adr_discovery/
├── cli.py argument parsing, exit codes. Nothing else.
├── cli.py process boundary: arguments, output files and exit codes
├── pipeline.py the composition root — the only file importing more than one stage
│
├── contracts/ the types the stages hand each other
Expand Down Expand Up @@ -170,12 +170,12 @@ A boundary described only in a document is a boundary that has already been cros

| Rule | The regression it prevents |
| --- | --- |
| Only `world/` imports `os`, `pathlib`, `subprocess`, `socket` | Probes quietly growing private file access, each with its own containment bug |
| Only `world/` and the CLI process boundary import host-facing modules such as `os`, `pathlib`, `subprocess` and `socket` | Probes quietly growing private file access, each with its own containment bug |
| No stage imports a sibling stage | The cycle that makes any one stage impossible to test alone |
| Only `pipeline.py` imports more than one stage | Execution order becoming an emergent property of the import graph |
| Outside stage internals and the CLI boundary, only `pipeline.py` imports more than one stage | Execution order becoming an emergent property of the import graph |
| `catalog/` imports nothing from the package | The landscape's weekly churn ending up on the release train |
| Every stage is a function from its input type to its output type, reading no module-level state | `PROJECT_ROOTS` in five files — the defect this rewrite exists for |
| Every stage returns its coverage alongside its result | A partial answer that reads as a clean machine |
| Stages hold no mutable module-level state | `PROJECT_ROOTS` in five files — the defect this rewrite exists for |
| Coverage-producing stages record gaps in the shared ledger, which the pipeline freezes into the snapshot | A partial answer that reads as a clean machine |

Every other guarantee in this file is a claim about intent. The import test is a claim about the code, it runs in under a second, and it fails on the pull request that would have reintroduced the problem.

Expand Down Expand Up @@ -247,15 +247,25 @@ Accuracy claims state which layer produced them. A suite whose worlds are built

### The two instruments

[tests/README.md](../tests/README.md) documents the end-to-end fidelity measurement: real tools installed on a clean guest per OS, a scan before and after, and a comparison of what was installed against what the collector reported. Because it installs real software, signs into real accounts and starts real listeners, it is not part of per-commit CI — it runs against a release candidate, when the catalog changes, and when a new OS version ships.
[tests/README.md](../tests/README.md) specifies the end-to-end fidelity
measurement: real tools installed on a clean guest per OS, a scan before and
after, and a comparison of what was installed against what the collector
reported. The complete three-OS workflow is not operational yet. Linux and
macOS drivers and a subset of executable recipes exist, but automated golden
restore is not wired up, there is no Windows driver, and the integrated harness
runner currently supports dry runs only.

The fast per-commit suite is a separate instrument, documented alongside it: synthetic endpoints built on disk, scanned by a real pipeline, in about four seconds on any CI box.
The fast per-commit suite is a separate instrument: synthetic endpoints are
built on disk and scanned through the real pipeline without requiring a guest.

The two are complementary and neither replaces the other. The fixture suite has a perfect oracle — it built the machine — but can only contain situations somebody imagined, so it catches regressions. The VM run has real input nobody predicted but a slower, costlier oracle, so it discovers defects. Every defect a VM run finds should be reduced to a fixture case, which is the intended flow of work between them.

The harness under `tests/` can validate manifests, synthesize and replay recorded runs, provision through its current guest drivers, and generate JSON and HTML scorecards. Only part of the install manifest currently has executable recipes; see [HARNESS.md](../tests/HARNESS.md) for current counts and known guest limitations.

Unit tests mirror the package tree — one directory per module, importing only that module. The arrangement is the assertion: a module that cannot be tested without standing up three others does not have a boundary, whatever the directory listing says.
Unit tests under `tests_unit/` use one focused test module per implementation
area. Import-graph tests enforce the stage boundaries directly; a module that
cannot be tested without standing up three others does not have a meaningful
boundary, whatever the directory listing says.

## Implementation progress

Expand Down
78 changes: 62 additions & 16 deletions Discovery/tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,26 @@ The measurement is a comparison between two lists:
compare ═══► TP / FP / FN per category
```

## Test environment
## Current harness status

Three virtual machines, one per operating system. Nothing is tested on a developer's own machine, because the whole method depends on knowing the complete contents of the endpoint.
The 120-entry manifest is the target corpus, not a claim that every entry can
be installed automatically today. The harness currently provides manifest
validation, synthetic run generation, replayable scoring, JSON and HTML
scorecards, and dry-run command planning. Four recipe families execute against
the guest-driver interface; on Linux, 64 of 105 applicable entries currently
have executable recipes.

Linux and macOS guest drivers exist, but neither has automated golden-snapshot
restoration wired up. There is no Windows guest driver yet, and the integrated
`tests.cli run` command supports only `--dry`. See [HARNESS.md](HARNESS.md) for
the current recipe, driver, and measured-run status.

## Target test environment

The complete fidelity measurement targets three virtual machines, one per
operating system. Nothing should be scored as a golden-endpoint run on a
developer's own machine, because the method depends on knowing the complete
contents of the endpoint.

| VM | Operating system | Purpose |
| --- | --- | --- |
Expand All @@ -30,9 +47,12 @@ VMs rather than containers or the host, for three reasons that matter to the res

1. **A known-clean baseline.** Every run starts from a snapshot with no AI tooling installed. Without that, a reported asset can't be attributed to the manifest.
2. **Rollback between runs.** Installing 30 tools mutates the machine permanently. A VM snapshot makes each run reproducible instead of cumulative.
3. **Real OS surfaces.** A container has no launchd, no Windows registry, no Task Scheduler, and no GUI app bundles. Those are exactly the surfaces four of the ten probes exist to read.
3. **Real OS surfaces.** A container has no launchd, no Windows registry, no Task Scheduler, and no GUI app bundles. Those are surfaces the collector and manifest are intended to exercise.

Each VM is provisioned with the OS defaults plus the runtimes the tools need (Node, Python, a browser, a JetBrains IDE where applicable) and **nothing else**. Those prerequisites are part of the baseline, not the manifest.
For a complete golden run, each VM must be provisioned with the OS defaults plus
the runtimes the tools need (Node, Python, a browser, and a JetBrains IDE where
applicable) and **nothing else**. Those prerequisites are part of the baseline,
not the manifest.

## Method

Expand All @@ -58,7 +78,12 @@ compare(delta.added, manifest) # this comparison is the test result

## The install manifest

This is the complete inventory the harness installs. Every row has a stable **id**, and the id is what the runner executes, what `manifest.actual.json` records an outcome against, and what a scorecard reports a miss under. Nothing is left as "and a few others" — an entry that is not listed here is not tested.
This is the complete inventory the harness intends to install and score. Every
row has a stable **id**, and the id is what the runner plans, what
`manifest.actual.json` records an outcome against, and what a scorecard reports
a miss under. Entries whose recipe family is not implemented are recorded as
unavailable rather than reported as successful; only entries recorded as
installed are scored for recall.

`catalog_id` is the join key between what we installed and what was reported. Rows with no `catalog_id` (MCP servers, skills, hooks) are matched by install path or launch identity instead.

Expand All @@ -74,7 +99,12 @@ Platform columns record where a vendor ships the tool. Availability is re-confir
| Agents | 12 | `AG-*` |
| Negative controls | 10 | `N-*` |

The AI-tool rows cover **all 42 entries in the catalog**, and that is a property worth keeping: a catalog entry with no manifest row is a tool the collector claims to recognize but that nothing ever verifies. Adding a catalog entry should mean adding a manifest row in the same change.
The manifest currently references 42 distinct `catalog_id` values, while the
collector catalog contains 35 entries. They are not synchronized: six current
catalog IDs have no manifest row, and 13 manifest IDs no longer exist under
those names in the catalog. Restoring one-to-one catalog coverage is open work.
Adding or renaming a catalog entry should include the corresponding manifest
change.

### Category 1 — AI tools

Expand Down Expand Up @@ -324,7 +354,9 @@ For each category, every manifest entry is matched to reported assets by `catalo

**Redaction**, checked over the whole snapshot rather than per asset. Every credential planted during installation is a unique canary string. The check is a search of the serialized snapshot for each canary; any hit is a critical failure regardless of the scores above.

**Errors.** `stats.error_count` must be zero, or every error must be explained by something the manifest deliberately created (a denied path, a permission the VM lacks). Unexplained errors fail the run.
**Errors.** The scorecard records total and unexplained errors. Every error must
be explained by something the manifest deliberately created (a denied path or
a permission the VM lacks); unexplained errors fail the gate.

## Reporting the result

Expand All @@ -341,26 +373,40 @@ One table per category per OS, plus a run summary:
| Agents | | | | | | | |
| Negative controls | — | — | | — | — | — | — |

Accompanied by: the baseline asset count, field accuracy per field, the review queue contents, the canary check verdict, `stats.error_count`, and wall-clock scan time per OS.
Accompanied by: the baseline asset count, field accuracy per field, the review
queue contents, the canary check verdict, total and unexplained error counts,
and wall-clock scan time per OS.

Every FP and FN is listed individually with the evidence the collector recorded, because the aggregate number is for tracking and the individual rows are what get fixed.

## Running it

```bash
# per VM, from a clean snapshot
adr-discovery --json > snapshot_before.json
<apply the manifest for this OS>
adr-discovery --json > snapshot_after.json
The implemented local workflows are:

```sh
python3 -m tests.cli check
python3 -m tests.cli synthesize runs/local --os linux
python3 -m tests.cli score runs/local --html runs/local/score.html
python3 -m tests.cli run --os linux --dry
```

Then score `snapshot_before.json`, `snapshot_after.json` and the recorded manifest.
The dry run records installation commands without touching a guest. Individual
Linux and macOS drivers can be used by the bootstrap tooling, but a complete
restore-install-scan-score workflow is not yet wired into `tests.cli run`.
Current driver-specific instructions and limitations are in
[HARNESS.md](HARNESS.md).

Because this run installs real software, signs into real accounts and starts real listeners, it is **not** part of per-commit CI. Run it against a release candidate, when the catalog changes, and when a new OS version ships.
Once the complete golden run is automated, it will install real software, sign
into real accounts, and start real listeners. That workflow should run against
a release candidate, when the catalog changes, and when a new OS version ships,
not as part of per-commit CI.

## Relationship to the fixture suite

This document describes the end-to-end fidelity measurement. The fast, per-commit suite is a different instrument and is documented separately in [FIXTURE_SUITE.md](FIXTURE_SUITE.md): 241 cases that build synthetic endpoints and run in about four seconds on any CI box.
This document describes the end-to-end fidelity measurement. The fast,
per-commit suite is a different instrument: collector module and pipeline tests
live under `adr_discovery/tests_unit/`, while harness validation and scoring
tests live directly under `tests/`. Run both with `pytest -q`.

The two are complementary, and neither replaces the other:

Expand Down