DR: how to implement complex templates?

[AlexanderLanin]

Model-complete Sphinx-Needs reports: decision and investigation

The candidate implementations are not inherently complex; several are quite small. The hard part is selecting an architecture that remains correct for future report dimensions.

The main investigation is split into one comment per phase; an additional assessment follows:

1. Phase 1 – Problem statement and decision criteria
2. Phase 2 – Established evidence and current implementations
3. Phase 3 – Proofs of concept, solution space and critical analysis
4. Phase 4 – Validation and explicit exclusions
5. Phase 5 – Recommendation and next decision
6. Additional assessment – Claude opinion

[AlexanderLanin]

Phase 1 – Problem statement and decision criteria

Decision to make

Choose an architecture for a model-driven verification report that has both:

1. the complete, resolved Sphinx-Needs model (including cross-worker and
   external Needs); and
2. ordinary reStructuredText sections, anchors, and local table-of-contents
   entries in the same build.

The decision is not whether a Jinja helper should be a function or a
precomputed value. That choice cannot change when the model is available. The
actual question is whether the report becomes normal source before Sphinx
reads the document, whether a deliberately version-pinned extension may
materialize an existing Doctree and refresh its local ToC after collection, or
whether upstream Sphinx/Sphinx-Needs grows a supported post-collection
report-and-ToC API.

The current implementation is evidence and a temporary workaround, not an
accepted design.

Decision criteria

The selected design must:

• retain parallel reading for the ordinary documentation build;
• use graph links, not Need-ID conventions or hardcoded component lists;
• resolve external Needs and calculated backlink fields from the complete
  model;
• let the report grow to further graph-derived dimensions without adding a
  parallel directive-option schema and one drift check per dimension;
• provide a real hierarchy to Sphinx and the theme, rather than a second HTML
  navigation menu;
• have explicit inputs and dependencies, deterministic output, and a clear
  incremental-build story; and
• keep the existing docs() build topology: this report must not introduce a
  second Sphinx model-export target, manifest pipeline, or generated-source
  mount solely to materialize its body; and
• use public, versioned APIs at the integration boundary. Dependency-private
  state may only be isolated behind a separately tested compatibility adapter.

Central phase conflict to resolve

There is no currently documented extension event that satisfies both timing
requirements in one ordinary Sphinx read pass:

• doctree-read and Sphinx's ToC collection happen while individual documents
  are read; a parallel worker does not yet have the integrated Need model.
• env-updated runs after all documents and worker environments are merged,
  but its supported return value merely marks documents for writing; it does
  not schedule a second source read.
• Sphinx-Needs documents get_needs_view(app) as a read-only view of fully
  resolved Needs for the write phase. Calling it may resolve and lock the
  Need data, so it is intentionally not a read-phase template API.

Consequently, a report rendered after collection can use the complete model,
but cannot become ordinary read-phase sections and ToC entries through a
public API. A controlled env-updated mutation can refresh the local ToC of
an existing document, but only through a Sphinx-version compatibility adapter;
it is not a second ordinary source-read boundary. A report rendered while
reading can become normal sections, but cannot safely depend on the complete
model. The present manual purge and re-read crosses this boundary, which
explains both why it works and why it is fragile.

Relevant contracts: Sphinx event callbacks,
Sphinx build-environment API,
Sphinx-Needs get_needs_view,
and Sphinx-Needs builder/export configuration.

Concrete use case

Running:

bazel run //:docs

must generate:

_build/baselibs_report.html

The source of the report is:

src/needs_templates/mod_ver_report_tiny.need

The desired result is the same report content and navigation as the reference
page:

https://eclipse-score.github.io/baselibs/pr-491/reporting/module_verification_report.html

The development build currently emits warnings from score_baselibs; those
warnings are deliberately outside the scope of this problem and should not be
used as evidence that the template architecture is wrong.

The document currently contains one report Need in
docs/baselibs_report.rst:

.. mod_ver_report:: Report for Test Module
   :id: mod_vrep__baselibs__test_mod
   :belongs_to: mod__baselibs
   :template: mod_ver_report_tiny

The real input is an integrated Need model containing the module, its feature,
components, requirements, architecture items, tests, and work products. Many
of those Needs come from mounted or external repositories.

Desired behavior

The report must be model-driven

The template must follow Need links and derive its sections from the Need
graph. In particular:

• component IDs must not be copied into the template as a manually maintained
  list;
• the component order should follow the order of the relevant Need links;
• the feature and component data should be resolved from Need relationships,
  not reconstructed from ID naming conventions where a relationship exists;
• filters should be able to use the generated backlink fields, for example
  includes_back, belongs_to_back, or satisfied_by_back;
• an external Need such as feat__baselibs is a real, known Need in the
  integrated build. The implementation must not make it appear absent merely
  because it was collected in another worker or loaded at another lifecycle
  stage.

The template currently has a linked_needs(need_id, link_name) helper. It
follows outgoing links and returns NeedItem objects. It is useful for an
ordinary, read-phase template, but it is not a solution for a model-complete
report: changing it into a precomputed attribute cannot move the model into
the earlier read phase.

The report must be a normal Sphinx page

The generated report should have:

• the same visible report content as the reference;
• working links to the actual Needs and documents;
• the usual theme navigation, including the local “On this page” navigation;
• navigation entries for dynamic component headings;
• real, working anchors for those entries;
• a meaningful heading hierarchy rather than a flat, manually assembled HTML
  menu.

An actual navigable menu is a requirement, not an optional enhancement. It
should use the normal Sphinx/theme navigation and must contain links to the
generated report sections, including sections for dynamically linked
components.

The report is currently rendered inside a Sphinx-Needs Need node. This is a
compatibility detail, not a selection criterion: any replacement must preserve
the report Need's metadata, links, targets, and normal Sphinx-Needs processing
and must regression-test dependent layouts/extracts. The visible report body
may move outside the node when that produces normal document structure.

Parallelism and build behavior are non-negotiable

Normal Sphinx parallel reading must remain enabled for the complete docs
build. Disabling parallelism globally makes the build too slow and is not an
acceptable solution.

Serializing the processing of this one report, or running a dedicated final
render step for it, is acceptable if that is required to obtain the complete
Need model and a correct menu. What is not acceptable is serializing the
entire documentation build: the hundreds of unrelated pages must continue to
be read and processed in parallel.

The solution should also preserve:

• incremental builds;
• deterministic output;
• safe environment merging between workers;
• HTML builds without breaking other builders or ordinary Need templates;
• a clean process model without hidden cross-build global state.

[AlexanderLanin]

Phase 2 – Established evidence and current implementations

Relevant lifecycle facts

The following behavior was observed while investigating the problem:

1. Sphinx reads source documents in parallel workers.
2. A Sphinx-Needs template is expanded while the source document is being read. At that point, a worker may not have all Needs from other workers.
3. Sphinx-Needs creates a Need node and parses template output into it with titles disabled (match_titles=False). RST headings inside that Need content therefore do not automatically become ordinary Sphinx sections.
4. Worker environments are merged later.
5. External Needs can be present in the integrated model, but their availability during the first worker-local template expansion must not be assumed.
6. Sphinx's normal ToC collector runs during the read phase. It does not automatically collect arbitrary sections nested in a Sphinx-Needs Need.
7. Sphinx-Needs resolves backlinks during later Need post-processing. Backlink fields are therefore safe inputs only once the resolved NeedsView is available.
8. Sphinx has no public event that both provides the fully merged NeedsView and starts a second ordinary source-read/ToC-collection pass for one document.

The central lifecycle problem is therefore real: a worker-local template render cannot safely discover the complete graph, while a late render has no ordinary public Sphinx API for adding sections and local ToC entries.

What has been tried or considered

1. Hardcoded component lists

A manually maintained list produces predictable sections, but it does not discover the report outline from the Need graph. It was rejected as the general graph-driven solution.

2. A dynamic template function

A helper such as:

{% set components = linked_needs(module_id, "includes") %}
{% for component in components %}
{{ component["title"] }}
{% endfor %}

removes hardcoded component IDs, but it still runs during the worker's first source read. The function cannot make an incomplete model complete.

3. A plain function in the render context

Sphinx serializes render configuration for parallel readers, so a context callable must be pickleable. A top-level callable class solves serialization, but not model completeness. This is an implementation detail, not a lifecycle solution.

4. Disable parallelism globally

This would hide the timing problem, but it would slow down the complete documentation build and is not acceptable.

5. PR #763 / the former G implementation

The former implementation rendered a marker during the normal read, then waited for environment merge, purged selected documents, called builder.read_doc() again, and patched env.tocs for navigation.

It demonstrated that the report can be produced, but it was a fragile lifecycle workaround:

• it re-entered source reading from env-updated;
• it called private or lifecycle-sensitive operations such as read_doc(), clear_doc(), and direct ToC-state mutation;
• it rendered and processed the document twice;
• it depended on event ordering and Sphinx-Needs post-processing;
• it did not provide a general, builder-neutral contract.

This implementation is retained only as historical context and a behavioral baseline. It is not the implementation proposed by PR #769.

6. Set the template environment at extension startup

app.env is not available during extension setup. Keeping it in module-global state after builder-inited is implicit process-local state and is not a satisfactory model API. The L+ implementation keeps report declarations, fingerprints, and reverse dependencies in the Sphinx build environment instead.

7. Extract the template code into a dedicated extension

The implementation was moved to:

src/extensions/score_sphinx_needs_templates/

This separates configuration, the L+ lifecycle integration, the template, and tests from the general score_sphinx_bundle. The separation improves ownership, but it does not remove the Sphinx phase boundary.

8. Use ordinary RST section headings directly inside a Need template

This does not work by itself because Sphinx-Needs parses Need content with match_titles=False. L+ therefore parses the final rendered RST as a controlled fragment and inserts its section nodes into the existing report document after the complete model is available.

9. Add a second HTML menu or patch env.tocs directly

The former G implementation used rubric-derived IDs and direct env.tocs mutation. This produces a visually plausible HTML menu, but it is flat, builder-specific, and coupled to collector internals. L+ instead reruns the supported collector implementation through one explicit, version-pinned compatibility adapter for the affected document only.

10. The L+ implementation in PR #769

The current proof of concept uses the template from src/needs_templates/mod_ver_report_tiny.need and follows Need links against the final resolved NeedsView.

The lifecycle is:

parallel source read
        |
        v
report Need and lightweight placeholder
        |
        v
worker environment merge
        |
        v
env-updated obtains the complete NeedsView
        |
        v
render selected Jinja template and parse the RST fragment
        |
        v
replace the placeholder in the cached report Doctree
        |
        v
refresh that document's local ToC through a version-pinned adapter

The report Need remains present for its metadata, links, targets, and normal Sphinx-Needs processing. The generated report body is materialized beside it. The implementation records graph-based reverse dependencies and template fingerprints for incremental invalidation.

The important improvement over PR #763 is that L+ does not manually purge and re-read the document, call read_doc() or clear_doc(), or append entries directly to env.tocs. The remaining private boundary is deliberately small: cached Doctree persistence and the Sphinx ToC collector are isolated in a compatibility adapter and currently pinned to Sphinx 9.x.

Current validation

The following checks pass:

bazel test //src/extensions/score_sphinx_needs_templates:report_pipeline_tests
bazel run //:docs

The HTML build produces the report with graph-derived sections and nested local navigation. The proof of concept has not yet completed the full acceptance gate from Phase 4: incremental-build equivalence, serial versus parallel output comparison, a non-HTML builder, and broader reference/link checks remain open.

Updated conclusion

L+ is a technically valid controlled solution for this report, but it is not a lightweight general-purpose extension. Sphinx does not currently provide a public API for model-complete late sections plus local ToC integration. L+ can contain that gap in a small version-pinned adapter, but it cannot make the coupling disappear.

If automatic discovery from the Need graph is mandatory, L+ is a reasonable prototype for the existing one-build docs() topology. If an explicit, reviewable component list is acceptable, PR #730 remains the simpler long-term solution because it creates ordinary sections during the read phase using public Sphinx APIs.

[AlexanderLanin]

Phase 3 – Proofs of concept, solution space and critical analysis

Proofs of concept

The immediate decision is between three concrete implementations:

• PR #730 introduces .. module-verification-report::. It creates the report outline during the normal read phase from an explicit :components: list, emits normal section nodes beside a generated mod_ver_report Need, and leaves data tables to late Sphinx-Needs filters. It is option K.
• PR #763 was the original option G proof of concept. It demonstrated graph-driven output, but used a manual purge/reread and direct ToC-state patching. It is now historical context, not the proposed implementation.
• PR #769 is the L+ proof of concept. It keeps the parallel read, records a report placeholder, renders the selected Jinja template against the complete merged NeedsView at env-updated, materializes the fragment in the existing report Doctree, and refreshes the local ToC through a bounded version-pinned adapter.

K / PR #767: read-phase implementation evidence

PR #767 makes the K boundary concrete. Its report body is parsed into the surrounding document as a sibling of the report Need, so headings are ordinary sections rather than content inside a Need node. Its rendering code is a model-independent string transformation; Need-dependent tables and references remain normal Sphinx-Needs directives resolved later. It deliberately has no env-updated rerender, no build-finished consistency pass, and no cross-worker registry. The scope check is a bidirectional metamodel graph check rather than lifecycle state in the reporting extension. Stable anchors are promoted during the read phase and are covered by multiple-report tests.

This is positive evidence for K as the least lifecycle-complex option, not proof that PR #767 is ready to merge. Its self-review identifies three bounded but real integration risks:

• ReportNeedDirective restores the Need type by changing the invoked directive name; this depends on a Sphinx-Needs implementation detail and adds a second directive name;
• the priority-900 config-inited registration depends on Sphinx-Needs registering its own directive at priority 500; and
• anchor promotion mutates Docutils document ID state and removes targets to avoid duplicate propagation.

Before using this implementation, prove directive resolution, root and nested placement, two reports at different heading depths, stable anchors, and the supported Docutils/Sphinx-Needs matrix. These are much narrower concerns than a late lifecycle re-entry, but they should remain visible in the comparison.

Decision criterion	PR #730 / option K	PR #763 / former option G	PR #769 / option L+
Report outline	Explicit :components: list at read time	Discovered from Need links after merge	Discovered from Need links after merge
Complete resolved model	Data widgets can resolve it; outline is declared	Yes in the second pass	Yes at env-updated
Normal sections and local ToC	Yes, through public read-phase parsing	Partial; rubric-derived private env.tocs patch	Section nodes plus a local ToC refresh adapter
Parallel ordinary docs	Yes	Yes, except selected reread	Yes; only the selected report Doctree is materialized after merge
External-Need handling	Tables can resolve external IDs; outline still needs them listed	Available in the post-merge model	Available in the resolved NeedsView and graph traversal
Main source of truth	Directive options plus Need data	Need graph plus lifecycle-private machinery	Need graph plus environment-owned declarations/dependencies
Main risk	Scope can drift from the graph	Re-entrant lifecycle and private ToC state	Version-pinned cached-Doctree/ToC compatibility boundary
Current status	Clean static-scope alternative	Historical behavioral baseline	Validated prototype; strict acceptance gate remains

PR #730 remains the cleanest Sphinx integration when a reviewable, manually declared report scope is acceptable. PR #763 proved the behavior but paid for it with an unbounded lifecycle workaround. PR #769 shows that L+ can retain graph-driven discovery within the existing one-build docs() topology while reducing the workaround to a contained compatibility adapter. That makes L+ technically viable, but not lightweight or equivalent to generated normal source.

Solution proposals

Comparative assessment

The options below deliberately separate the materialization, document-identity,
and ToC-integration axes. A check mark means the property follows from the
architecture; an asterisk means that it is possible only with an upstream API
that does not exist today.

Option	Concrete implementation	Complete resolved model	Normal Sphinx ToC	Parallel ordinary docs	Recommended use
A. Generated source from report manifest	—	Yes, with explicit integrated-model contract	Yes	Yes	Reject: second Sphinx/Bazel pipeline
B. Generated reports bundle	—	Yes	Yes	Yes	Reject: more packaging of A
C. Stand-alone graph compiler	—	Yes	Yes	Yes	Reusable report platform
D. Preload model in every worker	—	Potentially	No with the current template structure	No clean way	Reject
E. Reader ordering / report serialization	—	No clean way	Potentially	No	Reject
F. Upstream report + ToC API	—	Yes	Yes*	Yes	Strategic upstream investment
G. Current second pass	★ PR #763	Yes in practice	Partial, private patch	Yes, except selected reread	Temporary bridge
H. Write-phase node replacement	—	Yes	No	Yes	Builder-neutral late fragment only
L. Post-collection materializer	—	Yes	No	Yes	Self-contained late fragments only
L+. Materializer with ToC refresh	★ PR #769	Yes	Local sections through adapter†	Yes	Viable prototype; strict acceptance gate
I. html-collect-pages page	—	Yes	No	Yes	Separate HTML export only
J. Relax the normal-ToC requirement	—	Yes	Theme-specific only	Yes	Product decision, not an implementation fix
K. Hybrid directive	★ PR #730	Data widgets: yes; outline: no	Yes	Yes	Only if explicit component list is accepted

† L+ re-collects the existing document's local section tree through a
version-pinned compatibility adapter. It does not create new source documents
or grant late-generated content ordinary parse and reference semantics.

A. Alternative: generate normal report source in a separate model stage

Make the report body a generated .rst source file, not the rendered content
of a Need template. Preserve the report Need as a small, explicit metadata
Need on the page, but put the model-derived report body immediately after it
with an .. include:: of the generated file. This retains the report Need's
ID, fields, links, and Sphinx-Needs processing while making the report
headings normal document sections.

The build has two dependency-aware stages:

Need sources + mounted/external Need inputs
        |
        v
parallel model-collection action -> versioned report manifest
        |
        v
report renderer -> generated .rst with sections and Need references
        |
        v
normal parallel Sphinx build -> ToC, links, supported builders

This is deliberately two Sphinx collections on a cache miss: the model
action reads the baseline documentation with the report stub and exports the
Need model; the final action reads the documentation again with the generated
report body and writes the selected builder output. Manifest and RST rendering
are cheap pure actions between them. Bazel caches the model export, manifest,
and generated RST by their declared inputs, so an unchanged repeat does not
rerun those actions. It does not, however, share a Sphinx environment, parsed
Doctrees, or Need collection between the two Sphinx actions. A change to any
input of a monolithic model export can therefore incur the collection action
again, even when the generated report remains byte-identical.

This cost is the central operational trade-off against L+. It also violates
the adopted requirement to keep this report within the existing docs() build
topology. The pipeline remains a valid reference architecture if a future
product already requires the model export for other consumers, but it is not
the selected implementation path here. It is not correct to describe A merely
as one small additional render action.

The collection action must have an explicit integrated-model contract. The
repository's //:needs_json is not automatically such a contract: the
Sphinx-Needs builder filter normally excludes external Needs. Choose one of
these deliberate designs:

1. configure a report collector to export the required external Needs and
   resolved fields, with an ID-collision and external-URL policy; or
2. consume local :needs_json_file and every external needs.json as declared
   inputs and perform a specified merge, prefix, condition, and backlink
   resolution step in the manifest producer.

The manifest must be typed and versioned rather than leaking a Sphinx
environment pickle or private dictionaries. It must interpret exports produced
with needs_json_remove_defaults=True against the matching schema, so an
absent field is not accidentally treated as None; it must also preserve the
Sphinx-Needs semantics of conditional links before traversal. The renderer
follows the chosen, documented link order, validates every required Need, and
emits standard section syntax and :need: references. A missing linked Need
is a generation error with the source report and link field in its diagnostic;
it must not render as an empty section.

This is the only option that fulfils the complete-model and normal-ToC
requirements using today's public phase boundaries. Its costs are an explicit
intermediate artifact and a second, parallelisable build action. In Bazel,
those costs are normally acceptable: the generated source is cacheable and
the action inputs precisely describe when a report must change.

The graph collection must be acyclic. Concretely, replace the current complex
template with a non-dynamic report-Need stub in the collection source. The
stub supplies the report ID and link fields, but not its generated body. The
manifest action then produces the body consumed by the final documentation
action. Do not make :needs_json depend on a final HTML build that itself
depends on the generated report; that would create a Bazel/Sphinx cycle and
make incremental behaviour opaque.

Required compatibility check. The report Need remains the authoritative
Need object. Verify that its fields, links, targets, layout, extracts, and
consumer-facing references remain correct after the visual body becomes normal
document content. This is a regression obligation for every option, not a
criterion that chooses between A and F.

The semantic check must include builders and styling. Existing report content
may contain HTML-specific raw markup or CSS assumptions; moving it into normal
RST only preserves non-HTML builders when the generator uses builder-neutral
nodes/markup or supplies an intentional representation for each builder.

B. Variant of A: materialise reports as a generated documentation bundle

When reports are numerous, generated report source should be packaged as a
data-only docs_bundle and mounted into the final documentation source tree.
Each report becomes an ordinary generated .rst page beneath a well-defined
mount point. The bundle contract must provide an attachable entry document:
either generate an index.rst with its own toctree or set entry_doc to the
generated report page. The final Sphinx build then reads it as normal source,
so sections, targets, search indexing, and all supported builders work without
special theme code.

This variant has useful ownership boundaries: the report generator owns the
derived sources, while the consuming documentation project owns placement and
ToC attachment. It also permits a report package to be tested from its
manifest without running the complete site. The repository's docs_bundle
and docs(data=...) mechanisms are designed to stage generated/supporting
files; use the former when the report must travel with a mount.

It adds rules and staging paths, so it is excessive for exactly one small
report. It is preferable to ad-hoc source-tree writes: generated files must
be Bazel outputs, never be written back to docs/ during a build. Cross
references must be emitted in the final project's Sphinx source context, not
resolved while the bundle is generated; otherwise links to local Needs cannot
be validated by the final build.

This is primarily an own-page packaging model. A generated file mounted in a
data-only bundle is not automatically a reliable .. include:: immediately
after a Need in a host document: the include path must resolve identically for
workspace runs and sandboxed builds. Treat inline inclusion as a separately
tested adapter, not as an implied property of the bundle mechanism.

C. Alternative input architecture: a stand-alone canonical graph compiler

Instead of obtaining the manifest through a seed Sphinx build, define a
canonical, validated Need-graph IR and compile it from Need source and external
needs.json inputs. Both the report generator and, eventually, Sphinx-Needs
could consume that IR. The report renderer is then a conventional deterministic
compiler: graph IR plus report specification in, .rst and diagnostics out.

This is attractive if reports become a product area used outside Sphinx (for
example PDF, dashboards, or compliance exports), because traversal semantics,
ordering, and diagnostics have one owner. It also makes Bazel dependencies
very explicit and avoids using a Sphinx builder merely as a model exporter.

It is not a cheap replacement for proposal A. Parsing Need directives,
dynamic fields, variants, constraints, external imports, and backlink rules
outside Sphinx-Needs would duplicate a large semantic surface and risks two
different models. Adopt it only if the project commits to making this IR the
single authoritative model with conformance tests against Sphinx-Needs.
Otherwise, use Sphinx-Needs' exported model as the manifest source in A.

D. Rejected: give every parallel reader the complete model up front

At first sight, a pre-read collector that builds the Need graph and distributes
it to every worker seems to preserve the current template API: the workers
would render templates with a complete linked_needs context, and Sphinx
would collect the resulting headings normally. It does not form a clean
architecture in this project.

Local Needs are only known after their source documents are parsed. Building
the graph before that point therefore either implements a second parser for
Sphinx-Needs syntax (the duplication problem in C), or reads all documents
once before the normal reader. The latter is a two-stage build in disguise and
must be a separate Bazel action to have reliable invalidation and environment
merging. Passing a mutable BuildEnvironment to worker render context is also
not a public API and produces stale, pickle, and process-isolation hazards.

This option is acceptable only if a canonical external graph compiler already
exists; then it collapses into C followed by A. It must not be implemented as
a hidden global, worker-local cache or a custom pre-scan of .rst files.

E. Rejected: influence reader order or serialize the report document

Another apparent shortcut is to read all ordinary documents in parallel and
read report documents last, serially. Sphinx does provide the public
env-before-read-docs hook to reorder the initial document list, but that is
not a dependency barrier: with parallel reading Sphinx still partitions the
list into worker chunks, and worker environments are merged only after those
chunks complete. A nominally late report document therefore cannot see local
Needs from other workers without manually triggering the same merge/read
operations used by the current workaround. External Needs may already be
loaded at this point; the unresolved part is the locally read model.

Global -j 1 has the same logical effect but makes the entire documentation
set serial and violates the performance requirement. Isolating one document
in a separate Sphinx invocation is viable, but that is precisely the explicit
model-and-source pipeline in A, not an in-process ordering trick.

F. Strategic: request a supported upstream report feature

Propose a Sphinx-Needs feature for model-complete reports, rather than growing
another repository-local lifecycle workaround. The upstream API must cover two
separate products:

• a post-collection renderer supplied with an immutable, complete
  NeedsView, including external Needs and backlinks; and
• a way to contribute a structured section outline (title, stable target,
  level, and document) before Sphinx builds the document ToC, or a Sphinx
  upstream API that allows the same contribution later without editing
  env.tocs.

An API that only adds a Jinja context, a doctree-resolved hook, or an HTML
visitor is insufficient: it can produce content but cannot retrospectively
make it a normal source-level hierarchy. Likewise, a Sphinx-Needs-only
post-render hook cannot meet the ToC requirement unless it comes with an
explicit, supported ToC contract.

It is not an immediate delivery path: it needs an upstream design,
compatibility version, and tests for parallel reading, external Needs,
incremental builds, and non-HTML builders. Until such an API exists, proposal
A is the clean local solution.

G. PR #763: transitional containment with a second pass

The existing marked-template second pass may remain only as a time-boxed
bridge while proposal A is implemented or an upstream feature is evaluated.
Its contract should be deliberately narrow:

• only named report documents may opt in; ordinary Need templates remain
  read-phase templates;
• the second read happens once, after merge and before Need post-processing;
• model traversal is read-only and errors on absent graph targets;
• all use of SphinxNeedsData, manual purging, read_doc, and env.tocs is
  isolated in one adapter with pinned Sphinx/Sphinx-Needs compatibility tests;
• it is exercised with -j 1 and -j > 1, a clean build and an incremental
  rebuild, plus an external linked Need; and
• it has an owner and removal criterion.

This option deliberately does not become the long-term architecture. It
still relies on re-entering a lifecycle that Sphinx exposes no public event to
repeat, and its ToC patch is structurally flat.

It is also listener-order-sensitive: another earlier env-updated listener
can resolve and lock the Need view, causing this handler to skip its reread.
The builder's direct read_doc and environment clear_doc calls are not
public extension APIs, and the module-global template environment is
process/app state rather than environment-owned state. These are compatibility
risks even when the current event sequence happens to work.

H. Rejected for the stated requirements: write-phase node replacement

A custom placeholder node could be populated in doctree-resolved from
get_needs_view(app). It can be a builder-neutral late fragment when it emits
only already-supported Docutils nodes with no new Sphinx-level semantics. It
is not a solution here:
reference resolution and the normal ToC were already processed in the read
phase, and post-hoc sections have no supported way to join that hierarchy.
It must also record reverse Need-to-report dependencies and return affected
report documents for rewriting; otherwise an incremental build can retain a
stale late fragment after a linked Need changes.

L. Alternative: a Needextend-like post-collection materializer

needextend provides a useful lifecycle precedent for a different design. Its
directive is parsed during the read phase, but its modifications are not
applied immediately. Sphinx-Needs first merges the worker environments through
env-merge-info; at write-started, it post-processes the complete Need
collection, applies the recorded extensions, resolves functions and links,
and computes backlinks. The write phase then exposes the resulting immutable
NeedsView to Need roles, needtable, and needpie.

A complex_need_template directive could use the same boundary without
rereading a document:

1. During reading, register a placeholder and the report's reverse Need
   dependencies.
2. After the Need collection has been merged and post-processed, obtain the
   final NeedsView.
3. During doctree-resolved, replace the placeholder in the local report
   document with generated content derived from that model.

This avoids both the PR #763 lifecycle re-entry and the Bazel bootstrap cycle:
there is one Sphinx build, no generated source is fed back into the model
collection, and the materializer sees Needs collected by other parallel
workers. It also naturally observes needextend changes that were accepted
by the model.

The approach is not a way to create a normal source-level document hierarchy
after the fact. Sphinx's ToC collector has already built env.tocs during
reading, and existing toctree nodes are resolved before doctree-resolved is
emitted. Newly inserted sections therefore render in the page body but do not
automatically appear in the normal local ToC. Newly inserted pending
references or Sphinx-Needs nodes would also need their processing rerun or
would have to be emitted as already-resolved nodes.

This option is consequently a valid alternative only if the requirements are
relaxed to accept write-phase report content. It is attractive when keeping
docs() unchanged and avoiding a second Bazel action matter more than normal
section-level navigation, source-level cross-reference processing, and full
non-HTML parity. It must still persist and merge its reverse dependency map so
that changes to selected Needs, links, schemas, or external inputs invalidate
the report document during incremental builds.

Decision: keep L as a serious alternative to A, not as an implementation
of the normal-ToC requirement. Prototype it only with a small report that
emits self-contained Docutils nodes, and explicitly test needextend, local
and external Needs, -j 1/parallel writes, and incremental invalidation.

L+ Variant: post-collection materialization with ToC refresh

L becomes materially stronger if the placeholder is materialized at
env-updated, rather than at doctree-resolved. At that point all reader
workers have been merged, the final NeedsView can force the normal
post-collection processing, and Builder.write() has not started yet.

The extension can then load the already-read report doctree, replace its
placeholder with real section, title, target, and Need-rendering nodes, and
rebuild the local ToC for that document before writing starts. The existing
global toctree already points at the report document; only the report's local
section ToC needs to be refreshed. This is still one Sphinx build and does not
create a Bazel dependency cycle.

L+ therefore restores the most visible navigation property of generated RST
without rereading the document. It is nevertheless a controlled mutation of
Sphinx's cached doctree and collector state, not a public general-purpose
source parser boundary. The implementation must:

• materialize exactly once and guard clean and incremental builds;
• avoid builder.read_doc, env.clear_doc, and manual reader re-entry;
• use direct Docutils/Sphinx-Needs nodes or an explicitly isolated parser
  adapter for the generated body;
• isolate the ToC refresh behind a Sphinx-version compatibility adapter; and
• preserve the environment-owned reverse dependency map for incremental
  invalidation.

L+ can provide a normal local section ToC, but it does not make newly created
documents or arbitrary late source semantics. Generated pending references,
Need directives, non-HTML builders, and parallel write/cache behavior need
explicit tests. It is best understood as a Sphinx extension implementation
of the A contract, with a later model boundary, rather than as a fully
ordinary generated source file.

Decision: L+ is the selected implementation direction. It preserves the
existing docs() topology and avoids the reread in G; its remaining question
is whether the bounded Sphinx compatibility adapter can meet the acceptance
tests. Keep plain L only as the fallback for reports that do not need local
section navigation.

I. Rejected for normal documentation: publish a late HTML page

Sphinx's html-collect-pages event can add an HTML page after the model is
complete. A page context can be built from get_needs_view(app), so it is a
reasonable export mechanism for a self-contained dashboard, downloadable
report, or machine-oriented visualization.

It is not an ordinary Sphinx document: it is HTML-builder specific, has no
source document for the normal ToC collector, and cannot naturally participate
in source-level cross-reference resolution, non-HTML builders, or the report
Need's normal document semantics. It can receive the theme's global context and look
visually integrated, but it has no ordinary local ToC or active toctree
position; search indexing, object inventory, and linkcheck do not gain this
virtual page automatically. Relative URLs for nested page names and HTML
escaping are also the extension's responsibility. Keep this option only for
an explicitly separate HTML deliverable.

J. Product alternative: relax the normal-ToC requirement explicitly

If the product decision is that only HTML output matters and standard local
navigation is out of scope, retain a custom report navigation model: a
semantic report-heading node, stable anchors, and a theme adapter that renders
its tree. This is better than scraping rubric nodes because titles and levels
are explicit and testable.

It is still not Sphinx's normal ToC. It requires a versioned adapter for the
chosen theme (PyData in this repository) and separate behaviour for every
builder, and it keeps the
complete-model timing problem unless combined with G or a write-phase render.
Therefore this is a conscious requirement trade-off, not an alternative that
meets the requirements at the top of this document. It should be selected only
with product approval that standard local navigation and non-HTML parity are
out of scope.

K. PR #730: static report outline with late Sphinx-Needs widgets

PR #730 implements
another viable shape: a .. module-verification-report:: directive renders a
temporary plain Docutils container during the normal read phase, using
nested_parse_with_titles, and returns its children into the surrounding
document. Its generated reStructuredText therefore contains ordinary section
headings (and normal document navigation) plus a generated mod_ver_report
Need. The data-heavy portions are emitted as needtable and needpie
directives; Sphinx-Needs resolves their filters only after Need collection. The
generated Need and report body are siblings, not a report body nested inside a
Need. A build-finished consistency check verifies only that each listed
component belongs to the module's includes links and, when set, that the
requested feature belongs to each listed component.

This is a sound solution for a static report outline with dynamic table
contents. It keeps ordinary parallel reading, uses normal Sphinx headings,
and avoids the current manual reread and env.tocs patch. It is therefore
substantially cleaner than G for consumers willing to declare their report
structure.

It does not meet this investigation's model-driven-outline requirement:
:components: is mandatory and is a manually maintained list, component
titles are derived from ID prefixes instead of Need fields, and the directive
does not discover the component order from the module's includes links. Its
check does not detect a component that is present in includes but omitted
from :components:; it only rejects listed components that are not linked.
Even a stronger late check could not make an omitted component appear in the
already-read ToC. The report body also sits beside a generated report Need,
rather than inside that Need node.

Accordingly, K is not a replacement for A under the stated criteria. It is a
strong alternative only if the product explicitly accepts :components: as
the authoritative report-outline declaration and treats graph validation—not
graph discovery—as sufficient. That is not the expected direction for this
report: components are only the first graph-derived dimension. Features, work
products, or other model groups would require further directive options,
validation rules, ordering rules, and title policy, turning K into a second
manually maintained report model.

Bidirectional validation is still required if K is ever used: fail both when a
declared component is absent from module.includes and when an included
component is absent from :components:. It prevents drift for that one known
relation, but it does not make the outline model-driven or solve growth to
additional relations. In the static-scope case, adopt PR #730's directive and
add tests for both mismatch directions, an external Need in a needtable, and
section/anchor output under parallel reading.

Critical-risk analysis and evidence required

The following analysis is deliberately more demanding than a feature list. A
passing happy-path demo is not sufficient evidence for any option: the risks
are primarily invalidation, model semantics, phase ordering, and long-term API
ownership.

A. Generated source from a report manifest

The major risk is a hidden bootstrap cycle. The seed needs_json build must
read a report-Need stub but must not require the final generated report body;
the final documentation build may consume the body. If the seed and final
build have different Sphinx config, defines, mounts, or external-Needs inputs,
dynamic fields, variants, link targets, and visibility can disagree even when
both builds individually succeed. The manifest must therefore record a
fingerprint of every semantics-affecting input and fail when the final build
uses an incompatible configuration.

The word "complete" cannot be delegated to the existing :needs_json target
without a contract. Its default Sphinx-Needs builder filter excludes external
Needs, and its configured removal of default-valued fields makes an omitted
JSON field semantically different from an explicit None. The manifest
producer must either export the integrated model with an intentional external
Need filter, or merge the local export and every external JSON input itself.
It must apply the matching schema defaults and conditional-link semantics,
including any configured ID-prefix and URL policy.

Graph ordering requires special care. The requirement is source-link order,
but Sphinx-Needs may normalise link collections for deterministic processing.
The manifest contract must explicitly state which order it exports and test it
with non-alphabetical linked components. If the exported model no longer
retains declared order, the producer must export that order as a first-class
field; sorting IDs in the renderer is not an equivalent substitute.

There are two incremental-build limits. First, the model export is a separate
Sphinx collection: on a cache miss it reads the baseline sources in addition
to the final Sphinx action that reads the full report sources. Bazel can reuse
the export across invocations and consumers, but cannot reuse its Sphinx
environment in the final action. Second, if the renderer consumes one complete
needs.json, Bazel will normally execute it whenever that input changes,
including unrelated Need changes. The realistic target is a byte-identical
generated report and no invalidation of the final HTML when its transitive
fingerprint is unchanged—not necessarily zero renderer executions. Achieving
finer action-level invalidation needs the model producer to emit per-report
shards or a dependency index, which is an optimisation to justify with build
measurements. Generated-source diagnostics must map failures back to the
report ID and contributing Need IDs, otherwise authors receive only opaque
line numbers in a Bazel output tree.

Evidence before implementation: inspect one complete exported model for
external Needs, resolved backlinks, variants, and link order; define a schema
and compatibility fingerprint; prove seed/final equality for the actual report
dependencies (including builder-affecting settings such as variants and
mounts); and write clean/incremental/parallel tests that compare generated
source, not merely HTML screenshots.

B. Generated reports bundle

This inherits all risks from A and adds source-placement risk. A data-only
bundle is not automatically proof that its .rst files are discovered by the
final Sphinx source tree, mounted once, and attached to the intended toctree.
The bundle rules, mount manifest, and sphinx_docs source staging must agree
for both bazel run and sandboxed bazel build; a path that works only from a
workspace checkout is not sufficient.

A pure-data bundle also needs an explicit entry contract. Its default entry is
index, so a generator that emits only module-report.rst has no reliable
attachment point unless it creates an index/toctree stub or sets entry_doc
accordingly. Files are mounted as a directory, not inserted beside a host
Need; inline inclusion is a separate cross-mode path-resolution problem.

Ownership can also obscure dependencies: the report-producing package may use
Need IDs from the consuming package, while the consumer owns the final
cross-reference resolution. Keep the generated body free of resolved URLs and
emit only source-level Need references. Verify collision handling for generated
docnames, labels, anchors, static assets, and multiple mounted report bundles.

Evidence before implementation: an end-to-end fixture with two mounts,
one report, a local cross-reference, and an external Need; run it both through
the interactive target and the sandboxed needs_json/HTML targets.

C. Stand-alone canonical graph compiler

Its critical risk is semantic fork, not implementation size alone. Sphinx-
Needs has behaviour for dynamic values, filters, parts, imports, variants,
constraints, and backlinks. A compiler that approximates even one of those
rules can produce a report that contradicts Sphinx's own Need tables. Version
pinning does not solve this: every upgrade creates a compatibility surface
that must be revalidated.

The design is credible only if the graph IR becomes an owned product with an
explicit language specification and differential tests against Sphinx-Needs
for representative models, including errors. Otherwise the compiler becomes a
second undocumented parser whose apparent determinism hides correctness drift.

Evidence before investment: name a non-Sphinx consumer that cannot use A,
define IR ownership and upgrade policy, and demonstrate differential agreement
on imports, backlinks, conditional links, and variants. Without these, reject
C on lifecycle cost alone.

D. Preload the model in every reader

The central failure mode is two inconsistent models: a pre-scan model used for
headings and Sphinx-Needs' later model used for tables and validation. A small
source change can then affect only one model, particularly with includes,
collections, mounted sources, configuration-dependent options, or external
Needs. Transferring the model to workers also creates a large serialisation and
memory cost, and mutable shared state invalidates the extension's parallel-safe
claim.

Even a hypothetical complete preloaded model solves only data availability.
With the current template render location, output remains inside a
Sphinx-Needs Need node and is parsed with titles disabled; it cannot become
normal document sections or ToC entries. Solving that requires a separate
structural design such as A, K, or an upstream API.

No amount of locking fixes this because the ordering is wrong: the ordinary
reader has not yet established the data it is supposed to preload. The only
safe implementation is an independent prior action with a stable artifact,
which reduces D to A or C.

Decision: reject unless C has already been adopted; do not prototype an
in-process pre-scan or module-global worker cache.

E. Reader ordering or report serialisation

This assumes that document order can make worker data visible. It cannot:
env-before-read-docs can reorder Sphinx's initial document list, but parallel
reading still partitions it into worker chunks and Sphinx merges their
environments only after the read phase, not between selected documents. A
custom serial report chunk would additionally have to reproduce purging,
dependency tracking, collector events, and error handling, turning the
extension into a partial Sphinx builder. External Needs may be available
before reading; local Needs from other workers are not.

Serialising the complete build masks the model issue but imposes a permanent
performance regression. Serialising an isolated, separate Sphinx action is
valid only when its output is treated as an artifact, which again is A.

Decision: reject as an in-process design; measure it only if considering
the explicit seed action of A.

F. Upstream report-and-ToC API

The difficult question is API ownership across two projects. Sphinx-Needs can
own a resolved Need view, but Sphinx owns source reading, doctrees, and ToC
collection. A feature that promises post-collection content but delegates ToC
integration to consumers would merely standardise the current workaround.
Conversely, a Sphinx change must define persistence, invalidation, translation,
single-page builders, and interactions with other collectors.

An upstream proposal should contain a minimal contract, not a request for a
generic callback: immutable resolved model input; declarative structured
outline; stable generated-source identity; explicit dependency reporting; and
documented behaviour for every supported builder. It also needs a migration
path for cached environments and a version floor for dependent projects.

The proposal must additionally choose a real phase mechanism. The current
env-updated contract can mark documents for writing, not re-enter source
reading and ToC collection. Thus either Sphinx must own a declarative second
collection iteration—with purging, re-reading, collector execution,
invalidation, and convergence limits—or the generated source/manifest must be
available before the first reader pass. Sphinx-Needs alone cannot add normal
sections after Sphinx's collection phase.

Evidence before commitment: maintainer agreement that this belongs in the
respective project, an accepted API sketch covering ToC timing, and a supported
fallback for existing Sphinx versions. Until then, it has schedule risk with
no bounded delivery date.

G. PR #763 current reread and env.tocs patch

The current mechanism is vulnerable to re-entrancy: env-updated is a normal
post-read event, but the handler purges and reads documents again from inside
it. Other extensions may have already recorded read-phase data, while
Sphinx-Needs may be about to post-process or lock Needs. A guard that prevents
an infinite loop does not make those extensions' state reversible. The handler
also declares parallel-read safety even though the rerender is intentionally
main-process and builder-specific.

There is a concrete hidden order dependency: an earlier env-updated listener
may call get_needs_view() and lock post-processing, after which the current
handler returns without rerendering. Direct clear_doc and read_doc calls
also bypass parts of the normal Builder.read() protocol, while the
module-global template environment is cross-app/process state rather than
environment-owned data. A build that happens to pass once therefore does not
establish extension compatibility.

The ToC patch has a separate correctness defect: rubrics carry no section
level, and direct mutation of env.tocs/toc_num_entries copies collector
internals. Anchor collision handling, translated titles, incremental doctree
caching, nested reports, and non-HTML builders are all unproven. A successful
HTML page demonstrates only the happy path.

Containment requirements: pin Sphinx and Sphinx-Needs versions, add an
adapter boundary and a removal date, and test event traces as well as output
for clean and incremental -j 1/parallel builds. Include an intentionally
early env-updated view resolver, two reports, template changes, removed
reports, and two Sphinx apps in one interpreter. Do not add new report types
or features to this mechanism.

H. Write-phase node replacement

This option has a hard phase barrier. By doctree-resolved, Sphinx has applied
post-transforms and resolved references for the document being written. New
nodes may render, but newly created pending references, citations, sections,
and ToC relationships do not receive the same normal processing unless the
extension reimplements or reruns private transforms. Persisting the mutation
would also blur the distinction between the cached read doctree and a
builder-specific write doctree.

It is safe only when replacement emits self-contained nodes with no new
Sphinx-level structure or references. This is narrower than the report.

It also runs only for documents selected for writing. A correct incremental
implementation needs an environment-persisted, parallel-merge-safe reverse
map from each traversed Need to report documents and must mark those documents
for rewrite when the Need changes. Without it, a clean build may be correct
while an incremental build retains stale late content.

Decision: retain only as a documented pattern for simple late fragments;
do not use it for the report page.

L. Needextend-like post-collection materializer

The attractive property of this option is that it uses an existing semantic
boundary: needextend is collected in parallel, merged through
env-merge-info, and applied when Sphinx-Needs post-processes the complete
Need collection at write-started. A custom report placeholder could then
read the final NeedsView and replace itself while the report document is
being written. This gives it access to local and external Need data after
merge, including accepted needextend changes, without a second read pass or
a Bazel dependency cycle.

The main risk is that this boundary is a model boundary, not a document-tree
collection boundary. At doctree-resolved, Sphinx has already collected local
section ToCs and resolves existing toctrees before emitting the event. Late
sections are therefore body content only; they do not become ordinary
env.tocs entries. Pending references and Sphinx-Needs directives inserted at
that point also do not automatically receive their normal read/write
processing. Emitting self-contained Docutils nodes avoids some of this risk,
but no longer provides the full semantics of generated RST.

The implementation must also keep an environment-owned reverse dependency
map from selected Need IDs and external inputs to report documents. That map
must merge deterministically across workers and invalidate the report when a
linked Need, link condition, backlink, schema, or external export changes.
The final NeedsView must be treated as read-only; generated nodes must not
be stored back into the shared environment.

Evidence before commitment: a prototype must demonstrate that a
post-merge needextend changes the report, that local and external targets
are visible, and that the report is rebuilt after a linked Need changes under
both serial and parallel writes. It must also document the deliberate loss of
normal late-created ToC entries before this option can replace A.

L+. Needextend-like materializer with a pre-write ToC refresh

PR #769 validates this lifecycle as a narrow proof of concept. The materializer runs from env-updated, after worker environments have been merged and Sphinx-Needs exposes the complete resolved NeedsView. It replaces a placeholder in the existing report Doctree with the selected Jinja template rendered as RST.

This preserves the one-Sphinx-build docs() topology and avoids a second source read. The report outline is derived from Need links, including linked and external Needs. The report Need itself remains available for metadata, targets, links, and normal Sphinx-Needs processing. The materialized fragment contains real section nodes, and the local ToC is refreshed for that document.

The implementation deliberately does not call builder.read_doc(), env.clear_doc(), or append entries directly to env.tocs. It does, however, need a small compatibility boundary around cached Doctree persistence and the Sphinx ToC collector. That boundary is pinned to the supported Sphinx major version and must remain covered by compatibility tests.

The proof of concept passed the focused report-pipeline test and the complete HTML docs build. The Phase 4 acceptance gate is not yet complete: incremental invalidation, serial/parallel equivalence, a non-HTML builder, and broader reference/link behavior still need explicit validation.

Decision: L+ is technically valid and is the best fit when graph-driven discovery is mandatory within the existing one-build topology. It is not a lightweight general-purpose replacement for ordinary Sphinx source processing. If the report can accept an explicit reviewed component list, K / PR #730 remains simpler and preferable.

I. Late HTML page via html-collect-pages

The page has no source identity, so standard document-level facilities are
absent by design: no ToC collector entry, no source dependency graph, and no
normal resolution of RST cross-references. It can use the theme's global
context and look integrated, but it has no ordinary local ToC, active toctree
position, automatic search-index entry, inventory entry, or linkcheck
coverage. Relative URLs, anchors, escaping, and accessibility behaviour become
extension responsibilities. A static rendered page may look correct while
linkcheck, non-HTML builders, and theme upgrades silently diverge.

This is viable for a separately branded export whose links are deliberately
plain URLs and whose accessibility/navigation are tested as an HTML product.
It must not be presented as a normal Sphinx report page.

J. Theme-specific report navigation

This replaces a stable Sphinx semantic contract with a private theme contract.
The extension must define heading nesting, unique anchors, keyboard behaviour,
ARIA semantics, mobile layout, active-section tracking, translations, and
theme upgrades. It also has to decide what non-HTML builders and alternate
themes receive. Simply copying the visible theme menu is neither accessible
nor maintainable integration.

PyData is the active repository theme, so any adapter must target its actual
sidebar/scroll behaviour, not a generic or Furo-shaped menu. It must also
declare reverse dependencies from Need-graph changes to the report page;
write-phase/theme-context navigation otherwise becomes stale on incremental
builds even if a clean build is correct.

The approach is justified only if a report is explicitly an HTML application
rather than documentation. It requires product ownership for the navigation
component and visual/accessibility regression tests; without that, reject it.

K. PR #730 hybrid directive

K has a favourable phase split but a strong source-of-truth problem. The
directive's :components: list creates the section skeleton early enough for
the ToC, while needtable filters resolve their data later. If the list and
the graph disagree, the build-finished check reports only a listed component
that is absent from module.includes; it does not detect an included component
that is absent from the list. It is too late to create missing headings in
either case; under -W, users receive a failed build only for the directed
checks that exist, rather than an automatically corrected report. This may be
acceptable when the list is a deliberate reviewable report scope, but it is
not automatic graph coverage.

Its title and slug derivation are also naming-convention semantics. Renaming a
Need, using a non-standard external ID, or changing a title can leave headings
and data inconsistent. The directive should at least preserve explicit titles
in its declaration or consume them from a generated manifest. It must safely
quote IDs used in generated filter expressions and produce collision-resistant
anchors. In the reviewed PR, work-product selection additionally uses
normalised-ID substring matching rather than Need links, which can produce
false positives and false negatives; component overview sorting can also
diverge from the declared outline order.

Finally, nested_parse_with_titles makes heading level depend on the location
of the directive in the surrounding document. Tests must cover root and nested
placement, multiple reports in one document, references to generated anchors,
and the interaction of needtable with external Needs under parallel reading.
It must also test duplicate component/report IDs and normalised title-label
collisions, version-qualified component entries whose qualifier is discarded,
and safely quoted filter IDs. Coverage-file integration is another declared
build input; reading an LCOV path during directive execution without a tracked
dependency lets a changed file at the same path leave an incremental Sphinx
report stale and harms Bazel reproducibility.

Decision: K is a strong immediate option only when an explicit outline is
an intentional authoring contract. To evolve it toward graph discovery, feed
it the manifest from A; do not move graph traversal into its read-phase
directive.

[AlexanderLanin]

Phase 4 – Validation and explicit exclusions

Acceptance tests for any selected design

1. A module linked to two components generates two ordered component sections
   without their IDs appearing in the template or renderer configuration.
   Adding a second report dimension represented by a Need link (for example,
   ordered work products) requires neither a new directive option nor a
   dimension-specific consistency check.
2. An external feature such as feat__baselibs resolves through its graph
   link and produces a working Need reference.
3. Backlink-dependent report content is evaluated only after backlinks are
   resolved.
4. The local navigation contains nested, working anchors for the report and
   component sections, with no hand-authored HTML menu.
5. -j 1 and parallel reading produce equivalent report output and local
   navigation. A/B additionally produce byte-identical generated source; L+
   additionally produces an identical serialized local ToC.
6. Changing an unrelated document leaves the final report HTML unchanged;
   changing a transitive report Need or external-Needs input updates it. For
   A/B, the generated source is byte-identical for the unrelated change. The
   renderer may still execute when it consumes a monolithic model export.
7. HTML and one non-HTML builder complete without dependency-private API
   failures.
8. A clean build produces the report on its first invocation; no second
   invocation, warmed Sphinx environment, or source-tree write is required.
9. The selected model boundary has no feedback cycle. For A/B, the Bazel model
   export does not consume the final generated report, and the final docs
   action consumes only declared generated outputs.
10. A/B reject a manifest with an unsupported schema or producer version with
    a clear diagnostic rather than producing a plausible but incorrect report.
11. The decision record includes timings for a cold build, a changed transitive
    report Need, an unrelated Need, and a fully cached repeat. For A/B, report
    the model-collection and final-Sphinx portions separately against the
    current one-build baseline.

Non-solutions to avoid

Any proposed solution should explicitly reject or justify the following:

• disabling Sphinx parallelism for the whole docs build;
• hardcoding component IDs in the .need template;
• relying on a worker-local Need model being complete;
• manually purging and re-reading arbitrary documents unless this is a
  documented, supported Sphinx extension pattern;
• monkeypatching Sphinx-Needs internals without a strong compatibility plan;
• rendering a duplicate HTML-only menu instead of integrating with the normal
  ToC;
• inferring relationships from Need ID string conventions when graph links are
  available;
• silently hiding missing Needs or falling back to an empty report when the
  model is incomplete.

[AlexanderLanin]

Phase 5 – Final conclusion

The actual trade-off

No current option provides all three of the following properties through a supported, low-maintenance boundary:

1. an outline discovered from the complete, resolved Need graph;
2. ordinary source-level section and ToC semantics; and
3. the existing one-build docs() topology.

This is the decision. It is not solved by choosing a different Jinja helper or by treating a private API boundary as automatically unacceptable. The practical choice is which trade-off the report may make.

Option	What it preserves	What it costs or gives up
K / PR #730 and PR #767	Ordinary read-phase sections, anchors, nested local ToC, parallel reading, and no late Need-model access	The outline is an explicit, reviewable declaration rather than automatic graph discovery. Its small directive/Docutils integration boundary still needs pinning and tests.
G / PR #763	Graph-driven outline from the post-merge model, one build, and parallel reading of ordinary documentation	A second read is initiated from env-updated; navigation is a flat rubric-based addition to collector state; the current implementation is HTML-only.
L+ / PR #769	Graph-driven outline, one build, no second source read, and a refreshed local section tree	A materially more complex late-materialization design with a version-pinned compatibility boundary around cached Doctrees and ToC collection. It does not restore the ordinary source-read boundary or make the coupling disappear.
A/B	Complete model and ordinary generated-source semantics across builders, references, and indexing	A second model-collection/build stage, generated-source staging, and a versioned manifest contract.

Private Sphinx or Sphinx-Needs APIs are a maintenance consideration, not a veto in this repository. Their use must be isolated, version-pinned, and tested, but it should not by itself decide between G and L+. The more important distinction is architectural: G repeats a read-phase lifecycle from env-updated; L+ avoids that reread but pays for a more elaborate cached-Doctree and ToC integration.

Recommendation

Use K if an explicit, reviewed report outline is acceptable. It is the lowest lifecycle-complexity implementation that provides normal Sphinx document structure today. Treat the declared outline as an intentional authoring contract and validate it bidirectionally against the Need graph.

If automatic graph discovery is non-negotiable while the one-build topology must remain, neither G nor L+ is a generally clean solution. Keep both as tightly scoped experiments behind a pinned compatibility adapter. Select L+ only if the acceptance tests demonstrate that its additional mechanism delivers the required hierarchical local navigation and builder behaviour; its complexity is not justified merely by avoiding the reread in G. Select G only as a time-boxed bridge with an owner, version pins, an adapter boundary, a removal/review date, and the containment tests from Phase 3. Do not add further report features to either approach before that decision is made.

Choose A/B if ordinary generated-source semantics across builders, references, search, and indexing are a hard requirement that outweighs the additional build topology. Pursue F as the longer-term route if this class of report becomes a recurring product capability.

Required evidence before accepting G or L+

Both late approaches require the Phase 4 gate, including:

1. clean and incremental equivalence with serial and parallel reading;
2. an external-Need and post-merge needextend case;
3. stable anchors, nested local navigation, and cross-reference coverage;
4. a non-HTML builder; and
5. cold, incremental, unrelated-change, and cached-build measurements.

For G, additionally test an intentionally early env-updated NeedsView resolver, two reports, removed reports, template changes, and two Sphinx apps in one interpreter. For L+, additionally prove cached-Doctree persistence and local-ToC equivalence across the supported Sphinx version range.

Decision

L+ is a complex option, not a default and not a simple improvement over G: it trades lifecycle re-entry for a different compatibility boundary around cached Doctrees and ToC collection. K has the smallest lifecycle surface, but deliberately uses a static outline and retains narrow directive/Docutils compatibility obligations.

If the K authoring contract is unacceptable, the remaining decision is between the bounded coupling of G or L+ and the additional build topology of A/B. Each choice has an explicit cost; none is free.

[MaximilianSoerenPollak]

Additional assessment – Claude opinion

Claude Opinion:

Short version: the issue is optimizing the wrong variable. Nearly all the complexity in options A, G, L and L+ exists to satisfy one requirement — "the outline must be discovered from the Need graph, not declared" — and that requirement is the cheapest one to give up. If low build and maintenance cost is the goal, take PR #730 (option K), harden its consistency check, and drop PR #763 entirely.

Why K is the low-complexity answer

K already makes the correct split, and the issue undersells this. The report has two kinds of content: the outline (sections, anchors, ToC) which must exist at read time, and the data (tables, pie charts, backlinks) which must be resolved after merge. K puts the outline in a read-phase directive and delegates all data to needtable/needpie, which sphinx-needs already resolves post-collection. So the model-completeness problem is already solved for everything except the section headings.

The only thing that actually needs to move earlier in the lifecycle is a list of roughly ten component IDs. A, C and the manifest pipeline are a very large amount of machinery — a second Sphinx collection, a versioned schema, a fingerprint contract, external-needs merge policy, Bazel cycle avoidance — to move ten strings from a directive option into a generated file. That trade is not worth making for one report.

K also uses only public APIs, keeps parallel reading, gives you real sections with real anchors and a real nested local ToC for free, and produces no lifecycle re-entry. The remaining app.connect() calls are ordinary parallel-safety plumbing (init/purge/merge for one registry) — that's the standard pattern, not a smell.

Why G should not ship, even as a bridge

The "10x less code" screenshot is measuring the template, not the system. The .need template is small; the extension behind it purges documents, calls builder.read_doc() from inside env-updated, and writes directly into env.tocs and toc_num_entries. None of that is public API. Concretely that means every Sphinx or sphinx-needs bump is a potential silent breakage, and the failure mode is a report that renders but is wrong — not a crash. There's also a real ordering bug already identified: any other extension calling get_needs_view() earlier on env-updated locks post-processing and the reread silently doesn't happen.

For downstream consumers who pin docs-as-code and carry patches, that's a recurring tax with no bounded end date. And after paying it you still get a flat rubric-derived menu instead of a proper hierarchy — so it doesn't even meet the navigation requirement that motivated the work. A "temporary bridge" with private-API coupling in a shared tooling repo tends to become permanent.

The one change that closes K's real gap

K's genuine weakness is drift: a component added to mod__baselibs.includes but not to :components: goes unreported. The current check is one-directional. Make it bidirectional at build-finished — warn (error under -W) when a component in the module's includes is missing from :components:. That converts graph discovery into graph validation, which is a few dozen lines against a data structure you already have.

The resulting authoring contract is: add a component, the build fails, you add one line to the report. For a safety verification report, an explicitly declared and reviewed scope is arguably preferable anyway — it shows up in the diff and can be signed off, rather than silently changing because someone edited a link elsewhere.

Cleanups worth doing in #730 before merge

• Work-product selection uses normalized-ID substring matching. Replace with link traversal; substring matching will produce false positives eventually.
• Component titles derived from ID prefixes (replace("_"," ").title()) will look wrong for json, bit_manipulation, etc. Allow an explicit title per entry; keep derivation as fallback.
• The LCOV path is read without being a tracked Sphinx dependency — that breaks incremental correctness and Bazel reproducibility. Either declare it properly or split coverage out of this PR.
• nested_parse_with_titles makes heading level depend on placement. Test root and nested placement.
• Version qualifiers ([version==N]) are silently stripped. Warn instead of ignoring.

What to say about A, L/L+, C, F

Defer all of them. A is a defensible architecture but it's two Sphinx collections on a cache miss, plus a manifest schema you now own — pay that only when there are several reports, or when the declared component list actually becomes painful in practice. L/L+ trades one set of private-API risk for another and still can't create ordinary source semantics. C is a semantic fork of sphinx-needs. F has no delivery date.

If graph-driven outlines later become non-negotiable, the cheapest upgrade path is not the full manifest pipeline — it's feeding K's directive an ordered component list from a small generated file, keeping all rendering in the existing directive. That's a strictly smaller step than A and it's available whenever you want it.

---

No — it confirms it. That requirement is the strongest argument for #730 and it's what disqualifies #763.

The requirement translates into one hard technical constraint

Sections only become clickable, anchored, ToC-collected, cross-referenceable, search-indexed page structure if they exist as real section nodes when Sphinx's ToC collector runs on doctree-read. There is no supported way to add them later. Everything downstream — anchors, "On this page", :ref: targets, non-HTML builders, the search index, the objects inventory — hangs off that one moment.

That has a direct corollary: the report body cannot live inside the sphinx-needs Need node. Sphinx-needs parses Need content with match_titles=False, so headings in a .need template are structurally incapable of becoming sections. This isn't a bug to work around, it's the parse mode. Any design that renders the report inside the Need node has already lost the requirement before the model-completeness question even comes up.

What that eliminates

• complex templates #763 / G — the env.tocs patch produces a flat list scraped from rubric nodes. Rubrics carry no level, so you cannot get a hierarchy out of them. The entries also aren't real section targets, so :ref: to a component section doesn't work, the anchors are synthesized after parsing, and non-HTML builders get nothing. It looks like a menu in the PyData sidebar and isn't one. This requirement rules it out outright, not "as a bridge."
• L (doctree-resolved materializer) — same failure, more explicitly: the issue itself concedes late sections are body content only.
• L+ — refreshes the local section ToC via a version-pinned adapter. Even if it works, you get the sidebar back but not source-level cross-reference resolution or clean non-HTML behavior. It's the closest near-miss and it costs a compatibility adapter you maintain forever.
• I and J — never candidates once "like a normal RST page" is the bar.

What survives

Only the two options where the outline exists at read time: K (#730), where a directive emits sections during the read phase, and A, where the sections are in a generated .rst file that Sphinx reads normally.

Both give identical navigation quality. They differ only in where the component list comes from — a directive option versus a generated manifest — and A costs a second Sphinx collection, a versioned schema, an external-needs merge policy, and Bazel cycle discipline to produce it.

So the non-negotiable requirement narrows the field to exactly the two options I'd already ranked first and second, and the low-complexity preference picks between them the same way as before.

One thing to verify in #730 rather than assume

K gets this right by using nested_parse_with_titles and returning the children into the surrounding document, with the generated mod_ver_report Need as a sibling of the report body rather than its container. That sibling relationship is precisely what makes the sections normal — worth stating explicitly in the PR as a deliberate design decision, because a reviewer might otherwise ask for the body to be moved inside the Need and silently break the requirement.

Concrete acceptance tests to add, since "it looks right in the sidebar" is not evidence:

1. :ref: from another page to a generated component section resolves and links.
2. The local ToC is nested (report → components), not flat.
3. Anchors are stable across rebuilds and collision-free with two reports on one page.
4. Directive at root level and nested under an existing heading both produce correct heading depth.
5. One non-HTML builder (latex or singlehtml) completes with the sections intact.
6. The section structure is identical under -j 1 and parallel reading.

If #730 passes those, the requirement is met with public APIs and no compatibility adapter. That's the outcome you want, and it's still the cheapest one available.

[AlexanderLanin]

Summary

• G / complex templates #763 or L+ / feat: add L+ Jinja report materialization #769 are the best solution if we need to support multiple such templates. But will require additional effort to make them really polished.
• K Solution K (for templates) #767 / refactor(module-verification-report): split, tests, docs #730 make most sense if we have only 1-2 such templates

=> we start with #730 and apply learnings from this DR to it

note: regular sphinx needs templates are supported regardless of this decision