Skip to content

docs(ai-policy): ask for the reasoning effort, make the agent optional - #817

Open
MiMoHo wants to merge 1 commit into
nextcloud:masterfrom
MiMoHo:docs/ai-policy-trailer-model-effort
Open

MiMoHo wants to merge 1 commit into
nextcloud:masterfrom
MiMoHo:docs/ai-policy-trailer-model-effort

Conversation

@MiMoHo

@MiMoHo MiMoHo commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

The policy asks for Assisted-by: AGENT_NAME:MODEL_VERSION. This PR keeps the key and the AGENT_NAME:MODEL_VERSION part and changes two things: the reasoning effort is added wherever the tool offers a setting, and the agent name becomes optional.

Assisted-by: MODEL_VERSION (REASONING_EFFORT)
Assisted-by: AGENT_NAME:MODEL_VERSION (REASONING_EFFORT)

Tools without an effort setting leave it out, for example Assisted-by: Copilot:gpt-4o. Existing commits such as Assisted-by: ClaudeCode:claude-sonnet-4-6 don't need to be rewritten, and the AI Policy check doesn't change. Assisted-by: keeps covering any kind of help. For content an LLM generated, AI-Assisted-By: says so explicitly; the check has recognised it since #742.

The model stays required, because it is the part a reviewer can act on. With the same agent, SWE-agent resolved 18.0% of the SWE-bench Lite issues with GPT-4 Turbo and 13.0% with Claude 3 Opus, and its authors found Llama 3 and DeepSeek Coder "subpar" in that setting. The typical mistakes differ too: across 16 code models, the share of hallucinated package names averaged at least 5.2% for commercial and 21.7% for open-source models (Spracklen et al., USENIX Security 2025), which matters for every PR that adds a dependency. If problems pile up with one model, the trailer lets maintainers find the affected commits with git log --grep, look at them again and ask PRs made with that model for more, such as tests. For an open-weight model running locally, the policy now counts the quantization as part of the model version, for example qwen3-coder-30b-a3b-instruct-q4_k_m. 8-bit formats lose little or nothing (Kurtic et al., ACL 2025), but Llama 3 still degrades noticeably at very low bit widths (Huang et al., Visual Intelligence 2024).

The reasoning effort tells a reviewer more than the agent does, because the same model gives measurably different results depending on how much it is allowed to think. Anthropic reports that Opus 4.5 at medium effort matches Sonnet 4.5's best score on SWE-bench Verified and only pulls ahead, by 4.3 points, at its highest effort. Peer-reviewed work points the same way. On problems a smaller model can partly solve, extra compute at inference time can beat a 14 times larger model (Snell et al., ICLR 2025), and a 7B model with a better search strategy consistently outperformed a 34B model on the MATH benchmark (Wu et al., ICLR 2025). The model name alone is only half the information.

The agent, on the other hand, mostly describes the contributor's setup, and that is what attackers care about. Coding agents run with access to the shell, the source code and often credentials. In the Nx supply chain attack, malicious packages used locally installed AI command line tools to search machines for secrets, and Invariant Labs showed how a crafted GitHub issue can take over a coding agent. A permanent, public record of which agent each GitHub user runs makes it easier to pick targets. Making it optional costs reviewers little: in nextcloud/server, the 26 commits from the last twelve months that use AI-Assisted-By: already name only the model.

Some objections I expect:

  • "The trailer should stay generic." It does. Assisted-by: and AGENT_NAME:MODEL_VERSION stay; the effort is added and the agent becomes optional.
  • "The Linux kernel dropped agent and model in August 2026." It now asks only for Assisted-by: LLM, because naming models gives proprietary vendors free advertising. I see the model as a fact about how a change was made, the same way the bug report template asks for the server and PHP version. The policy's examples already name open-weight models, Devstral and Qwen3-Coder, and Nextcloud's own Ethical AI Rating asks whether a model is freely available. That question needs the model name.
  • "The agent affects quality too." It can. SWE-agent (NeurIPS 2024) showed that the interface an agent gets changes how many issues it solves. But a name like "Copilot" doesn't say which version, settings, tools or prompts were used, so a reviewer can't judge much from it. Anyone who wants to name the agent still can.
  • "More effort isn't always better." True. Gema et al. (TMLR 2025) built tasks where longer reasoning makes models worse. That is a reason to record the effort, not to leave it out.
  • "Not every tool has a reasoning effort setting." Then the effort is left out, for example Assisted-by: Copilot:gpt-4o.
  • "@llvm asks for the name of the code assistant." LLVM says its labelling is "intended to facilitate reviews, and not to track which parts of LLVM are generated". Model and effort serve that purpose better than the agent's name.
  • "The tool's terms matter for licensing." The model already names the provider whose terms apply. The licensing section makes checking them the contributor's job ("Ignorance of AI-generated provenance is not a defense"), and so does the Linux Foundation guidance.

I also replaced qwen3-coder-32b in the examples. I couldn't find a Qwen3-Coder model with 32B parameters, so the example now uses Qwen3-Coder-30B-A3B-Instruct with a 4-bit quantization.

Like #816, this PR reduces what contributors have to make public about themselves: there their real name, here the tools they run. Each of the two can be merged on its own.

🤖 AI (if applicable)

  • The content of this PR was assisted using AI

The model helped me draft this change. I reviewed and edited everything. The commit carries the same line as a trailer.

Assisted-by: Claude Opus 5.5 (xhigh)

Comment thread AGENT_MD_SNIPPETS.md Outdated
### What this agent must always do

- Add an `Assisted-by: AGENT_NAME:MODEL_VERSION` git trailer to every commit containing AI-assisted content.
- Add an `AI-Assisted-By: MODEL (REASONING_EFFORT)` git trailer to every commit containing AI-assisted content, for example `AI-Assisted-By: Claude Opus 4.5 (medium)`. Ask the contributor for the reasoning effort if you do not know it. Do not add your own agent name unless the contributor asks for it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the policy is also interested in the harness used, but I agree that it's confusing that we're saying AGENT_NAME when we mean HARNESS_NAME

Suggested change
- Add an `AI-Assisted-By: MODEL (REASONING_EFFORT)` git trailer to every commit containing AI-assisted content, for example `AI-Assisted-By: Claude Opus 4.5 (medium)`. Ask the contributor for the reasoning effort if you do not know it. Do not add your own agent name unless the contributor asks for it.
- Add an `AI-Assisted-By: HARNESS:MODEL (REASONING_EFFORT)` git trailer to every commit containing AI-assisted content, for example `AI-Assisted-By: Claude code:Claude Opus 4.5 (medium)`. Ask the contributor for the reasoning effort if you do not know it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my own interpretation, I might be wrong the maintainers can correct me on it

@MiMoHo
MiMoHo force-pushed the docs/ai-policy-trailer-model-effort branch from 49ca625 to b610600 Compare October 17, 2026 21:56

@AndyScherzinger AndyScherzinger left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd currently veto this change and would like to stick to what we have.

AI- adds no extra value, it just makes the trailer more specific and limited to AI use. Linux kernel discusses dropping it completely which I think is not a good idea given the EU AI Act.

The trailers goal is to know about the IF and WHAT, not the "HOW"

So in its current form it is generic enough to also server other attributions outside of AI use.

In a sense this is somewhat meta data, so changing the format is not desired.

@MiMoHo MiMoHo changed the title docs(ai-policy): use AI-Assisted-By with model and reasoning effort docs(ai-policy): ask for the reasoning effort, make the agent optional Oct 18, 2026
@MiMoHo
MiMoHo force-pushed the docs/ai-policy-trailer-model-effort branch from b610600 to 56c1f58 Compare October 18, 2026 17:51
@MiMoHo

MiMoHo commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @AndyScherzinger. I've cut this down: the key and the AGENT_NAME:MODEL_VERSION part stay, existing commits don't need to change, and the check is unchanged. Two things change: the reasoning effort is added wherever the tool offers a setting, and the agent name becomes optional, so it's Assisted-by: MODEL_VERSION (REASONING_EFFORT) or Assisted-by: AGENT_NAME:MODEL_VERSION (REASONING_EFFORT). Assisted-by: keeps covering any kind of help; for content an LLM generated, AI-Assisted-By: says so explicitly, and the check already recognises it.

The effort is the "how", but it changes the "what": the same model gives measurably different results depending on it. The agent mostly tells attackers which tool runs on a contributor's machine, so I'd make it optional. The 26 commits with AI-Assisted-By: in nextcloud/server from the last twelve months already name only the model.

I'd keep the model required, even though the kernel dropped it in August. With the same agent, SWE-agent solved 18.0% of SWE-bench Lite with GPT-4 Turbo and 13.0% with Claude 3 Opus, and the share of hallucinated package names was about four times higher on average for open-source than for commercial models. If problems accumulate with a model, or one quantization of an open-weight model, the trailer lets us find those commits and ask for more, such as tests.

On the AI Act: the trailer stays, but the Act doesn't require it from contributors. Art. 50 puts machine-readable marking on the providers of AI systems. Deployers only have to disclose AI-generated text if it is published to inform the public on matters of public interest, like news, and not even then if a human reviewed it and takes editorial responsibility. A commit or PR description isn't that kind of text, and its author reviews it anyway.

Ask for the reasoning effort wherever the tool offers a setting and
make the agent name optional, so the trailer reads
`Assisted-by: MODEL_VERSION (REASONING_EFFORT)` or
`Assisted-by: AGENT_NAME:MODEL_VERSION (REASONING_EFFORT)`. Existing
commits with AGENT_NAME:MODEL_VERSION stay as they are. For an
open-weight model run locally, the model version includes the
quantization.

The agent mostly describes the contributor's setup, and a public record
of which agent each contributor runs helps attackers more than
reviewers. The model, its quantization and the reasoning effort change
what the contribution looks like, so they tell a reviewer more.

Mention AI-Assisted-By: for content an LLM generated; the AI Policy
check has recognised it since nextcloud#742. Replace qwen3-coder-32b in the
examples, which does not exist, with qwen3-coder-30b-a3b-instruct-q4_k_m.

Assisted-by: Claude Opus 5.5 (xhigh)
Signed-off-by: MiMoHo <37556964+MiMoHo@users.noreply.github.com>
@MiMoHo
MiMoHo force-pushed the docs/ai-policy-trailer-model-effort branch from 56c1f58 to 264548d Compare October 18, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants