Repository navigation
Conversation
444ec7f to
bea5a22
Compare
sannewielinga-kai
left a comment
There was a problem hiding this comment.
Thanks for turning the security session into settings already!! I tested the files in a copy with Claude Code 2.1.319. I checked the Codex rules offline with codex execpolicy check (0.169).
I do think the cross-agent review should probably be taken out of this PR and designed separately. Because in my test it lets Codex run a command outside both sandboxes (Codex's sandbox protects .git, .agents, .codex and .aws, but not .claude). So Codex can add an apiKeyHelper to .claude/settings.json. It can then run the allowed claude --safe-mode --print …, which the allow rule runs outside the sandbox without a prompt. In my test, that command ran the planted helper. In the other direction, Codex's read-only sandbox can read every file, including .env files and keys in the home folder.
Maybe also add an allow list?
"allow": ["Bash(pnpm validate)", "Bash(pnpm lint)", "Bash(pnpm typecheck)", "Bash(pnpm format:check)"]Also noticed the agent can't run the template's own checks in a Python project. Claude: uv can't write its cache inside the sandbox (error: Could not acquire lock … Operation not permitted … ~/.cache/uv). allowUnsandboxedCommands: false then stops the retry. uv run, uvx and the pre-commit hooks all need that cache. A project cache worked in my test: cache-dir = ".uv-cache" under [tool.uv], plus .uv-cache/ in .gitignore. pre-commit keeps its own cache in ~/.cache/pre-commit, so it needs the same
- Codex:
uvxis forbidden, and the Python checks all run throughuvx. get-startedstep 4 deletes folders and runs installs, which are now denied or forbidden. A line like this in the skill would stop the agent from looking for another command:- Some steps are blocked by `.claude/settings.json` or `.codex/rules/default.rules`, such as deleting `templates/` or installing dependencies. Give the developer the exact command to run in their own terminal, and wait. Do not look for another command that does the same.
- Could you run
get-startedonce per agent and stack with these settings, and put the result in the PR body?
The README should probably also say that echo $OPENAI_API_KEY still prints a key from the environment (tested this as well). I don't think this can be covered currently by the printenv and env rules. And read(./.env.*) also blocks .env.example, which both templates commit on purpose. Is that intended?
Finally I think it's smart to add the Claude Code and Codex versions to this PR body, because their behaviors change so much between versions.
| "Read(~/.npmrc)", | ||
| "Read(~/.pypirc)", | ||
| "Read(~/.netrc)", | ||
| "Read(~/.config/gh/**)", |
There was a problem hiding this comment.
This is missing Codex's login and git's plain text credential store. And Edit(./.codex/**) also stops Claude from changing the Codex rules. While testing I saw Claude appended to default.rules (from codex) without a prompt and this rule appeared to block it
| "Read(~/.config/gh/**)", | |
| "Read(~/.config/gh/**)", | |
| "Read(~/.codex/auth.json)", | |
| "Read(~/.git-credentials)", | |
| "Edit(./.codex/**)", |
There was a problem hiding this comment.
but then we also wouldn't want to allow claude to change its own settings no? So we should add a deny "Edit(./.claude/**)" as well?
There was a problem hiding this comment.
Hmm yes but it appears to only be possible in auto mode and bypass mode, not in the other modes. But indeed to be safe let's add it
| ] | ||
| }, | ||
| "sandbox": { | ||
| "enabled": true, |
There was a problem hiding this comment.
Found these settings too, lmk what you think. Not sure how intrusive these would be.
"autoAllowBashIfSandboxed" to false forces you to review and approve shell cmds even if they are safely fenced, an "failIfUnavailable" true makes cc exit at startup if the sandbox can't start.
| "enabled": true, | |
| "enabled": true, | |
| "autoAllowBashIfSandboxed": false, | |
| "failIfUnavailable": true, |
There was a problem hiding this comment.
autoAllowBashIfSandboxed: yes I like this. This would not change anything for the deny and ask rules, but move some commands from automatically being allowed to the auto-review.
|
|
||
| For this reason, this template uses `forbidden` for these commands instead of `prompt`. A matching command is blocked in both manual and automatic approval modes. Auto-review cannot approve it, and a matching `allow` rule does not override it. Run a needed blocked operation yourself in a separate terminal. The [Claude review command](#using-claude-and-codex-together) remains an explicit `allow` exception. | ||
|
|
||
| These rules do not make all other commands safe. They match literal command prefixes, so shell wrappers, alternate executable paths, and commands run by other tools can bypass them. They do not replace the sandbox, and they do not prevent reads of `.env` files, keys, or credentials. Keep secrets outside the agent's accessible environment. Project settings are defaults that users can override; use managed requirements for enforced restrictions. The project must be trusted for its rules to load; restart Codex after changes. See the [Codex rules](https://learn.chatgpt.com/docs/agent-configuration/rules) and [Auto-review documentation](https://learn.chatgpt.com/docs/sandboxing/auto-review). |
There was a problem hiding this comment.
Missing two (of which one very unfortunate) limits here. Let's assume everyone will read this 🙃
| These rules do not make all other commands safe. They match literal command prefixes, so shell wrappers, alternate executable paths, and commands run by other tools can bypass them. They do not replace the sandbox, and they do not prevent reads of `.env` files, keys, or credentials. Keep secrets outside the agent's accessible environment. Project settings are defaults that users can override; use managed requirements for enforced restrictions. The project must be trusted for its rules to load; restart Codex after changes. See the [Codex rules](https://learn.chatgpt.com/docs/agent-configuration/rules) and [Auto-review documentation](https://learn.chatgpt.com/docs/sandboxing/auto-review). | |
| These rules do not make all other commands safe. They match literal command prefixes, so shell wrappers, alternate executable paths, and commands run by other tools can bypass them. Codex also does not split a command that uses a redirect, a wildcard, or a `FOO=bar` assignment, so no rule matches a command such as `rm -rf build/*`. The rules do not replace the sandbox, and they do not prevent reads of `.env` files, keys, or credentials. The Codex sandbox protects `.codex/` but not `.claude/`, so Codex can change the Claude settings. Keep secrets outside the agent's accessible environment. Project settings are defaults that users can override; use managed requirements for enforced restrictions. The project must be trusted for its rules to load; restart Codex after changes. See the [Codex rules](https://learn.chatgpt.com/docs/agent-configuration/rules) and [Auto-review documentation](https://learn.chatgpt.com/docs/sandboxing/auto-review). |
There was a problem hiding this comment.
i guess it would make sense to disallow both claude and codex to change claude and codex settings what do you think?
And yeah this isn't 100% water tight. If you have a Fable or Astra that is super determined to run a command but is constantly blocked (maybe in goal mode or loop or something) it would still find a way. We can add this to AGENTS.md, but another level of security could be to add a custom pre-tool-call hook that monitors the tool calls and block there as well
There was a problem hiding this comment.
Yeah agreed, I would maybe also add .agents/ too, because both agents load skills from it and a skill can also define hooks. I don't think I would do a custom hook to be honest, it sees the same command text as the rules so if you have an agent that puts the action inside a script or test it will probably get past both. Hooks also seem to run outside the sandbox so we'd need to also protect that file.
There was a problem hiding this comment.
if we add .agents, they won't be able to make skills themselves. Not a big deal but a bit inconvenient
Co-authored-by: sannewielinga-kai <sanne.wielinga@kickstartai.org>
…pre-commit in the sandbox
|
about the Read(./.env.*) yeah that's on purpose so that it doesn't read .env.local or something like that. |
sannewielinga-kai
left a comment
There was a problem hiding this comment.
Alrighty did some more testing! I tested with Claude Code 2.1.325 and 2.1.327, and with Codex 0.169 through codex sandbox. I saw that cross-agent exceptions are gone, .claude/ and .codex/ are read-only for both agents & every Claude deny and ask command is also forbidden in Codex 👍
What I think is left is mostly one problem, as agents can still write a few files that later run outside the sandbox.
- An agent can add a hook to a skill in
.agents/, Claude runs that hook outside the sandbox when the skill is used - An agent can write hooks in
.husky/, git runs them outside the sandbox at the nextgit pullorfetchin Claude, or at the next commit in a terminal - With
pre-committhe git hook starts a Python that lives in.cache/uvand agents can change files there. This resulted from the project cache I suggested lol. I feel like the more we introduce the more other things appear/break down - Claude can also still read Codex's login and git's saved passwords
Curious what you guys think! We probably need to wrap this up before it becomes too complicated and just test it and see what we run into
Move rm and local git commands from deny to ask, ask only before recursive deletes, start Claude in acceptEdits mode, block file-tool writes outside the project, and disable bypass mode. Mirror the delete rules in Codex. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No description provided.