+ 
commons
An R and Python package that helps data scientists build trustworthy data analysis agents.
commons
install.packages("commons")To install the Python package from PyPI, run:
+To install the Python package from PyPI, run:
pip install commonsGet started
To get started with the R package, check out the introductory vignette. The package ships with an agent skill to help you hook your trusted code and context up to the agent.
-The Python package is based on the same ideas, so the linked vignette from the R package will give you a sense of where the package is going. (The resulting apps will look very similar regardless of whether you use R or Python—they literally share the same CSS!) Check out the package README to learn more.
+To get started with the Python package, check out the introduction. (The resulting apps will look very similar regardless of whether you use R or Python—they literally share the same CSS!)
+
+Although the agent had to decide *which* trusted calculation to run. It did not have to decide *what code to write*, reducing degrees of freedom and allowing the agent to use code that you already vetted.
+
+However, we also expect users to ask questions that stray from the “happy path.” For those, the agent searches for additional context and writes custom code (currently SQL, but soon custom Python code as well). Answers from this path either include a blue quote-mark citation marker if a citation was found in the added context, or display the yellow exclamation provenance marker for an `Untrusted` outcome.
+
+## Trust flow
+
+A `commons` agent uses trusted calculations when it can. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it calls that calculation, and the answer shows the green check-shield provenance marker for the `Verified answer` outcome.
+
+If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL (or soon Python) code to answer the user’s question. These answers either include blue quote-mark citation markers that open details about verified sources or display the yellow exclamation provenance marker for an `Untrusted` outcome.
+
+The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or Python, it can also include supporting text quoted from a trusted source. If `commons` verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the answer displays blue quote-mark citation markers that open the source details. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays the yellow exclamation provenance marker
+
+The table below lists how each provenance outcome can occur:
+
+| How the answer is produced | Provenance outcome |
+|---|---|
+| A trusted Python [measure](#semantic-layer) | `Verified answer` 
or `Untrusted` 
|
+| No data tool used (for example, the agent already had the information, or no accessible information answered the question) | No provenance outcome |
+
+The agent does not decide the provenance outcome. `commons` derives it from the tools the agent called and the citations it made.
+
+## Information layers
+
+`commons` has two primary layers of information: the **semantic layer** and the **context layer**. The semantic layer holds trusted calculations, ideally lifted from reliable code that you already use. The context layer holds background information. The agent uses this information to decide what custom SQL to write and how to interpret results. Data dictionaries span the two layers.
+
+| Layer | Sources | Role |
+|---|---|---|
+| Semantic layer | Measures in `.py` files and [`definitions`](#definitions) in `data-dict.yaml` | Provides trusted calculations. |
+| Context layer | Markdown files and descriptive fields in [`data-dict.yaml`](#data-dictionaries) | Informs custom SQL and guides interpretation. |
+
+### Semantic layer
+
+The most direct way to add a trusted calculation is a **measure**. A measure is a Python function decorated with `@commons.measure`. The decorator needs a description, from its `description` argument or from the docstring of the function. When measures exist, the agent searches for a measure that matches the question. If it finds one, it calls that measure, with arguments if the measure takes any.
+
+A measure can take two kinds of parameter in its signature. A parameter declared as `Annotated[T, Field(description=...)]` gets its value from the model, which reads the description to decide what value to pass. A parameter declared as `commons.Injected[T]` gets its value from `commons`. The model never sees this parameter: `commons` passes the connection of the data source that has the same name as the parameter. A parameter with no description and no `Injected` annotation will result in an error.
+
+As an example, in this measure, `commons` passes `biodiversity` and the model passes `site`:
+
+```python
+@commons.measure(description="Species observed at one site.")
+def species_at_site(
+ biodiversity: commons.Injected[Any], # commons passes the connection
+ site: Annotated[str, Field(description="Site name.")], # the model passes a value
+) -> Any:
+ ...
+```
+
+When a user asks "Which species were observed at Oak Bluff?", the model reads the description of `site` and passes `"Oak Bluff"`. `commons` passes the connection of the data source named `biodiversity`, and the measure runs with both.
+
+`data-dict.yaml` files also add to the semantic layer through [`definitions`](#definitions).
+
+### Context layer
+
+The context layer holds unstructured text from Markdown files and the descriptive fields of `data-dict.yaml`. commons indexes this text and retrieves the parts that match a question. Facts that every conversation needs belong in `instructions`, not in the context layer.
+
+### Examples
+
+Here are short examples of a data dictionary, a measure file, and a context document for the biodiversity example:
+
+#### Data dictionary
+
+`dictionaries/biodiversity.yaml`
+
+```yaml
+tables:
+ - name: observations
+ description: Species observations by nature preserve.
+ columns:
+ - name: count
+ description: Individuals observed during surveys.
+```
+
+#### Measure file
+
+`measures/biodiversity.py`
+
+```python
+from typing import Annotated, Any
+
+from pydantic import Field
+
+import commons
+
+
+@commons.measure(description="Species richness by site.")
+def biodiversity_by_site(
+ biodiversity: commons.Injected[Any],
+ site: Annotated[str, Field(description="Site name.")],
+) -> Any:
+ return biodiversity.execute(
+ "SELECT COUNT(DISTINCT species) AS species_richness "
+ "FROM observations WHERE obs_site = ?",
+ [site],
+ ).fetchdf()
+```
+
+The `biodiversity` parameter receives the connection of the data source named `biodiversity`. The model supplies `site`.
+
+#### Context document
+
+`context/biodiversity.md`
+
+```markdown
+# Interpreting survey results
+
+Observed individuals reflect organisms recorded during surveys. They are not a complete population census of a nature preserve.
+```
+
+## Data sources
+
+One of the primary decisions you’ll need to make when building a `commons` agent is which data sources to grant the agent access to. Each data source combines the underlying data with the tables to expose to the agent. It can also include a data dictionary describing those tables and trusted calculations on them.
+
+Data sources are created with `commons.data_source()`. The data can be named `pandas` or `polars` data frames, a [pins](https://rstudio.github.io/pins-python/) board, or a SQLAlchemy `Engine`. `commons` loads data frames and pins into an in-process DuckDB database. `commons` queries a database engine directly without copying data into a local database.
+
+For example, the following code creates a data source from two data frames. Each name becomes a table that the agent can query. `dictionary` is an optional path to a `data-dict.yaml` file.
+
+```python
+biodiversity = commons.data_source(
+ observations=observations,
+ site_area=site_area,
+ dictionary="dictionaries/biodiversity.yaml",
+)
+```
+
+For an engine or a pins board, the `tables` argument selects the tables to expose. An engine takes a list of table names:
+
+```python
+import sqlalchemy
+
+engine = sqlalchemy.create_engine("duckdb:///surveys.duckdb")
+surveys = commons.data_source(engine, tables=["observations", "sites"])
+```
+
+With a pins board, `tables` maps each table name the agent sees to the pin that supplies the data:
+
+```python
+surveys = commons.data_source(
+ board,
+ tables={"observations": "survey-observations", "sites": "survey-sites"},
+)
+```
+
+A Snowflake or Databricks engine also imports its catalog, so commons can resolve the selection against the warehouse and check access. `exclude` drops objects from the catalog listing by glob:
+
+```python
+warehouse = commons.data_source(engine, exclude=["TMP_*"])
+```
+
+If the catalog is too large for the system prompt, the agent gets a `search_catalog` tool instead of a table listing. The system prompt reports only the number of selected catalog objects and tells the agent to call `search_catalog` before `describe_table`. The agent then searches the catalog for objects that match the question and describes only those before it writes SQL.
+
+### Data dictionaries
+
+A data dictionary provides structured documentation for one data source. Use it to state what each table represents, the meaning and type of each column, relationships between tables, and glossary terms. It also holds trusted `definitions`. `commons` reads the [`data-dict.yaml` specification](https://data-dict.tidyverse.org/).
+
+A `commons` agent uses a data dictionary in three ways:
+
+- Dataset-level descriptions give broad context that is always available. Glossary terms go in the system prompt as space allows.
+- When the agent first uses a documented table in a conversation, it receives the description, columns, relationships, and glossary terms of that table.
+- Descriptive fields, such as `description` and `details`, are part of the context layer.
+
+#### Definitions
+
+**Definitions** are named, governed expressions attached to tables in `data-dict.yaml`. They let an agent reuse trusted metrics, filters, and derived values, so they add to the semantic layer.
+
+Each definition is an expression in the [data-dict expression language](https://data-dict.tidyverse.org/expressions.html), not in the SQL dialect of your database:
+
+```yaml
+tables:
+ - name: observations
+ columns:
+ - name: count
+ type: number
+ definitions:
+ - name: total_individuals
+ label: Total individuals observed
+ description: Sum of the individuals recorded in surveys.
+ expr: SUM(count)
+```
+
+There are three kinds of definition. A definition can take part in a trusted metric calculation, or the agent can use it in custom SQL:[^definition-sql]
+
+| Kind | Example | Use in a trusted metric calculation |
+|---|---|---|
+| Metric | `SUM(n)` | Computes the metric |
+| Filter | `status = 'active'` | Restricts rows or provides a grouping dimension |
+| Derived value | `price * quantity` | Provides a grouping dimension |
+
+`commons` infers the kind from the expression. An aggregate or constant expression is a metric. A row-level Boolean expression is a filter. Every other row-level expression is a derived value.
+
+[^definition-sql]: In custom SQL, the agent writes a definition as a `{{name}}` token, and `commons` expands it to the SQL compiled for the data source. This is still custom SQL, so the provenance outcome is `Cited` or `Untrusted`, not `Verified answer`.
+
+See the [DevRel Agent's `data-dict.yaml`](https://github.com/posit-dev/devrel-agent/blob/main/dictionaries/devrel.data-dict.yaml) for examples of definitions.
+
+When a data source is constructed, `commons` validates each definition and compiles it to the source’s SQL dialect.
+
+## Project directory organization
+
+A `commons` agent is easiest to maintain when each piece has its own file:
+
+```text
+.
+|-- app.py
+|-- agent.py
+|-- pyproject.toml
+|-- AGENTS.md # or the file your coding agent reads (for example, CLAUDE.md)
+|-- instructions.md
+|-- dictionaries/
+| `-- biodiversity.yaml
+|-- measures/
+| `-- biodiversity.py
+`-- context/
+ `-- context.md
+```
+
+## Constructing the agent
+
+Use `commons.Commons()` to construct an agent. Pass it a `chatlas.Chat` and one or more data sources, plus any semantic and context layers. You can also optionally append information to the `commons` agent system prompt using the `instructions` argument.
+
+```python
+import chatlas
+import commons
+
+biodiversity = commons.data_source(
+ observations=observations,
+ site_area=site_area,
+ dictionary="dictionaries/biodiversity.yaml",
+)
+
+agent = commons.Commons(
+ client=chatlas.ChatAnthropic(model="claude-sonnet-5"),
+ data_sources={"biodiversity": biodiversity},
+ semantic_layer=commons.semantic_layer("measures"),
+ context_layer=commons.context_layer(files=["context/context.md"]),
+ instructions="instructions.md",
+)
+
+agent.chat("How many species were observed at Oak Bluff?")
+```
+
+`data_sources` is one `DataSource` or a mapping of name to `DataSource`. Name the sources when a measure takes an injected connection, because commons injects by name. `semantic_layer()` accepts measures, modules, or paths to `.py` files and directories. `context_layer()` accepts a list of file paths.
+
+Underneath, a `Commons` agent inherits from a `chatlas.Chat` object, but builds its own system prompt and tools. As such, it follows the `Chat` API and provides the `chat()` and `stream_async()` methods as ways to ask a question. The other `chatlas` entry points, such as `chat_async()`, `stream()`, and `chat_structured()`, are not supported and will raise `NotImplementedError` at this moment.
+
+### Chat UI
+
+The above examples will build a terminal-based agent without any UI attached. If you want to
+use the agent in an interactive web UI, the `commons.ui` module will put the agent behind a
+[Shiny](https://shiny.posit.co/py/) chat. This requires installing the `shiny` extra group.
+
+When developing locally, you can use `commons.ui.app(agent)` to return a complete Shiny app:
+
+```python
+app = commons.ui.app(agent)
+```
+
+`commons.ui.app()` shares one agent across every session, so it suits one visitor at a time.
+For a deployed app, you should instead build the page with `commons.ui.theme()` and construct
+the agent inside the Shiny server function, so each session gets its own agent:
+
+```python
+import shinychat
+from shiny import App
+
+app_ui = shinychat.page_chat("Biodiversity", id="chat", theme=commons.ui.theme())
+
+
+def app_server(input, output, session):
+ # The same construction as above; building it per session keeps each
+ # session's citation and provenance state separate.
+ commons.ui.server(
+ "chat",
+ commons.Commons(
+ client=chatlas.ChatAnthropic(model="claude-sonnet-5"),
+ data_sources={"biodiversity": biodiversity},
+ semantic_layer=commons.semantic_layer("measures"),
+ context_layer=commons.context_layer(files=["context/context.md"]),
+ instructions="instructions.md",
+ ),
+ )
+
+
+app = App(app_ui, app_server)
+```
+
+## Example application
+
+[`demo.py`](https://github.com/posit-dev/commons/blob/main/pkg-py/demo.py), in the Python package's directory, is a fuller worked example: an agent over made-up forest canopy data with a semantic layer of measures and a context layer. Run it with `uv run shiny run demo.py` for the chat UI, or `uv run python demo.py` to ask the same questions from the terminal. [`demo.ipynb`](https://github.com/posit-dev/commons/blob/main/pkg-py/demo.ipynb) is the same agent in a notebook, with cells for reading what the agent registered and adding a measure of your own.
diff --git a/pkg-py/docs/governance.qmd b/pkg-py/docs/governance.qmd
new file mode 100644
index 00000000..ff010930
--- /dev/null
+++ b/pkg-py/docs/governance.qmd
@@ -0,0 +1,49 @@
+---
+title: "Security and governance for commons agents"
+---
+
+`commons` allows users to build and deploy agents that can run SQL queries against live databases and (soon) execute arbitrary Python code. This raises several security and governance questions: How do you make sure that an agent cannot delete production data? Can model-generated Python code interfere with a [Posit Connect](https://posit.co/products/enterprise/connect) deployment? Can an agent show users tables, rows, or business context that they shouldn't be able to see?
+
+This page explains the boundaries that `commons` provides and the responsibilities that remain with the application author and server administrator.
+
+## The `commons` harness
+
+A `commons` agent is a `chatlas.Chat` object that carries a system prompt and a set of tools. The system prompt lists the available data sources and business context. Tools let the model retrieve more context, call a trusted calculation, or run a SQL query.
+
+We assume that a model might make any request allowed by its tools. Application security should therefore not depend on the model following an instruction like “never reveal sensitive data.” Instead, only give an agent access to data that the current user of the application is allowed to see. The system prompt, tool arguments, and tool results are also sent to the model provider. The agent must therefore not have access to data that you [do not trust that provider to process](https://posit.co/blog/trust-llm-tools).
+
+## SQL code execution
+
+### Destructive actions
+
+The agent can run one read-only statement at a time. `commons` parses each statement with [sqlglot](https://github.com/tobymao/sqlglot) and checks its structure against an allowlist of statement forms that only read, such as `SELECT` and the set operations. `commons` refuses every other form, stacked statements, and statements that `sqlglot` cannot parse, so it fails safe/closed.
+
+For data frames and pins, which `commons` loads into its own DuckDB database, it also hardens the connection. It disables community and unsigned extensions, automatic extension install and load, external access, and the local filesystem. Then it sets `lock_configuration`, so a query cannot turn those settings back on.
+
+These checks provide defense in depth, but they are not a database sandbox. When you supply a database connection, `commons` queries it as-is, and a statement that the parser accepts runs with all the permissions of that connection. The primary safeguard against destructive SQL is therefore database-enforced read-only access, and you should open the connection as a read-only user or role.
+
+### Data access
+
+The `tables` argument to `data_source()` controls which tables `commons` describes to the model, but it is not an authorization boundary. SQL written by the agent can query any object that the connection can reach.
+
+On Posit Connect, [viewer OAuth integrations](https://docs.posit.co/connect/admin/access-controls/) can give an interactive application the current viewer's Snowflake or Databricks credentials. If the application creates its engine from those credentials, the warehouse continues to enforce that viewer's existing access policies, including row-level and column-level security.
+
+For a Snowflake or Databricks source, `commons` snapshots the connection’s principal and namespace when it creates a Snowflake or Databricks data source, and its active and secondary roles as well on Snowflake, and rejects subsequent operations if that identity changes.
+
+Viewer credentials are not automatic. `commons` uses the engine that the application supplies. When using viewer credentials, you should create the engine and the `commons` agent inside the Shiny server function so that each session has the correct database identity.
+
+When viewer credentials are not available, it is recommended to use a service account with access only to the data that the application needs. Every viewer then has the same database permissions, so share the application only with users who are allowed that access.
+
+## Python code execution
+
+The Python agent currently has no tool that runs code written by the model, but this feature is expected in the near future. Currently, there is no concern for Python `commons` agents executing arbitrary Python code. When the feature is added, it will generally follow the R package's [R code execution](../r/articles/governance.html#r-code-execution) design.
+
+## Permissioning facts
+
+The rows returned by SQL are not the only sensitive information available to an agent. The description and details of each data dictionary, and its glossary entries up to a size cap, go into the system prompt. The full dictionary entry for a table, with a summary of sampled rows, goes to the model when a table is first described or queried. Context documents and the rest of the dictionary prose are available to the model through search.
+
+Because of this, you should only include facts that you can share with both the application's viewers and its model provider. If one audience must not see a fact, use separate applications with separate context, or put the facts behind viewer credentials. Asking the model to hide the fact from that audience is not a reliable means of preventing unauthorized access to the information.
+
+## Telemetry
+
+The Python implementation of `commons` currently does not support telemetry or conversation logging, though this feature is planned to be added in the near future. When it is added, it will generally follow the R package's [Logging trajectories](../r/articles/governance.html#logging-trajectories) design.
diff --git a/pkg-py/docs/index.qmd b/pkg-py/docs/index.qmd
new file mode 100644
index 00000000..671b16dd
--- /dev/null
+++ b/pkg-py/docs/index.qmd
@@ -0,0 +1,56 @@
+---
+title: "commons"
+---
+
+```{=html}
+