diff --git a/agents/lightdash-mcp.mdx b/agents/lightdash-mcp.mdx index a479ca98..8e0879bd 100644 --- a/agents/lightdash-mcp.mdx +++ b/agents/lightdash-mcp.mdx @@ -267,7 +267,7 @@ When the header is set: - The `list_projects` tool is hidden from `tools/list`, so the AI assistant isn't steered toward another project. - Access to the project is still enforced by your user, API key, or service account's permissions — the header can't be used to access projects the caller can't already see. -This is useful when you want each integration, workspace, or environment to be locked to one project (for example, a separate connector per project) without relying on the AI assistant to pick the right project. +This is useful when you want each integration, workspace, or environment to be locked to one project (for example, a separate connector per project) without relying on the AI assistant to pick the right project. A user chooses this for their own integration; it isn't an admin-enforced restriction. To restrict which projects MCP clients can reach for everyone in your organization, see [Restrict MCP to specific projects](#restrict-mcp-to-specific-projects). The header value must be a valid project UUID. You can find the project UUID in the Lightdash URL when viewing a project (`/projects//...`). @@ -398,6 +398,15 @@ Organization admins can turn MCP on or off for the whole organization with **All **Settings → MCP → Connect** always shows. **General** and **Analytics** show only when [AI features are enabled](/agents/set-up-agents#enable-ai-features) for your organization (on a self-hosted instance, [enable AI Analyst](/self-host/enterprise-features/ai-agents#enable-ai-analyst)). **General** needs an organization admin or developer role. **Analytics** also shows to people who can manage agents in at least one project. +### Restrict MCP to specific projects + +The **Project access** card in **Settings → MCP → General** lets organization admins choose which projects MCP clients can reach, separately from each user's project access in the Lightdash app. It's disabled while **Allow MCP access** is off. + +- **All projects** (the default): MCP clients can reach any project the connecting user, API key, or service account already has access to in Lightdash. +- **Only selected projects**: MCP clients can only reach the projects you add to **Allowed projects**. Picking this mode requires selecting at least one project before it applies. A new project stays unavailable to MCP clients until an admin adds it here, even if users already have access to it in the app. If you remove every project from the list, MCP clients can't access any project — the card shows a "No projects allowed" warning until you add one back. + +This setting only limits what MCP clients can reach; it doesn't change a user's access in the Lightdash app, and it doesn't grant access either — a caller still needs its own permissions on a project to use it over MCP. It's the admin-enforced counterpart to a user [pinning a project with the `X-Lightdash-Project` header](#pin-a-project-with-the-x-lightdash-project-header) on their own integration. + ## What it can do ### Core capabilities @@ -424,7 +433,7 @@ An active project must be set before MCP can retrieve any data. Your AI assistan #### Query execution -- **Run metric query** - Execute queries using your semantic layer's metrics and dimensions. Completed results include a best-effort `exploreUrl` so you can open the query in Lightdash to keep exploring. +- **Run metric query** - Execute queries using your semantic layer's metrics and dimensions. Returns up to 5,000 rows by default; a requested limit above that is clamped to the cap without the response saying so. Completed results include a best-effort `exploreUrl` so you can open the query in Lightdash to keep exploring. MCP has no export or download tool — for a larger or complete extract, open the `exploreUrl` and [download the results](/explore/share-charts#download-results-or-a-chart-image) with **All results** (bounded by the instance's [export limits](/workspace-admin/export-limits)), or set up a [scheduled delivery](/explore/create-scheduled-deliveries) as CSV or Excel. - **Render chart** - Render a visualization (tables, bar charts, line charts, pie charts, and more) for a completed metric query result in MCP App-capable clients (e.g., Claude). Call this after a metric query finishes when you want a visual chart; the query is not re-executed. It can also render with one of the project's [custom chart types](#chart-types). SQL Runner results are not supported by `render_chart`. - **Run SQL** - Execute arbitrary SQL queries directly against the project's data warehouse. Useful for ad-hoc analysis or queries that don't fit the explore-based model. Returns up to 500 rows by default (configurable up to 5,000). Completed results include a best-effort `sqlRunnerUrl` so you can open the compiled SQL in [SQL Runner](/explore/sql-runner). - **Get query result** - Poll for the result of a long-running query. `run_metric_query` and `run_sql` each wait about 50 seconds for the query to finish; if it hasn't, the response has no `exploreUrl` (or `sqlRunnerUrl`) yet — just a `queryUuid` to poll. Call `get_query_result` with that `queryUuid` until the warehouse completes the query — so long queries don't time out the session. The completed (`done`) response includes the same best-effort `exploreUrl` (or `sqlRunnerUrl` for `run_sql` queries) as a synchronously-completed call. diff --git a/agents/set-up-agents.mdx b/agents/set-up-agents.mdx index d3fe375c..73a9d32a 100644 --- a/agents/set-up-agents.mdx +++ b/agents/set-up-agents.mdx @@ -151,16 +151,18 @@ To run Ask AI on Claude models in your own AWS account and keep inference in one - **Name** — how projects refer to the credential, for example `Europe (Frankfurt)`. - **AWS region** — Tokyo, Osaka, Frankfurt, Ireland, N. Virginia or Oregon. Enable the models you want in that region of your AWS account first. -- **Inference geography** — where Bedrock may process requests. It defaults to the narrowest choice for the region (Japan, United States or Europe); **Global** widens it. +- **Inference geography** — where Bedrock may process requests; the AWS region above only decides which endpoint receives them. It defaults to the narrowest choice for the region: a Tokyo or Osaka credential can keep requests inside Japan (the Tokyo and Osaka inference profiles only), and a US or EU credential keeps requests inside that geography. **Global** widens it to any supported region worldwide. Pick the narrowest geography your data-residency or compliance review requires. - **Bedrock API key** — a Bedrock API key, not an AWS access key pair. - **Allowed models** — the Claude models users can pick. -You can add several credentials. The organization default serves everything without its own selection, including Slack routing and thread titles. Organization admins and developers can point a project at another credential in **Project settings** → **AI region**, and, when there's more than one credential, an agent under **AI provider credential** in its settings. +You can add several credentials. The first one you create becomes the organization default unless you clear **Set as organization default** in the Add credential dialog, so you can register a credential without putting it into use. The default serves everything without its own project-level selection, including Slack routing and thread titles. Organization admins and developers can point a project at another credential in **Project settings** → **AI region**, and, when there's more than one credential, an agent under **AI provider credential** in its settings. If custom AI providers aren't enabled for your organization, **Project settings** → **AI region** shows *Custom AI providers are not enabled for this organization.* A project already pointed at another credential then has no AI features until you clear the selection or custom providers are turned on. While a default Bedrock credential is set, Bedrock is the only provider Ask AI uses and nothing falls back to other providers. Agents set to another provider's model stop working until you change their model, and verified-answer search and [AI writeback](/agents/ai-writeback) are unavailable. +You can also list, create, update and delete Bedrock credentials through the API, under `/api/v1/ai/provider-credentials`, authenticated with a [personal access token](/personal-settings/create-personal-tokens). See the [API reference](/api-reference/v1/introduction) — this is the route to manage credentials from Terraform or another IaC tool. + #### Key-gated models Some models are hidden by default and only become selectable once your BYO key can access them. Currently this applies to **Claude Opus 4.8** — after you save an Anthropic key, Lightdash verifies access against Anthropic's models API and unlocks the preset automatically. diff --git a/data-apps.mdx b/data-apps.mdx index 93869656..5d2f0ce0 100644 --- a/data-apps.mdx +++ b/data-apps.mdx @@ -154,7 +154,7 @@ To duplicate, you need **view** access on the source app and **create** permissi The inspector is a tabbed network panel that captures everything the running app talks to. Open it from **View network** in the preview header. -- **Queries** - every metric query the app runs against your semantic layer, including the underlying query definition, the fields it's pulling, and the rows it gets back. Live-linked charts show up here too. Use it to verify the app is asking for the right data, debug unexpected numbers, or sanity-check performance. +- **Queries** - every metric query the app runs against your semantic layer, including the underlying query definition, the fields it's pulling, and the rows it gets back. Live-linked charts show up here too. Use it to verify the app is asking for the right data, debug unexpected numbers, sanity-check performance, or spot a query returning far more rows than the app needs (see [App is slow to load](#app-is-slow-to-load)). - **Requests** - every external fetch the app makes at runtime through a linked external connection, with the method, path, status, response, and round-trip duration. Use it to confirm the app is calling the right endpoint and that the response shape matches what the UI expects. The inspector only ever sees the connection alias and path - credentials are injected server-side, so they're never displayed here. Both tabs are always visible so you can see at a glance whether the app has run any queries or external requests yet. Entries are held in-memory and cleared when you refresh the preview or load a new version unless you turn on the **Persist** toggle in the panel. @@ -357,6 +357,7 @@ On self-hosted instances, `LIGHTDASH_APP_CAPTURES_ENABLED=false` turns this off - **Describe drill-downs in full.** Say what a click opens, what it shows, and what data it needs - for example, "clicking a month opens a dialog with that month's daily order count". - **Use the network inspector early.** If something looks off, check the queries and external requests the app is running before tweaking the prompt. - **Build what-if inputs on existing metrics.** The app gets each query's rows in its own code, so a slider or input that scales a result (for example, a growth rate applied to revenue) can work on any metric you already have, with no dbt changes. You only need [parameters](/semantic-layer/parameters) when the input should change the query's SQL itself; the app then passes the value with the query. +- **Keep query results small.** An app tile loads a query's results from the backend 500 rows at a time, so a query returning tens of thousands of rows makes hundreds of sequential requests before the app can render - even when the warehouse query itself runs in seconds. Aggregate, filter, or lower the query's row limit so it returns hundreds of rows rather than tens of thousands. If you need full row-level detail, show a summary first and load the detailed rows on demand, for example filtered to whatever the user clicks. ## Troubleshooting @@ -373,6 +374,19 @@ To fix it, allowlist Lightdash in Malwarebytes Browser Guard: 3. Add the Lightdash domain you're using (for example, your Lightdash Cloud subdomain or your self-hosted Lightdash URL). 4. Reload the page. +### App is slow to load + +A data app loads every row a query returns before it renders anything. Results come back in pages of 500 rows, fetched one after another, so load time grows with how many rows the query returns - not with how long the warehouse takes to run it. A query that returns around 91,000 rows takes roughly 180 sequential page requests to finish loading, which can take minutes even when the warehouse answers the query itself in about 15 seconds. + +Open the [network inspector](#network-inspector)'s Queries tab and check each query's row count to find the one that's slowing the app down. + +To fix it, return fewer rows from the query itself: + +- Ask the agent to aggregate or filter the data in the query, rather than fetching everything and filtering in the app. +- Lower the query's `.limit()` so it returns hundreds of rows, not tens of thousands. + +If the app needs row-level detail, ask the agent to show a summary first and load the detailed rows separately - for example, only the rows for whatever the user clicks on. + ## Self-hosting Data apps are enabled by default on Lightdash Cloud. If you're running a self-hosted enterprise instance, see [data apps on self-hosted Lightdash](/self-host/enterprise-features/data-apps) for the configuration you'll need to add — a [sandbox provider](/self-host/customize-deployment/sandboxes), a Claude provider, and an S3 bucket for the built artifacts. diff --git a/embed/reference.mdx b/embed/reference.mdx index 7a846501..0f7db74d 100644 --- a/embed/reference.mdx +++ b/embed/reference.mdx @@ -864,6 +864,14 @@ Configure embed scopes in the **Embedding** section of a [custom role](/workspac Permissions from the actor's organization and project roles are additive. Removing a scope from one role does not revoke a grant from another role. When testing a restrictive custom role, make sure the actor has no other role that still grants the capability. +### Supporting many embedded end users + +The actor doesn't have to be one Lightdash user per embedded end user. When your app has a handful of permission profiles shared by many end users, create one service account per profile — for example `viewer`, `analyst`, `admin` — give each a custom role with the embed scopes for that profile, and have your backend pick the matching `serviceAccountUserUuid` when it signs each end user's token. A new end user reuses an existing profile's service account: there's no Lightdash user to create and no code change. + +If your embedded end users are themselves Lightdash users, look up the signed-in user's Lightdash user UUID instead and set it as `writeActions.userUuid` when you sign their token. + +Either pattern depends on signing the JWT per session or request, with a short `expiresIn`, rather than generating one token once and reusing it — that's what lets your backend resolve each request to the matching actor as end users and profiles change. The viewer's [`user.externalId`](#user-metadata) can still carry your own user identifier for tracking; it isn't the role-bearing actor. + ### Dashboard capabilities In `'roles'` mode, only the actor's embed scopes grant capabilities. JWT interactivity flags are ignored whether they are `true`, `false`, or omitted. For example, CSV export requires `view:EmbedCsvExport` even when the token contains `canExportCsv: true`. In `'default'` mode, the scope is ignored and the JWT flag decides. diff --git a/explore/metrics-catalog/curate-the-catalog.mdx b/explore/metrics-catalog/curate-the-catalog.mdx index 6630f9e3..e313e053 100644 --- a/explore/metrics-catalog/curate-the-catalog.mdx +++ b/explore/metrics-catalog/curate-the-catalog.mdx @@ -4,7 +4,7 @@ description: "Browse, explore, and curate your metrics in the Metrics Catalog" doc-type: tutorial --- -## Browsing the catalog +## Browsing the catalog (test) Browse all the project metrics you have access to (based on permissions and user-attributes) in the **Metrics Catalog**. diff --git a/self-host/customize-deployment/environment-variables.mdx b/self-host/customize-deployment/environment-variables.mdx index 41ff9d0f..1d3e3284 100644 --- a/self-host/customize-deployment/environment-variables.mdx +++ b/self-host/customize-deployment/environment-variables.mdx @@ -79,6 +79,7 @@ Control how many rows a query can return, page sizes, exports, and chart-version | `LIGHTDASH_CSV_CELLS_LIMIT` | Max cells on CSV file exports (default=100000) | | `LIGHTDASH_DASHBOARD_MAX_TILES_PER_TAB` | Maximum number of tiles allowed per tab on a dashboard (default=50) | | `LIGHTDASH_DASHBOARD_MAX_TABS_PER_DASHBOARD` | Maximum number of tabs allowed per dashboard (default=20) | +| `LIGHTDASH_DASHBOARD_MAX_CONCURRENT_TILE_QUERIES` | Caps how many dashboard tile queries run concurrently instead of every tile firing its query to the warehouse as soon as the dashboard loads. Tiles currently visible on screen are started first; others wait for a free slot. A query started by a data refetch (for example, after a filter change) also waits for a free slot like any other tile query. Unset by default (no cap) | | `MAX_DOWNLOADS_AS_CODE` | Maximum number of downloads as code (default=100) | ## Scheduler @@ -342,6 +343,49 @@ These variables are **deprecated**; use the `RESULTS_S3_*` versions instead. +### Audit log events + +Set `LIGHTDASH_LOG_LEVEL=AUDIT` (or `DEBUG`) to write audit log entries alongside your application logs. Each entry has an `actor` (the user, [impersonated session](/support/support-access#how-impersonation-works), service account, or anonymous viewer that performed the action), an `action`, a `resource` (a `type` and a `metadata` object with the details of the change), and a `status` of `allowed`, `denied`, or `allowed-bypass`. + +Today, audit logging covers: + +- **Group membership changes**, including changes made through SCIM, logged as `resource.metadata.event: "group.membership_changed"` with the `operation` (`add`, `remove`, `replace`, `delete`) and the affected `memberUuids`. +- **Organization- and project-level role-set changes** — `organization_role_set.replaced`, `project_role_set.replaced`, and `project_group_role_set.replaced` — logged with the role set `before` and `after` the change, and the users or groups `added` and `removed`. +- **Direct access grants, revokes, and role changes** on dashboards, charts, and apps — `direct_access.grant`, `direct_access.revoke`, `direct_access.role_change`, `direct_access.reset`, and `direct_access.replace` — logged with `beforeRole` and `afterRole` on the resource. + +Some older, single-field actions — for example the legacy per-user organization role update, or a user joining an organization — don't yet have a dedicated event carrying the old and new value. Lightdash audits every permission check automatically as a generic `allowed`/`denied` entry, independent of whether a business-level audit event is also written for that action, so these actions still appear in the audit log as a permission check rather than as a change with before and after values. + + +Audit logging runs on every Lightdash Cloud instance, but the resulting entries currently live only in Lightdash's own infrastructure — there's no way to export or view them from a Cloud account today. The log level and output variables above configure a self-hosted deployment's own log output. + + +### Audit log events + +Lightdash writes audit log events to the application log, at the `audit` log level. `audit` is below `info`, so set `LIGHTDASH_LOG_LEVEL=audit` or `LIGHTDASH_LOG_LEVEL=debug` to write them — with the default level, Lightdash does not write them. Set `LIGHTDASH_LOG_FORMAT=json` to get each entry as structured JSON instead of a plain-text line. + +Each entry has: + +- `id` and `timestamp`. +- `actor`: `type` (`session`, `pat`, `oauth`, `service-account` or `anonymous`), the user `uuid`, `email`, `organizationRole`, and `impersonatedBy` when the action ran under [impersonation](/support/support-access). +- `action` and `resource` (`type`, `organizationUuid`, `projectUuid` when relevant, and `metadata`). +- `context`: `ip`, `userAgent` and `requestId`. +- `status`: `allowed`, `denied` or `allowed-bypass`. + +If your log indexer maps the `actor` field as text, set `LIGHTDASH_LOG_AUDIT_ACTOR_AS_STRING=true` to serialize it to a JSON string instead of a nested object. + +Today, these access changes write an explicit event that names the target and, where a role changed, the before and after value: + +| Change | `action` | `resource.metadata` | +| --- | --- | --- | +| Group membership change (project UI or SCIM) | `update` on resource type `Group`, with `resource.metadata.event = group.membership_changed` | `groupUuid`, `operation` (`add`, `remove`, `replace` or `delete`), `memberUuids` | +| Organization role set replaced | `organization_role_set.replaced` | target `userUuid`, `before`, `after`, `added`, `removed` | +| Project role set replaced | `project_role_set.replaced` | target `userUuid`, `before`, `after`, `added`, `removed` | +| Project group role set replaced | `project_group_role_set.replaced` | target `groupUuid`, `before`, `after`, `added`, `removed` | +| Direct access grant, revoke or role change on a dashboard, chart, SQL chart, document or app | `direct_access.grant`, `direct_access.revoke` or `direct_access.role_change` (plus `direct_access.reset` and `direct_access.replace` for bulk changes) | `resourceUuid`, `principalType`, `principalUuid`, `beforeRole`, `afterRole` | +| Login, logout, leaving an organization, impersonation start and stop | `login`, `logout`, `leave_organization`, `impersonation_start`, `impersonation_stop` | login provider and similar context | + +Other access changes — a user joining an organization, a single role assignment, adding, updating or removing a user's project access, and adding or removing a user's space access — currently only write the permission-check entry for the request that made the change (`actor`, `action`, `resource`, and `status` of `allowed` or `denied`). That entry doesn't name the user whose access changed or the role involved. + ## Prometheus @@ -404,7 +448,7 @@ Enterprise-only. For setup steps (which provider to pick, how to point at an LLM | `AI_COPILOT_TELEMETRY_ENABLED` | Enables telemetry for AI Copilot (default=false) | | `AI_COPILOT_THREAD_DUMP_ENABLED` | Enables the [AI thread debug dump](/support/ai-thread-debug-dump) download action for organization AI admins (default=false) | | `AI_COPILOT_REQUIRES_FEATURE_FLAG` | Requires a feature flag to use AI Copilot (default=false) | -| `AI_COPILOT_MAX_QUERY_LIMIT` | Maximum number of rows returned in AI-generated queries (default=5000) | +| `AI_COPILOT_MAX_QUERY_LIMIT` | Maximum number of rows returned in AI-generated queries and by the MCP `run_metric_query` tool (default=5000) | | `AI_COPILOT_RUN_SQL_MAX_LIMIT` | Maximum number of rows the AI Analyst `run_sql` tool can return. Falls back to `AI_COPILOT_MAX_QUERY_LIMIT`, then to 5000. | | `AI_VERIFIED_ANSWER_SIMILARITY_THRESHOLD` | Similarity threshold (0-1) for verified answer matching (default=0.6) | | `AI_COPILOT_ALLOWED_PROJECT_UUID` | Comma-separated list of project UUIDs allowed to use AI Copilot. When set, AI Copilot is disabled for all other projects. | diff --git a/self-host/customize-deployment/sandboxes.mdx b/self-host/customize-deployment/sandboxes.mdx index db8b0005..5b3b8dc2 100644 --- a/self-host/customize-deployment/sandboxes.mdx +++ b/self-host/customize-deployment/sandboxes.mdx @@ -116,6 +116,42 @@ This is the **recommended sandbox provider for customers deploying Lightdash on it keeps the sandbox boundary inside your existing AWS account and avoids sending agent workloads or repository contents to a third-party service. + +Registering a Lambda MicroVM image uses AWS's `create-microvm-image`: a build role, a +`codeArtifact` zip staged in S3, the AWS-managed `al2023` base image, `ARM_64`, 4 GB +memory, and the agent's `/ready` hook on port 8080. That call returns an **image ARN**, +and the ARN is the only thing Lightdash itself needs — the backend passes it to +`RunMicrovm` and never builds, pulls, or otherwise touches ECR or S3 at runtime. Any +pipeline that produces a `codeArtifact` zip AWS can build into a MicroVM image works, +with or without ECR. + + +Lightdash's own build script (`./sandboxes/microvm-agent/build-microvm-image.sh`, see +[Prerequisites](#prerequisites)) pushes a prebuilt image to ECR and builds the registered +image `FROM` it, to avoid rebuilding the npm/dbt toolchain on every update. That's the +script's own shortcut, not an AWS or Lightdash requirement. + + +```mermaid +sequenceDiagram + participant Build as Your build pipeline + participant ECR as ECR (optional) + participant S3 as S3 (build-time staging) + participant Reg as create-microvm-image + participant Backend as Lightdash backend / worker + participant Lambda as Lambda MicroVMs control plane + + Note over Build,Reg: Build time — once per image version + Build->>ECR: push prebuilt image (optional shortcut) + Build->>S3: upload codeArtifact zip + Build->>Reg: create-microvm-image (al2023 base, ARM_64, 4 GB, /ready on 8080) + Reg-->>Build: image ARN + + Note over Backend,Lambda: Runtime — every sandbox launch + Backend->>Lambda: RunMicrovm(imageArn) + Lambda-->>Backend: microVM endpoint + token +``` + ### Prerequisites Provision these with your own IaC, in the region you set as `LAMBDA_MICROVM_REGION` and in @@ -125,14 +161,33 @@ so the resources can sit in a separate account when the backend's role is in tha for example an IRSA role there that trusts your cluster's OIDC provider: - **Two MicroVM images** — one for data app generation and one for AI writeback (they - bundle different toolchains). Build them from the Dockerfiles in the Lightdash repo - (`sandboxes/data-apps/`, `sandboxes/ai-writeback/`, and the exec agent in - `sandboxes/microvm-agent/`), push them to ECR, and register each as a Lambda MicroVM + bundle different toolchains), built from the Dockerfiles in the Lightdash repo + (`sandboxes/data-apps/`, `sandboxes/ai-writeback/`) and registered as a Lambda MicroVM image on the AWS-managed `al2023` base — `ARM_64`, 4 GB memory, with the agent's `/ready` hook on port 8080. Each registration returns an **image ARN** for the config - below. The data app Dockerfile copies `lightdash-query-sdk.tgz`, a generated artifact - that a fresh checkout does not contain — build and pack it into `sandboxes/data-apps/` - first with `./sandboxes/data-apps/pack-query-sdk.sh`, using the toolchain listed in + below. + + The recommended way to produce and refresh these ARNs is + `sandboxes/microvm-agent/build-microvm-image.sh` in the Lightdash repo, which runs the + whole pipeline: it builds the local Docker image for the feature you pass it + (`data-app` or `writeback`), provisions the ECR repository, build IAM role, and S3 + staging bucket if they don't exist yet, and pushes the local image to ECR — Lambda + MicroVM registration, like Lambda container-image functions generally, only builds + from an ECR-hosted base. It then assembles a thin `FROM ` Dockerfile that + adds the exec agent (`sandboxes/microvm-agent/`), zips it, and uploads it to S3 as the + build's `codeArtifact` — that zip is a thin wrapper around the ECR image, not the full + application. It calls AWS's `create-microvm-image` (or `update-microvm-image` when the + image already exists), polls the build to an active version, and prints the image ARN + to set as `LAMBDA_MICROVM_DATA_APP_IMAGE_ARN` or `LAMBDA_MICROVM_AI_WRITEBACK_IMAGE_ARN`. + Run it with `./sandboxes/microvm-agent/build-microvm-image.sh data-app` and + `./sandboxes/microvm-agent/build-microvm-image.sh writeback`. None of this runs at + deploy time: the Lightdash backend never builds images or touches ECR or S3 — it only + calls AWS's `RunMicrovm` with the image ARN you already registered. + + The data app Dockerfile copies `lightdash-query-sdk.tgz`, a generated artifact that a + fresh checkout does not contain. `build-microvm-image.sh` packs it for you; building + the Dockerfile directly instead needs + `./sandboxes/data-apps/pack-query-sdk.sh` first, using the toolchain listed in [Build the gateway image](#2-build-the-gateway-image) under the Cloud Run provider. The writeback Dockerfile has no such prerequisite. - **Control-plane permissions on your backend's existing IAM role** — add `RunMicrovm`, diff --git a/self-host/enterprise-features/mcp.mdx b/self-host/enterprise-features/mcp.mdx index 97136901..9171e2e6 100644 --- a/self-host/enterprise-features/mcp.mdx +++ b/self-host/enterprise-features/mcp.mdx @@ -40,6 +40,8 @@ MCP_RUN_SQL_MAX_LIMIT=20000 Requests that ask for more rows are clamped to this limit. +`run_metric_query` has no separate MCP variable — its row cap is `AI_COPILOT_MAX_QUERY_LIMIT` (default 5000), the same instance-wide setting that limits AI Analyst queries (see [AI Analyst environment variables](/self-host/customize-deployment/environment-variables#ai-analyst)). It isn't configurable per organization, and a requested limit above the cap is clamped without the response saying so. + To reduce the filter-related query schema sent to the model, enable [compact filter expressions](/self-host/enterprise-features/ai-agents#compact-filter-expressions). That section covers deployment and client-refresh requirements. ## MCP URL format @@ -68,13 +70,29 @@ If your Lightdash instance sits behind a firewall or VPN, ensure the MCP client The client needs `/api/v1/mcp`, the OAuth endpoints under `/api/v1/oauth/`, and the `/.well-known/oauth-*` discovery paths. Some clients connect from their vendor's cloud rather than from the user's machine, so an instance that's only reachable on your internal network works from one client and fails from another. For those clients, allow the vendor's published outbound IP ranges to reach these paths. +Allow more than `GET`: Claude's cloud calls `POST /api/v1/oauth/token` and `POST /api/v1/oauth/register`, and it needs `POST`, `GET` (it opens an SSE stream), and `DELETE` on `/api/v1/mcp` itself. A `GET`-only or exact-path allowlist rule lets OAuth discovery pass but leaves the connection stuck after you click **Continue connecting** in Claude. + To check that the endpoint is reachable from a given network, run: ```bash curl -i https:///api/v1/mcp ``` -A reachable instance returns `401 Unauthorized` with a `WWW-Authenticate` header pointing at your `SITE_URL`. If that URL isn't the public HTTPS address of your instance, OAuth discovery fails. +A reachable instance returns `401 Unauthorized` with a `WWW-Authenticate` header pointing at your `SITE_URL`. If that URL isn't the public HTTPS address of your instance, OAuth discovery fails. Also check discovery directly: + +```bash +curl https:///.well-known/oauth-authorization-server +``` + +Its `issuer` and `token_endpoint` should both be your instance's public HTTPS address (`SITE_URL`), not an internal hostname. + +### Outbound egress for OAuth client ID metadata documents + +Lightdash 2.580.0 and later accepts [OAuth client ID metadata documents](https://datatracker.ietf.org/doc/draft-ietf-oauth-client-id-metadata-document/) as a way for MCP clients to identify themselves, instead of registering a client first. Claude uses this: it presents an `https` URL as its `client_id`, for example `https://claude.ai/oauth/mcp-oauth-client-metadata`. When a user authorizes that client, the Lightdash backend fetches the document at that URL itself, over HTTPS, with no redirects and a 5-second timeout — so your backend pods need direct outbound HTTPS access to `claude.ai`, not routed through a proxy that blocks it. If that fetch fails, authorization fails even though the consent page rendered correctly. + + + Versions 2.580.0–2.589.3 reject client ID metadata documents that advertise extra grant types, which Claude's document does — you need 2.589.4 or later. + ## Troubleshooting @@ -83,3 +101,5 @@ A reachable instance returns `401 Unauthorized` with a `WWW-Authenticate` header | `/api/v1/mcp` returns an error or 404 | `MCP_ENABLED` isn't set to `true`, or `LIGHTDASH_LICENSE_KEY` is missing or invalid | | OAuth flow fails | The Lightdash instance isn't reachable from the browser performing the OAuth redirect | | AI assistant can't connect | The MCP URL is wrong or the instance isn't accessible over HTTPS | +| Connection hangs after **Continue connecting** | The allowlist only permits `GET` or an exact path on `/api/v1/mcp` — allow `POST`, `GET`, and `DELETE` on `/api/v1/mcp`, and `POST` on `/api/v1/oauth/token` and `/api/v1/oauth/register` | +| Consent page renders, but **Authorize** returns a 400 | Check you're on Lightdash 2.589.4 or later and that backend pods have outbound HTTPS access to the client's `client_id` host (e.g. `claude.ai`). If `client_id` is an opaque ID rather than a URL, the client registered dynamically instead — check `POST /api/v1/oauth/register` is allowed through and look at the backend logs | diff --git a/semantic-layer/caching.mdx b/semantic-layer/caching.mdx index 94067111..546c4b30 100644 --- a/semantic-layer/caching.mdx +++ b/semantic-layer/caching.mdx @@ -5,7 +5,6 @@ description: "Cut the number of queries Lightdash runs against your warehouse wi import { FeatureBadge } from '/snippets/feature-badge.jsx'; - Caching is available on Lightdash Cloud and on self-hosted instances with a valid Enterprise license. Filter value caching is available on all Cloud plans; results caching requires Enterprise. Lightdash supports two types of caching: @@ -47,14 +46,15 @@ These Lightdash features use caching (if it's enabled on your instance): - **Saved Charts** are cached based on the last refresh in any context (edit mode, view mode, dashboard refresh, etc.), but queries made while editing are NOT cached or pulled from cache. - **Dashboard tiles** (internal and embedded) will use the cache from saved charts they reference. If charts only exist on a single dashboard, they will refresh whenever you click the Refresh button on the dashboard. Embedded dashboards have no Refresh button, so embedded viewers see cached results until they expire. To control how fresh embedded data is, set the project's [cache duration](#cache-duration-per-project); a shorter duration sends more queries to the warehouse. -- **Scheduled Deliveries** generate and use cached results for the saved chart or dashboard they belong to. -- **Google Sheets syncs** also use cached results. If you need syncs to always return fresh data, be aware that enabling caching will cause syncs to deliver cached results until the cache expires. - **SQL runner queries** including saved SQL charts and dashboard SQL chart tiles are cached through the same execution path as metric-based charts. Note that the SQL runner does not have a UI button to force-refresh or invalidate the cache — results remain cached until the cache expires. - **Metrics Catalog** queries go through the same async query service and are cached like any other query. - **[Data apps](/data-apps)** run every metric query through the same async query service and are cached like any other query, including live-linked charts. The **Refresh** button in the app header (while building or viewing) invalidates the cache for that app and re-queries the warehouse fresh for the rest of the session, the same way the dashboard refresh button does. Because the cache key is project ID plus generated SQL, apps with lots of interactive filter controls produce more distinct cache entries than an equivalent dashboard — the first hit on each filter combination still goes to the warehouse. These Lightdash features DO NOT use caching: +- **Scheduled Deliveries** (Slack, email, webhook) always bypass the cache and run a fresh query against the warehouse, even when caching is enabled for the project. +- **Google Sheets syncs** always bypass the cache and run a fresh query against the warehouse, even when caching is enabled for the project. + - **Editing a saved chart** — actions in edit mode (such as changing columns, filters, or other query parameters) explicitly bypass the cache to ensure you always see fresh results while building a query. - **[External connections](/data-apps/external-connections) in data apps** — requests to third-party HTTP APIs are runtime fetches through the Lightdash proxy and are not covered by results caching. @@ -112,6 +112,23 @@ A cached result is served as stored until it expires, even one cached while the There's no button in the Lightdash UI to invalidate a single Saved Chart's cache outside of a dashboard. You can do this through the API: set `invalidateCache: true` on a request to the [execute saved chart](/api-reference/v2/execute-saved-chart) or [execute dashboard chart](/api-reference/v2/execute-dashboard-chart) endpoint to bypass the cache and re-run that chart's query, which repopulates its cache entry. +#### Clearing cached results for a project + +Project admins can clear every cached result for a project from **Project settings → Results caching**: + +- **Clear cache** invalidates every cached result in the project immediately. The next load of each chart, dashboard, or API query runs against the warehouse and repopulates the cache. +- **Clear on a schedule** runs the same clear automatically — daily or weekly, at one or more hours, in the project's scheduler timezone. + +To invalidate the cache from a script or a data pipeline, call one of these endpoints with a [personal access token](/personal-settings/create-personal-tokens) (`Authorization: ApiKey `). All three require project admin access; query history and existing result files are kept, and the next run of the affected chart, dashboard, or project queries the warehouse and repopulates the cache: + +| Scope | Endpoint | +| --- | --- | +| Project | `POST /api/v1/projects/{projectUuid}/results-cache/invalidate` | +| Dashboard | `POST /api/v2/dashboards/{dashboardUuid}/results-cache/invalidate` | +| Saved chart | `POST /api/v2/saved/{chartUuid}/results-cache/invalidate` | + +Call the project-level endpoint from a data pipeline once a run finishes, so the next person to open a chart or dashboard gets fresh results instead of what was cached before the run. See the [API reference](/api-reference/v1/introduction) for full request and response details. + ## Results caching vs pre-aggregates Lightdash has two independent systems for speeding up queries: **results caching** (documented above) and **pre-aggregates**. They work differently and are designed to be used together, not as replacements for each other. diff --git a/semantic-layer/yaml.mdx b/semantic-layer/yaml.mdx index cc5a487b..ae2c2551 100644 --- a/semantic-layer/yaml.mdx +++ b/semantic-layer/yaml.mdx @@ -38,6 +38,10 @@ Lightdash reads `.yml` and `.yaml` files recursively. It also supports `lightdas The model name does not need to match its filename. Lightdash tracks the source path so write-back updates the original file, including models nested in subdirectories. Keep native model files inside the configured project directory. +### Combining models from multiple sources + +A native project is a single directory with one `lightdash.config.yml`, and native YAML has no equivalent of dbt packages or `dbt deps` for pulling in models from other repositories. If your semantic models live across several source repos, add a CI step that checks out or copies each source's model files into the one project's `models/` directory before running `lightdash deploy` or `lightdash lint` — a subfolder per source works, since Lightdash reads `.yml` and `.yaml` files recursively. Every model still needs a project-wide unique `name`; deploys and refreshes fail on duplicate names once the sources are merged. + ## Recommended: build with an AI coding agent The fastest way to get started with Lightdash YAML is to let an AI coding agent (Claude Code, Cursor, etc.) do the heavy lifting for you. diff --git a/workflow/content-as-code.mdx b/workflow/content-as-code.mdx index 23948c9c..d0c04b5a 100644 --- a/workflow/content-as-code.mdx +++ b/workflow/content-as-code.mdx @@ -88,6 +88,10 @@ content_as_code: To publish a draft, a developer or admin opens **Project settings** → **Content review**, picks the draft, and clicks **Write back to repo**. Lightdash opens a pull request with the draft's content; merge it and run `lightdash upload`, for example from your deploy CI job, to publish the change. **Dismiss** closes a draft without writing it back. **Content review** appears only on projects connected to GitHub or GitLab. +Sync applies to the whole project — there's no per-dashboard or per-chart toggle in the UI. To stop content being Git-backed, remove `content_as_code.sync: true` from `lightdash.config.yml` (or set it to `false`) and run `lightdash upload` again. This stamps sync off on the project: UI saves to that content publish normally from then on, and no further drafts are created. + +With sync on, a full `lightdash upload` of a directory also releases any Git-backed chart or dashboard whose file is no longer in that directory, and dismisses its open drafts. Only content with a recorded file path is released this way; a filtered upload (`-c`, `-d`) never releases content. + ## `lightdash download` From the [Lightdash CLI](/workflow/cli/reference), you can use the command `lightdash download` to download all of the charts and dashboards from your Lightdash project as code. All of the charts and dashboards will be written as .yml files to a `lightdash` directory wherever you're running the command. diff --git a/workspace-admin/appearance.mdx b/workspace-admin/appearance.mdx index c700ea72..bb82b7f3 100644 --- a/workspace-admin/appearance.mdx +++ b/workspace-admin/appearance.mdx @@ -5,6 +5,8 @@ description: "Set the default chart colors for your organization so everything y doc-type: reference --- +import { FeatureBadge } from '/snippets/feature-badge.jsx'; + To customize the colors used by default in your organization, head to the `Organization settings` --> `Appearance` and you can update any and all of your default colors used in your organization. ## Viewing available palettes @@ -65,20 +67,28 @@ In the chart editor, the palette picker is at the top of the **Series** tab (car ## Brand appearance -Under `Organization settings` --> `Appearance`, organization admins can also configure a **Brand appearance** for the org. This stores your brand's domain, logos, fonts, and colors alongside the color palettes above. The saved brand colors feed the **Brand** and **Accent** themes of a homepage [Call to action](/explore/homepage#call-to-action), and **Generate palette** under **Chart palette** expands them into a chart palette you can save. The brand doesn't restyle the Lightdash app itself, such as the login page or navigation bar. +Under `Organization settings` --> `Appearance`, organization admins can also configure a **Brand appearance** for the org. This stores your brand's domain, name, logos, fonts, and colors alongside the color palettes above. The saved brand colors feed the **Brand** and **Accent** themes of a homepage [Call to action](/explore/homepage#call-to-action), and **Generate palette** under **Chart palette** expands them into a chart palette you can save — brand colors only ever generate chart palettes and CTA themes; they don't recolor the rest of the app, and the navigation bar stays dark regardless of your brand colors. + +With a valid license, organization admins can also set a **brand name** and a **logo** to rebrand the app itself: + +- The logo replaces the Lightdash logo in the main app's navigation bar and is used as the browser tab icon (favicon). +- The brand name replaces "Lightdash" in the browser tab title. + +Embedded dashboards and the SDK keep their own embedding configuration and are unaffected by the organization's brand name and logo. - Brand appearance relies on a [Brandfetch](https://brandfetch.com/) integration to detect a brand from its domain. On Lightdash Cloud this is configured for you. On self-hosted or OSS deployments, set the `BRANDFETCH_API_KEY` environment variable to a Brandfetch API key. Without it, the section is hidden unless the organization already has a saved brand. The `hasBrandfetch` flag on `GET /health` indicates whether the current instance can detect new brands. + Brand appearance can use a [Brandfetch](https://brandfetch.com/) integration to detect a brand from its domain, but Brandfetch is optional: you can set the brand name and logo directly without it. On Lightdash Cloud, Brandfetch is configured for you. On self-hosted or OSS deployments, set the `BRANDFETCH_API_KEY` environment variable to a Brandfetch API key to enable it. The `hasBrandfetch` flag on `GET /health` indicates whether the current instance can detect new brands from a domain. Without a valid Enterprise license, Brandfetch, or an already-saved brand, the **Brand appearance** section is hidden. ### Editing your brand Open `Organization settings` --> `Appearance` --> **Brand appearance**. You'll see a form containing: -- **Domain** — your brand's domain (for example, `ecom-store.com`). Used as the identifier when fetching brand information. -- **Colors** — the palette detected for your brand. Each color has a hex value and a classification: `brand`, `accent`, `dark`, `light`, or `other`. You can add new colors, remove existing ones, edit a hex value, or re-classify a color. +- **Domain** — your brand's domain (for example, `ecom-store.com`). Used to fetch brand information from Brandfetch, when that integration is enabled. +- **Name** — your brand's name. Requires a valid Enterprise license; it's used for the browser tab title. +- **Colors** — the palette detected for your brand, or added manually. Each color has a hex value and a classification: `brand`, `accent`, `dark`, `light`, or `other`. You can add new colors, remove existing ones, edit a hex value, or re-classify a color. - **Fonts** — the title and body font families associated with your brand. These are shown as read-only for now; a font picker is not yet available. -- **Logos** — the light and dark logos for your brand. Logos are populated by the Brandfetch integration only — direct logo upload is not exposed in the UI. +- **Logos** — the light and dark logos for your brand. Requires a valid Enterprise license. Upload an image (PNG, JPG, SVG, WebP, or GIF, up to 512 KB) or paste an image URL — Brandfetch isn't required. ### Refreshing from Brandfetch