diff --git a/.yfm b/.yfm index 2bc24e5..664e147 100644 --- a/.yfm +++ b/.yfm @@ -7,6 +7,10 @@ vars: product_description: "Observability Data Lake Engine" version: "0.1.0" repo_url: "https://github.com/icegatetech/icegate" + # Rust toolchain floor. Churns on every edition/MSRV bump and is quoted in 12 prose + # places across en/fr/ru, so it is the highest-churn literal in the corpus after the + # product name. Code blocks keep the literal (see AGENTS.md). + rust_version: "1.92.0" license: "Apache 2.0" # Markdown parsing options diff --git a/AGENTS.md b/AGENTS.md index 0b4e8a7..7db258d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,16 @@ npm run build:fr # Build French only npm run build:ru # Build Russian only ``` +**Do not drop `--static-content` from the build scripts.** Its help text ("allow loading custom +resources into statically generated pages") undersells it: without the flag Diplodoc ships every +page as an empty `
` with the real content parked in a `diplodoc-state` JSON blob, +so a crawler that does not run JavaScript sees ~4 words, no `

`, and — because the TOC is +rendered client-side too — no links to follow. Ahrefs found 4 of this site's pages for exactly +that reason. With the flag, pages ship prerendered (~670 words and a real `

` on a typical +page) and the client bundle still hydrates on top, so nothing about the reading experience +changes. Removing it breaks search and AI-crawler visibility site-wide, silently and with a +green build. + ## Project Structure ``` @@ -32,6 +42,7 @@ npm run build:ru # Build Russian only ├── ru/ # Russian documentation ├── llms.txt # LLM context file — overview with key examples ├── llms-full.txt # LLM context file — complete documentation content +├── robots.txt # Crawler policy; copied to the build root by `npm run build` ├── presets.yaml # Build presets (default, development, production) ├── .yfm # Diplodoc configuration (vars, langs, settings) └── .yfmlint # Linter rules configuration @@ -72,7 +83,31 @@ When updating documentation, regenerate `llms-full.txt` after changes. `llms.txt ## Writing Documentation -- Use variables from `.yfm` vars section: `{{product_name}}`, `{{version}}`, `{{repo_url}}` +- **Use the `.yfm` vars in prose** — every mention outside code. The corpus is converted, so a + rename or version bump is a one-line edit in `.yfm` rather than a find-and-replace across + three languages. In use today: `{{product_name}}` (266 sites), `{{rust_version}}` (15), + `{{repo_url}}` (5), `{{license}}` (4). +- **Adding a var is only worth it when the value appears in prose.** Measure before you add: + ports are the cautionary case — `3100` appears 37 times in prose but 131 times inside code + blocks, and since code must stay literal, a `{{loki_port}}` var would let prose and the + adjacent `curl` command disagree after a change. That is strictly worse than a literal, + because it *looks* single-sourced. Same verdict for `{{version}}` and the Helm OCI ref: code + only, so they stay defined but unused. `{{product_description}}` is title-case and every + prose site is mid-sentence lowercase, so it does not fit either. +- **Never substitute inside code.** Fenced blocks, inline code, link targets and HTML + attributes keep the literal name — commands, image tags, hostnames (`icegate-query`), + datasource UIDs and `github.com/icegatetech/icegate` are identifiers, not prose, and a reader + copy-pasting `{{product_name}}` into a shell gets nothing useful. +- **`llms.txt` and `llms-full.txt` must contain the literal name, never a variable.** The build + `cp`s them into `./build` verbatim, so yfm never renders them — a `{{product_name}}` there + ships raw to the LLM consumers the files exist for. 23 of them were doing exactly that. +- The per-language scripts pass `-c ./.yfm`, and **the `./` is load-bearing**. `--help` says + relative config paths resolve from the execution directory and "other" paths from `--input`; + a bare `.yfm` counts as "other", so it resolves to `en/.yfm`, silently finds nothing, and the + build emits 121 "Variable not found" warnings while shipping raw `{{product_name}}` to disk. + `./.yfm` resolves from the repo root and works. `../.yfm` fails outright (ENOENT one + directory above the repo). Verify a change here by grepping the output for `{{`, not by + trusting the exit code — a config that fails to load is a warning, not an error. - HTML is allowed (`allowHTML: true`) - Files must end with newline (MD047 enforced) - Line length not enforced (MD013 disabled) @@ -86,7 +121,10 @@ When updating documentation, regenerate `llms-full.txt` after changes. `llms.txt - Primary installation method: **Helm chart** (`oci://ghcr.io/icegatetech/charts/icegate`) - Development environment: **Skaffold** (`skaffold dev`) with Kustomize overlays - Docker Compose available as alternative for local development -- Rust 1.92.0+ (2024 edition), 6 workspace crates: common, queue, query, ingest, maintain, jobmanager +- Rust 1.92.0+ (2024 edition), 6 workspace crates: common, catalog-s3, queue, query, ingest, maintain. `jobmanager` is **not** a workspace crate — it lives in `icegatetech/jobmanager` and is consumed as a git-pinned dependency +- Default catalog backend is IceGate's own S3 catalog (`backend: !s3`, state in `root.json`), not Nessie. Nessie/Glue/S3 Tables are alternatives +- Default object store is **RustFS** (S3-compatible), not MinIO +- Shift (WAL → Iceberg) lives in the **ingest** crate; compaction, orphan GC, and the LLM pricing crawler live in **maintain** - Metrics port: **9091** (not 9090). Prometheus API port is 9090. - Real environment variables: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `RUST_LOG` diff --git a/assets/c4/structurizr-CatalogComponents.png b/assets/c4/structurizr-CatalogComponents.png new file mode 100644 index 0000000..47b1d83 Binary files /dev/null and b/assets/c4/structurizr-CatalogComponents.png differ diff --git a/assets/c4/structurizr-Containers.png b/assets/c4/structurizr-Containers.png index f60615b..2c7206e 100644 Binary files a/assets/c4/structurizr-Containers.png and b/assets/c4/structurizr-Containers.png differ diff --git a/assets/c4/structurizr-IngestComponents.png b/assets/c4/structurizr-IngestComponents.png index 7e9017f..cc4f1e2 100644 Binary files a/assets/c4/structurizr-IngestComponents.png and b/assets/c4/structurizr-IngestComponents.png differ diff --git a/assets/c4/structurizr-IngestionFlow.png b/assets/c4/structurizr-IngestionFlow.png new file mode 100644 index 0000000..3e91d8d Binary files /dev/null and b/assets/c4/structurizr-IngestionFlow.png differ diff --git a/assets/c4/structurizr-MaintainComponents.png b/assets/c4/structurizr-MaintainComponents.png index 1235180..882f29c 100644 Binary files a/assets/c4/structurizr-MaintainComponents.png and b/assets/c4/structurizr-MaintainComponents.png differ diff --git a/assets/c4/structurizr-MaintenanceFlow.png b/assets/c4/structurizr-MaintenanceFlow.png new file mode 100644 index 0000000..8724abf Binary files /dev/null and b/assets/c4/structurizr-MaintenanceFlow.png differ diff --git a/assets/c4/structurizr-QueryComponents.png b/assets/c4/structurizr-QueryComponents.png index e63c024..0ba179f 100644 Binary files a/assets/c4/structurizr-QueryComponents.png and b/assets/c4/structurizr-QueryComponents.png differ diff --git a/assets/c4/structurizr-QueryFlow.png b/assets/c4/structurizr-QueryFlow.png new file mode 100644 index 0000000..50c6e13 Binary files /dev/null and b/assets/c4/structurizr-QueryFlow.png differ diff --git a/assets/c4/structurizr-QueueComponents.png b/assets/c4/structurizr-QueueComponents.png index e45764c..4713318 100644 Binary files a/assets/c4/structurizr-QueueComponents.png and b/assets/c4/structurizr-QueueComponents.png differ diff --git a/assets/c4/structurizr-SystemContext.png b/assets/c4/structurizr-SystemContext.png index f5ce2cc..ab713f8 100644 Binary files a/assets/c4/structurizr-SystemContext.png and b/assets/c4/structurizr-SystemContext.png differ diff --git a/c4/README.md b/c4/README.md index 7988b63..99db593 100644 --- a/c4/README.md +++ b/c4/README.md @@ -44,31 +44,57 @@ After running `make png`, the following files are created in `../assets/c4/`: | File | Description | |------|-------------| | `structurizr-SystemContext.png` | System context - IceGate and external systems | -| `structurizr-Containers.png` | Container diagram - Services and storage | +| `structurizr-Containers.png` | Container diagram - Services, libraries and storage | | `structurizr-IngestComponents.png` | Ingest Service internal components | | `structurizr-QueryComponents.png` | Query Service internal components | | `structurizr-QueueComponents.png` | Queue Library internal components | | `structurizr-MaintainComponents.png` | Maintain Service internal components | +| `structurizr-CatalogComponents.png` | S3 Catalog internal components | +| `structurizr-IngestionFlow.png` | Ingestion process (sequence) - OTLP request through shift to a committed snapshot | +| `structurizr-QueryFlow.png` | Query process (sequence) - LogQL request to a merged WAL and Iceberg result | +| `structurizr-MaintenanceFlow.png` | Maintenance process (sequence) - migration, compaction, orphan GC, pricing crawler | ## Workspace Structure ``` workspace.dsl ├── Model -│ ├── External Systems (OTel Collector, Grafana, Trino) +│ ├── External Systems (OTel Collector/SDK, Grafana, BI & SQL clients, +│ │ Prometheus, tracing backend, Trino, LLM pricing feeds) │ └── IceGate System -│ ├── Ingest Service (OTLP handlers, compactor) -│ ├── Query Service (Loki/Prometheus/Tempo APIs) -│ ├── Maintain Service (schema migrations) -│ ├── Queue Library (WAL on S3) -│ ├── Common Library (shared code) -│ └── Storage (Queue, Iceberg, Catalog) +│ ├── Ingest Service (OTLP handlers, transform, WAL writer, shift) +│ ├── Query Service (Loki/Prometheus/Tempo/Flight SQL, LogQL, TraceQL) +│ ├── Maintain Service (migrate, compaction, orphan GC, pricing crawler) +│ ├── S3 Catalog (root.json CAS catalog; optional REST server) +│ ├── Queue Library (Parquet WAL on S3) +│ ├── Common Library (schemas, storage cache, sort-merge, memory guard) +│ └── Storage (Queue/WAL, Iceberg, Catalog, Job state) └── Views ├── SystemContext ├── Containers - └── Component diagrams (per service) + ├── Component diagrams (Ingest, Query, Maintain, Queue, Catalog) + └── Process diagrams (Ingestion, Query, Maintenance) ``` +## Process Diagrams + +The three `*Flow` views are Structurizr [dynamic views](https://docs.structurizr.com/dsl/language#dynamic-view). +Two things to know before editing them: + +- **Every step must correspond to a relationship that already exists in the model.** A dynamic + view may give that relationship a step-specific description, but it cannot invent an edge — + the DSL fails with `A relationship between X and Y does not exist in model`. When a flow needs + a step you have not modelled, add the relationship to the `model` block first. +- **They render as UML sequence diagrams**, via the per-view + `properties { "plantuml.sequenceDiagram" "true" }`. Without it the exporter falls back to the + numbered box layout, which turns into unreadable long-arc spaghetti past about ten steps. + `autoLayout` is kept on each view as the fallback for that case. + +Structurizr's parallel-block syntax (`{ { … } { … } }`) is deliberately unused: both PlantUML +exporters flatten it into duplicate step numbers with no visual grouping, so concurrent loops +read as one pipeline. The maintenance view names the owning loop in each step description +instead. + ## Interactive Editing For the best editing experience, use Structurizr Lite: diff --git a/c4/workspace.dsl b/c4/workspace.dsl index f6ea7a7..1eb0bfb 100644 --- a/c4/workspace.dsl +++ b/c4/workspace.dsl @@ -7,93 +7,176 @@ workspace "IceGate" "Observability Data Lake Engine" { } model { - # External actors - otelCollector = softwareSystem "OpenTelemetry Collector" "Sends telemetry data via OTLP protocol" "External" - grafana = softwareSystem "Grafana" "Visualization and dashboards" "External" + # External actors and systems + otelCollector = softwareSystem "OpenTelemetry Collector / SDK" "Instrumented applications and collectors sending telemetry over OTLP" "External" + grafana = softwareSystem "Grafana" "Dashboards over the Loki and Tempo APIs, with trace-to-logs correlation" "External" + sqlClients = softwareSystem "BI and SQL Clients" "JDBC, ODBC and ADBC clients (DBeaver, Superset, Tableau, dbt) using Arrow Flight SQL" "External" + prometheusServer = softwareSystem "Prometheus" "Scrapes the services' own metrics endpoints" "External" + tracingBackend = softwareSystem "Tracing Backend" "Jaeger or any OTLP endpoint receiving IceGate's own traces" "External" + trino = softwareSystem "Trino" "Legacy SQL analytics; reads Iceberg only through an external REST catalog" "External" + pricingFeeds = softwareSystem "LLM Pricing Feeds" "OpenRouter and LiteLLM rate cards crawled into icegate.prices" "External" # IceGate system - icegate = softwareSystem "IceGate" "Observability data lake engine storing logs, traces, metrics, and events in Apache Iceberg" { - # Containers - ingestService = container "Ingest Service" "Receives OTLP data, buffers in queue, and compacts to Iceberg" "Rust / Axum / Tonic" { - otlpHttpHandler = component "OTLP HTTP Handler" "Receives OTLP data over HTTP/protobuf" "Axum" - otlpGrpcHandler = component "OTLP gRPC Handler" "Receives OTLP data over gRPC" "Tonic" - recordTransformer = component "Record Transformer" "Converts OTLP to Arrow RecordBatch" "Rust" + icegate = softwareSystem "IceGate" "Observability data lake engine storing logs, spans, events, metrics and LLM operations in Apache Iceberg" { + + # --------------------------------------------------------------- + # Services (deployable binaries) + # --------------------------------------------------------------- + ingestService = container "Ingest Service" "Receives OTLP, writes the Parquet WAL, and shifts WAL segments into Iceberg" "Rust / Axum / Tonic" { + otlpHttpHandler = component "OTLP HTTP Handler" "Receives OTLP over HTTP/protobuf on :4318 and reports partial success" "Axum" + otlpGrpcHandler = component "OTLP gRPC Handler" "Receives OTLP over gRPC on :4317 and reports partial success" "Tonic" + recordTransformer = component "Record Transformer" "Converts OTLP logs, spans, metrics and LLM operations into Arrow RecordBatches" "Rust / Arrow" + walWriter = component "WAL Writer" "Sorts batches by the table sort order and submits them to the queue" "Rust" + shiftPlanner = component "Shift Planner" "Groups WAL segments into per-table shift plans and schedules them as jobs" "Rust / jobmanager" + shiftExecutor = component "Shift Executor" "K-way merges sorted WAL row groups into Iceberg data files" "Rust / Parquet" + commitRunner = component "Commit Runner" "Commits shifted data files as an Iceberg snapshot carrying the WAL offset" "Rust / Iceberg" + } + + queryService = container "Query Service" "Loki, Prometheus, Tempo and Arrow Flight SQL APIs over the merged WAL and Iceberg view" "Rust / Axum / Tonic / DataFusion" { + lokiApi = component "Loki API" "LogQL query and label endpoints on :3100" "Axum" + prometheusApi = component "Prometheus API" "PromQL routes on :9090; handlers still return 501 Not Implemented" "Axum" + tempoApi = component "Tempo API" "TraceQL search and trace-by-id endpoints on :3200" "Axum" + flightSqlServer = component "Flight SQL Server" "Read-only Arrow Flight SQL on :8815; tenant taken from x-scope-orgid" "Tonic / Arrow Flight SQL" + logqlEngine = component "LogQL Parser and Planner" "Parses LogQL and lowers it into DataFusion plans" "Rust" + traceqlEngine = component "TraceQL Parser and Planner" "Parses TraceQL and lowers it into DataFusion plans" "Rust" + tenantCatalog = component "Tenant Catalog" "Enforces row-level tenant_id and hides the column from SQL sessions" "Rust" + queryEngine = component "Query Engine" "DataFusion session whose catalog provider merges WAL segments with Iceberg tables at the committed WAL offset" "DataFusion" } - queryService = container "Query Service" "Provides Loki/Prometheus/Tempo-compatible query APIs" "Rust / Axum / DataFusion" { - lokiApi = component "Loki API" "LogQL query endpoint" "Axum" - prometheusApi = component "Prometheus API" "PromQL query endpoint" "Axum" - tempoApi = component "Tempo API" "Trace query endpoint" "Axum" - logqlParser = component "LogQL Parser" "Parses LogQL into AST" "ANTLR4" - logqlPlanner = component "LogQL Planner" "Converts LogQL AST to DataFusion plans" "Rust" - queryEngine = component "Query Engine" "DataFusion-based query execution" "DataFusion" + maintainService = container "Maintain Service" "Schema migrations plus long-running compaction, orphan GC and pricing jobs" "Rust / CLI / jobmanager" { + migrator = component "Schema Migrator" "Creates the Iceberg tables (maintain migrate create)" "Rust / Iceberg" + dataCompactor = component "Data Compactor" "Rewrites small Parquet data files into fewer, larger sorted ones" "Rust / jobmanager" + manifestCompactor = component "Manifest Compactor" "Rewrites fragmented Iceberg manifests" "Rust / Iceberg" + orphanGc = component "Orphan GC" "Deletes objects the current table metadata no longer references, past a grace period" "Rust / jobmanager" + pricingCrawler = component "Pricing Crawler" "Crawls LLM rate cards and appends changed rates to icegate.prices" "Rust / reqwest" } - maintainService = container "Maintain Service" "Schema migrations and data maintenance" "Rust / CLI" { - migrator = component "Schema Migrator" "Creates and upgrades Iceberg tables" "Rust" - shifter = component "Queue Shifter" "Background task that combine segments to write to Iceberg" "Rust" + # --------------------------------------------------------------- + # Libraries linked into the services + # --------------------------------------------------------------- + s3Catalog = container "S3 Catalog" "Iceberg catalog held in a root.json object updated by compare-and-swap. Default backend, linked into every service, optionally served standalone as an Iceberg REST API on :8181" "Rust Library / Axum" "Library" { + catalogRestApi = component "Catalog REST API" "Iceberg REST /v1 config, namespace and table endpoints" "Axum" + catalogService = component "Catalog Service" "iceberg::Catalog implementation over the root.json state" "Rust" + catalogCache = component "Cached Storage" "Conditional-read root cache plus an LRU of immutable table metadata" "Rust" + catalogStorage = component "S3 Catalog Storage" "Conditional reads and CAS writes of catalog objects, with retries" "Rust / AWS SDK" } - queueLib = container "Queue Library" "Generic WAL-based queue with Parquet on object storage" "Rust Library" { - queueWriter = component "Queue Writer" "Writes Parquet segments to S3" "Rust" - queueReader = component "Queue Reader" "Reads Parquet segments from S3" "Rust" + queueLib = container "Queue Library" "Durable Parquet WAL on object storage with exactly-once, offset-ordered writes" "Rust Library" "Library" { + queueChannel = component "Write Channel" "Bounded channel that sheds load with a retryable 429 instead of buffering to OOM" "Rust / Tokio" + queueAccumulator = component "Accumulator" "Batches rows until the flush size or interval is reached" "Rust" + queueWriter = component "Queue Writer" "Writes Parquet segments with If-None-Match for exactly-once semantics" "Rust / Parquet" + queueReader = component "Queue Reader" "Reads Parquet segments and caches parsed metadata" "Rust / Parquet" } - commonLib = container "Common Library" "Shared utilities, schemas, and abstractions" "Rust Library" + commonLib = container "Common Library" "Table schemas, catalog and storage builders, foyer read cache, sort-merge primitives, memory-pressure guard, metrics and tracing" "Rust Library" "Library" - # External storage - queueStorage = container "Queue Storage" "Queue segments (Parquet WAL files)" "MinIO / S3" "Database" - icebergStorage = container "Iceberg Storage" "Iceberg data files and manifests" "MinIO / S3" "Database" - catalogStore = container "Catalog Store" "Iceberg catalog metadata" "Nessie / REST" "Catalog" + # --------------------------------------------------------------- + # Object storage + # --------------------------------------------------------------- + queueStorage = container "Queue Storage (WAL)" "Parquet WAL segments, reclaimed by an object lifecycle rule rather than deleted by IceGate" "RustFS / S3" "Database" + icebergStorage = container "Iceberg Storage" "Iceberg data files, manifests and table metadata" "RustFS / S3" "Database" + catalogStore = container "Catalog Store" "Catalog state: root.json on S3 by default, or an external Nessie, AWS Glue or S3 Tables catalog" "S3 / Nessie / Glue / S3 Tables" "Catalog" + jobStore = container "Job State Store" "jobmanager job and task state for shift, compaction, GC and pricing" "RustFS / S3" "Database" } - # Relationships - External to IceGate - otelCollector -> icegate.ingestService "Sends OTLP logs/traces/metrics" "HTTP/gRPC" - grafana -> icegate.queryService "Queries via Loki/Prometheus/Tempo APIs" "HTTP" + # Relationships - external to IceGate + otelCollector -> icegate.ingestService "Sends OTLP logs, spans and metrics" "HTTP :4318 / gRPC :4317" + grafana -> icegate.queryService "Queries logs and traces, tenant from X-Scope-OrgID" "HTTP :3100 / :3200" + sqlClients -> icegate.queryService "Runs read-only SQL" "Arrow Flight SQL :8815" + + # The same edges at component granularity, so the component and dynamic + # views show where a request actually lands. Structurizr keeps the + # container-level relationship above for the container and context + # views rather than drawing a second arrow. + otelCollector -> icegate.ingestService.otlpHttpHandler "Sends OTLP data" "HTTP :4318" + otelCollector -> icegate.ingestService.otlpGrpcHandler "Sends OTLP data" "gRPC :4317" + grafana -> icegate.queryService.lokiApi "Queries logs" "HTTP :3100" + grafana -> icegate.queryService.tempoApi "Queries traces" "HTTP :3200" + sqlClients -> icegate.queryService.flightSqlServer "Runs read-only SQL" "Arrow Flight SQL :8815" + prometheusServer -> icegate.ingestService "Scrapes metrics" "HTTP :9091" + prometheusServer -> icegate.queryService "Scrapes metrics" "HTTP :9091" + prometheusServer -> icegate.maintainService "Scrapes metrics" "HTTP :9091" + trino -> icegate.catalogStore "Reads table metadata (REST catalog only)" "Iceberg REST" + trino -> icegate.icebergStorage "Reads data files" "S3" + + # Relationships - IceGate to external + icegate.ingestService -> tracingBackend "Exports its own traces" "OTLP" + icegate.queryService -> tracingBackend "Exports its own traces" "OTLP" + icegate.maintainService -> tracingBackend "Exports its own traces" "OTLP" + icegate.maintainService -> pricingFeeds "Fetches LLM rate cards" "HTTPS" + + # Relationships - services to libraries + icegate.ingestService -> icegate.queueLib "Writes and reads WAL segments" "Rust API" + icegate.ingestService -> icegate.s3Catalog "Loads and commits table metadata" "Rust API" + icegate.ingestService -> icegate.commonLib "Schemas, storage, sort-merge, memory guard" "Rust API" + + icegate.queryService -> icegate.queueLib "Reads WAL segments" "Rust API" + icegate.queryService -> icegate.s3Catalog "Loads table metadata" "Rust API" + icegate.queryService -> icegate.commonLib "Schemas, cached storage, memory guard" "Rust API" - # Relationships - Internal containers - icegate.ingestService -> icegate.queueLib "Writes and reads queue" "Rust API" - icegate.ingestService -> icegate.queueStorage "Reads/writes queue segments" "S3" - icegate.ingestService -> icegate.icebergStorage "Writes compacted data" "S3" - icegate.ingestService -> icegate.catalogStore "Commits snapshots" "REST" - icegate.ingestService -> icegate.commonLib "Uses schemas" "Rust API" + icegate.maintainService -> icegate.s3Catalog "Creates tables and commits snapshots" "Rust API" + icegate.maintainService -> icegate.commonLib "Schemas, storage, manifest scan, sort-merge" "Rust API" - icegate.queryService -> icegate.queueStorage "Reads queue segments" "S3" - icegate.queryService -> icegate.icebergStorage "Reads Iceberg data" "S3" - icegate.queryService -> icegate.catalogStore "Reads table metadata" "REST" - icegate.queryService -> icegate.queueLib "Reads queue" "Rust API" - icegate.queryService -> icegate.commonLib "Uses schemas" "Rust API" + # Relationships - services to storage + icegate.ingestService -> icegate.icebergStorage "Writes shifted data files" "S3" + icegate.ingestService -> icegate.jobStore "Persists shift job state" "S3" + icegate.queryService -> icegate.icebergStorage "Reads data files through the foyer cache" "S3" + icegate.maintainService -> icegate.icebergStorage "Rewrites and deletes data files" "S3" + icegate.maintainService -> icegate.jobStore "Persists compaction, GC and pricing job state" "S3" - icegate.maintainService -> icegate.icebergStorage "Manages table storage" "S3" - icegate.maintainService -> icegate.catalogStore "Creates/updates tables" "REST" - icegate.maintainService -> icegate.commonLib "Uses schemas" "Rust API" + icegate.queueLib -> icegate.queueStorage "Reads and writes Parquet segments" "S3" + icegate.s3Catalog -> icegate.catalogStore "Reads and CAS-writes catalog state" "S3" # Component relationships - Ingest Service - icegate.ingestService.otlpHttpHandler -> icegate.ingestService.recordTransformer "Parsed OTLP" "Rust" - icegate.ingestService.otlpGrpcHandler -> icegate.ingestService.recordTransformer "Parsed OTLP" "Rust" - icegate.ingestService.recordTransformer -> icegate.queueLib.queueWriter "RecordBatch" "Rust API" + icegate.ingestService.otlpHttpHandler -> icegate.ingestService.recordTransformer "Decoded OTLP request" "Rust" + icegate.ingestService.otlpGrpcHandler -> icegate.ingestService.recordTransformer "Decoded OTLP request" "Rust" + icegate.ingestService.recordTransformer -> icegate.ingestService.walWriter "RecordBatch per signal" "Rust" + icegate.ingestService.walWriter -> icegate.queueLib.queueChannel "Submits sorted row groups" "Rust API" + icegate.ingestService.shiftPlanner -> icegate.queueLib.queueReader "Lists WAL segments and their bounds" "Rust API" + icegate.ingestService.shiftPlanner -> icegate.jobStore "Persists plan and shift tasks" "S3" + icegate.ingestService.shiftPlanner -> icegate.ingestService.shiftExecutor "Shift tasks" "Rust" + icegate.ingestService.shiftExecutor -> icegate.queueLib.queueReader "Reads sorted row groups" "Rust API" + icegate.ingestService.shiftExecutor -> icegate.icebergStorage "Writes Iceberg data files" "S3" + icegate.ingestService.shiftExecutor -> icegate.ingestService.commitRunner "Written data files" "Rust" + icegate.ingestService.commitRunner -> icegate.s3Catalog.catalogService "Commits a snapshot with the WAL offset" "Rust API" # Component relationships - Queue Library - icegate.queueLib.queueWriter -> icegate.queueStorage "Writes segments" "S3" - icegate.queueLib.queueReader -> icegate.queueStorage "Reads segments" "S3" + icegate.queueLib.queueChannel -> icegate.queueLib.queueAccumulator "Write requests" "Rust" + icegate.queueLib.queueAccumulator -> icegate.queueLib.queueWriter "Flushed batches" "Rust" + icegate.queueLib.queueWriter -> icegate.queueStorage "Writes Parquet segments" "S3" + icegate.queueLib.queueReader -> icegate.queueStorage "Reads Parquet segments" "S3" # Component relationships - Query Service - icegate.queryService.lokiApi -> icegate.queryService.queryEngine "LogQL query" "Rust" - icegate.queryService.prometheusApi -> icegate.queryService.queryEngine "PromQL query" "Rust" - icegate.queryService.tempoApi -> icegate.queryService.queryEngine "Trace query" "Rust" - icegate.queryService.queryEngine -> icegate.queryService.logqlParser "Parses LogQL" "Rust" - icegate.queryService.queryEngine -> icegate.queryService.logqlPlanner "Converts AST to plan" "Rust" - icegate.queryService.queryEngine -> icegate.queueStorage "Reads queue" "S3" - icegate.queryService.queryEngine -> icegate.icebergStorage "Reads Iceberg" "S3" - icegate.queryService.queryEngine -> icegate.catalogStore "Reads metadata" "REST" - icegate.queryService.queryEngine -> icegate.queueLib.queueReader "Reads segments" "Rust API" + icegate.queryService.lokiApi -> icegate.queryService.logqlEngine "LogQL query" "Rust" + icegate.queryService.tempoApi -> icegate.queryService.traceqlEngine "TraceQL query" "Rust" + icegate.queryService.flightSqlServer -> icegate.queryService.tenantCatalog "SQL statement and tenant" "Rust" + icegate.queryService.logqlEngine -> icegate.queryService.queryEngine "DataFusion plan" "Rust" + icegate.queryService.traceqlEngine -> icegate.queryService.queryEngine "DataFusion plan" "Rust" + icegate.queryService.tenantCatalog -> icegate.queryService.queryEngine "Tenant-scoped session" "Rust" + icegate.queryService.queryEngine -> icegate.queueLib.queueReader "Reads WAL segments past the committed offset" "Rust API" + icegate.queryService.queryEngine -> icegate.s3Catalog.catalogService "Loads table metadata" "Rust API" + icegate.queryService.queryEngine -> icegate.icebergStorage "Reads data files" "S3" # Component relationships - Maintain Service - icegate.maintainService.migrator -> icegate.catalogStore "Creates tables" "REST" - icegate.maintainService.migrator -> icegate.icebergStorage "Initializes storage" "S3" - icegate.maintainService.shifter -> icegate.queueLib.queueReader "Reads queue segments" "Rust API" - icegate.maintainService.shifter -> icegate.icebergStorage "Writes Iceberg data" "S3" - icegate.maintainService.shifter -> icegate.catalogStore "Commits snapshots" "REST" + icegate.maintainService.migrator -> icegate.s3Catalog.catalogService "Creates the icegate tables" "Rust API" + icegate.maintainService.migrator -> icegate.icebergStorage "Writes initial table metadata" "S3" + icegate.maintainService.dataCompactor -> icegate.icebergStorage "Rewrites small data files" "S3" + icegate.maintainService.dataCompactor -> icegate.s3Catalog.catalogService "Commits rewrite snapshots" "Rust API" + icegate.maintainService.dataCompactor -> icegate.jobStore "Persists compaction job state" "S3" + icegate.maintainService.manifestCompactor -> icegate.icebergStorage "Rewrites manifests" "S3" + icegate.maintainService.manifestCompactor -> icegate.s3Catalog.catalogService "Commits rewritten manifests" "Rust API" + icegate.maintainService.orphanGc -> icegate.s3Catalog.catalogService "Reads the referenced file set" "Rust API" + icegate.maintainService.orphanGc -> icegate.icebergStorage "Lists and deletes unreferenced objects" "S3" + icegate.maintainService.orphanGc -> icegate.jobStore "Persists GC job state" "S3" + icegate.maintainService.pricingCrawler -> pricingFeeds "Fetches rate cards" "HTTPS" + icegate.maintainService.pricingCrawler -> icegate.s3Catalog.catalogService "Appends changed rates to icegate.prices" "Rust API" + icegate.maintainService.pricingCrawler -> icegate.jobStore "Persists crawler job state" "S3" + + # Component relationships - S3 Catalog + icegate.s3Catalog.catalogRestApi -> icegate.s3Catalog.catalogService "Catalog operations" "Rust" + icegate.s3Catalog.catalogService -> icegate.s3Catalog.catalogCache "Loads and saves catalog state" "Rust" + icegate.s3Catalog.catalogCache -> icegate.s3Catalog.catalogStorage "Conditional reads, CAS writes" "Rust" + icegate.s3Catalog.catalogStorage -> icegate.catalogStore "root.json and table metadata" "S3" } views { @@ -127,6 +210,82 @@ workspace "IceGate" "Observability Data Lake Engine" { autoLayout } + component icegate.s3Catalog "CatalogComponents" "S3 Catalog components" { + include * + autoLayout + } + + # Process views. Each renders as a UML sequence diagram rather than the + # default numbered box layout — `plantuml.sequenceDiagram` is what + # switches the exporter over, and the box layout turns into unreadable + # long-arc spaghetti once a flow passes ten steps. `autoLayout` stays as + # the fallback for anyone who turns the property off. + # A dynamic view may reuse a model relationship with a step-specific + # description, but the relationship itself must already exist in the + # model — the DSL fails the build otherwise. + dynamic icegate.ingestService "IngestionFlow" "Ingestion: from an OTLP request to a committed Iceberg snapshot" { + otelCollector -> icegate.ingestService.otlpHttpHandler "Posts an OTLP export request" + icegate.ingestService.otlpHttpHandler -> icegate.ingestService.recordTransformer "Decoded resource/scope/record tree" + icegate.ingestService.recordTransformer -> icegate.ingestService.walWriter "One Arrow RecordBatch per signal" + icegate.ingestService.walWriter -> icegate.queueLib.queueChannel "Row groups sorted by the table sort order" + icegate.queueLib.queueChannel -> icegate.queueLib.queueAccumulator "Write request, or a retryable 429 when full" + icegate.queueLib.queueAccumulator -> icegate.queueLib.queueWriter "Batch, at the flush size or interval" + icegate.queueLib.queueWriter -> icegate.queueStorage "Writes the segment with If-None-Match, then the request is acknowledged" + icegate.ingestService.shiftPlanner -> icegate.queueLib.queueReader "Lists segments past the last committed offset" + icegate.ingestService.shiftPlanner -> icegate.jobStore "Claims a shift task by compare-and-swap" + icegate.ingestService.shiftPlanner -> icegate.ingestService.shiftExecutor "Dispatches the shift task" + icegate.ingestService.shiftExecutor -> icegate.queueLib.queueReader "Reads the task's sorted row groups" + icegate.ingestService.shiftExecutor -> icegate.icebergStorage "Writes k-way merged Iceberg data files" + icegate.ingestService.shiftExecutor -> icegate.ingestService.commitRunner "Hands over the written data files" + icegate.ingestService.commitRunner -> icegate.s3Catalog.catalogService "Commits a snapshot recording the WAL offset" + autoLayout + properties { + "plantuml.sequenceDiagram" "true" + } + } + + dynamic icegate.queryService "QueryFlow" "Query: from a LogQL request to a merged WAL and Iceberg result" { + grafana -> icegate.queryService.lokiApi "Sends a LogQL range query with X-Scope-OrgID" + icegate.queryService.lokiApi -> icegate.queryService.logqlEngine "Raw LogQL expression" + icegate.queryService.logqlEngine -> icegate.queryService.queryEngine "Parsed AST, lowered to a DataFusion plan" + icegate.queryService.queryEngine -> icegate.s3Catalog.catalogService "Loads the current table metadata" + icegate.s3Catalog.catalogService -> icegate.s3Catalog.catalogCache "Requests the catalog root" + icegate.s3Catalog.catalogCache -> icegate.s3Catalog.catalogStorage "Conditional read; Not Modified serves the cached root" + icegate.s3Catalog.catalogStorage -> icegate.catalogStore "Reads root.json and table metadata" + icegate.queryService.queryEngine -> icegate.icebergStorage "Scans matching data files through the foyer cache" + icegate.queryService.queryEngine -> icegate.queueLib.queueReader "Reads WAL segments past the committed offset" + icegate.queueLib.queueReader -> icegate.queueStorage "Reads segments; merged with the Iceberg side" + autoLayout + properties { + "plantuml.sequenceDiagram" "true" + } + } + + # The three loops below are independent and run on their own schedules; + # the step numbers order each loop, not the loops against each other. + # Structurizr's parallel-block syntax is deliberately not used: the + # PlantUML exporters flatten it to duplicate step numbers with no + # visual grouping, which reads as a single pipeline — worse than + # naming the loop in every step description. + dynamic icegate.maintainService "MaintenanceFlow" "Maintenance: a one-shot migration, then three job loops on independent schedules" { + icegate.maintainService.migrator -> icegate.s3Catalog.catalogService "Migration (one-shot): creates the icegate tables" + icegate.maintainService.dataCompactor -> icegate.jobStore "Compaction loop: claims a task by compare-and-swap" + icegate.maintainService.dataCompactor -> icegate.icebergStorage "Compaction loop: rewrites small data files into larger sorted ones" + icegate.maintainService.dataCompactor -> icegate.s3Catalog.catalogService "Compaction loop: commits a rewrite snapshot" + icegate.maintainService.manifestCompactor -> icegate.icebergStorage "Compaction loop: repacks fragmented manifests" + icegate.maintainService.manifestCompactor -> icegate.s3Catalog.catalogService "Compaction loop: commits the rewritten manifest list" + icegate.maintainService.orphanGc -> icegate.jobStore "GC loop: claims a sweep task" + icegate.maintainService.orphanGc -> icegate.s3Catalog.catalogService "GC loop: reads the set of referenced files" + icegate.maintainService.orphanGc -> icegate.icebergStorage "GC loop: deletes unreferenced objects past the grace period" + icegate.maintainService.pricingCrawler -> icegate.jobStore "Pricing loop: claims a crawl task" + icegate.maintainService.pricingCrawler -> pricingFeeds "Pricing loop: fetches the OpenRouter and LiteLLM rate cards" + icegate.maintainService.pricingCrawler -> icegate.s3Catalog.catalogService "Pricing loop: appends changed rates to icegate.prices" + autoLayout + properties { + "plantuml.sequenceDiagram" "true" + } + } + styles { element "Software System" { background #1168bd @@ -144,6 +303,9 @@ workspace "IceGate" "Observability Data Lake Engine" { background #85bbf0 color #000000 } + element "Library" { + shape Component + } element "Database" { shape Cylinder } diff --git a/en/api-reference/loki.md b/en/api-reference/loki.md index f7ed07c..bece77e 100644 --- a/en/api-reference/loki.md +++ b/en/api-reference/loki.md @@ -1,11 +1,14 @@ --- title: Loki API Reference -description: Loki-compatible HTTP API endpoints +description: Loki-compatible HTTP API endpoints served by {{product_name}} --- # Loki API Reference -IceGate provides a Loki-compatible HTTP API for querying logs. +{{product_name}} provides a Loki®-compatible HTTP API for querying logs, served on port 3100. The endpoints +documented below are the ones implemented — this is a subset of Loki's API, not a complete +reimplementation, so anything not listed here should be assumed unimplemented. See +[Trademarks](../trademarks.md) for attribution. ## Base URL @@ -220,7 +223,7 @@ curl -G http://localhost:3100/loki/api/v1/series \ ### Explain -Get query execution plan (IceGate extension). +Get query execution plan ({{product_name}} extension). **Endpoint:** `GET /loki/api/v1/explain` @@ -269,5 +272,5 @@ All errors return a JSON response: ## Next Steps - Learn [LogQL Querying](../guides/querying.md) -- Explore the [Prometheus API](prometheus.md) +- Explore the [Prometheus API](prometheus.md) — planned, not implemented yet - See [Tempo API](tempo.md) for traces diff --git a/en/api-reference/otlp.md b/en/api-reference/otlp.md index 5c8b2e3..d0beef5 100644 --- a/en/api-reference/otlp.md +++ b/en/api-reference/otlp.md @@ -5,7 +5,7 @@ description: OpenTelemetry Protocol endpoints for data ingestion # OTLP Ingestion API -IceGate accepts observability data via the OpenTelemetry Protocol (OTLP). Both HTTP and gRPC transports are supported. +{{product_name}} accepts observability data via the OpenTelemetry Protocol (OTLP). Both HTTP and gRPC transports are supported. ## Protocols @@ -291,7 +291,7 @@ service: ## Load Testing with IceGen -[IceGen](https://github.com/icegatetech/icegen) is a high-performance OpenTelemetry log generator for testing IceGate ingestion. +[IceGen](https://github.com/icegatetech/icegen) is a high-performance OpenTelemetry log generator for testing {{product_name}} ingestion. ### Install diff --git a/en/api-reference/prometheus.md b/en/api-reference/prometheus.md index 44addf6..66be752 100644 --- a/en/api-reference/prometheus.md +++ b/en/api-reference/prometheus.md @@ -1,11 +1,22 @@ --- title: Prometheus API Reference -description: Prometheus-compatible HTTP API endpoints +description: Planned Prometheus-compatible HTTP API — not yet implemented --- # Prometheus API Reference -IceGate provides a Prometheus-compatible HTTP API for querying metrics. +{% note warning %} + +**Not implemented yet.** The routes below are mounted on port 9090, but every one of them returns +`501 Not Implemented`; only `/-/ready` responds. This page documents the *planned* surface so that +integrators can see where it is heading — do not build against it yet. + +For metrics queries today, use [Arrow Flight SQL](../guides/querying.md) against the same data. + +{% endnote %} + +This is the planned shape of {{product_name}}'s Prometheus®-compatible HTTP API for querying metrics. See +[Trademarks](../trademarks.md) for attribution. ## Base URL @@ -25,7 +36,7 @@ X-Scope-OrgID: my-tenant {% note warning %} -The Prometheus API is currently under development. Basic endpoints are available but full PromQL support is planned for future releases. +None of these endpoints are implemented. Every one returns `501 Not Implemented`, including the metadata endpoints; only `/-/ready` responds. PromQL is not parsed at all yet. {% endnote %} @@ -107,7 +118,7 @@ curl -G http://localhost:9090/api/v1/series \ ## Metric Types -IceGate stores all OpenTelemetry metric types: +{{product_name}} stores all OpenTelemetry metric types: | Metric Type | Description | |-------------|-------------| diff --git a/en/api-reference/tempo.md b/en/api-reference/tempo.md index 77cfb13..65bd31c 100644 --- a/en/api-reference/tempo.md +++ b/en/api-reference/tempo.md @@ -1,11 +1,16 @@ --- title: Tempo API Reference -description: Tempo-compatible HTTP API endpoints +description: Tempo-compatible HTTP API endpoints served by {{product_name}} --- # Tempo API Reference -IceGate provides a Tempo-compatible HTTP API for querying distributed traces. +{{product_name}} provides a Tempo®-compatible HTTP API for querying distributed traces, served on port 3200. +The endpoints documented below are the ones implemented — this is a subset of Tempo's API, not a +complete reimplementation, so anything not listed here should be assumed unimplemented. TraceQL is +supported for `/api/search`; TraceQL features that are not yet implemented return +`501 Not Implemented` rather than silently returning wrong results. See +[Trademarks](../trademarks.md) for attribution. ## Base URL @@ -25,7 +30,7 @@ X-Scope-OrgID: my-tenant {% note warning %} -The Tempo API is currently under development. Basic trace retrieval is available but TraceQL support is planned for future releases. +The Tempo API implements a subset of Tempo's HTTP read API. Trace retrieval and `/api/search` are available, and TraceQL is supported for search — TraceQL features that are not yet implemented return `501 Not Implemented` rather than silently returning wrong results. {% endnote %} @@ -136,7 +141,7 @@ curl http://localhost:3200/api/search/tag/service.name/values \ ## Span Data Model -Spans stored in IceGate include: +Spans stored in {{product_name}} include: | Field | Type | Description | |-------|------|-------------| @@ -157,4 +162,4 @@ Spans stored in IceGate include: - Learn about [Data Ingestion](../guides/ingestion.md) - Explore the [Loki API](loki.md) for logs -- See [Prometheus API](prometheus.md) for metrics +- See [Prometheus API](prometheus.md) for the planned metrics API (not implemented yet) diff --git a/en/architecture/data-model.md b/en/architecture/data-model.md index 6291445..c50767d 100644 --- a/en/architecture/data-model.md +++ b/en/architecture/data-model.md @@ -1,11 +1,11 @@ --- title: Data Model -description: IceGate Iceberg table schemas for observability data +description: {{product_name}} Iceberg table schemas for observability data --- # Data Model -IceGate stores observability data in four Apache Iceberg tables: logs, spans, events, and metrics. +{{product_name}} stores observability data in five tenant-scoped Apache Iceberg tables — logs, spans, events, metrics, and operations — plus one global reference table, prices. ## Table Overview @@ -15,12 +15,14 @@ IceGate stores observability data in four Apache Iceberg tables: logs, spans, ev | `spans` | Distributed trace spans | Request tracing | | `events` | Semantic events | Business events, alerts | | `metrics` | All metric types | Performance monitoring | +| `operations` | LLM and agent operations | Token usage, cost, prompt and completion capture | +| `prices` | Global LLM rate card (no `tenant_id`) | Reference rates for costing `operations` | ## Common Design Patterns ### Multi-Tenancy -All tables use identity partitioning on `tenant_id`: +The five tenant-scoped tables use identity partitioning on `tenant_id`. `prices` is reference data shared by every tenant, so it carries no `tenant_id` and is partitioned differently: ```sql partitioning = ARRAY['tenant_id', 'account_id', 'day(timestamp)'] @@ -264,6 +266,125 @@ CREATE TABLE metrics ( | `exponential_histogram` | `count`, `sum`, `scale`, `zero_count`, `positive_*`, `negative_*` | | `summary` | `count`, `sum`, `quantile_values` | +## Operations Table + +LLM and agent operations, following the OpenTelemetry generative-AI semantic conventions. + +```sql +CREATE TABLE operations ( + tenant_id VARCHAR NOT NULL, + conversation_id VARCHAR, + + -- identity + trace_id VARBINARY NOT NULL, + span_id VARBINARY NOT NULL, + parent_span_id VARBINARY, + service_name VARCHAR, + scope_name VARCHAR, + scope_version VARCHAR, + + -- timing + timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + end_timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + duration_micros BIGINT NOT NULL, + ingested_timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + + operation_name VARCHAR NOT NULL, + + -- provider and model + provider_name VARCHAR, + request_model VARCHAR, + response_model VARCHAR, + response_id VARCHAR, + + -- sampling parameters + temperature DOUBLE, + top_p DOUBLE, + top_k BIGINT, + max_tokens BIGINT, + frequency_penalty DOUBLE, + presence_penalty DOUBLE, + seed BIGINT, + stream BOOLEAN, + choice_count BIGINT, + output_type VARCHAR, + reasoning_effort VARCHAR, + + time_to_first_chunk_ms BIGINT, + + -- token usage + input_tokens BIGINT, + output_tokens BIGINT, + total_tokens BIGINT, + reasoning_tokens BIGINT, + cache_creation_input_tokens BIGINT, + cache_read_input_tokens BIGINT, + + user_id VARCHAR, + + -- tool calls + tool_name VARCHAR, + tool_call_id VARCHAR, + tool_type VARCHAR, + tool_description VARCHAR, + + data_source_id VARCHAR, + embedding_dimensions INTEGER, + + -- server and status + server_address VARCHAR, + server_port INTEGER, + status_code INTEGER, + status_message VARCHAR, + error_type VARCHAR, + + -- agent and workflow + agent_id VARCHAR, + agent_name VARCHAR, + agent_version VARCHAR, + agent_description VARCHAR, + workflow_name VARCHAR, + + -- content, JSON-encoded + input_messages VARCHAR, + output_messages VARCHAR, + system_instructions VARCHAR, + tool_definitions VARCHAR, + tool_call_arguments VARCHAR, + tool_call_result VARCHAR, + + stop_sequences ARRAY(VARCHAR), + finish_reasons ARRAY(VARCHAR), + encoding_formats ARRAY(VARCHAR) +) +``` + +**Partitioning:** `tenant_id` (identity), `day(timestamp)` + +**Sorting:** `trace_id`, `timestamp DESC` — clusters a trace's operations together, recent first + +The six `VARCHAR` content columns (`input_messages`, `output_messages`, `system_instructions`, `tool_definitions`, `tool_call_arguments`, `tool_call_result`) hold JSON-encoded payloads rather than parsed structures, so prompt and completion shapes can vary per provider without a schema change. + +## Prices Table + +A global LLM rate card, populated by the Maintain service's pricing crawler from the OpenRouter and LiteLLM feeds. + +Unlike the five telemetry tables it carries **no `tenant_id`** — rates are reference data, identical for every tenant. It is an append-only observation log: a row is written only when a rate first differs from the previous one for its key, and `valid_to` is derived at query time. + +**Key:** `(provider, model, service_tier, region, min_input_tokens, valid_from)` + +Context tiers and service tiers live in the key rather than in extra columns, so the rate columns stay flat as the card grows. Rate columns are `DECIMAL(38, 10)` rather than floating point — money has to be exact, and binary `f64` cannot represent a value like `0.075` or sum it without drift. + +### Joining Prices to Operations + +The query engine exposes a derived view, `prices_effective`, which adds `valid_to` — the next revision's `valid_from` for the same key, `NULL` for the row currently in effect. It is a DataFusion object, so the Loki, Tempo, and Flight SQL paths see it; Trino reads the Iceberg catalog directly and does not, which is why the raw table stays self-sufficient. + +{% note warning %} + +{{product_name}} does not compute cost, and `operations` does not carry the full pricing key. It records `provider_name` and `request_model`, which line up with `prices.provider` and `prices.model`, but nothing for `service_tier`, `region`, or `min_input_tokens`. A cost query has to supply those three from deployment knowledge — a fixed tier and region per account, say. Treat such a join as an estimate parameterised by your own assumptions, not a derivation the schema guarantees. + +{% endnote %} + ## Query Examples ### Logs Query diff --git a/en/architecture/overview.md b/en/architecture/overview.md index 2bcc190..b9bce73 100644 --- a/en/architecture/overview.md +++ b/en/architecture/overview.md @@ -1,11 +1,11 @@ --- title: Architecture Overview -description: IceGate system architecture and components +description: {{product_name}} system architecture and components --- # Architecture Overview -IceGate is an observability data lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables with DataFusion as the query engine. +{{product_name}} is an observability data lake engine that stores logs, traces, metrics, events, and LLM operations in Apache Iceberg tables with DataFusion as the query engine. ## Design Principles @@ -43,13 +43,18 @@ The Write-Ahead Log (WAL) stores data as Parquet files organized for compatibili **Purpose:** Execute queries against logs, traces, metrics, and events - **Engine:** Apache DataFusion + Apache Arrow -- **APIs:** Loki (3100), Prometheus (9090), Tempo (3200) -- **Query Languages:** LogQL, PromQL (planned), TraceQL (planned) +- **APIs:** Loki (3100), Tempo (3200), Arrow Flight SQL (8815); Prometheus (9090) serves routes but its handlers still return `501 Not Implemented` +- **Query Languages:** LogQL, TraceQL, SQL; PromQL planned +- **Multi-tenancy:** Tenant taken from the `X-Scope-OrgID` header, or the `x-scope-orgid` gRPC metadata for Flight SQL + +Arrow Flight SQL is strictly read-only — DDL and DML are rejected — and enforces `tenant_id` at the row level on every scan, so JDBC, ODBC, and ADBC clients query `iceberg.icegate.` with no {{product_name}}-specific client code. The query service reads from both: - **WAL**: For real-time data (seconds-old) -- **Iceberg Tables**: For historical data (compacted) +- **Iceberg Tables**: For historical data (shifted and compacted) + +The boundary between the two is the WAL offset recorded in the Iceberg snapshot summary, so a row is read from exactly one side and never counted twice. ### Maintain Service @@ -57,10 +62,25 @@ The query service reads from both: **Purpose:** Data lifecycle and optimization operations -- **Compaction:** Merge small WAL files into optimized Iceberg tables -- **TTL:** Expire and delete old data based on retention policies -- **Optimization:** Rewrite data files for better query performance -- **Cleanup:** Remove orphaned files and expired snapshots +- **Schema migration:** Create the Iceberg tables (`maintain migrate create`) +- **Data compaction:** Rewrite small Parquet data files into fewer, larger sorted ones +- **Manifest compaction:** Repack fragmented Iceberg manifests +- **Orphan GC:** Delete objects the current table metadata no longer references, once past a grace period +- **Pricing crawler:** Crawl LLM rate cards from external feeds into the global `icegate.prices` table + +Compaction, GC, and the pricing crawler each run as jobs whose state lives in object storage, under their own job-state prefix. + +### Catalog + +![Catalog Components](../../assets/c4/structurizr-CatalogComponents.png) + +**Purpose:** Organize the data lake with ACID transactions, without a dedicated OLTP database + +- **Default backend:** {{product_name}}'s own S3 catalog — catalog state is a `root.json` object updated by compare-and-swap +- **Alternative backends:** REST (Nessie), AWS S3 Tables, AWS Glue +- **Deployment:** Linked into Ingest, Query, and Maintain by default; optionally deployed standalone as an Iceberg REST server on port 8181 + +A conditional read keeps the cached catalog root fresh; table metadata is immutable per location, so it is cached unconditionally in an LRU. ### Alert Service (Planned) @@ -79,15 +99,21 @@ The query service reads from both: | Memory Format | Apache Arrow 57.0 | Zero-copy data processing | | Storage Format | Apache Parquet 57.0 | Columnar storage with ZSTD compression | | Ingestion | OpenTelemetry 0.31 | Standard observability protocol (gRPC + HTTP) | -| Catalog | Nessie, AWS S3 Tables, AWS Glue | Iceberg REST catalog backends | -| Job Manager | icegate-jobmanager | S3-based shift job state management | +| SQL Interface | Arrow Flight SQL 57.0 | Read-only SQL for JDBC, ODBC, and ADBC clients | +| Catalog | S3 catalog (default), Nessie, AWS S3 Tables, AWS Glue | Iceberg catalog backends; the default keeps state in object storage | +| Object Storage | RustFS, or any S3-compatible store | WAL segments, Iceberg data, catalog state, job state | +| Job Manager | jobmanager (separate repository) | S3-based job state for shift, compaction, GC, and pricing | | Caching | foyer 0.22 | Hybrid memory + disk cache for S3 reads | -| Language | Rust 1.92+ (2024 edition) | Memory-safe, high-performance runtime | +| Language | Rust {{rust_version}}+ (2024 edition) | Memory-safe, high-performance runtime | ## Data Flow ### Ingestion Flow +![Ingestion Sequence](../../assets/c4/structurizr-IngestionFlow.png) + +Steps 1-7 are the write path, acknowledged once the WAL segment lands. Steps 8-14 are shift, which runs independently of the request. + 1. Client sends OTLP data to Ingest service 2. Ingest validates and transforms data 3. Data written to WAL as Parquet files @@ -95,6 +121,8 @@ The query service reads from both: ### Query Flow +![Query Sequence](../../assets/c4/structurizr-QueryFlow.png) + 1. Client sends query to Query service 2. Query parsed and planned by DataFusion 3. Data read from Iceberg tables and/or WAL @@ -106,8 +134,17 @@ The query service reads from both: 2. Groups segments into shift tasks 3. Reads WAL files in parallel, merges and re-partitions data 4. Writes optimized Iceberg data files -5. Commits new snapshot to catalog -6. Deletes processed WAL segments +5. Commits a new snapshot to the catalog, recording the last committed WAL offset in the snapshot summary + +Shift never deletes WAL segments. They are reclaimed by an object lifecycle rule on the queue bucket, and the offset in the snapshot summary is what lets shift resume where it left off. + +That makes the lifecycle expiration a durability parameter, not housekeeping: a segment has to outlive the commit that covers it. If shift is delayed or failing when the rule fires, segments whose offsets were never committed are deleted and the data is gone. See [Data Retention](../guides/data-retention.md) for sizing. + +### Maintenance Flow + +![Maintenance Sequence](../../assets/c4/structurizr-MaintenanceFlow.png) + +Migration is a one-shot job. Compaction, orphan GC, and the pricing crawler are independent loops on their own schedules — the step numbers order each loop, not the loops against each other. Each claims work under its own job-state prefix, so the loops never fight over task ownership. They still share the tables underneath — compaction commits rewrite snapshots while GC deletes unreferenced objects — which is why GC only removes files older than its grace period and commits use optimistic concurrency, retrying on conflict. ## Scalability @@ -115,7 +152,7 @@ The query service reads from both: - **Ingest:** Scale replicas for higher throughput - **Query:** Scale replicas for concurrent queries -- **Maintain:** Single instance (leader election) +- **Maintain:** Scale replicas for more rewrite throughput — workers share job state in object storage with compare-and-swap and commit with optimistic concurrency, so parallel instances are safe. Prefer raising in-process worker count first; returns taper as replicas grow, since all workers on a table contend on one job-state object. ### Storage Scaling diff --git a/en/cookbooks/centralized-logging.md b/en/cookbooks/centralized-logging.md index e98c2f6..f086304 100644 --- a/en/cookbooks/centralized-logging.md +++ b/en/cookbooks/centralized-logging.md @@ -1,6 +1,6 @@ --- title: Centralized Logging for Microservices -description: Set up centralized log collection from microservices into IceGate +description: Set up centralized log collection from microservices into {{product_name}} --- # Centralized Logging for Microservices diff --git a/en/cookbooks/observability-correlation.md b/en/cookbooks/observability-correlation.md index e0fce11..4278822 100644 --- a/en/cookbooks/observability-correlation.md +++ b/en/cookbooks/observability-correlation.md @@ -1,6 +1,6 @@ --- title: Cross-Signal Correlation -description: Correlate logs, traces, and metrics across observability pillars in IceGate +description: Correlate logs, traces, and metrics across observability pillars in {{product_name}} --- # Cross-Signal Correlation @@ -9,7 +9,7 @@ This cookbook shows how to correlate data across logs, traces, and metrics in {{ {% note warning %} -This guide uses the Loki API (fully implemented) and the Tempo API (basic trace retrieval and search available; TraceQL planned). The Prometheus API is under development — use LogQL metric queries as an alternative for log-based metrics. +This guide uses the Loki API (fully implemented) and the Tempo API (retrieval and search available; TraceQL supported, unimplemented features return `501`). The Prometheus API is NOT implemented — every route returns `501` except `/-/ready`; use LogQL metric queries as an alternative for log-based metrics. {% endnote %} diff --git a/en/cookbooks/traces-end-to-end.md b/en/cookbooks/traces-end-to-end.md index 9888667..11747df 100644 --- a/en/cookbooks/traces-end-to-end.md +++ b/en/cookbooks/traces-end-to-end.md @@ -1,6 +1,6 @@ --- title: End-to-End Distributed Tracing -description: Instrument services, send traces to IceGate, and query them via the Tempo API +description: Instrument services, send traces to {{product_name}}, and query them via the Tempo API --- # End-to-End Distributed Tracing @@ -9,7 +9,7 @@ This cookbook walks through instrumenting services with OpenTelemetry, sending t {% note warning %} -The Tempo API is currently under development. Basic trace retrieval by ID and search by tags are available. TraceQL query language support is planned for future releases. +The Tempo-compatible API implements a subset of Tempo's API: trace retrieval by ID, TraceQL search via `/api/search`, and tag discovery (v1 and v2). TraceQL features that are not yet implemented return `501 Not Implemented` rather than a wrong result. See the [Tempo API reference](../api-reference/tempo.md) for the endpoints served today. {% endnote %} @@ -288,7 +288,7 @@ datasources: Then in Grafana: -1. Go to **Explore** > select **IceGate Traces** +1. Go to **Explore** > select **{{product_name}} Traces** 2. Enter a service name in the search field 3. Click a trace to view its span waterfall diagram 4. Inspect individual spans for attributes and timing diff --git a/en/development/building.md b/en/development/building.md index 5b45a88..5a75365 100644 --- a/en/development/building.md +++ b/en/development/building.md @@ -1,17 +1,17 @@ --- title: Building from Source -description: Build IceGate from source code +description: Build {{product_name}} from source code --- # Building from Source -This guide covers building IceGate from source for development and production. +This guide covers building {{product_name}} from source for development and production. ## Prerequisites ### Required -- **Rust** >= 1.92.0 (for Rust 2024 edition support) +- **Rust** >= {{rust_version}} (for Rust 2024 edition support) - **Cargo** (included with Rust) - **Git** @@ -96,17 +96,17 @@ debug = true ## Workspace Structure -IceGate uses a Cargo workspace: +{{product_name}} uses a Cargo workspace: ```text Cargo.toml (workspace) ├── crates/ │ ├── icegate-common/Cargo.toml +│ ├── icegate-catalog-s3/Cargo.toml │ ├── icegate-queue/Cargo.toml │ ├── icegate-query/Cargo.toml │ ├── icegate-ingest/Cargo.toml -│ ├── icegate-maintain/Cargo.toml -│ └── icegate-jobmanager/Cargo.toml +│ └── icegate-maintain/Cargo.toml ``` Build individual crates: @@ -194,7 +194,7 @@ make ci ### Compilation Errors -1. Ensure Rust version >= 1.92.0: +1. Ensure Rust version >= {{rust_version}}: ```bash rustup update diff --git a/en/development/contributing.md b/en/development/contributing.md index 7bae867..4677884 100644 --- a/en/development/contributing.md +++ b/en/development/contributing.md @@ -1,11 +1,11 @@ --- title: Contributing -description: How to contribute to IceGate development +description: How to contribute to {{product_name}} development --- # Contributing -We welcome contributions to IceGate! This guide explains how to get started. +We welcome contributions to {{product_name}}! This guide explains how to get started. ## Ways to Contribute @@ -19,7 +19,7 @@ We welcome contributions to IceGate! This guide explains how to get started. ### Prerequisites -- Rust >= 1.92.0 +- Rust >= {{rust_version}} - Docker and Docker Compose - Git @@ -100,13 +100,15 @@ This runs: ``` crates/ ├── icegate-common/ # Shared infrastructure (catalog, storage, metrics, tracing) +├── icegate-catalog-s3/ # S3-backed Iceberg catalog (default) and its REST server ├── icegate-queue/ # Write-ahead log (Parquet on object storage) -├── icegate-query/ # Query service (Loki/Prometheus/Tempo APIs) -├── icegate-ingest/ # Ingest service (OTLP HTTP/gRPC) -├── icegate-maintain/ # Maintenance operations (schema migration) -└── icegate-jobmanager/ # Shift job state management +├── icegate-query/ # Query service (Loki/Tempo/Flight SQL; Prometheus routes 501) +├── icegate-ingest/ # Ingest service (OTLP HTTP/gRPC, WAL, shift) +└── icegate-maintain/ # Migration, compaction, orphan GC, pricing crawler ``` +The job/task framework is not a workspace crate: it lives in `icegatetech/jobmanager` and is consumed as a git-pinned dependency. + See [Architecture](../architecture/overview.md) for details. ## Pull Request Guidelines diff --git a/en/development/patterns.md b/en/development/patterns.md index 6e74141..80883a8 100644 --- a/en/development/patterns.md +++ b/en/development/patterns.md @@ -1,11 +1,11 @@ --- title: Development Patterns -description: Standard patterns used across the IceGate codebase +description: Standard patterns used across the {{product_name}} codebase --- # Development Patterns -This document defines the standard patterns used across the IceGate codebase for config, errors, HTTP routes, handlers, and services. +This document defines the standard patterns used across the {{product_name}} codebase for config, errors, HTTP routes, handlers, and services. ## 1. Config Pattern diff --git a/en/development/setup.md b/en/development/setup.md index 2224ba2..a3bb35d 100644 --- a/en/development/setup.md +++ b/en/development/setup.md @@ -1,15 +1,15 @@ --- title: Development Setup -description: Set up a local IceGate development environment +description: Set up a local {{product_name}} development environment --- # Development Setup -This guide covers setting up a local IceGate development environment for contributing code, running tests, and debugging. +This guide covers setting up a local {{product_name}} development environment for contributing code, running tests, and debugging. ## Prerequisites -- **Rust** >= 1.92.0 (Rust 2024 edition) +- **Rust** >= {{rust_version}} (Rust 2024 edition) - **Docker** (for building container images) - **Git** - A local Kubernetes cluster (for Skaffold) @@ -58,7 +58,7 @@ You need a local Kubernetes cluster. Options: ### Run with Skaffold ```bash -# Default profile (local k8s with MinIO + Nessie) +# Default profile (local k8s with RustFS + the built-in S3 catalog) skaffold dev # OrbStack profile @@ -75,7 +75,7 @@ skaffold dev -p k3s-external-s3 Skaffold uses Kustomize overlays that compose multiple Helm charts: -**IceGate namespace (`icegate`):** +**{{product_name}} namespace (`icegate`):** | Component | Description | |-----------|-------------| @@ -87,22 +87,21 @@ Skaffold uses Kustomize overlays that compose multiple Helm charts: | Component | Description | |-----------|-------------| -| MinIO | S3-compatible storage with buckets: `warehouse`, `queue`, `jobs` | -| Nessie | Iceberg REST catalog with RocksDB persistence | +| RustFS | S3-compatible storage with buckets: `warehouse`, `queue`, `jobs` | **Observability namespace (`observability`):** | Component | Description | |-----------|-------------| | Prometheus | Metrics collection (kube-prometheus-stack) | -| Grafana | Dashboards with pre-built IceGate Ingest and Query panels | -| Jaeger | Distributed tracing for IceGate services | +| Grafana | Dashboards with pre-built {{product_name}} Ingest and Query panels | +| Jaeger | Distributed tracing for {{product_name}} services | ### Skaffold Profiles | Profile | Overlay | Use Case | |---------|---------|----------| -| (default) | `skaffold` | Local development with MinIO + Nessie | +| (default) | `skaffold` | Local development with RustFS + the built-in S3 catalog | | `orbstack` | `orbstack` | OrbStack Kubernetes (macOS) | | `aws-glue` | `aws-glue` | AWS Glue catalog (pushes images) | | `k3s-external-s3` | `external-s3` | External S3 + Nessie (pushes images) | @@ -155,10 +154,9 @@ make down | Service | Port | Description | |---------|------|-------------| -| MinIO | 9000, 9001 | S3-compatible storage + console | -| Nessie | 19120 | Iceberg REST catalog | +| RustFS | 9000, 9001 | S3-compatible storage + console | | Ingest | 4317, 4318 | OTLP gRPC and HTTP receivers | -| Query | 3100, 9090, 3200 | Loki, Prometheus, Tempo APIs | +| Query | 3100, 9090, 3200, 8815 | Loki, Tempo, Arrow Flight SQL APIs; Prometheus routes return 501 except `/-/ready` | | Grafana | 3000 | Dashboards | Docker Compose profiles add optional services: @@ -167,7 +165,7 @@ Docker Compose profiles add optional services: |---------|----------| | `load` | otelgen (log load generator) | | `monitoring` | Jaeger (16686), Prometheus (9092), node-exporter, cAdvisor | -| `analytics` | Trino SQL engine (8082) | +| `analytics` | Nessie (19120) and Trino SQL engine (8082) | ### Docker Build @@ -188,11 +186,11 @@ docker build -t icegate/query:dev \ ## Environment Variables -For local development with MinIO: +For local development with RustFS: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/en/faq.md b/en/faq.md index d73ab34..9389017 100644 --- a/en/faq.md +++ b/en/faq.md @@ -1,36 +1,36 @@ --- title: FAQ -description: Frequently asked questions about IceGate +description: Frequently asked questions about {{product_name}} --- # Frequently Asked Questions ## General -### What is IceGate? +### What is {{product_name}}? -IceGate is an observability data lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables. It provides Loki, Prometheus, and Tempo-compatible APIs for querying. +{{product_name}} is an observability data lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables. It provides Loki- and Tempo-compatible APIs for querying, plus Arrow Flight SQL. A Prometheus-compatible API is [planned but not implemented yet](api-reference/prometheus.md) — see the [Loki](api-reference/loki.md) and [Tempo](api-reference/tempo.md) references for what is served today. -### What makes IceGate different? +### What makes {{product_name}} different? - **Open Standards**: Built entirely on Apache Iceberg, Arrow, Parquet, and OpenTelemetry -- **Cost-Effective**: Uses object storage (S3/MinIO) instead of expensive databases +- **Cost-Effective**: Uses object storage (S3 or RustFS) instead of expensive databases - **ACID Transactions**: Full transaction support without a dedicated OLTP database - **Compute-Storage Separation**: Scale processing and storage independently ### What is the current status? -IceGate is in **alpha** development. Core features work, but APIs may change. +{{product_name}} is in **alpha** development. Core features work, but APIs may change. -### What license is IceGate under? +### What license is {{product_name}} under? -Apache License 2.0. +{{license}}. ## Getting Started ### What are the minimum requirements? -- Rust 1.92.0+ +- Rust {{rust_version}}+ - Docker (for development environment) - S3-compatible object storage @@ -40,7 +40,7 @@ See the [Installation](getting-started/installation.md) guide and [Quick Start]( ### Do I need Kubernetes? -No. IceGate can run with Docker Compose for smaller deployments. Kubernetes is recommended for production. +No. {{product_name}} can run with Docker Compose for smaller deployments. Kubernetes is recommended for production. ## Data and Storage @@ -97,7 +97,7 @@ Not yet implemented: ### Can I use Grafana? -Yes! IceGate provides Loki-compatible APIs that work with Grafana's Loki data source. +Yes. {{product_name}} provides Loki-compatible APIs that work with Grafana's Loki data source, and Tempo-compatible APIs for the Tempo data source. Both implement a subset of the upstream API, so check the [Loki](api-reference/loki.md) and [Tempo](api-reference/tempo.md) references if a panel depends on a specific endpoint. The Prometheus data source will not work yet — that API is planned. ## Multi-Tenancy @@ -111,11 +111,11 @@ Yes. Queries only access data for the tenant specified in the header. Data is ph ### Can I have multiple tenants in one deployment? -Yes. IceGate is designed as a multi-tenant system. +Yes. {{product_name}} is designed as a multi-tenant system. ## Performance -### How does IceGate scale? +### How does {{product_name}} scale? - **Ingest**: Horizontal scaling for write throughput - **Query**: Horizontal scaling for concurrent queries @@ -137,7 +137,7 @@ The Ingest service's built-in shift process automatically compacts WAL files int ## Operations -### How do I monitor IceGate? +### How do I monitor {{product_name}}? - Prometheus metrics exposed on each service - Health check endpoints @@ -207,4 +207,4 @@ See [Contributing Guide](development/contributing.md). We welcome: ### Where do I report issues? -GitHub Issues: [https://github.com/icegatetech/icegate/issues](https://github.com/icegatetech/icegate/issues) +GitHub Issues: [{{repo_url}}/issues]({{repo_url}}/issues) diff --git a/en/getting-started/configuration.md b/en/getting-started/configuration.md index ecca91a..42f5484 100644 --- a/en/getting-started/configuration.md +++ b/en/getting-started/configuration.md @@ -1,6 +1,6 @@ --- title: Configuration -description: Configure IceGate components +description: Configure {{product_name}} components --- # Configuration @@ -42,24 +42,48 @@ The `catalog` section configures the Apache Iceberg catalog. It is shared by all ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 ``` ### Catalog Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| -| `backend` | enum | No | `memory` | Catalog backend type (see below) | +| `backend` | enum | Yes | — | Catalog backend type (see below). No default — the field is required | | `warehouse` | string | Yes | — | Warehouse location (e.g., `s3://warehouse/`) | | `properties` | map | No | `{}` | Additional catalog-specific properties | | `cache` | object | No | — | IO cache configuration (see [Cache Configuration](#cache-configuration)) | ### Catalog Backends +#### S3 Catalog (Default) + +{{product_name}}'s own catalog. Catalog state is a `root.json` object in object storage, updated by compare-and-swap, so no external catalog service is required: + +```yaml +catalog: + backend: !s3 + warehouse: catalog + warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 +``` + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `warehouse` (inside `!s3`) | string | Yes | Object-storage key prefix holding the catalog state | +| `properties.bucket` | string | Yes | Bucket holding the catalog state | +| `properties.region` | string | Yes | Region for the catalog's S3 client | +| `properties.endpoint` | string | No | Custom endpoint for S3-compatible storage. Omit for real AWS S3 | + #### REST Catalog (Nessie) ```yaml @@ -115,9 +139,13 @@ The optional `cache` section enables a foyer hybrid cache (memory + disk) to red ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -141,21 +169,21 @@ catalog: The `storage` section configures the object storage backend. Shared by all services. -### S3 / S3-Compatible (MinIO) +### S3 / S3-Compatible (RustFS) ```yaml storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `bucket` | string | Yes | — | S3 bucket name | | `region` | string | Yes | — | AWS region | -| `endpoint` | string | No | — | Custom endpoint URL for S3-compatible storage (MinIO, etc.) | +| `endpoint` | string | No | — | Custom endpoint URL for S3-compatible storage (RustFS, etc.) | ### Local Filesystem @@ -184,17 +212,19 @@ Full reference for the Ingest service (`ingest run -c ingest.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 queue: common: @@ -223,7 +253,7 @@ shift: poll_interval_ms: 1000 iteration_interval_millisecs: 30000 storage: - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 bucket: jobs prefix: shifter region: us-east-1 @@ -322,11 +352,13 @@ Full reference for the Query service (`query run -c query.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -336,7 +368,7 @@ storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 engine: batch_size: 8192 @@ -406,7 +438,7 @@ When `engine.wal_query_enabled` is `true`, the query service reads both committe | `loki.enabled` | bool | `true` | Enable Loki-compatible log query API | | `loki.host` | string | `0.0.0.0` | Bind address | | `loki.port` | integer | `3100` | Loki API port | -| `prometheus.enabled` | bool | `true` | Enable Prometheus-compatible metrics API | +| `prometheus.enabled` | bool | `true` | Serve the Prometheus query API. Routes are registered, but every handler except `/-/ready` returns `501 Not Implemented` — PromQL is not implemented yet. This is not the metrics endpoint; that is the `metrics` block on port 9091 | | `prometheus.host` | string | `0.0.0.0` | Bind address | | `prometheus.port` | integer | `9090` | Prometheus API port | | `tempo.enabled` | bool | `true` | Enable Tempo-compatible trace API | @@ -419,17 +451,19 @@ The Maintain service only requires catalog and storage configuration: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` ### Maintain CLI @@ -499,8 +533,8 @@ make run-analytics-release Environment variables for local development: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/en/getting-started/installation.md b/en/getting-started/installation.md index 008828e..c0d09f9 100644 --- a/en/getting-started/installation.md +++ b/en/getting-started/installation.md @@ -1,21 +1,21 @@ --- title: Installation -description: Install IceGate on Kubernetes with Helm +description: Install {{product_name}} on Kubernetes with Helm --- # Installation -IceGate is deployed on Kubernetes using Helm charts, with Kustomize overlays for environment-specific customizations. +{{product_name}} is deployed on Kubernetes using Helm charts, with Kustomize overlays for environment-specific customizations. ## Prerequisites - **Kubernetes** >= 1.28 with **Helm 3** -- **Object Storage:** AWS S3 or S3-compatible (MinIO) -- **Iceberg Catalog:** Nessie (REST), AWS S3 Tables, or AWS Glue +- **Object Storage:** AWS S3 or S3-compatible (RustFS) +- **Iceberg Catalog:** the built-in S3 catalog (default, no external service), or Nessie (REST), AWS S3 Tables, or AWS Glue ## Helm Chart -The Helm chart deploys all IceGate components: Ingest, Query, and a Migrate job (schema creation as a pre-install/pre-upgrade hook). +The Helm chart deploys all {{product_name}} components: Ingest, Query, and a Migrate job (schema creation as a pre-install/pre-upgrade hook). ### Install from OCI Registry @@ -41,24 +41,24 @@ helm install icegate ./icegate/config/helm/icegate \ {% note info %} -Helm values use camelCase and flat keys (e.g., `backend: rest` + `rest.uri`). The chart translates these into the native serde tagged enum config format (`backend: !rest`) that IceGate binaries expect. See [Configuration](configuration.md) for the native config reference. +Helm values use camelCase and flat keys (e.g., `backend: s3` + `s3.warehouse`). The chart translates these into the native serde tagged enum config format (`backend: !s3`) that {{product_name}} binaries expect. See [Configuration](configuration.md) for the native config reference. {% endnote %} -A minimal `values.yaml` for a REST catalog (Nessie) with S3-compatible storage: +A minimal `values.yaml` using the default built-in S3 catalog with S3-compatible storage. No external catalog service is involved — the catalog state is a `root.json` object in the warehouse bucket: ```yaml catalog: - backend: rest - rest: - uri: http://nessie:19120/iceberg + backend: s3 + s3: + warehouse: catalog warehouse: "s3://warehouse/" storage: s3: bucket: warehouse region: us-east-1 - endpoint: "http://minio:9000" + endpoint: "http://rustfs:9000" queue: common: @@ -69,6 +69,28 @@ aws: region: us-east-1 ``` +### REST Catalog (Nessie) + +Use this only if you already run a Nessie or other Iceberg REST catalog — it adds an external service the default deployment does not need: + +```yaml +catalog: + backend: rest + rest: + uri: http://nessie:19120/iceberg + warehouse: "s3://warehouse/" + +storage: + s3: + bucket: warehouse + region: us-east-1 + endpoint: "http://rustfs:9000" + +aws: + existingSecret: icegate-aws-credentials + region: us-east-1 +``` + ### AWS Glue Catalog ```yaml @@ -109,9 +131,9 @@ aws: | Value | Default | Description | |-------|---------|-------------| -| `catalog.backend` | `rest` | Catalog type: `rest`, `s3tables`, or `glue` | +| `catalog.backend` | `s3` | Catalog type: `s3`, `rest`, `s3tables`, or `glue` | | `storage.s3.bucket` | `warehouse` | S3 bucket name | -| `storage.s3.endpoint` | `""` | Custom S3 endpoint (MinIO). Omit for real AWS S3 | +| `storage.s3.endpoint` | `""` | Custom S3 endpoint (RustFS). Omit for real AWS S3 | | `aws.existingSecret` | `""` | Secret with `aws-access-key-id` and `aws-secret-access-key` keys | | `query.replicaCount` | `1` | Query service replicas | | `ingest.replicaCount` | `1` | Ingest service replicas | @@ -130,19 +152,19 @@ aws: ## Kustomize Overlays -For environment-specific customizations, IceGate provides Kustomize overlays that compose the Helm chart with infrastructure dependencies. +For environment-specific customizations, {{product_name}} provides Kustomize overlays that compose the Helm chart with infrastructure dependencies. ### Available Overlays | Overlay | Description | Infrastructure | |---------|-------------|----------------| -| `skaffold` | Local development with Skaffold | MinIO, Nessie, observability stack | -| `orbstack` | OrbStack container runtime | MinIO, Nessie, observability stack | -| `aws-glue` | AWS Glue catalog | Observability stack (no MinIO/Nessie) | -| `aws-s3tables` | AWS S3 Tables catalog | Observability stack (no MinIO/Nessie) | -| `external-s3` | External S3 + Nessie catalog | Nessie, observability stack (no MinIO) | +| `skaffold` | Local development with Skaffold | RustFS, observability stack | +| `orbstack` | OrbStack container runtime | RustFS, observability stack | +| `aws-glue` | AWS Glue catalog | Observability stack (external S3) | +| `aws-s3tables` | AWS S3 Tables catalog | Observability stack (external S3) | +| `external-s3` | External S3 + Nessie catalog | Nessie, observability stack | -All overlays share a common base (`config/kustomize/base/`) that deploys the observability stack: Prometheus (kube-prometheus-stack), Grafana with pre-built IceGate dashboards, and Jaeger for distributed tracing. +All overlays share a common base (`config/kustomize/base/`) that deploys the observability stack: Prometheus (kube-prometheus-stack), Grafana with pre-built {{product_name}} dashboards, and Jaeger for distributed tracing. ### Usage @@ -159,7 +181,7 @@ skaffold dev Each overlay contains: - `kustomization.yaml` — declares Helm charts and patches -- `values-icegate.yaml` — IceGate Helm values for this environment +- `values-icegate.yaml` — {{product_name}} Helm values for this environment - `secret-aws.yaml` — AWS credentials Secret (edit before applying) To create a custom overlay: diff --git a/en/getting-started/quickstart.md b/en/getting-started/quickstart.md index dcee48a..b9a9620 100644 --- a/en/getting-started/quickstart.md +++ b/en/getting-started/quickstart.md @@ -1,21 +1,21 @@ --- title: Quick Start -description: Ingest and query your first observability data with IceGate +description: Ingest and query your first observability data with {{product_name}} --- # Quick Start -This guide walks you through ingesting logs, traces, and metrics into IceGate and querying them via the API and Grafana. +This guide walks you through ingesting logs, traces, and metrics into {{product_name}} and querying them via the API and Grafana. {% note info %} -This guide assumes IceGate is already running. See [Installation](installation.md) for Helm deployment or [Development Setup](../development/setup.md) for a local environment. +This guide assumes {{product_name}} is already running. See [Installation](installation.md) for Helm deployment or [Development Setup](../development/setup.md) for a local environment. {% endnote %} ## Ingest Logs -IceGate accepts data via the OpenTelemetry Protocol (OTLP) on the Ingest service. +{{product_name}} accepts data via the OpenTelemetry Protocol (OTLP) on the Ingest service. ### Send Logs via OTLP HTTP @@ -140,7 +140,8 @@ curl -X POST http://localhost:4318/v1/metrics \ ## Query Logs with LogQL -IceGate provides a Loki-compatible API on the Query service (port 3100). +{{product_name}} provides a Loki-compatible API on the Query service (port 3100) — a subset of Loki's API, +listed in the [API reference](../api-reference/loki.md). ### Basic Log Query @@ -219,9 +220,9 @@ curl -G http://localhost:3100/loki/api/v1/series \ ## Using Grafana -IceGate is compatible with Grafana's Loki data source for log visualization and dashboarding. +{{product_name}} is compatible with Grafana's Loki data source for log visualization and dashboarding. -### Add IceGate as a Data Source +### Add {{product_name}} as a Data Source 1. Open Grafana (default: [http://localhost:3000](http://localhost:3000)) 2. Go to **Connections** > **Data sources** > **Add data source** @@ -255,11 +256,11 @@ IceGate is compatible with Grafana's Loki data source for log visualization and ### Pre-Built Dashboards -If deployed with the Kustomize overlays or Docker Compose, Grafana comes pre-configured with IceGate dashboards for Ingest and Query service metrics. +If deployed with the Kustomize overlays or Docker Compose, Grafana comes pre-configured with {{product_name}} dashboards for Ingest and Query service metrics. ## Using the OpenTelemetry Collector -For production workloads, use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to forward data from your applications to IceGate: +For production workloads, use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to forward data from your applications to {{product_name}}: ```yaml # otel-collector-config.yaml @@ -286,7 +287,7 @@ service: ## Multi-Tenancy -IceGate isolates data by tenant using the `X-Scope-OrgID` header. Each tenant's data is physically partitioned. +{{product_name}} isolates data by tenant using the `X-Scope-OrgID` header. Each tenant's data is physically partitioned. ```bash # Ingest for tenant "team-a" diff --git a/en/guides/data-retention.md b/en/guides/data-retention.md index ddf1f3d..25ca57d 100644 --- a/en/guides/data-retention.md +++ b/en/guides/data-retention.md @@ -1,6 +1,6 @@ --- title: Data Retention -description: Configure data lifecycle, retention policies, and storage management in IceGate +description: Configure data lifecycle, retention policies, and storage management in {{product_name}} --- # Data Retention @@ -19,20 +19,32 @@ Each stage has independent retention controls. ## WAL Retention -WAL segments are automatically deleted after the shift process compacts them into Iceberg tables. For the queue bucket, configure an object storage lifecycle rule as a safety net: +Shift does not delete WAL segments after committing them to Iceberg — a lifecycle rule on the queue bucket is what reclaims them, so configure one: -### MinIO Lifecycle Rule +{% note warning %} + +Size the expiration from your worst-case unshifted-WAL window, not for convenience. A segment is only safe to expire once shift has committed it and recorded its offset in an Iceberg snapshot. If shift is stopped, backlogged, or recovering for longer than the expiration, the rule deletes segments whose offsets were never committed. The snapshot offset only tells shift where to resume — it cannot rebuild a deleted segment, so that is acknowledged data lost. One day suits the demo stack; choose yours from how long ingest can plausibly run without a successful shift commit, and alert on shift lag rather than relying on the rule to stay ahead of it. + +{% endnote %} + +The bucket in both commands below is the one from `queue.common.base_path` (`s3://queue/` by default). Substitute your own if you changed it — a rule applied to the wrong bucket leaves the real WAL bucket unmanaged. + +### RustFS (and other S3-compatible stores) + +RustFS speaks the S3 API, so the same `aws s3api` call the project's own bootstrap uses works against it: ```bash -# Set 1-day TTL on queue bucket -mc ilm rule add --expire-days 1 myminio/queue +# Set 1-day TTL on the queue bucket +aws --endpoint-url http://localhost:9000 s3api put-bucket-lifecycle-configuration \ + --bucket queue \ + --lifecycle-configuration '{"Rules":[{"ID":"expire-1d","Status":"Enabled","Filter":{"Prefix":""},"Expiration":{"Days":1}}]}' ``` ### AWS S3 Lifecycle Rule ```bash aws s3api put-bucket-lifecycle-configuration \ - --bucket icegate-queue \ + --bucket queue \ --lifecycle-configuration '{ "Rules": [{ "ID": "expire-wal-segments", @@ -212,13 +224,21 @@ Enable S3 versioning for point-in-time recovery of the warehouse bucket: ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` ### Catalog Backup -Back up the Nessie catalog (RocksDB storage): +On the default S3 catalog there is no service to stop and no database to dump — the catalog is `root.json` plus the table metadata files, in the warehouse bucket. Enabling versioning on that bucket (above) already gives point-in-time recovery. For an off-site copy, sync the catalog prefix: + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup-$(date +%Y%m%d)/ +``` + +Each object is replaced atomically — `root.json` by compare-and-swap, metadata files never in place — so no single object is ever copied half-written. The *set* is a different matter: `sync` lists and then copies, so commits landing during the run can leave the copy mixing catalog generations. For a point-in-time copy, read a single version from the versioned bucket, or take the copy while writes are quiesced, and verify it by restoring to a scratch prefix before relying on it. + +If you run the REST catalog backend instead, back up Nessie's RocksDB storage: ```bash # Stop Nessie diff --git a/en/guides/grafana-integration.md b/en/guides/grafana-integration.md index 9f3ea6f..d127348 100644 --- a/en/guides/grafana-integration.md +++ b/en/guides/grafana-integration.md @@ -1,11 +1,11 @@ --- title: Grafana Integration -description: Set up Grafana to query logs, traces, and metrics from IceGate +description: Set up Grafana to query logs, traces, and metrics from {{product_name}} --- # Grafana Integration -This guide covers connecting Grafana to all three {{product_name}} query APIs: Loki (logs), Tempo (traces), and Prometheus (metrics). +This guide covers connecting Grafana to the {{product_name}} query APIs: Loki (logs) and Tempo (traces), both implemented, plus Prometheus (metrics), which is planned and not yet functional. ## Prerequisites @@ -67,7 +67,7 @@ datasources: {% note warning %} -The Tempo API provides basic trace retrieval and search. TraceQL support is planned for future releases. +The Tempo API provides trace retrieval and search, and TraceQL is supported for `/api/search`; TraceQL features that are not yet implemented return `501 Not Implemented`. {% endnote %} @@ -106,11 +106,11 @@ datasources: ### Prometheus Data Source (Metrics) -{{product_name}} implements the Grafana Prometheus API on port **9090**. - {% note warning %} -The Prometheus query API is currently under development. Metadata endpoints (labels, series) are available, but PromQL queries are not yet supported. Use the Loki API with LogQL metric queries as an alternative for log-based metrics. +**The Prometheus data source will not work yet.** {{product_name}} mounts the Prometheus API routes on port **9090**, but every one of them — including the metadata endpoints (`labels`, `series`, `label/{name}/values`) — returns `501 Not Implemented`. Only `/-/ready` responds. + +Use the Loki data source with LogQL metric queries for log-based metrics, or Arrow Flight SQL for general-purpose SQL over the same data. The steps below are recorded for when the API lands. {% endnote %} @@ -253,12 +253,12 @@ Create a dashboard with three panels: ### Trace Explorer -1. Navigate to **Explore** > select **IceGate Traces** +1. Navigate to **Explore** > select **{{product_name}} Traces** 2. Search by service name: enter `service.name=my-service` in the tags field 3. Filter by minimum duration: set `minDuration` to `100ms` 4. Click a trace to view its span waterfall -## Using IceGate as a Drop-In for Existing Grafana +## Using {{product_name}} as a Drop-In for Existing Grafana If you have an existing Grafana setup with Loki, you can point it at {{product_name}} by changing only the data source URL: @@ -268,7 +268,7 @@ If you have an existing Grafana setup with Loki, you can point it at {{product_n 4. Add the `X-Scope-OrgID` header if not already present 5. Click **Save & Test** -Your existing dashboards, alerting rules, and saved queries will continue to work because {{product_name}} implements the same Loki API. +Dashboards, alerting rules, and saved queries keep working as long as they stay within the endpoints and LogQL features {{product_name}} implements — it serves a subset of the Loki read API, not all of it. Check the [Loki API reference](../api-reference/loki.md) and the [LogQL implementation status](querying.md) for anything a panel depends on, and re-test alert rules after switching. {% note info %} @@ -281,8 +281,8 @@ LogQL metric queries (`rate()`, `count_over_time()`, `sum by()`, etc.) are suppo | API | Port | Grafana Data Source Type | Status | |-----|------|--------------------------|--------| | Loki (logs) | 3100 | Loki | Fully implemented | -| Tempo (traces) | 3200 | Tempo | Basic retrieval and search (TraceQL planned) | -| Prometheus (metrics) | 9090 | Prometheus | Metadata only (PromQL planned) | +| Tempo (traces) | 3200 | Tempo | Retrieval and search; TraceQL supported (unimplemented features return `501`) | +| Prometheus (metrics) | 9090 | Prometheus | Planned; every route returns `501` except `/-/ready` | ## Next Steps diff --git a/en/guides/ingestion.md b/en/guides/ingestion.md index 5717a62..ee19fe2 100644 --- a/en/guides/ingestion.md +++ b/en/guides/ingestion.md @@ -1,11 +1,11 @@ --- title: Data Ingestion -description: Ingest logs, traces, and metrics into IceGate +description: Ingest logs, traces, and metrics into {{product_name}} --- # Data Ingestion -IceGate accepts observability data via the OpenTelemetry Protocol (OTLP). This guide covers how to ingest logs, traces, and metrics. +{{product_name}} accepts observability data via the OpenTelemetry Protocol (OTLP). This guide covers how to ingest logs, traces, and metrics. ## Supported Protocols @@ -49,7 +49,7 @@ curl -X POST http://localhost:4318/v1/logs \ ### Using OpenTelemetry SDKs -Configure your OpenTelemetry SDK to send logs to IceGate: +Configure your OpenTelemetry SDK to send logs to {{product_name}}: ```python # Python example @@ -70,7 +70,7 @@ logger_provider.add_log_record_processor( ## Ingesting Traces -Send distributed trace spans to IceGate: +Send distributed trace spans to {{product_name}}: ```bash curl -X POST http://localhost:4318/v1/traces \ @@ -133,7 +133,7 @@ curl -X POST http://localhost:4318/v1/metrics \ ## Tenant Identification -IceGate is multi-tenant. Specify the tenant using the `X-Scope-OrgID` header: +{{product_name}} is multi-tenant. Specify the tenant using the `X-Scope-OrgID` header: ```bash curl -X POST http://localhost:4318/v1/logs \ @@ -151,7 +151,7 @@ curl -X POST http://localhost:4318/v1/logs \ ## Delivery Guarantees -IceGate provides **exactly-once delivery** semantics: +{{product_name}} provides **exactly-once delivery** semantics: - Data is durably written to object storage before acknowledgment - Idempotent writes prevent duplicates diff --git a/en/guides/multi-tenancy.md b/en/guides/multi-tenancy.md index df88065..afea62d 100644 --- a/en/guides/multi-tenancy.md +++ b/en/guides/multi-tenancy.md @@ -1,11 +1,11 @@ --- title: Multi-Tenancy -description: Configure and use multi-tenant isolation in IceGate +description: Configure and use multi-tenant isolation in {{product_name}} --- # Multi-Tenancy -IceGate is designed as a multi-tenant system, providing data isolation between different organizations or teams. +{{product_name}} is designed as a multi-tenant system, providing data isolation between different organizations or teams. ## Tenant Identification diff --git a/en/guides/performance-tuning.md b/en/guides/performance-tuning.md index 2066cf7..9de4ecd 100644 --- a/en/guides/performance-tuning.md +++ b/en/guides/performance-tuning.md @@ -1,6 +1,6 @@ --- title: Performance Tuning -description: Optimize IceGate ingestion throughput, query performance, and compaction +description: Optimize {{product_name}} ingestion throughput, query performance, and compaction --- # Performance Tuning diff --git a/en/guides/querying.md b/en/guides/querying.md index 6a1ace2..6e82024 100644 --- a/en/guides/querying.md +++ b/en/guides/querying.md @@ -5,7 +5,11 @@ description: Query logs, traces, and metrics with LogQL, PromQL, and TraceQL # Querying Data -IceGate provides Loki, Prometheus, and Tempo-compatible APIs for querying observability data. +{{product_name}} provides Loki- and Tempo-compatible APIs for querying observability data, plus Arrow Flight +SQL for general-purpose SQL. The +[Prometheus-compatible API](../api-reference/prometheus.md) is planned and not implemented yet — +query metrics through Flight SQL in the meantime. The API reference pages are authoritative on +which endpoints are served today. ## LogQL for Logs diff --git a/en/index.yaml b/en/index.yaml index 3ece66d..0596ab5 100644 --- a/en/index.yaml +++ b/en/index.yaml @@ -2,7 +2,14 @@ title: IceGate Documentation description: | An Observability Data Lake engine designed to be fast, easy-to-use, cost-effective, scalable, and fault-tolerant. meta: - title: IceGate - Observability Data Lake Engine + # Rendered as " | ", so this plus " | IceGate Documentation" is the whole + # <title> tag — keep the pair under 60 characters or Google truncates it. Naming the product + # here as well as in `title` would spend 8 of those characters saying "IceGate" twice. + title: Observability Data Lake Engine + # Leading pages emit no <meta name="description"> from the `description` field above — that + # one only renders as body copy. Without this key the landing page of each language ships + # without a description at all. + description: Documentation for IceGate, an open-source observability data lake engine — install, query, and operate it on Apache Iceberg, Arrow and Parquet. links: - title: Getting Started description: Install IceGate and run your first queries in minutes @@ -11,7 +18,7 @@ links: description: Learn how to ingest data, query logs, and configure multi-tenancy href: guides/ingestion.md - title: API Reference - description: Loki, Prometheus, and Tempo compatible APIs + description: Loki®- and Tempo®-compatible APIs (Prometheus® planned) href: api-reference/loki.md - title: Architecture description: Understand IceGate's compute-storage separation design diff --git a/en/operations/deployment.md b/en/operations/deployment.md index 1c6f8d4..e924248 100644 --- a/en/operations/deployment.md +++ b/en/operations/deployment.md @@ -1,16 +1,16 @@ --- title: Deployment -description: Deploy IceGate in production environments +description: Deploy {{product_name}} in production environments --- # Deployment -This guide covers deploying IceGate in production environments. +This guide covers deploying {{product_name}} in production environments. ## Prerequisites -- **Object Storage:** S3, MinIO, or S3-compatible storage -- **Iceberg Catalog:** Nessie (REST), AWS S3 Tables, or AWS Glue +- **Object Storage:** S3, RustFS, or S3-compatible storage +- **Iceberg Catalog:** the built-in S3 catalog (default), or Nessie (REST), AWS S3 Tables, or AWS Glue - **Docker/Kubernetes:** For container orchestration ## Architecture Considerations @@ -21,7 +21,7 @@ This guide covers deploying IceGate in production environments. |-----------|---------|-------| | Ingest | Horizontal | Scale for write throughput | | Query | Horizontal | Scale for query concurrency | -| Maintain | Single leader | Coordinates compaction | +| Maintain | Horizontal | Workers coordinate through job state in object storage (compare-and-swap) | ### Resource Requirements @@ -50,7 +50,7 @@ This guide covers deploying IceGate in production environments. The project includes Docker Compose profiles for different deployment scenarios: ```bash -# Core services: MinIO, Nessie, Ingest, Query, Maintain +# Core services: RustFS, Ingest, Query, Maintain make run-core-release # Core + load generator for testing @@ -66,26 +66,19 @@ make run-analytics-release ```yaml # docker-compose.yml services: - minio: - image: minio/minio:latest - command: server /data --console-address ":9001" + rustfs: + image: rustfs/rustfs:1.0.0-beta.8 environment: - MINIO_ROOT_USER: ${S3_ACCESS_KEY} - MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY} + RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" volumes: - - minio-data:/data + - rustfs-data:/data ports: - - "9000:9000" - - "9001:9001" - - nessie: - image: projectnessie/nessie:latest - environment: - NESSIE_VERSION_STORE_TYPE: ROCKSDB - volumes: - - nessie-data:/data - ports: - - "19120:19120" + - "9000:9000" # S3 API + - "9001:9001" # Console ingest: image: icegate/ingest:latest @@ -100,8 +93,7 @@ services: - "4318:4318" # OTLP HTTP - "9091:9091" # Prometheus metrics depends_on: - - minio - - nessie + - rustfs query: image: icegate/query:latest @@ -116,9 +108,9 @@ services: - "3100:3100" # Loki API - "9090:9090" # Prometheus API - "3200:3200" # Tempo API + - "8815:8815" # Arrow Flight SQL depends_on: - - minio - - nessie + - rustfs maintain: image: icegate/maintain:latest @@ -128,12 +120,10 @@ services: volumes: - ./config/maintain.yaml:/etc/icegate/maintain.yaml:ro depends_on: - - minio - - nessie + - rustfs volumes: - minio-data: - nessie-data: + rustfs-data: query-cache: ``` @@ -165,7 +155,7 @@ docker build -t icegate/maintain:latest \ ### Helm Charts -IceGate includes Helm charts for Kubernetes deployment: +{{product_name}} includes Helm charts for Kubernetes deployment: ```bash # Install from local charts @@ -187,7 +177,7 @@ Pre-built Kustomize overlays are available for common scenarios: | `orbstack` | OrbStack container runtime | | `aws-glue` | AWS Glue catalog integration | | `aws-s3tables` | AWS S3 Tables catalog integration | -| `external-s3` | External S3 storage (not MinIO) | +| `external-s3` | External S3 storage with a Nessie catalog | ```bash # Apply with kustomize @@ -205,13 +195,13 @@ storage: region: us-east-1 ``` -### MinIO +### RustFS (S3-compatible) ```yaml storage: backend: !s3 bucket: warehouse - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 region: us-east-1 ``` @@ -227,11 +217,11 @@ storage: | Query replica fails | Reduced query capacity | Load balancer routes to healthy replicas | | Maintain/Shift | WAL segments accumulate | Restarts and resumes from last committed snapshot | | Object storage (S3) | Service outage | WAL writes fail with 503; clients should retry | -| Catalog (Nessie) | Cannot commit new data or read metadata | Queries fail; data in WAL is preserved | +| Catalog | Cannot commit new data or read metadata | Queries fail; data in WAL is preserved | ### Durability Guarantees -- **WAL persistence**: All ingested data is written to object storage (S3/MinIO) before acknowledgment. Data survives node failures. +- **WAL persistence**: All ingested data is written to object storage (S3 or RustFS) before acknowledgment. Data survives node failures. - **Exactly-once delivery**: The ingest service acknowledges only after WAL write completes. - **Immutable segments**: WAL segments are append-only Parquet files. Once written, they cannot be corrupted by subsequent operations. - **Iceberg snapshots**: Each shift operation creates an atomic Iceberg snapshot. Failed shifts do not corrupt existing data. @@ -321,7 +311,7 @@ readinessProbe: ### Metrics -IceGate services expose Prometheus metrics on a dedicated port (default: 9091): +{{product_name}} services expose Prometheus metrics on a dedicated port (default: 9091): - Ingest metrics: `http://ingest:9091/metrics` - Query metrics: `http://query:9091/metrics` @@ -338,7 +328,7 @@ metrics: ### Self-Observability with Tracing -IceGate can export its own traces via OTLP for debugging: +{{product_name}} can export its own traces via OTLP for debugging: ```yaml tracing: @@ -362,7 +352,7 @@ environment: ### Network Security - Use TLS for all external connections -- Restrict access to MinIO/Nessie from internal network only +- Restrict access to object storage and any external catalog from internal network only - Use network policies in Kubernetes ### Authentication diff --git a/en/operations/maintenance.md b/en/operations/maintenance.md index 7e70d9c..5ee284b 100644 --- a/en/operations/maintenance.md +++ b/en/operations/maintenance.md @@ -1,6 +1,6 @@ --- title: Maintenance -description: Maintain IceGate for optimal performance +description: Maintain {{product_name}} for optimal performance --- # Maintenance @@ -19,7 +19,7 @@ maintain migrate create -c maintain.yaml ### Schema Upgrades -Upgrade existing table schemas when updating IceGate: +Upgrade existing table schemas when updating {{product_name}}: ```bash maintain migrate upgrade -c maintain.yaml @@ -52,8 +52,9 @@ The Ingest service automatically shifts WAL data into optimized Iceberg tables v 3. Reads WAL Parquet files in parallel 4. Merges and re-partitions data 5. Writes optimized Iceberg data files -6. Commits new snapshot to catalog -7. Deletes processed WAL segments +6. Commits a new snapshot to the catalog, recording the last committed WAL offset in the snapshot summary + +Shift does not delete WAL segments. An object lifecycle rule on the queue bucket reclaims them, and the offset in the snapshot summary is what lets shift resume where it left off. ### Tuning Shift Performance @@ -144,7 +145,15 @@ curl http://localhost:4318/health ### Catalog Backup -Nessie stores catalog metadata. Back up the RocksDB data: +On the default S3 catalog the metadata is `root.json` plus the table metadata files in the warehouse bucket, so a backup is a copy of that prefix — there is no service to stop: + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup/ +``` + +`sync` is not an atomic snapshot: it lists, then copies, and commits landing in between can leave the copy mixing catalog generations. For a point-in-time copy, use bucket versioning (below) and read a single version, or take the copy while writes are quiesced. Verify any backup by restoring it to a scratch prefix and listing the tables before relying on it. + +If you run the REST catalog backend instead, back up Nessie's RocksDB data: ```bash # Stop Nessie @@ -178,7 +187,7 @@ Enable versioning on your S3 bucket for point-in-time recovery: ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` diff --git a/en/operations/troubleshooting.md b/en/operations/troubleshooting.md index 4748495..1f64205 100644 --- a/en/operations/troubleshooting.md +++ b/en/operations/troubleshooting.md @@ -1,6 +1,6 @@ --- title: Troubleshooting -description: Diagnose and resolve common IceGate issues +description: Diagnose and resolve common {{product_name}} issues --- # Troubleshooting @@ -61,15 +61,15 @@ docker compose logs -f maintain **Symptoms:** -- "Connection refused" to MinIO +- "Connection refused" to the object store - S3 authentication errors **Solutions:** -1. Verify MinIO is running: +1. On a local RustFS deployment, verify the object store is running. The readiness path is RustFS's own — on AWS S3 or another provider, skip to step 3 instead: ```bash - curl http://localhost:9000/minio/health/ready + curl http://localhost:9000/health/ready ``` 2. Check credentials: @@ -79,7 +79,7 @@ docker compose logs -f maintain echo $AWS_SECRET_ACCESS_KEY ``` -3. Test S3 connection: +3. Test the S3 connection. Drop `--endpoint-url` when the backend is real AWS S3: ```bash aws s3 ls --endpoint-url http://localhost:9000 @@ -94,19 +94,31 @@ docker compose logs -f maintain **Solutions:** -1. Verify Nessie is running: +1. On the default S3 catalog, confirm the catalog state object is readable — there is no catalog service to check: ```bash - curl http://localhost:19120/api/v1/trees + aws --endpoint-url http://localhost:9000 s3 ls s3://warehouse/catalog/root.json ``` + A missing `root.json` means migration never ran. Run `maintain migrate create` before anything else. + 2. Check catalog configuration: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 + ``` + +3. On the REST backend only, verify Nessie is running: + + ```bash + curl http://localhost:19120/api/v1/trees ``` ## Query Issues @@ -273,10 +285,10 @@ If issues persist: docker stats > stats.txt ``` -2. Check [GitHub Issues](https://github.com/icegatetech/icegate/issues) +2. Check [GitHub Issues]({{repo_url}}/issues) 3. Include: - - IceGate version + - {{product_name}} version - Configuration (sanitized) - Error messages - Steps to reproduce diff --git a/en/toc.yaml b/en/toc.yaml index a03bb9d..9ef96d3 100644 --- a/en/toc.yaml +++ b/en/toc.yaml @@ -96,3 +96,6 @@ items: - name: FAQ href: faq.md + + - name: Trademarks + href: trademarks.md diff --git a/en/trademarks.md b/en/trademarks.md new file mode 100644 index 0000000..d4d5c08 --- /dev/null +++ b/en/trademarks.md @@ -0,0 +1,37 @@ +--- +title: Trademarks +description: Third-party trademark attribution for the projects {{product_name}} builds on and interoperates with +--- + +# Trademarks + +{{product_name}} is developed by TripleCloud and released under the {{license}} licence. + +This documentation names third-party projects in order to describe, factually, which formats +{{product_name}} writes and which wire protocols its APIs implement. Such nominative use does not +imply any affiliation with, endorsement by, or sponsorship from the owners of those marks. + +Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL +and associated project logos are either registered trademarks or trademarks of The Apache Software +Foundation in the United States and/or other countries. + +OpenTelemetry® and Prometheus® are registered trademarks of The Linux Foundation. + +Grafana®, Loki® and Tempo® are registered trademarks of Raintank, Inc. dba Grafana Labs. + +{{product_name}} is not affiliated with, endorsed by, or sponsored by any of these organizations. +All other trademarks are the property of their respective owners. + +## What "compatible" means here + +Where this documentation describes an API as Loki- or Tempo-compatible, it means {{product_name}} +implements a subset of that project's HTTP read API — enough to serve the endpoints documented in +the [Loki](api-reference/loki.md) and [Tempo](api-reference/tempo.md) API references, not a +complete reimplementation. + +The Prometheus-compatible API is **planned, not implemented**: every route returns +`501 Not Implemented` except `/-/ready`, which responds. Its +[reference page](api-reference/prometheus.md) documents an intended surface, not a working one. + +The API reference pages are the authoritative statement of what works today: if an endpoint or +parameter is not listed there, assume it is not implemented yet. diff --git a/fr/api-reference/loki.md b/fr/api-reference/loki.md index 8a3c197..abb8bd7 100644 --- a/fr/api-reference/loki.md +++ b/fr/api-reference/loki.md @@ -1,11 +1,14 @@ --- title: Référence API Loki -description: Points de terminaison HTTP de l'API compatible Loki +description: Points de terminaison HTTP de l'API compatible Loki servis par {{product_name}} --- # Référence API Loki -IceGate fournit une API HTTP compatible Loki pour interroger les logs. +{{product_name}} fournit une API HTTP compatible Loki® pour interroger les logs, servie sur le port 3100. +Les points de terminaison documentés ci-dessous sont ceux implémentés — il s'agit d'un +sous-ensemble de l'API de Loki, et non d'une réimplémentation complète : tout ce qui n'y figure pas +doit être considéré comme non implémenté. Voir [Marques](../trademarks.md) pour l'attribution. ## URL de Base @@ -216,7 +219,7 @@ curl -G http://localhost:3100/loki/api/v1/series \ ### Explain -Obtenir le plan d'exécution d'une requête (extension IceGate). +Obtenir le plan d'exécution d'une requête (extension {{product_name}}). **Point de terminaison :** `GET /loki/api/v1/explain` @@ -265,5 +268,5 @@ Toutes les erreurs retournent une réponse JSON : ## Étapes Suivantes - Apprenez le [Requêtage LogQL](../guides/querying.md) -- Explorez l'[API Prometheus](prometheus.md) +- Explorez l'[API Prometheus](prometheus.md) — prévue, pas encore implémentée - Voir l'[API Tempo](tempo.md) pour les traces diff --git a/fr/api-reference/otlp.md b/fr/api-reference/otlp.md index 9d6fffe..7620310 100644 --- a/fr/api-reference/otlp.md +++ b/fr/api-reference/otlp.md @@ -5,7 +5,7 @@ description: Points d'accès OpenTelemetry Protocol pour l'ingestion de données # API d'Ingestion OTLP -IceGate accepte les données d'observabilité via le protocole OpenTelemetry (OTLP). Les transports HTTP et gRPC sont pris en charge. +{{product_name}} accepte les données d'observabilité via le protocole OpenTelemetry (OTLP). Les transports HTTP et gRPC sont pris en charge. ## Protocoles diff --git a/fr/api-reference/prometheus.md b/fr/api-reference/prometheus.md index 2769053..48af75a 100644 --- a/fr/api-reference/prometheus.md +++ b/fr/api-reference/prometheus.md @@ -1,6 +1,6 @@ --- title: Référence API Prometheus -description: Points de terminaison HTTP de l'API compatible Prometheus +description: API compatible Prometheus prévue — pas encore implémentée --- # Référence API Prometheus @@ -11,7 +11,19 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -IceGate fournit une API HTTP compatible Prometheus pour interroger les métriques. +{% note warning %} + +**Pas encore implémentée.** Les routes ci-dessous sont montées sur le port 9090, mais chacune +d'elles renvoie `501 Not Implemented` ; seul `/-/ready` répond. Cette page documente la surface +*prévue* afin que les intégrateurs voient la direction prise — ne développez pas encore dessus. + +Pour interroger les métriques aujourd'hui, utilisez +[Arrow Flight SQL](../guides/querying.md) sur les mêmes données. + +{% endnote %} + +Voici la forme prévue de l'API HTTP compatible Prometheus® d'{{product_name}} pour interroger les métriques. +Voir [Marques](../trademarks.md) pour l'attribution. ## URL de Base @@ -21,7 +33,7 @@ http://localhost:9090 ## État de l'Implémentation -L'API Prometheus est actuellement en développement. +Aucun de ces points de terminaison n'est implémenté : ils renvoient tous `501 Not Implemented`. Seul `/-/ready` répond. ## Étapes Suivantes diff --git a/fr/api-reference/tempo.md b/fr/api-reference/tempo.md index 8ce8200..5888a82 100644 --- a/fr/api-reference/tempo.md +++ b/fr/api-reference/tempo.md @@ -1,6 +1,6 @@ --- title: Référence API Tempo -description: Points de terminaison HTTP de l'API compatible Tempo +description: Points de terminaison HTTP de l'API compatible Tempo servis par {{product_name}} --- # Référence API Tempo @@ -11,7 +11,12 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -IceGate fournit une API HTTP compatible Tempo pour interroger les traces distribuées. +{{product_name}} fournit une API HTTP compatible Tempo® pour interroger les traces distribuées, servie sur le +port 3200. Les points de terminaison documentés ci-dessous sont ceux implémentés — il s'agit d'un +sous-ensemble de l'API de Tempo, et non d'une réimplémentation complète : tout ce qui n'y figure pas +doit être considéré comme non implémenté. TraceQL est pris en charge pour `/api/search` ; les +fonctionnalités TraceQL non encore implémentées renvoient `501 Not Implemented` plutôt que des +résultats erronés. Voir [Marques](../trademarks.md) pour l'attribution. ## URL de Base diff --git a/fr/architecture/data-model.md b/fr/architecture/data-model.md index b9dc92e..58e361f 100644 --- a/fr/architecture/data-model.md +++ b/fr/architecture/data-model.md @@ -1,6 +1,6 @@ --- title: Modèle de Données -description: Schémas des tables Iceberg IceGate pour les données d'observabilité +description: Schémas des tables Iceberg {{product_name}} pour les données d'observabilité --- # Modèle de Données @@ -11,7 +11,7 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -IceGate stocke les données d'observabilité dans quatre tables Apache Iceberg. +{{product_name}} stocke les données d'observabilité dans cinq tables Apache Iceberg par tenant — logs, spans, events, metrics et operations — plus une table de référence globale, prices. ## Vue d'Ensemble des Tables @@ -21,12 +21,14 @@ IceGate stocke les données d'observabilité dans quatre tables Apache Iceberg. | `spans` | Spans de traces distribuées | Traçage des requêtes | | `events` | Événements sémantiques | Événements métier | | `metrics` | Tous types de métriques | Monitoring de performance | +| `operations` | Opérations LLM et agents | Usage de tokens, coût, capture des prompts et complétions | +| `prices` | Grille tarifaire LLM globale (sans `tenant_id`) | Tarifs de référence pour chiffrer `operations` | ## Patterns Communs ### Multi-Tenancy -Toutes les tables utilisent le partitionnement par `tenant_id`. +Les cinq tables par tenant utilisent le partitionnement par identité sur `tenant_id`. `prices` est une donnée de référence partagée par tous les tenants : elle ne porte pas de `tenant_id` et est partitionnée différemment. ### Stockage des Attributs diff --git a/fr/architecture/overview.md b/fr/architecture/overview.md index 856c96f..b3d41c8 100644 --- a/fr/architecture/overview.md +++ b/fr/architecture/overview.md @@ -1,11 +1,11 @@ --- title: Vue d'Ensemble de l'Architecture -description: Architecture système et composants IceGate +description: Architecture système et composants {{product_name}} --- # Vue d'Ensemble de l'Architecture -IceGate est un moteur de lac de données d'observabilité qui stocke les logs, traces, métriques et événements dans des tables Apache Iceberg avec DataFusion comme moteur de requêtes. +{{product_name}} est un moteur de lac de données d'observabilité qui stocke les logs, traces, métriques, événements et opérations LLM dans des tables Apache Iceberg avec DataFusion comme moteur de requêtes. ## Principes de Conception @@ -43,13 +43,18 @@ Le Write-Ahead Log (WAL) stocke les données sous forme de fichiers Parquet orga **Objectif :** Exécuter des requêtes sur les logs, traces, métriques et événements - **Moteur :** Apache DataFusion + Apache Arrow -- **APIs :** Loki (3100), Prometheus (9090), Tempo (3200) -- **Langages de Requête :** LogQL, PromQL (planifié), TraceQL (planifié) +- **APIs :** Loki (3100), Tempo (3200), Arrow Flight SQL (8815) ; Prometheus (9090) expose ses routes mais ses handlers retournent encore `501 Not Implemented` +- **Langages de Requête :** LogQL, TraceQL, SQL ; PromQL planifié +- **Multi-tenance :** Le tenant provient de l'en-tête `X-Scope-OrgID`, ou des métadonnées gRPC `x-scope-orgid` pour Flight SQL + +Arrow Flight SQL est strictement en lecture seule — DDL et DML sont rejetés — et applique `tenant_id` au niveau des lignes à chaque scan, de sorte que les clients JDBC, ODBC et ADBC interrogent `iceberg.icegate.<table>` sans code client spécifique à {{product_name}}. Le service query lit depuis les deux sources : - **WAL** : Pour les données en temps réel (vieilles de quelques secondes) -- **Tables Iceberg** : Pour les données historiques (compactées) +- **Tables Iceberg** : Pour les données historiques (shiftées et compactées) + +La frontière entre les deux est l'offset WAL enregistré dans le résumé du snapshot Iceberg : une ligne est donc lue d'un seul côté et n'est jamais comptée deux fois. ### Service Maintain @@ -57,10 +62,25 @@ Le service query lit depuis les deux sources : **Objectif :** Opérations de cycle de vie et d'optimisation des données -- **Compaction :** Fusion des petits fichiers WAL en tables Iceberg optimisées -- **TTL :** Expiration et suppression des anciennes données -- **Optimisation :** Réécriture des fichiers pour de meilleures performances -- **Nettoyage :** Suppression des fichiers orphelins +- **Migration de schéma :** Création des tables Iceberg (`maintain migrate create`) +- **Compaction des données :** Réécriture des petits fichiers Parquet en fichiers triés moins nombreux et plus volumineux +- **Compaction des manifests :** Regroupement des manifests Iceberg fragmentés +- **GC des orphelins :** Suppression des objets que les métadonnées courantes de la table ne référencent plus, une fois le délai de grâce écoulé +- **Crawler de tarifs :** Collecte des grilles tarifaires LLM depuis des flux externes vers la table globale `icegate.prices` + +La compaction, le GC et le crawler de tarifs s'exécutent chacun comme des jobs dont l'état réside dans le stockage objet, sous leur propre préfixe d'état de jobs. + +### Catalogue + +![Composants Catalogue](../../assets/c4/structurizr-CatalogComponents.png) + +**Objectif :** Organiser le lac de données avec des transactions ACID, sans base de données OLTP dédiée + +- **Backend par défaut :** Le catalogue S3 propre à {{product_name}} — l'état du catalogue est un objet `root.json` mis à jour par compare-and-swap +- **Backends alternatifs :** REST (Nessie), AWS S3 Tables, AWS Glue +- **Déploiement :** Lié à Ingest, Query et Maintain par défaut ; optionnellement déployé de manière autonome comme serveur REST Iceberg sur le port 8181 + +Une lecture conditionnelle maintient à jour la racine du catalogue en cache ; les métadonnées de table sont immuables par emplacement et sont donc mises en cache inconditionnellement dans un LRU. ### Service Alert (Planifié) @@ -79,15 +99,21 @@ Le service query lit depuis les deux sources : | Format Mémoire | Apache Arrow 57.0 | Traitement de données sans copie | | Format de Stockage | Apache Parquet 57.0 | Stockage en colonnes avec compression ZSTD | | Ingestion | OpenTelemetry 0.31 | Protocole d'observabilité standard (gRPC + HTTP) | -| Catalogue | Nessie, AWS S3 Tables, AWS Glue | Backends de catalogue REST Iceberg | -| Job Manager | icegate-jobmanager | Gestion de l'état des jobs shift basée sur S3 | +| Interface SQL | Arrow Flight SQL 57.0 | SQL en lecture seule pour les clients JDBC, ODBC et ADBC | +| Catalogue | Catalogue S3 (par défaut), Nessie, AWS S3 Tables, AWS Glue | Backends de catalogue Iceberg ; celui par défaut conserve son état dans le stockage objet | +| Stockage Objet | RustFS, ou tout stockage compatible S3 | Segments WAL, données Iceberg, état du catalogue, état des jobs | +| Job Manager | jobmanager (dépôt séparé) | État des jobs shift, compaction, GC et tarifs, basé sur S3 | | Cache | foyer 0.22 | Cache hybride mémoire + disque pour les lectures S3 | -| Langage | Rust 1.92+ (édition 2024) | Runtime haute performance et sûr en mémoire | +| Langage | Rust {{rust_version}}+ (édition 2024) | Runtime haute performance et sûr en mémoire | ## Flux de Données ### Flux d'Ingestion +![Séquence d'Ingestion](../../assets/c4/structurizr-IngestionFlow.png) + +Les étapes 1 à 7 constituent le chemin d'écriture, acquitté dès que le segment WAL est écrit. Les étapes 8 à 14 sont le shift, qui s'exécute indépendamment de la requête. + 1. Le client envoie des données OTLP au service Ingest 2. Ingest valide et transforme les données 3. Les données sont écrites dans le WAL sous forme de fichiers Parquet @@ -95,6 +121,8 @@ Le service query lit depuis les deux sources : ### Flux de Requêtes +![Séquence de Requête](../../assets/c4/structurizr-QueryFlow.png) + 1. Le client envoie une requête au service Query 2. La requête est analysée et planifiée par DataFusion 3. Les données sont lues depuis les tables Iceberg et/ou le WAL @@ -106,8 +134,17 @@ Le service query lit depuis les deux sources : 2. Regroupe les segments en tâches shift 3. Lit les fichiers WAL en parallèle, fusionne et re-partitionne les données 4. Écrit les fichiers de données Iceberg optimisés -5. Valide un nouveau snapshot dans le catalogue -6. Supprime les segments WAL traités +5. Valide un nouveau snapshot dans le catalogue, en enregistrant le dernier offset WAL validé dans le résumé du snapshot + +Le shift ne supprime jamais les segments WAL. Ils sont récupérés par une règle de cycle de vie objet sur le bucket de la queue, et c'est l'offset du résumé du snapshot qui permet au shift de reprendre là où il s'était arrêté. + +L'expiration du cycle de vie est donc un paramètre de durabilité, pas une simple tâche d'entretien : un segment doit survivre au commit qui le couvre. Si le shift est retardé ou en échec au moment où la règle s'applique, les segments dont l'offset n'a jamais été validé sont supprimés et les données sont perdues. Voir [Rétention des Données](../guides/data-retention.md) pour le dimensionnement. + +### Flux de Maintenance + +![Séquence de Maintenance](../../assets/c4/structurizr-MaintenanceFlow.png) + +La migration est un job unique. La compaction, le GC des orphelins et le crawler de tarifs sont des boucles indépendantes suivant leurs propres cadences — les numéros d'étape ordonnent chaque boucle, pas les boucles entre elles. Chacune réserve son travail sous son propre préfixe d'état de jobs : les boucles ne se disputent donc jamais la propriété des tâches. Elles partagent malgré tout les tables sous-jacentes — la compaction valide des snapshots de réécriture pendant que le GC supprime des objets non référencés — d'où le fait que le GC ne supprime que les fichiers plus anciens que son délai de grâce et que les commits utilisent la concurrence optimiste, avec réessai en cas de conflit. ## Évolutivité @@ -115,7 +152,7 @@ Le service query lit depuis les deux sources : - **Ingest :** Augmenter le nombre de réplicas pour un débit plus élevé - **Query :** Augmenter le nombre de réplicas pour les requêtes concurrentes -- **Maintain :** Instance unique (élection de leader) +- **Maintain :** Augmenter le nombre de réplicas pour plus de débit de réécriture — les workers partagent leur état de jobs dans le stockage objet via compare-and-swap et valident en concurrence optimiste, les instances parallèles sont donc sûres. Privilégier d'abord l'augmentation du nombre de workers en processus ; les gains s'amenuisent à mesure que les réplicas augmentent, tous les workers d'une table étant en contention sur un unique objet d'état de jobs. ### Mise à l'Échelle du Stockage diff --git a/fr/cookbooks/centralized-logging.md b/fr/cookbooks/centralized-logging.md index 32421de..8c253e9 100644 --- a/fr/cookbooks/centralized-logging.md +++ b/fr/cookbooks/centralized-logging.md @@ -1,6 +1,6 @@ --- title: Journalisation Centralisée pour Microservices -description: Mettre en place une journalisation centralisée avec OpenTelemetry Collector et IceGate +description: Mettre en place une journalisation centralisée avec OpenTelemetry Collector et {{product_name}} --- # Journalisation Centralisée pour Microservices diff --git a/fr/cookbooks/observability-correlation.md b/fr/cookbooks/observability-correlation.md index f8492ee..9c5d4f5 100644 --- a/fr/cookbooks/observability-correlation.md +++ b/fr/cookbooks/observability-correlation.md @@ -1,6 +1,6 @@ --- title: Corrélation des Signaux d'Observabilité -description: Corréler les logs et les traces dans IceGate pour un diagnostic efficace +description: Corréler les logs et les traces dans {{product_name}} pour un diagnostic efficace --- # Corrélation des Signaux d'Observabilité diff --git a/fr/cookbooks/traces-end-to-end.md b/fr/cookbooks/traces-end-to-end.md index 193b897..143e5de 100644 --- a/fr/cookbooks/traces-end-to-end.md +++ b/fr/cookbooks/traces-end-to-end.md @@ -1,6 +1,6 @@ --- title: Traçage Distribué de Bout en Bout -description: Instrumenter les services et interroger les traces via l'API Tempo d'IceGate +description: Instrumenter les services et interroger les traces via l'API Tempo d'{{product_name}} --- # Traçage Distribué de Bout en Bout diff --git a/fr/development/building.md b/fr/development/building.md index 41a5ff6..c99c7bc 100644 --- a/fr/development/building.md +++ b/fr/development/building.md @@ -1,17 +1,17 @@ --- title: Compilation -description: Compiler IceGate à partir du code source +description: Compiler {{product_name}} à partir du code source --- # Compilation à partir du Code Source -Ce guide couvre la compilation d'IceGate à partir du code source pour le développement et la production. +Ce guide couvre la compilation d'{{product_name}} à partir du code source pour le développement et la production. ## Prérequis ### Requis -- **Rust** >= 1.92.0 (pour le support de l'édition Rust 2024) +- **Rust** >= {{rust_version}} (pour le support de l'édition Rust 2024) - **Cargo** (inclus avec Rust) - **Git** @@ -96,17 +96,17 @@ debug = true ## Structure du Workspace -IceGate utilise un workspace Cargo : +{{product_name}} utilise un workspace Cargo : ```text Cargo.toml (workspace) ├── crates/ │ ├── icegate-common/Cargo.toml +│ ├── icegate-catalog-s3/Cargo.toml │ ├── icegate-queue/Cargo.toml │ ├── icegate-query/Cargo.toml │ ├── icegate-ingest/Cargo.toml -│ ├── icegate-maintain/Cargo.toml -│ └── icegate-jobmanager/Cargo.toml +│ └── icegate-maintain/Cargo.toml ``` Compiler des crates individuels : @@ -194,7 +194,7 @@ make ci ### Erreurs de Compilation -1. Vérifiez que la version de Rust est >= 1.92.0 : +1. Vérifiez que la version de Rust est >= {{rust_version}} : ```bash rustup update diff --git a/fr/development/contributing.md b/fr/development/contributing.md index ec5f728..5fd4bda 100644 --- a/fr/development/contributing.md +++ b/fr/development/contributing.md @@ -1,11 +1,11 @@ --- title: Contribuer -description: Comment contribuer au développement d'IceGate +description: Comment contribuer au développement d'{{product_name}} --- # Contribuer -Nous accueillons les contributions à IceGate ! Ce guide explique comment commencer. +Nous accueillons les contributions à {{product_name}} ! Ce guide explique comment commencer. ## Façons de Contribuer @@ -19,7 +19,7 @@ Nous accueillons les contributions à IceGate ! Ce guide explique comment commen ### Prérequis -- Rust >= 1.92.0 +- Rust >= {{rust_version}} - Docker et Docker Compose - Git @@ -100,13 +100,15 @@ Cela exécute : ``` crates/ ├── icegate-common/ # Infrastructure partagée (catalogue, stockage, métriques, traçage) +├── icegate-catalog-s3/ # Catalogue Iceberg sur S3 (par défaut) et son serveur REST ├── icegate-queue/ # Write-ahead log (Parquet sur stockage objet) -├── icegate-query/ # Service Query (APIs Loki/Prometheus/Tempo) -├── icegate-ingest/ # Service Ingest (OTLP HTTP/gRPC) -├── icegate-maintain/ # Opérations de maintenance (migration de schéma) -└── icegate-jobmanager/ # Gestion de l'état des jobs shift +├── icegate-query/ # Service Query (Loki/Tempo/Flight SQL ; routes Prometheus 501) +├── icegate-ingest/ # Service Ingest (OTLP HTTP/gRPC, WAL, shift) +└── icegate-maintain/ # Migration, compaction, GC des orphelins, crawler de tarifs ``` +Le framework de jobs/tâches n'est pas un crate du workspace : il réside dans `icegatetech/jobmanager` et est consommé comme dépendance git épinglée. + Voir l'[Architecture](../architecture/overview.md) pour les détails. ## Directives pour les Pull Requests diff --git a/fr/development/patterns.md b/fr/development/patterns.md index d40c9ba..fda2b0a 100644 --- a/fr/development/patterns.md +++ b/fr/development/patterns.md @@ -1,6 +1,6 @@ --- title: Patterns de Développement -description: Patterns standards utilisés dans le codebase IceGate +description: Patterns standards utilisés dans le codebase {{product_name}} --- # Patterns de Développement @@ -11,7 +11,7 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -Ce document définit les patterns standards utilisés dans le codebase IceGate pour la configuration, les erreurs, les routes HTTP, les handlers et les services. +Ce document définit les patterns standards utilisés dans le codebase {{product_name}} pour la configuration, les erreurs, les routes HTTP, les handlers et les services. ## Étapes Suivantes diff --git a/fr/development/setup.md b/fr/development/setup.md index b92ef7f..de64cfd 100644 --- a/fr/development/setup.md +++ b/fr/development/setup.md @@ -1,15 +1,15 @@ --- title: Environnement de Développement -description: Configurer un environnement de développement local pour IceGate +description: Configurer un environnement de développement local pour {{product_name}} --- # Environnement de Développement -Ce guide couvre la configuration d'un environnement de développement local IceGate pour contribuer au code, exécuter les tests et déboguer. +Ce guide couvre la configuration d'un environnement de développement local {{product_name}} pour contribuer au code, exécuter les tests et déboguer. ## Prérequis -- **Rust** >= 1.92.0 (édition Rust 2024) +- **Rust** >= {{rust_version}} (édition Rust 2024) - **Docker** (pour la construction des images de conteneurs) - **Git** - Un cluster Kubernetes local (pour Skaffold) @@ -58,7 +58,7 @@ Vous avez besoin d'un cluster Kubernetes local. Options : ### Exécuter avec Skaffold ```bash -# Profil par défaut (k8s local avec MinIO + Nessie) +# Profil par défaut (k8s local avec RustFS + le catalogue S3 intégré) skaffold dev # Profil OrbStack @@ -75,7 +75,7 @@ skaffold dev -p k3s-external-s3 Skaffold utilise des overlays Kustomize qui composent plusieurs charts Helm : -**Namespace IceGate (`icegate`) :** +**Namespace {{product_name}} (`icegate`) :** | Composant | Description | |-----------|-------------| @@ -87,22 +87,21 @@ Skaffold utilise des overlays Kustomize qui composent plusieurs charts Helm : | Composant | Description | |-----------|-------------| -| MinIO | Stockage compatible S3 avec les buckets : `warehouse`, `queue`, `jobs` | -| Nessie | Catalogue Iceberg REST avec persistance RocksDB | +| RustFS | Stockage compatible S3 avec les buckets : `warehouse`, `queue`, `jobs` | **Namespace Observabilité (`observability`) :** | Composant | Description | |-----------|-------------| | Prometheus | Collecte de métriques (kube-prometheus-stack) | -| Grafana | Tableaux de bord avec panneaux IceGate Ingest et Query pré-configurés | -| Jaeger | Traçage distribué pour les services IceGate | +| Grafana | Tableaux de bord avec panneaux {{product_name}} Ingest et Query pré-configurés | +| Jaeger | Traçage distribué pour les services {{product_name}} | ### Profils Skaffold | Profil | Overlay | Cas d'utilisation | |--------|---------|-------------------| -| (défaut) | `skaffold` | Développement local avec MinIO + Nessie | +| (défaut) | `skaffold` | Développement local avec RustFS + le catalogue S3 intégré | | `orbstack` | `orbstack` | Kubernetes OrbStack (macOS) | | `aws-glue` | `aws-glue` | Catalogue AWS Glue (pousse les images) | | `k3s-external-s3` | `external-s3` | S3 externe + Nessie (pousse les images) | @@ -155,10 +154,9 @@ make down | Service | Port | Description | |---------|------|-------------| -| MinIO | 9000, 9001 | Stockage compatible S3 + console | -| Nessie | 19120 | Catalogue Iceberg REST | +| RustFS | 9000, 9001 | Stockage compatible S3 + console | | Ingest | 4317, 4318 | Récepteurs OTLP gRPC et HTTP | -| Query | 3100, 9090, 3200 | APIs Loki, Prometheus, Tempo | +| Query | 3100, 9090, 3200, 8815 | APIs Loki, Tempo, Arrow Flight SQL ; les routes Prometheus retournent 501 sauf `/-/ready` | | Grafana | 3000 | Tableaux de bord | Les profils Docker Compose ajoutent des services optionnels : @@ -167,7 +165,7 @@ Les profils Docker Compose ajoutent des services optionnels : |--------|----------| | `load` | otelgen (générateur de charge de logs) | | `monitoring` | Jaeger (16686), Prometheus (9092), node-exporter, cAdvisor | -| `analytics` | Moteur SQL Trino (8082) | +| `analytics` | Nessie (19120) et moteur SQL Trino (8082) | ### Build Docker @@ -188,11 +186,11 @@ docker build -t icegate/query:dev \ ## Variables d'Environnement -Pour le développement local avec MinIO : +Pour le développement local avec RustFS : ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/fr/faq.md b/fr/faq.md index 2a33157..532a890 100644 --- a/fr/faq.md +++ b/fr/faq.md @@ -1,6 +1,6 @@ --- title: FAQ -description: Questions fréquemment posées sur IceGate +description: Questions fréquemment posées sur {{product_name}} --- # Questions Fréquemment Posées @@ -13,20 +13,20 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez ## Général -### Qu'est-ce qu'IceGate ? +### Qu'est-ce qu'{{product_name}} ? -IceGate est un moteur de lac de données d'observabilité qui stocke les logs, traces, métriques et événements dans des tables Apache Iceberg. +{{product_name}} est un moteur de lac de données d'observabilité qui stocke les logs, traces, métriques et événements dans des tables Apache Iceberg. -### Qu'est-ce qui rend IceGate différent ? +### Qu'est-ce qui rend {{product_name}} différent ? - **Standards Ouverts** : Construit sur Apache Iceberg, Arrow, Parquet et OpenTelemetry -- **Économique** : Utilise le stockage objet (S3/MinIO) +- **Économique** : Utilise le stockage objet (S3 ou RustFS) - **Transactions ACID** : Support complet des transactions - **Séparation Calcul-Stockage** : Mise à l'échelle indépendante ### Quel est le statut actuel ? -IceGate est en développement **alpha**. +{{product_name}} est en développement **alpha**. ## Démarrage diff --git a/fr/getting-started/configuration.md b/fr/getting-started/configuration.md index 8aa5b60..b28a212 100644 --- a/fr/getting-started/configuration.md +++ b/fr/getting-started/configuration.md @@ -1,6 +1,6 @@ --- title: Configuration -description: Configurer les composants IceGate +description: Configurer les composants {{product_name}} --- # Configuration @@ -42,24 +42,48 @@ La section `catalog` configure le catalogue Apache Iceberg. Elle est partagée p ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 ``` ### Paramètres du Catalogue | Paramètre | Type | Requis | Défaut | Description | |-----------|------|--------|--------|-------------| -| `backend` | enum | Oui | `memory` | Type de backend du catalogue (voir ci-dessous) | +| `backend` | enum | Oui | — | Type de backend du catalogue (voir ci-dessous). Pas de valeur par défaut : le champ est requis | | `warehouse` | string | Oui | — | Emplacement de l'entrepôt (ex. `s3://warehouse/`) | | `properties` | map | Non | `{}` | Propriétés supplémentaires spécifiques au catalogue | | `cache` | object | Non | — | Configuration du cache IO (voir [Configuration du Cache](#configuration-du-cache)) | ### Backends du Catalogue +#### Catalogue S3 (par défaut) + +Le catalogue propre à {{product_name}}. L'état du catalogue est un objet `root.json` dans le stockage objet, mis à jour par compare-and-swap : aucun service de catalogue externe n'est requis. + +```yaml +catalog: + backend: !s3 + warehouse: catalog + warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 +``` + +| Paramètre | Type | Requis | Description | +|-----------|------|--------|-------------| +| `warehouse` (dans `!s3`) | string | Oui | Préfixe de clé du stockage objet contenant l'état du catalogue | +| `properties.bucket` | string | Oui | Bucket contenant l'état du catalogue | +| `properties.region` | string | Oui | Région du client S3 du catalogue | +| `properties.endpoint` | string | Non | Endpoint personnalisé pour un stockage compatible S3. Omettre pour le vrai AWS S3 | + #### REST Catalog (Nessie) ```yaml @@ -115,9 +139,13 @@ La section optionnelle `cache` active un cache hybride foyer (mémoire + disque) ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -141,21 +169,21 @@ catalog: La section `storage` configure le backend de stockage objet. Partagée par tous les services. -### S3 / Compatible S3 (MinIO) +### S3 / Compatible S3 (RustFS) ```yaml storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` | Paramètre | Type | Requis | Défaut | Description | |-----------|------|--------|--------|-------------| | `bucket` | string | Oui | — | Nom du bucket S3 | | `region` | string | Oui | — | Région AWS | -| `endpoint` | string | Non | — | URL de point de terminaison personnalisée pour le stockage compatible S3 (MinIO, etc.) | +| `endpoint` | string | Non | — | URL de point de terminaison personnalisée pour le stockage compatible S3 (RustFS, etc.) | ### Système de Fichiers Local @@ -184,17 +212,19 @@ Référence complète pour le service Ingest (`ingest run -c ingest.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 queue: common: @@ -223,7 +253,7 @@ shift: poll_interval_ms: 1000 iteration_interval_millisecs: 30000 storage: - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 bucket: jobs prefix: shifter region: us-east-1 @@ -322,11 +352,13 @@ Référence complète pour le service Query (`query run -c query.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -336,7 +368,7 @@ storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 engine: batch_size: 8192 @@ -406,7 +438,7 @@ Lorsque `engine.wal_query_enabled` est `true`, le service query lit à la fois l | `loki.enabled` | bool | `true` | Activer l'API de requête de logs compatible Loki | | `loki.host` | string | `0.0.0.0` | Adresse d'écoute | | `loki.port` | integer | `3100` | Port de l'API Loki | -| `prometheus.enabled` | bool | `true` | Activer l'API de métriques compatible Prometheus | +| `prometheus.enabled` | bool | `true` | Servir l'API de requêtes Prometheus. Les routes sont enregistrées, mais tout handler sauf `/-/ready` retourne `501 Not Implemented` — PromQL n'est pas encore implémenté. Ce n'est pas l'endpoint de métriques : celui-ci est le bloc `metrics` sur le port 9091 | | `prometheus.host` | string | `0.0.0.0` | Adresse d'écoute | | `prometheus.port` | integer | `9090` | Port de l'API Prometheus | | `tempo.enabled` | bool | `true` | Activer l'API de traces compatible Tempo | @@ -419,17 +451,19 @@ Le service Maintain nécessite uniquement la configuration du catalogue et du st ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` ### CLI Maintain @@ -500,8 +534,8 @@ make run-analytics-release Variables d'environnement pour le développement local : ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/fr/getting-started/installation.md b/fr/getting-started/installation.md index 5168c1f..79604ce 100644 --- a/fr/getting-started/installation.md +++ b/fr/getting-started/installation.md @@ -1,21 +1,21 @@ --- title: Installation -description: Installer IceGate sur Kubernetes avec Helm +description: Installer {{product_name}} sur Kubernetes avec Helm --- # Installation -IceGate est déployé sur Kubernetes en utilisant des charts Helm, avec des overlays Kustomize pour les personnalisations spécifiques à l'environnement. +{{product_name}} est déployé sur Kubernetes en utilisant des charts Helm, avec des overlays Kustomize pour les personnalisations spécifiques à l'environnement. ## Prérequis - **Kubernetes** >= 1.28 avec **Helm 3** -- **Stockage objet :** AWS S3 ou compatible S3 (MinIO) -- **Catalogue Iceberg :** Nessie (REST), AWS S3 Tables ou AWS Glue +- **Stockage objet :** AWS S3 ou compatible S3 (RustFS) +- **Catalogue Iceberg :** le catalogue S3 intégré (par défaut, sans service externe), ou Nessie (REST), AWS S3 Tables ou AWS Glue ## Helm Chart -Le chart Helm déploie tous les composants IceGate : Ingest, Query et un job Migrate (création du schéma en tant que hook pre-install/pre-upgrade). +Le chart Helm déploie tous les composants {{product_name}} : Ingest, Query et un job Migrate (création du schéma en tant que hook pre-install/pre-upgrade). ### Installation depuis le registre OCI @@ -41,24 +41,24 @@ helm install icegate ./icegate/config/helm/icegate \ {% note info %} -Les valeurs Helm utilisent le camelCase et des clés plates (ex. `backend: rest` + `rest.uri`). Le chart traduit ces valeurs dans le format natif de configuration serde tagged enum (`backend: !rest`) attendu par les binaires IceGate. Voir [Configuration](configuration.md) pour la référence de configuration native. +Les valeurs Helm utilisent le camelCase et des clés plates (ex. `backend: s3` + `s3.warehouse`). Le chart traduit ces valeurs dans le format natif de configuration serde tagged enum (`backend: !s3`) attendu par les binaires {{product_name}}. Voir [Configuration](configuration.md) pour la référence de configuration native. {% endnote %} -Un fichier `values.yaml` minimal pour un catalogue REST (Nessie) avec stockage compatible S3 : +Un fichier `values.yaml` minimal utilisant le catalogue S3 intégré par défaut avec un stockage compatible S3. Aucun service de catalogue externe n'intervient — l'état du catalogue est un objet `root.json` dans le bucket warehouse : ```yaml catalog: - backend: rest - rest: - uri: http://nessie:19120/iceberg + backend: s3 + s3: + warehouse: catalog warehouse: "s3://warehouse/" storage: s3: bucket: warehouse region: us-east-1 - endpoint: "http://minio:9000" + endpoint: "http://rustfs:9000" queue: common: @@ -69,6 +69,28 @@ aws: region: us-east-1 ``` +### Catalogue REST (Nessie) + +À utiliser uniquement si vous exploitez déjà Nessie ou un autre catalogue REST Iceberg — cela ajoute un service externe dont le déploiement par défaut n'a pas besoin : + +```yaml +catalog: + backend: rest + rest: + uri: http://nessie:19120/iceberg + warehouse: "s3://warehouse/" + +storage: + s3: + bucket: warehouse + region: us-east-1 + endpoint: "http://rustfs:9000" + +aws: + existingSecret: icegate-aws-credentials + region: us-east-1 +``` + ### Catalogue AWS Glue ```yaml @@ -109,9 +131,9 @@ aws: | Valeur | Défaut | Description | |--------|--------|-------------| -| `catalog.backend` | `rest` | Type de catalogue : `rest`, `s3tables` ou `glue` | +| `catalog.backend` | `s3` | Type de catalogue : `s3`, `rest`, `s3tables` ou `glue` | | `storage.s3.bucket` | `warehouse` | Nom du bucket S3 | -| `storage.s3.endpoint` | `""` | Endpoint S3 personnalisé (MinIO). Omettre pour AWS S3 réel | +| `storage.s3.endpoint` | `""` | Endpoint S3 personnalisé (RustFS). Omettre pour AWS S3 réel | | `aws.existingSecret` | `""` | Secret contenant les clés `aws-access-key-id` et `aws-secret-access-key` | | `query.replicaCount` | `1` | Réplicas du service Query | | `ingest.replicaCount` | `1` | Réplicas du service Ingest | @@ -130,19 +152,19 @@ aws: ## Overlays Kustomize -Pour les personnalisations spécifiques à l'environnement, IceGate fournit des overlays Kustomize qui composent le chart Helm avec les dépendances d'infrastructure. +Pour les personnalisations spécifiques à l'environnement, {{product_name}} fournit des overlays Kustomize qui composent le chart Helm avec les dépendances d'infrastructure. ### Overlays disponibles | Overlay | Description | Infrastructure | |---------|-------------|----------------| -| `skaffold` | Développement local avec Skaffold | MinIO, Nessie, stack d'observabilité | -| `orbstack` | Runtime de conteneurs OrbStack | MinIO, Nessie, stack d'observabilité | -| `aws-glue` | Catalogue AWS Glue | Stack d'observabilité (sans MinIO/Nessie) | -| `aws-s3tables` | Catalogue AWS S3 Tables | Stack d'observabilité (sans MinIO/Nessie) | -| `external-s3` | S3 externe + catalogue Nessie | Nessie, stack d'observabilité (sans MinIO) | +| `skaffold` | Développement local avec Skaffold | RustFS, stack d'observabilité | +| `orbstack` | Runtime de conteneurs OrbStack | RustFS, stack d'observabilité | +| `aws-glue` | Catalogue AWS Glue | Stack d'observabilité (S3 externe) | +| `aws-s3tables` | Catalogue AWS S3 Tables | Stack d'observabilité (S3 externe) | +| `external-s3` | S3 externe + catalogue Nessie | Nessie, stack d'observabilité | -Tous les overlays partagent une base commune (`config/kustomize/base/`) qui déploie la stack d'observabilité : Prometheus (kube-prometheus-stack), Grafana avec des tableaux de bord IceGate pré-configurés et Jaeger pour le traçage distribué. +Tous les overlays partagent une base commune (`config/kustomize/base/`) qui déploie la stack d'observabilité : Prometheus (kube-prometheus-stack), Grafana avec des tableaux de bord {{product_name}} pré-configurés et Jaeger pour le traçage distribué. ### Utilisation @@ -159,7 +181,7 @@ skaffold dev Chaque overlay contient : - `kustomization.yaml` — déclare les charts Helm et les patches -- `values-icegate.yaml` — valeurs Helm IceGate pour cet environnement +- `values-icegate.yaml` — valeurs Helm {{product_name}} pour cet environnement - `secret-aws.yaml` — Secret des identifiants AWS (à modifier avant application) Pour créer un overlay personnalisé : diff --git a/fr/getting-started/quickstart.md b/fr/getting-started/quickstart.md index 83d38ca..18c4cd5 100644 --- a/fr/getting-started/quickstart.md +++ b/fr/getting-started/quickstart.md @@ -1,21 +1,21 @@ --- title: Guide de Démarrage -description: Ingérer et interroger vos premières données d'observabilité avec IceGate +description: Ingérer et interroger vos premières données d'observabilité avec {{product_name}} --- # Guide de Démarrage -Ce guide vous accompagne dans l'ingestion de logs, traces et métriques dans IceGate, ainsi que dans leur interrogation via l'API et Grafana. +Ce guide vous accompagne dans l'ingestion de logs, traces et métriques dans {{product_name}}, ainsi que dans leur interrogation via l'API et Grafana. {% note info %} -Ce guide suppose qu'IceGate est déjà en cours d'exécution. Consultez [Installation](installation.md) pour le déploiement Helm ou [Environnement de développement](../development/setup.md) pour un environnement local. +Ce guide suppose qu'{{product_name}} est déjà en cours d'exécution. Consultez [Installation](installation.md) pour le déploiement Helm ou [Environnement de développement](../development/setup.md) pour un environnement local. {% endnote %} ## Ingérer des Logs -IceGate accepte les données via le protocole OpenTelemetry (OTLP) sur le service d'ingestion. +{{product_name}} accepte les données via le protocole OpenTelemetry (OTLP) sur le service d'ingestion. ### Envoyer des Logs via OTLP HTTP @@ -140,7 +140,8 @@ curl -X POST http://localhost:4318/v1/metrics \ ## Interroger les Logs avec LogQL -IceGate fournit une API compatible Loki sur le service de requête (port 3100). +{{product_name}} fournit une API compatible Loki sur le service de requête (port 3100) — un sous-ensemble de +l'API de Loki, listé dans la [référence des API](../api-reference/loki.md). ### Requête de Logs Basique @@ -219,9 +220,9 @@ curl -G http://localhost:3100/loki/api/v1/series \ ## Utiliser Grafana -IceGate est compatible avec la source de données Loki de Grafana pour la visualisation et la création de tableaux de bord. +{{product_name}} est compatible avec la source de données Loki de Grafana pour la visualisation et la création de tableaux de bord. -### Ajouter IceGate comme Source de Données +### Ajouter {{product_name}} comme Source de Données 1. Ouvrez Grafana (par défaut : [http://localhost:3000](http://localhost:3000)) 2. Allez dans **Connections** > **Data sources** > **Add data source** @@ -255,11 +256,11 @@ IceGate est compatible avec la source de données Loki de Grafana pour la visual ### Tableaux de Bord Préconfigurés -Si déployé avec les overlays Kustomize ou Docker Compose, Grafana est préconfiguré avec des tableaux de bord IceGate pour les métriques des services d'ingestion et de requête. +Si déployé avec les overlays Kustomize ou Docker Compose, Grafana est préconfiguré avec des tableaux de bord {{product_name}} pour les métriques des services d'ingestion et de requête. ## Utiliser l'OpenTelemetry Collector -Pour les charges de travail de production, utilisez l'[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) pour transférer les données de vos applications vers IceGate : +Pour les charges de travail de production, utilisez l'[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) pour transférer les données de vos applications vers {{product_name}} : ```yaml # otel-collector-config.yaml @@ -286,7 +287,7 @@ service: ## Multi-Tenancy -IceGate isole les données par tenant à l'aide de l'en-tête `X-Scope-OrgID`. Les données de chaque tenant sont physiquement partitionnées. +{{product_name}} isole les données par tenant à l'aide de l'en-tête `X-Scope-OrgID`. Les données de chaque tenant sont physiquement partitionnées. ```bash # Ingestion pour le tenant "team-a" diff --git a/fr/guides/data-retention.md b/fr/guides/data-retention.md index 380cfb1..649596d 100644 --- a/fr/guides/data-retention.md +++ b/fr/guides/data-retention.md @@ -1,6 +1,6 @@ --- title: Rétention des Données -description: Configurer le cycle de vie des données, les politiques de rétention et la gestion du stockage dans IceGate +description: Configurer le cycle de vie des données, les politiques de rétention et la gestion du stockage dans {{product_name}} --- # Rétention des Données diff --git a/fr/guides/grafana-integration.md b/fr/guides/grafana-integration.md index 1e6a8ca..3b79844 100644 --- a/fr/guides/grafana-integration.md +++ b/fr/guides/grafana-integration.md @@ -1,6 +1,6 @@ --- title: Intégration Grafana -description: Configurer Grafana pour interroger les logs, traces et métriques depuis IceGate +description: Configurer Grafana pour interroger les logs, traces et métriques depuis {{product_name}} --- # Intégration Grafana @@ -11,7 +11,7 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -Ce guide explique comment connecter Grafana aux trois API de requête de {{product_name}} : Loki pour les logs (port 3100), Tempo pour les traces (port 3200) et Prometheus pour les métriques (port 9090). Vous apprendrez à configurer chaque source de données et à vérifier la connectivité. +Ce guide explique comment connecter Grafana aux API de requête de {{product_name}} : Loki pour les logs (port 3100) et Tempo pour les traces (port 3200), toutes deux implémentées, ainsi que Prometheus pour les métriques (port 9090), qui est prévu mais pas encore fonctionnel — toutes ses routes renvoient `501 Not Implemented`. Vous apprendrez à configurer chaque source de données et à vérifier la connectivité. ## Étapes Suivantes diff --git a/fr/guides/ingestion.md b/fr/guides/ingestion.md index 9bf8059..599b2ad 100644 --- a/fr/guides/ingestion.md +++ b/fr/guides/ingestion.md @@ -1,6 +1,6 @@ --- title: Ingestion de Données -description: Ingérer des logs, traces et métriques dans IceGate +description: Ingérer des logs, traces et métriques dans {{product_name}} --- # Ingestion de Données @@ -11,7 +11,7 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -IceGate accepte les données d'observabilité via le protocole OpenTelemetry (OTLP). +{{product_name}} accepte les données d'observabilité via le protocole OpenTelemetry (OTLP). ## Protocoles Supportés @@ -22,7 +22,7 @@ IceGate accepte les données d'observabilité via le protocole OpenTelemetry (OT ## Identification du Tenant -IceGate est multi-tenant. Spécifiez le tenant avec l'en-tête `X-Scope-OrgID` : +{{product_name}} est multi-tenant. Spécifiez le tenant avec l'en-tête `X-Scope-OrgID` : ```bash curl -X POST http://localhost:4318/v1/logs \ diff --git a/fr/guides/multi-tenancy.md b/fr/guides/multi-tenancy.md index 66eefb1..cdf9504 100644 --- a/fr/guides/multi-tenancy.md +++ b/fr/guides/multi-tenancy.md @@ -1,6 +1,6 @@ --- title: Multi-Tenancy -description: Configurer et utiliser l'isolation multi-tenant dans IceGate +description: Configurer et utiliser l'isolation multi-tenant dans {{product_name}} --- # Multi-Tenancy @@ -11,7 +11,7 @@ Cette page est en cours de traduction. Pour la documentation complète, veuillez {% endnote %} -IceGate est conçu comme un système multi-tenant, fournissant une isolation des données entre différentes organisations ou équipes. +{{product_name}} est conçu comme un système multi-tenant, fournissant une isolation des données entre différentes organisations ou équipes. ## Identification du Tenant diff --git a/fr/guides/performance-tuning.md b/fr/guides/performance-tuning.md index b9c067b..2a38e7a 100644 --- a/fr/guides/performance-tuning.md +++ b/fr/guides/performance-tuning.md @@ -1,6 +1,6 @@ --- title: Optimisation des Performances -description: Optimiser le débit d'ingestion, les performances de requêtes et la compaction dans IceGate +description: Optimiser le débit d'ingestion, les performances de requêtes et la compaction dans {{product_name}} --- # Optimisation des Performances diff --git a/fr/guides/querying.md b/fr/guides/querying.md index 8cb62f5..2241dd6 100644 --- a/fr/guides/querying.md +++ b/fr/guides/querying.md @@ -5,7 +5,11 @@ description: Interroger les logs, traces et métriques avec LogQL, PromQL et Tra # Interrogation des Données -IceGate fournit des APIs compatibles Loki, Prometheus et Tempo pour interroger les données d'observabilité. +{{product_name}} fournit des APIs compatibles Loki et Tempo pour interroger les données d'observabilité, +ainsi qu'Arrow Flight SQL pour du SQL généraliste. +L'[API compatible Prometheus](../api-reference/prometheus.md) est prévue mais pas encore +implémentée — interrogez les métriques via Flight SQL en attendant. Les pages de référence des API +font foi sur les points de terminaison servis aujourd'hui. ## LogQL pour les Logs diff --git a/fr/index.yaml b/fr/index.yaml index 7a31cd3..a173752 100644 --- a/fr/index.yaml +++ b/fr/index.yaml @@ -2,7 +2,9 @@ title: Documentation IceGate description: | Un moteur de lac de données d'observabilité conçu pour être rapide, facile à utiliser, économique, évolutif et tolérant aux pannes. meta: - title: IceGate - Moteur de Lac de Données d'Observabilité + # See en/index.yaml for why the product name is dropped here and why `description` is needed. + title: Lac de Données d'Observabilité + description: Documentation d'IceGate, moteur open source de lac de données d'observabilité — installation, requêtes et exploitation sur Apache Iceberg, Arrow et Parquet. links: - title: Démarrage Rapide description: Installez IceGate et exécutez vos premières requêtes en quelques minutes @@ -11,7 +13,7 @@ links: description: Apprenez à ingérer des données, interroger les logs et configurer le multi-tenancy href: guides/ingestion.md - title: Référence API - description: APIs compatibles Loki, Prometheus et Tempo + description: APIs compatibles Loki® et Tempo® (Prometheus® prévu) href: api-reference/loki.md - title: Architecture description: Comprendre l'architecture de séparation calcul-stockage d'IceGate diff --git a/fr/operations/deployment.md b/fr/operations/deployment.md index f705302..5befa15 100644 --- a/fr/operations/deployment.md +++ b/fr/operations/deployment.md @@ -1,16 +1,16 @@ --- title: Déploiement -description: Déployer IceGate en environnements de production +description: Déployer {{product_name}} en environnements de production --- # Déploiement -Ce guide couvre le déploiement d'IceGate en environnements de production. +Ce guide couvre le déploiement d'{{product_name}} en environnements de production. ## Prérequis -- **Stockage Objet :** S3, MinIO ou stockage compatible S3 -- **Catalogue Iceberg :** Nessie (REST), AWS S3 Tables ou AWS Glue +- **Stockage Objet :** S3, RustFS ou stockage compatible S3 +- **Catalogue Iceberg :** le catalogue S3 intégré (par défaut), ou Nessie (REST), AWS S3 Tables ou AWS Glue - **Docker/Kubernetes :** Pour l'orchestration des conteneurs ## Considérations d'Architecture @@ -21,7 +21,7 @@ Ce guide couvre le déploiement d'IceGate en environnements de production. |-----------|-----------------|-------| | Ingest | Horizontale | Mise à l'échelle pour le débit d'écriture | | Query | Horizontale | Mise à l'échelle pour la concurrence des requêtes | -| Maintain | Leader unique | Coordonne la compaction | +| Maintain | Horizontale | Les workers se coordonnent via l'état des jobs dans le stockage objet (compare-and-swap) | ### Exigences en Ressources @@ -50,7 +50,7 @@ Ce guide couvre le déploiement d'IceGate en environnements de production. Le projet inclut des profils Docker Compose pour différents scénarios de déploiement : ```bash -# Services principaux : MinIO, Nessie, Ingest, Query, Maintain +# Services principaux : RustFS, Ingest, Query, Maintain make run-core-release # Services principaux + générateur de charge pour les tests @@ -66,26 +66,19 @@ make run-analytics-release ```yaml # docker-compose.yml services: - minio: - image: minio/minio:latest - command: server /data --console-address ":9001" + rustfs: + image: rustfs/rustfs:1.0.0-beta.8 environment: - MINIO_ROOT_USER: ${S3_ACCESS_KEY} - MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY} + RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" volumes: - - minio-data:/data + - rustfs-data:/data ports: - - "9000:9000" - - "9001:9001" - - nessie: - image: projectnessie/nessie:latest - environment: - NESSIE_VERSION_STORE_TYPE: ROCKSDB - volumes: - - nessie-data:/data - ports: - - "19120:19120" + - "9000:9000" # S3 API + - "9001:9001" # Console ingest: image: icegate/ingest:latest @@ -100,8 +93,7 @@ services: - "4318:4318" # OTLP HTTP - "9091:9091" # Prometheus metrics depends_on: - - minio - - nessie + - rustfs query: image: icegate/query:latest @@ -116,9 +108,9 @@ services: - "3100:3100" # Loki API - "9090:9090" # Prometheus API - "3200:3200" # Tempo API + - "8815:8815" # Arrow Flight SQL depends_on: - - minio - - nessie + - rustfs maintain: image: icegate/maintain:latest @@ -128,12 +120,10 @@ services: volumes: - ./config/maintain.yaml:/etc/icegate/maintain.yaml:ro depends_on: - - minio - - nessie + - rustfs volumes: - minio-data: - nessie-data: + rustfs-data: query-cache: ``` @@ -165,7 +155,7 @@ docker build -t icegate/maintain:latest \ ### Helm Charts -IceGate inclut des Helm charts pour le déploiement Kubernetes : +{{product_name}} inclut des Helm charts pour le déploiement Kubernetes : ```bash # Installation depuis les charts locaux @@ -187,7 +177,7 @@ Des overlays Kustomize pré-construits sont disponibles pour les scénarios cour | `orbstack` | Runtime de conteneurs OrbStack | | `aws-glue` | Intégration avec le catalogue AWS Glue | | `aws-s3tables` | Intégration du catalogue AWS S3 Tables | -| `external-s3` | Stockage S3 externe (pas MinIO) | +| `external-s3` | Stockage S3 externe avec un catalogue Nessie | ```bash # Appliquer avec kustomize @@ -205,13 +195,13 @@ storage: region: us-east-1 ``` -### MinIO +### RustFS (compatible S3) ```yaml storage: backend: !s3 bucket: warehouse - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 region: us-east-1 ``` @@ -242,7 +232,7 @@ Tous les services exposent des points de terminaison de santé : ### Métriques -Les services IceGate exposent des métriques Prometheus sur un port dédié (par défaut : 9091) : +Les services {{product_name}} exposent des métriques Prometheus sur un port dédié (par défaut : 9091) : - Métriques Ingest : `http://ingest:9091/metrics` - Métriques Query : `http://query:9091/metrics` @@ -259,7 +249,7 @@ metrics: ### Auto-Observabilité avec le Traçage -IceGate peut exporter ses propres traces via OTLP pour le débogage : +{{product_name}} peut exporter ses propres traces via OTLP pour le débogage : ```yaml tracing: @@ -284,7 +274,7 @@ environment: ### Sécurité Réseau - Utilisez TLS pour toutes les connexions externes -- Restreignez l'accès à MinIO/Nessie au réseau interne uniquement +- Restreignez l'accès au stockage objet et à tout catalogue externe au réseau interne uniquement - Utilisez des politiques réseau dans Kubernetes ### Authentification diff --git a/fr/operations/maintenance.md b/fr/operations/maintenance.md index 55df1d3..743e584 100644 --- a/fr/operations/maintenance.md +++ b/fr/operations/maintenance.md @@ -1,6 +1,6 @@ --- title: Maintenance -description: Maintenir IceGate pour des performances optimales +description: Maintenir {{product_name}} pour des performances optimales --- # Maintenance @@ -19,7 +19,7 @@ maintain migrate create -c maintain.yaml ### Mises à Niveau de Schéma -Mettre à niveau les schémas de tables existants lors de la mise à jour d'IceGate : +Mettre à niveau les schémas de tables existants lors de la mise à jour d'{{product_name}} : ```bash maintain migrate upgrade -c maintain.yaml @@ -52,8 +52,9 @@ Le service Ingest transfère automatiquement les données WAL vers des tables Ic 3. Lit les fichiers WAL Parquet en parallèle 4. Fusionne et re-partitionne les données 5. Écrit les fichiers de données Iceberg optimisés -6. Valide un nouveau snapshot dans le catalogue -7. Supprime les segments WAL traités +6. Valide un nouveau snapshot dans le catalogue, en enregistrant le dernier offset WAL validé dans le résumé du snapshot + +Le shift ne supprime pas les segments WAL. Une règle de cycle de vie objet sur le bucket de la queue les récupère, et c'est l'offset du résumé du snapshot qui permet au shift de reprendre là où il s'était arrêté. ### Optimisation des Performances du Shift @@ -146,7 +147,15 @@ curl http://localhost:4318/health ### Sauvegarde du Catalogue -Nessie stocke les métadonnées du catalogue. Sauvegardez les données RocksDB : +Avec le catalogue S3 par défaut, les métadonnées sont `root.json` et les fichiers de métadonnées de table dans le bucket warehouse : une sauvegarde est donc une copie de ce préfixe, sans service à arrêter : + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup/ +``` + +`sync` n'est pas un instantané atomique : il liste puis copie, et des commits survenant entre-temps peuvent produire une copie mélangeant plusieurs générations du catalogue. Pour une copie à un instant donné, utilisez le versioning du bucket (ci-dessous) en lisant une seule version, ou effectuez la copie pendant que les écritures sont suspendues. Vérifiez toute sauvegarde en la restaurant sur un préfixe de test et en listant les tables avant de vous y fier. + +Si vous utilisez le backend de catalogue REST, sauvegardez les données RocksDB de Nessie : ```bash # Arrêter Nessie @@ -180,7 +189,7 @@ Activez le versioning sur votre bucket S3 pour la récupération à un point dan ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` diff --git a/fr/operations/troubleshooting.md b/fr/operations/troubleshooting.md index 712ba5a..e62c56f 100644 --- a/fr/operations/troubleshooting.md +++ b/fr/operations/troubleshooting.md @@ -1,6 +1,6 @@ --- title: Dépannage -description: Diagnostiquer et résoudre les problèmes courants IceGate +description: Diagnostiquer et résoudre les problèmes courants {{product_name}} --- # Dépannage @@ -61,15 +61,15 @@ docker compose logs -f maintain **Symptômes :** -- "Connection refused" vers MinIO +- "Connection refused" vers le stockage objet - Erreurs d'authentification S3 **Solutions :** -1. Vérifiez que MinIO est en cours d'exécution : +1. Sur un déploiement RustFS local, vérifiez que le stockage objet fonctionne. Le chemin de disponibilité est propre à RustFS — sur AWS S3 ou un autre fournisseur, passez directement à l'étape 3 : ```bash - curl http://localhost:9000/minio/health/ready + curl http://localhost:9000/health/ready ``` 2. Vérifiez les identifiants : @@ -79,7 +79,7 @@ docker compose logs -f maintain echo $AWS_SECRET_ACCESS_KEY ``` -3. Testez la connexion S3 : +3. Testez la connexion S3. Retirez `--endpoint-url` si le backend est le vrai AWS S3 : ```bash aws s3 ls --endpoint-url http://localhost:9000 @@ -94,19 +94,31 @@ docker compose logs -f maintain **Solutions :** -1. Vérifiez que Nessie est en cours d'exécution : +1. Avec le catalogue S3 par défaut, vérifiez que l'objet d'état du catalogue est lisible — il n'y a aucun service de catalogue à contrôler : ```bash - curl http://localhost:19120/api/v1/trees + aws --endpoint-url http://localhost:9000 s3 ls s3://warehouse/catalog/root.json ``` + Un `root.json` absent signifie que la migration n'a jamais été exécutée. Lancez `maintain migrate create` avant toute autre chose. + 2. Vérifiez la configuration du catalogue : ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 + ``` + +3. Uniquement avec le backend REST, vérifiez que Nessie est en cours d'exécution : + + ```bash + curl http://localhost:19120/api/v1/trees ``` ## Problèmes de Requêtes @@ -273,10 +285,10 @@ Si les problèmes persistent : docker stats > stats.txt ``` -2. Consultez les [GitHub Issues](https://github.com/icegatetech/icegate/issues) +2. Consultez les [GitHub Issues]({{repo_url}}/issues) 3. Incluez : - - Version d'IceGate + - Version d'{{product_name}} - Configuration (nettoyée) - Messages d'erreur - Étapes pour reproduire diff --git a/fr/toc.yaml b/fr/toc.yaml index 79647c8..4e0beda 100644 --- a/fr/toc.yaml +++ b/fr/toc.yaml @@ -96,3 +96,6 @@ items: - name: FAQ href: faq.md + + - name: Marques + href: trademarks.md diff --git a/fr/trademarks.md b/fr/trademarks.md new file mode 100644 index 0000000..31afb2c --- /dev/null +++ b/fr/trademarks.md @@ -0,0 +1,38 @@ +--- +title: Marques +description: Attribution des marques tierces citées dans la documentation {{product_name}} +--- + +# Marques + +{{product_name}} est développé par TripleCloud et distribué sous licence {{license}}. + +Cette documentation cite des projets tiers afin de décrire, de manière factuelle, les formats que +{{product_name}} écrit et les protocoles que ses API implémentent. Cet usage nominatif n'implique +aucune affiliation avec les titulaires de ces marques, ni approbation ou parrainage de leur part. + +Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL +ainsi que les logos des projets associés sont des marques déposées ou des marques de The Apache +Software Foundation aux États-Unis et/ou dans d'autres pays. + +OpenTelemetry® et Prometheus® sont des marques déposées de The Linux Foundation. + +Grafana®, Loki® et Tempo® sont des marques déposées de Raintank, Inc. dba Grafana Labs. + +{{product_name}} n'est ni affilié à ces organisations, ni approuvé ou parrainé par elles. Toutes +les autres marques appartiennent à leurs titulaires respectifs. + +## Ce que « compatible » signifie ici + +Lorsque cette documentation décrit une API comme compatible Loki ou Tempo, cela signifie que +{{product_name}} implémente un sous-ensemble de l'API HTTP de lecture du projet concerné — de quoi +servir les points de terminaison documentés dans les références [Loki](api-reference/loki.md) et [Tempo](api-reference/tempo.md), et +non une réimplémentation complète. + +L’API compatible Prometheus est **prévue, pas implémentée** : toutes ses routes renvoient +`501 Not Implemented`, à l’exception de `/-/ready`, qui répond. Sa +[page de référence](api-reference/prometheus.md) documente une surface envisagée, et non une +surface fonctionnelle. + +Les pages de référence des API font foi sur ce qui fonctionne aujourd'hui : si un point de +terminaison ou un paramètre n'y figure pas, considérez qu'il n'est pas encore implémenté. diff --git a/llms-full.txt b/llms-full.txt index 68d3c51..8157b7a 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -15,8 +15,8 @@ IceGate is deployed on Kubernetes using Helm charts, with Kustomize overlays for ## Prerequisites - **Kubernetes** >= 1.28 with **Helm 3** -- **Object Storage:** AWS S3 or S3-compatible (MinIO) -- **Iceberg Catalog:** Nessie (REST), AWS S3 Tables, or AWS Glue +- **Object Storage:** AWS S3 or S3-compatible (RustFS) +- **Iceberg Catalog:** the built-in S3 catalog (default, no external service), or Nessie (REST), AWS S3 Tables, or AWS Glue ## Helm Chart @@ -46,24 +46,24 @@ helm install icegate ./icegate/config/helm/icegate \ {% note info %} -Helm values use camelCase and flat keys (e.g., `backend: rest` + `rest.uri`). The chart translates these into the native serde tagged enum config format (`backend: !rest`) that IceGate binaries expect. See [Configuration](configuration.md) for the native config reference. +Helm values use camelCase and flat keys (e.g., `backend: s3` + `s3.warehouse`). The chart translates these into the native serde tagged enum config format (`backend: !s3`) that IceGate binaries expect. See [Configuration](configuration.md) for the native config reference. {% endnote %} -A minimal `values.yaml` for a REST catalog (Nessie) with S3-compatible storage: +A minimal `values.yaml` using the default built-in S3 catalog with S3-compatible storage. No external catalog service is involved — the catalog state is a `root.json` object in the warehouse bucket: ```yaml catalog: - backend: rest - rest: - uri: http://nessie:19120/iceberg + backend: s3 + s3: + warehouse: catalog warehouse: "s3://warehouse/" storage: s3: bucket: warehouse region: us-east-1 - endpoint: "http://minio:9000" + endpoint: "http://rustfs:9000" queue: common: @@ -74,6 +74,28 @@ aws: region: us-east-1 ``` +### REST Catalog (Nessie) + +Use this only if you already run a Nessie or other Iceberg REST catalog — it adds an external service the default deployment does not need: + +```yaml +catalog: + backend: rest + rest: + uri: http://nessie:19120/iceberg + warehouse: "s3://warehouse/" + +storage: + s3: + bucket: warehouse + region: us-east-1 + endpoint: "http://rustfs:9000" + +aws: + existingSecret: icegate-aws-credentials + region: us-east-1 +``` + ### AWS Glue Catalog ```yaml @@ -114,9 +136,9 @@ aws: | Value | Default | Description | |-------|---------|-------------| -| `catalog.backend` | `rest` | Catalog type: `rest`, `s3tables`, or `glue` | +| `catalog.backend` | `s3` | Catalog type: `s3`, `rest`, `s3tables`, or `glue` | | `storage.s3.bucket` | `warehouse` | S3 bucket name | -| `storage.s3.endpoint` | `""` | Custom S3 endpoint (MinIO). Omit for real AWS S3 | +| `storage.s3.endpoint` | `""` | Custom S3 endpoint (RustFS). Omit for real AWS S3 | | `aws.existingSecret` | `""` | Secret with `aws-access-key-id` and `aws-secret-access-key` keys | | `query.replicaCount` | `1` | Query service replicas | | `ingest.replicaCount` | `1` | Ingest service replicas | @@ -141,11 +163,11 @@ For environment-specific customizations, IceGate provides Kustomize overlays tha | Overlay | Description | Infrastructure | |---------|-------------|----------------| -| `skaffold` | Local development with Skaffold | MinIO, Nessie, observability stack | -| `orbstack` | OrbStack container runtime | MinIO, Nessie, observability stack | -| `aws-glue` | AWS Glue catalog | Observability stack (no MinIO/Nessie) | -| `aws-s3tables` | AWS S3 Tables catalog | Observability stack (no MinIO/Nessie) | -| `external-s3` | External S3 + Nessie catalog | Nessie, observability stack (no MinIO) | +| `skaffold` | Local development with Skaffold | RustFS, observability stack | +| `orbstack` | OrbStack container runtime | RustFS, observability stack | +| `aws-glue` | AWS Glue catalog | Observability stack (external S3) | +| `aws-s3tables` | AWS S3 Tables catalog | Observability stack (external S3) | +| `external-s3` | External S3 + Nessie catalog | Nessie, observability stack | All overlays share a common base (`config/kustomize/base/`) that deploys the observability stack: Prometheus (kube-prometheus-stack), Grafana with pre-built IceGate dashboards, and Jaeger for distributed tracing. @@ -198,7 +220,7 @@ curl http://localhost:3100/ready # Configuration -{{product_name}} uses YAML or TOML configuration files. The format is auto-detected by file extension (`.yaml`/`.yml` for YAML, `.toml` for TOML). +IceGate uses YAML or TOML configuration files. The format is auto-detected by file extension (`.yaml`/`.yml` for YAML, `.toml` for TOML). ## CLI Usage @@ -235,24 +257,48 @@ The `catalog` section configures the Apache Iceberg catalog. It is shared by all ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 ``` ### Catalog Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| -| `backend` | enum | No | `memory` | Catalog backend type (see below) | +| `backend` | enum | Yes | — | Catalog backend type (see below). No default — the field is required | | `warehouse` | string | Yes | — | Warehouse location (e.g., `s3://warehouse/`) | | `properties` | map | No | `{}` | Additional catalog-specific properties | | `cache` | object | No | — | IO cache configuration (see [Cache Configuration](#cache-configuration)) | ### Catalog Backends +#### S3 Catalog (Default) + +IceGate's own catalog. Catalog state is a `root.json` object in object storage, updated by compare-and-swap, so no external catalog service is required: + +```yaml +catalog: + backend: !s3 + warehouse: catalog + warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 +``` + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `warehouse` (inside `!s3`) | string | Yes | Object-storage key prefix holding the catalog state | +| `properties.bucket` | string | Yes | Bucket holding the catalog state | +| `properties.region` | string | Yes | Region for the catalog's S3 client | +| `properties.endpoint` | string | No | Custom endpoint for S3-compatible storage. Omit for real AWS S3 | + #### REST Catalog (Nessie) ```yaml @@ -308,9 +354,13 @@ The optional `cache` section enables a foyer hybrid cache (memory + disk) to red ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -334,21 +384,21 @@ catalog: The `storage` section configures the object storage backend. Shared by all services. -### S3 / S3-Compatible (MinIO) +### S3 / S3-Compatible (RustFS) ```yaml storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `bucket` | string | Yes | — | S3 bucket name | | `region` | string | Yes | — | AWS region | -| `endpoint` | string | No | — | Custom endpoint URL for S3-compatible storage (MinIO, etc.) | +| `endpoint` | string | No | — | Custom endpoint URL for S3-compatible storage (RustFS, etc.) | ### Local Filesystem @@ -377,17 +427,19 @@ Full reference for the Ingest service (`ingest run -c ingest.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 queue: common: @@ -416,7 +468,7 @@ shift: poll_interval_ms: 1000 iteration_interval_millisecs: 30000 storage: - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 bucket: jobs prefix: shifter region: us-east-1 @@ -515,11 +567,13 @@ Full reference for the Query service (`query run -c query.yaml`). ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -529,7 +583,7 @@ storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 engine: batch_size: 8192 @@ -599,7 +653,7 @@ When `engine.wal_query_enabled` is `true`, the query service reads both committe | `loki.enabled` | bool | `true` | Enable Loki-compatible log query API | | `loki.host` | string | `0.0.0.0` | Bind address | | `loki.port` | integer | `3100` | Loki API port | -| `prometheus.enabled` | bool | `true` | Enable Prometheus-compatible metrics API | +| `prometheus.enabled` | bool | `true` | Serve the Prometheus query API. Routes are registered, but every handler except `/-/ready` returns `501 Not Implemented` — PromQL is not implemented yet. This is not the metrics endpoint; that is the `metrics` block on port 9091 | | `prometheus.host` | string | `0.0.0.0` | Bind address | | `prometheus.port` | integer | `9090` | Prometheus API port | | `tempo.enabled` | bool | `true` | Enable Tempo-compatible trace API | @@ -612,17 +666,19 @@ The Maintain service only requires catalog and storage configuration: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` ### Maintain CLI @@ -692,8 +748,8 @@ make run-analytics-release Environment variables for local development: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` @@ -1356,11 +1412,11 @@ curl http://localhost:3100/loki/api/v1/label/service_name/values \ # Grafana Integration -This guide covers connecting Grafana to all three {{product_name}} query APIs: Loki (logs), Tempo (traces), and Prometheus (metrics). +This guide covers connecting Grafana to the IceGate query APIs: Loki (logs) and Tempo (traces), both implemented, plus Prometheus (metrics), which is planned and not yet functional. ## Prerequisites -- {{product_name}} Query service running (see [Installation](../getting-started/installation.md)) +- IceGate Query service running (see [Installation](../getting-started/installation.md)) - Grafana 10+ ([grafana.com/oss](https://grafana.com/oss/grafana/)) ## Verify Query Service Health @@ -1384,7 +1440,7 @@ All endpoints should return HTTP 200. ### Loki Data Source (Logs) -{{product_name}} implements the Grafana Loki API on port **3100**. +IceGate implements the Grafana Loki API on port **3100**. 1. Go to **Connections** > **Data sources** > **Add data source** 2. Select **Loki** @@ -1414,11 +1470,11 @@ datasources: ### Tempo Data Source (Traces) -{{product_name}} implements the Grafana Tempo API on port **3200**. +IceGate implements the Grafana Tempo API on port **3200**. {% note warning %} -The Tempo API provides basic trace retrieval and search. TraceQL support is planned for future releases. +The Tempo API provides trace retrieval and search, and TraceQL is supported for /api/search; TraceQL features that are not yet implemented return 501 Not Implemented. {% endnote %} @@ -1457,11 +1513,11 @@ datasources: ### Prometheus Data Source (Metrics) -{{product_name}} implements the Grafana Prometheus API on port **9090**. +IceGate mounts the Grafana Prometheus API routes on port **9090**, but they are not implemented: every route returns 501 Not Implemented except /-/ready. {% note warning %} -The Prometheus query API is currently under development. Metadata endpoints (labels, series) are available, but PromQL queries are not yet supported. Use the Loki API with LogQL metric queries as an alternative for log-based metrics. +The Prometheus query API is NOT implemented. Every route returns 501 Not Implemented, including the metadata endpoints; only /-/ready responds. PromQL is not parsed at all yet. Use the Loki API with LogQL metric queries as an alternative for log-based metrics. {% endnote %} @@ -1541,7 +1597,7 @@ datasources: ### Logs to Traces -{{product_name}} stores `trace_id` and `span_id` fields in log records. Configure Grafana to link from log lines to traces: +IceGate stores `trace_id` and `span_id` fields in log records. Configure Grafana to link from log lines to traces: 1. In the Loki data source settings, go to **Derived fields** 2. Add a derived field: @@ -1611,7 +1667,7 @@ Create a dashboard with three panels: ## Using IceGate as a Drop-In for Existing Grafana -If you have an existing Grafana setup with Loki, you can point it at {{product_name}} by changing only the data source URL: +If you have an existing Grafana setup with Loki, you can point it at IceGate by changing only the data source URL: 1. Go to **Connections** > **Data sources** 2. Edit your existing Loki data source @@ -1619,7 +1675,7 @@ If you have an existing Grafana setup with Loki, you can point it at {{product_n 4. Add the `X-Scope-OrgID` header if not already present 5. Click **Save & Test** -Your existing dashboards, alerting rules, and saved queries will continue to work because {{product_name}} implements the same Loki API. +Dashboards, alerting rules, and saved queries keep working as long as they stay within the endpoints and LogQL features IceGate implements — it serves a subset of the Loki read API, not all of it. Check the [Loki API reference](../api-reference/loki.md) and the [LogQL implementation status](querying.md) for anything a panel depends on, and re-test alert rules after switching. {% note info %} @@ -1632,8 +1688,8 @@ LogQL metric queries (`rate()`, `count_over_time()`, `sum by()`, etc.) are suppo | API | Port | Grafana Data Source Type | Status | |-----|------|--------------------------|--------| | Loki (logs) | 3100 | Loki | Fully implemented | -| Tempo (traces) | 3200 | Tempo | Basic retrieval and search (TraceQL planned) | -| Prometheus (metrics) | 9090 | Prometheus | Metadata only (PromQL planned) | +| Tempo (traces) | 3200 | Tempo | Retrieval and search; TraceQL supported (unimplemented features return 501) | +| Prometheus (metrics) | 9090 | Prometheus | Planned; every route returns 501 except /-/ready | ## Next Steps @@ -1730,7 +1786,7 @@ datasources: ## Architecture: Multi-Tenant Deployment -A multi-tenant {{product_name}} deployment uses a single cluster shared by all tenants. Data isolation is enforced at the storage layer: +A multi-tenant IceGate deployment uses a single cluster shared by all tenants. Data isolation is enforced at the storage layer: ``` Tenant A ──┐ ┌── Iceberg partition: tenant_id="tenant-a" @@ -1764,7 +1820,7 @@ Compare to dedicated-table approaches: | Separate tables per tenant | Medium | High (schema management per tenant) | Physical | | Separate clusters per tenant | High | Very high | Full | -{{product_name}} uses the shared tables approach, which is optimal for SaaS and platform use cases where many tenants share similar data shapes. +IceGate uses the shared tables approach, which is optimal for SaaS and platform use cases where many tenants share similar data shapes. ## Concrete Example: Three Tenants @@ -1872,7 +1928,7 @@ Consider implementing per-tenant limits: # Performance Tuning -This guide covers tuning {{product_name}} for high-volume workloads across ingestion, compaction, and query paths. +This guide covers tuning IceGate for high-volume workloads across ingestion, compaction, and query paths. ## Architecture Overview @@ -2145,11 +2201,11 @@ catalog: # Data Retention -This guide covers managing the data lifecycle in {{product_name}}, from WAL segments through Iceberg table maintenance. +This guide covers managing the data lifecycle in IceGate, from WAL segments through Iceberg table maintenance. ## Data Lifecycle -Data in {{product_name}} moves through three stages: +Data in IceGate moves through three stages: 1. **WAL (Write-Ahead Log)** — temporary Parquet files in object storage, written by Ingest 2. **Iceberg tables** — optimized, partitioned Parquet files managed by Apache Iceberg @@ -2159,20 +2215,32 @@ Each stage has independent retention controls. ## WAL Retention -WAL segments are automatically deleted after the shift process compacts them into Iceberg tables. For the queue bucket, configure an object storage lifecycle rule as a safety net: +Shift does not delete WAL segments after committing them to Iceberg — a lifecycle rule on the queue bucket is what reclaims them, so configure one: + +{% note warning %} -### MinIO Lifecycle Rule +Size the expiration from your worst-case unshifted-WAL window, not for convenience. A segment is only safe to expire once shift has committed it and recorded its offset in an Iceberg snapshot. If shift is stopped, backlogged, or recovering for longer than the expiration, the rule deletes segments whose offsets were never committed. The snapshot offset only tells shift where to resume — it cannot rebuild a deleted segment, so that is acknowledged data lost. One day suits the demo stack; choose yours from how long ingest can plausibly run without a successful shift commit, and alert on shift lag rather than relying on the rule to stay ahead of it. + +{% endnote %} + +The bucket in both commands below is the one from `queue.common.base_path` (`s3://queue/` by default). Substitute your own if you changed it — a rule applied to the wrong bucket leaves the real WAL bucket unmanaged. + +### RustFS (and other S3-compatible stores) + +RustFS speaks the S3 API, so the same `aws s3api` call the project's own bootstrap uses works against it: ```bash # Set 1-day TTL on queue bucket -mc ilm rule add --expire-days 1 myminio/queue +aws --endpoint-url http://localhost:9000 s3api put-bucket-lifecycle-configuration \ + --bucket queue \ + --lifecycle-configuration '{"Rules":[{"ID":"expire-1d","Status":"Enabled","Filter":{"Prefix":""},"Expiration":{"Days":1}}]}' ``` ### AWS S3 Lifecycle Rule ```bash aws s3api put-bucket-lifecycle-configuration \ - --bucket icegate-queue \ + --bucket queue \ --lifecycle-configuration '{ "Rules": [{ "ID": "expire-wal-segments", @@ -2352,13 +2420,21 @@ Enable S3 versioning for point-in-time recovery of the warehouse bucket: ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` ### Catalog Backup -Back up the Nessie catalog (RocksDB storage): +On the default S3 catalog there is no service to stop and no database to dump — the catalog is `root.json` plus the table metadata files, in the warehouse bucket. Enabling versioning on that bucket (above) already gives point-in-time recovery. For an off-site copy, sync the catalog prefix: + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup-$(date +%Y%m%d)/ +``` + +Because `root.json` is replaced by compare-and-swap, a copy taken mid-commit is still a consistent earlier version rather than a torn write. + +If you run the REST catalog backend instead, back up Nessie's RocksDB storage: ```bash # Stop Nessie @@ -2388,7 +2464,7 @@ docker start nessie # Centralized Logging for Microservices -This cookbook walks through setting up centralized log collection from multiple microservices into {{product_name}} using the OpenTelemetry Collector. +This cookbook walks through setting up centralized log collection from multiple microservices into IceGate using the OpenTelemetry Collector. ## Architecture @@ -2422,7 +2498,7 @@ This cookbook walks through setting up centralized log collection from multiple ## Step 1: Deploy the OpenTelemetry Collector -The Collector acts as a central aggregation point, decoupling your services from {{product_name}}. +The Collector acts as a central aggregation point, decoupling your services from IceGate. ```yaml # otel-collector-config.yaml @@ -2544,7 +2620,7 @@ exporter, _ := otlploggrpc.New(ctx, ### Direct Ingestion (without Collector) -For simple setups, send logs directly to {{product_name}}: +For simple setups, send logs directly to IceGate: ```bash curl -X POST http://localhost:4318/v1/logs \ @@ -2627,7 +2703,7 @@ curl http://localhost:3100/loki/api/v1/label/service_name/values \ ## Step 4: Set Up Grafana -Add {{product_name}} as a Loki data source in Grafana: +Add IceGate as a Loki data source in Grafana: ```yaml # grafana/provisioning/datasources/icegate.yaml @@ -2681,7 +2757,7 @@ Each team queries only their own data. See [Multi-Tenancy](../guides/multi-tenan # End-to-End Distributed Tracing -This cookbook walks through instrumenting services with OpenTelemetry, sending trace data to {{product_name}} via OTLP, and retrieving traces via the Tempo-compatible API. +This cookbook walks through instrumenting services with OpenTelemetry, sending trace data to IceGate via OTLP, and retrieving traces via the Tempo-compatible API. {% note warning %} @@ -2973,7 +3049,7 @@ See [Grafana Integration](../guides/grafana-integration.md) for cross-signal lin ## Span Data Model -Spans stored in {{product_name}} include: +Spans stored in IceGate include: | Field | Type | Description | |-------|------|-------------| @@ -2997,17 +3073,17 @@ Spans stored in {{product_name}} include: # Cross-Signal Correlation -This cookbook shows how to correlate data across logs, traces, and metrics in {{product_name}} to quickly move from an alert to a root cause. +This cookbook shows how to correlate data across logs, traces, and metrics in IceGate to quickly move from an alert to a root cause. {% note warning %} -This guide uses the Loki API (fully implemented) and the Tempo API (basic trace retrieval and search available; TraceQL planned). The Prometheus API is under development — use LogQL metric queries as an alternative for log-based metrics. +This guide uses the Loki API (fully implemented) and the Tempo API (retrieval and search available; TraceQL supported, unimplemented features return 501). The Prometheus API is NOT implemented — every route returns 501 except /-/ready; use LogQL metric queries as an alternative for log-based metrics. {% endnote %} ## How Correlation Works -{{product_name}} stores all observability signals in Apache Iceberg tables with shared fields that enable cross-signal linking: +IceGate stores all observability signals in Apache Iceberg tables with shared fields that enable cross-signal linking: | Field | Present In | Purpose | |-------|-----------|---------| @@ -4027,7 +4103,7 @@ X-Scope-OrgID: my-tenant {% note warning %} -The Tempo API is currently under development. Basic trace retrieval is available but TraceQL support is planned for future releases. +The Tempo API implements a subset of Tempo's HTTP read API. Trace retrieval and /api/search are available, and TraceQL is supported for search — TraceQL features that are not yet implemented return 501 Not Implemented rather than silently returning wrong results. {% endnote %} @@ -4164,7 +4240,7 @@ Spans stored in IceGate include: # Architecture Overview -IceGate is an observability data lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables with DataFusion as the query engine. +IceGate is an observability data lake engine that stores logs, traces, metrics, events, and LLM operations in Apache Iceberg tables with DataFusion as the query engine. ## Design Principles @@ -4202,13 +4278,18 @@ The Write-Ahead Log (WAL) stores data as Parquet files organized for compatibili **Purpose:** Execute queries against logs, traces, metrics, and events - **Engine:** Apache DataFusion + Apache Arrow -- **APIs:** Loki (3100), Prometheus (9090), Tempo (3200) -- **Query Languages:** LogQL, PromQL (planned), TraceQL (planned) +- **APIs:** Loki (3100), Tempo (3200), Arrow Flight SQL (8815); Prometheus (9090) serves routes but its handlers still return `501 Not Implemented` +- **Query Languages:** LogQL, TraceQL, SQL; PromQL planned +- **Multi-tenancy:** Tenant taken from the `X-Scope-OrgID` header, or the `x-scope-orgid` gRPC metadata for Flight SQL + +Arrow Flight SQL is strictly read-only — DDL and DML are rejected — and enforces `tenant_id` at the row level on every scan, so JDBC, ODBC, and ADBC clients query `iceberg.icegate.<table>` with no IceGate-specific client code. The query service reads from both: - **WAL**: For real-time data (seconds-old) -- **Iceberg Tables**: For historical data (compacted) +- **Iceberg Tables**: For historical data (shifted and compacted) + +The boundary between the two is the WAL offset recorded in the Iceberg snapshot summary, so a row is read from exactly one side and never counted twice. ### Maintain Service @@ -4216,10 +4297,25 @@ The query service reads from both: **Purpose:** Data lifecycle and optimization operations -- **Compaction:** Merge small WAL files into optimized Iceberg tables -- **TTL:** Expire and delete old data based on retention policies -- **Optimization:** Rewrite data files for better query performance -- **Cleanup:** Remove orphaned files and expired snapshots +- **Schema migration:** Create the Iceberg tables (`maintain migrate create`) +- **Data compaction:** Rewrite small Parquet data files into fewer, larger sorted ones +- **Manifest compaction:** Repack fragmented Iceberg manifests +- **Orphan GC:** Delete objects the current table metadata no longer references, once past a grace period +- **Pricing crawler:** Crawl LLM rate cards from external feeds into the global `icegate.prices` table + +Compaction, GC, and the pricing crawler each run as jobs whose state lives in object storage, under their own job-state prefix. + +### Catalog + +![Catalog Components](../../assets/c4/structurizr-CatalogComponents.png) + +**Purpose:** Organize the data lake with ACID transactions, without a dedicated OLTP database + +- **Default backend:** IceGate's own S3 catalog — catalog state is a `root.json` object updated by compare-and-swap +- **Alternative backends:** REST (Nessie), AWS S3 Tables, AWS Glue +- **Deployment:** Linked into Ingest, Query, and Maintain by default; optionally deployed standalone as an Iceberg REST server on port 8181 + +A conditional read keeps the cached catalog root fresh; table metadata is immutable per location, so it is cached unconditionally in an LRU. ### Alert Service (Planned) @@ -4238,8 +4334,10 @@ The query service reads from both: | Memory Format | Apache Arrow 57.0 | Zero-copy data processing | | Storage Format | Apache Parquet 57.0 | Columnar storage with ZSTD compression | | Ingestion | OpenTelemetry 0.31 | Standard observability protocol (gRPC + HTTP) | -| Catalog | Nessie, AWS S3 Tables, AWS Glue | Iceberg REST catalog backends | -| Job Manager | icegate-jobmanager | S3-based shift job state management | +| SQL Interface | Arrow Flight SQL 57.0 | Read-only SQL for JDBC, ODBC, and ADBC clients | +| Catalog | S3 catalog (default), Nessie, AWS S3 Tables, AWS Glue | Iceberg catalog backends; the default keeps state in object storage | +| Object Storage | RustFS, or any S3-compatible store | WAL segments, Iceberg data, catalog state, job state | +| Job Manager | jobmanager (separate repository) | S3-based job state for shift, compaction, GC, and pricing | | Caching | foyer 0.22 | Hybrid memory + disk cache for S3 reads | | Language | Rust 1.92+ (2024 edition) | Memory-safe, high-performance runtime | @@ -4247,6 +4345,10 @@ The query service reads from both: ### Ingestion Flow +![Ingestion Sequence](../../assets/c4/structurizr-IngestionFlow.png) + +Steps 1-7 are the write path, acknowledged once the WAL segment lands. Steps 8-14 are shift, which runs independently of the request. + 1. Client sends OTLP data to Ingest service 2. Ingest validates and transforms data 3. Data written to WAL as Parquet files @@ -4254,6 +4356,8 @@ The query service reads from both: ### Query Flow +![Query Sequence](../../assets/c4/structurizr-QueryFlow.png) + 1. Client sends query to Query service 2. Query parsed and planned by DataFusion 3. Data read from Iceberg tables and/or WAL @@ -4265,8 +4369,17 @@ The query service reads from both: 2. Groups segments into shift tasks 3. Reads WAL files in parallel, merges and re-partitions data 4. Writes optimized Iceberg data files -5. Commits new snapshot to catalog -6. Deletes processed WAL segments +5. Commits a new snapshot to the catalog, recording the last committed WAL offset in the snapshot summary + +Shift never deletes WAL segments. They are reclaimed by an object lifecycle rule on the queue bucket, and the offset in the snapshot summary is what lets shift resume where it left off. + +That makes the lifecycle expiration a durability parameter, not housekeeping: a segment has to outlive the commit that covers it. If shift is delayed or failing when the rule fires, segments whose offsets were never committed are deleted and the data is gone. See [Data Retention](../guides/data-retention.md) for sizing. + +### Maintenance Flow + +![Maintenance Sequence](../../assets/c4/structurizr-MaintenanceFlow.png) + +Migration is a one-shot job. Compaction, orphan GC, and the pricing crawler are independent loops on their own schedules — the step numbers order each loop, not the loops against each other. Each claims work under its own job-state prefix, so the loops never fight over task ownership. They still share the tables underneath — compaction commits rewrite snapshots while GC deletes unreferenced objects — which is why GC only removes files older than its grace period and commits use optimistic concurrency, retrying on conflict. ## Scalability @@ -4274,7 +4387,7 @@ The query service reads from both: - **Ingest:** Scale replicas for higher throughput - **Query:** Scale replicas for concurrent queries -- **Maintain:** Single instance (leader election) +- **Maintain:** Scale replicas for more rewrite throughput — workers share job state in object storage with compare-and-swap and commit with optimistic concurrency, so parallel instances are safe. Prefer raising in-process worker count first; returns taper as replicas grow, since all workers on a table contend on one job-state object. ### Storage Scaling @@ -4291,7 +4404,7 @@ The query service reads from both: # Data Model -IceGate stores observability data in four Apache Iceberg tables: logs, spans, events, and metrics. +IceGate stores observability data in five tenant-scoped Apache Iceberg tables — logs, spans, events, metrics, and operations — plus one global reference table, prices. ## Table Overview @@ -4301,12 +4414,14 @@ IceGate stores observability data in four Apache Iceberg tables: logs, spans, ev | `spans` | Distributed trace spans | Request tracing | | `events` | Semantic events | Business events, alerts | | `metrics` | All metric types | Performance monitoring | +| `operations` | LLM and agent operations | Token usage, cost, prompt and completion capture | +| `prices` | Global LLM rate card (no `tenant_id`) | Reference rates for costing `operations` | ## Common Design Patterns ### Multi-Tenancy -All tables use identity partitioning on `tenant_id`: +The five tenant-scoped tables use identity partitioning on `tenant_id`. `prices` is reference data shared by every tenant, so it carries no `tenant_id` and is partitioned differently: ```sql partitioning = ARRAY['tenant_id', 'account_id', 'day(timestamp)'] @@ -4550,6 +4665,125 @@ CREATE TABLE metrics ( | `exponential_histogram` | `count`, `sum`, `scale`, `zero_count`, `positive_*`, `negative_*` | | `summary` | `count`, `sum`, `quantile_values` | +## Operations Table + +LLM and agent operations, following the OpenTelemetry generative-AI semantic conventions. + +```sql +CREATE TABLE operations ( + tenant_id VARCHAR NOT NULL, + conversation_id VARCHAR, + + -- identity + trace_id VARBINARY NOT NULL, + span_id VARBINARY NOT NULL, + parent_span_id VARBINARY, + service_name VARCHAR, + scope_name VARCHAR, + scope_version VARCHAR, + + -- timing + timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + end_timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + duration_micros BIGINT NOT NULL, + ingested_timestamp TIMESTAMP(6) WITH TIME ZONE NOT NULL, + + operation_name VARCHAR NOT NULL, + + -- provider and model + provider_name VARCHAR, + request_model VARCHAR, + response_model VARCHAR, + response_id VARCHAR, + + -- sampling parameters + temperature DOUBLE, + top_p DOUBLE, + top_k BIGINT, + max_tokens BIGINT, + frequency_penalty DOUBLE, + presence_penalty DOUBLE, + seed BIGINT, + stream BOOLEAN, + choice_count BIGINT, + output_type VARCHAR, + reasoning_effort VARCHAR, + + time_to_first_chunk_ms BIGINT, + + -- token usage + input_tokens BIGINT, + output_tokens BIGINT, + total_tokens BIGINT, + reasoning_tokens BIGINT, + cache_creation_input_tokens BIGINT, + cache_read_input_tokens BIGINT, + + user_id VARCHAR, + + -- tool calls + tool_name VARCHAR, + tool_call_id VARCHAR, + tool_type VARCHAR, + tool_description VARCHAR, + + data_source_id VARCHAR, + embedding_dimensions INTEGER, + + -- server and status + server_address VARCHAR, + server_port INTEGER, + status_code INTEGER, + status_message VARCHAR, + error_type VARCHAR, + + -- agent and workflow + agent_id VARCHAR, + agent_name VARCHAR, + agent_version VARCHAR, + agent_description VARCHAR, + workflow_name VARCHAR, + + -- content, JSON-encoded + input_messages VARCHAR, + output_messages VARCHAR, + system_instructions VARCHAR, + tool_definitions VARCHAR, + tool_call_arguments VARCHAR, + tool_call_result VARCHAR, + + stop_sequences ARRAY(VARCHAR), + finish_reasons ARRAY(VARCHAR), + encoding_formats ARRAY(VARCHAR) +) +``` + +**Partitioning:** `tenant_id` (identity), `day(timestamp)` + +**Sorting:** `trace_id`, `timestamp DESC` — clusters a trace's operations together, recent first + +The six `VARCHAR` content columns (`input_messages`, `output_messages`, `system_instructions`, `tool_definitions`, `tool_call_arguments`, `tool_call_result`) hold JSON-encoded payloads rather than parsed structures, so prompt and completion shapes can vary per provider without a schema change. + +## Prices Table + +A global LLM rate card, populated by the Maintain service's pricing crawler from the OpenRouter and LiteLLM feeds. + +Unlike the five telemetry tables it carries **no `tenant_id`** — rates are reference data, identical for every tenant. It is an append-only observation log: a row is written only when a rate first differs from the previous one for its key, and `valid_to` is derived at query time. + +**Key:** `(provider, model, service_tier, region, min_input_tokens, valid_from)` + +Context tiers and service tiers live in the key rather than in extra columns, so the rate columns stay flat as the card grows. Rate columns are `DECIMAL(38, 10)` rather than floating point — money has to be exact, and binary `f64` cannot represent a value like `0.075` or sum it without drift. + +### Joining Prices to Operations + +The query engine exposes a derived view, `prices_effective`, which adds `valid_to` — the next revision's `valid_from` for the same key, `NULL` for the row currently in effect. It is a DataFusion object, so the Loki, Tempo, and Flight SQL paths see it; Trino reads the Iceberg catalog directly and does not, which is why the raw table stays self-sufficient. + +{% note warning %} + +IceGate does not compute cost, and `operations` does not carry the full pricing key. It records `provider_name` and `request_model`, which line up with `prices.provider` and `prices.model`, but nothing for `service_tier`, `region`, or `min_input_tokens`. A cost query has to supply those three from deployment knowledge — a fixed tier and region per account, say. Treat such a join as an estimate parameterised by your own assumptions, not a derivation the schema guarantees. + +{% endnote %} + ## Query Examples ### Logs Query @@ -4600,8 +4834,8 @@ This guide covers deploying IceGate in production environments. ## Prerequisites -- **Object Storage:** S3, MinIO, or S3-compatible storage -- **Iceberg Catalog:** Nessie (REST), AWS S3 Tables, or AWS Glue +- **Object Storage:** S3, RustFS, or S3-compatible storage +- **Iceberg Catalog:** the built-in S3 catalog (default), or Nessie (REST), AWS S3 Tables, or AWS Glue - **Docker/Kubernetes:** For container orchestration ## Architecture Considerations @@ -4612,7 +4846,7 @@ This guide covers deploying IceGate in production environments. |-----------|---------|-------| | Ingest | Horizontal | Scale for write throughput | | Query | Horizontal | Scale for query concurrency | -| Maintain | Single leader | Coordinates compaction | +| Maintain | Horizontal | Workers coordinate through job state in object storage (compare-and-swap) | ### Resource Requirements @@ -4641,7 +4875,7 @@ This guide covers deploying IceGate in production environments. The project includes Docker Compose profiles for different deployment scenarios: ```bash -# Core services: MinIO, Nessie, Ingest, Query, Maintain +# Core services: RustFS, Ingest, Query, Maintain make run-core-release # Core + load generator for testing @@ -4657,26 +4891,19 @@ make run-analytics-release ```yaml # docker-compose.yml services: - minio: - image: minio/minio:latest - command: server /data --console-address ":9001" - environment: - MINIO_ROOT_USER: ${S3_ACCESS_KEY} - MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY} - volumes: - - minio-data:/data - ports: - - "9000:9000" - - "9001:9001" - - nessie: - image: projectnessie/nessie:latest + rustfs: + image: rustfs/rustfs:1.0.0-beta.8 environment: - NESSIE_VERSION_STORE_TYPE: ROCKSDB + RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" volumes: - - nessie-data:/data + - rustfs-data:/data ports: - - "19120:19120" + - "9000:9000" # S3 API + - "9001:9001" # Console ingest: image: icegate/ingest:latest @@ -4691,8 +4918,7 @@ services: - "4318:4318" # OTLP HTTP - "9091:9091" # Prometheus metrics depends_on: - - minio - - nessie + - rustfs query: image: icegate/query:latest @@ -4707,9 +4933,9 @@ services: - "3100:3100" # Loki API - "9090:9090" # Prometheus API - "3200:3200" # Tempo API + - "8815:8815" # Arrow Flight SQL depends_on: - - minio - - nessie + - rustfs maintain: image: icegate/maintain:latest @@ -4719,12 +4945,10 @@ services: volumes: - ./config/maintain.yaml:/etc/icegate/maintain.yaml:ro depends_on: - - minio - - nessie + - rustfs volumes: - minio-data: - nessie-data: + rustfs-data: query-cache: ``` @@ -4778,7 +5002,7 @@ Pre-built Kustomize overlays are available for common scenarios: | `orbstack` | OrbStack container runtime | | `aws-glue` | AWS Glue catalog integration | | `aws-s3tables` | AWS S3 Tables catalog integration | -| `external-s3` | External S3 storage (not MinIO) | +| `external-s3` | External S3 storage with a Nessie catalog | ```bash # Apply with kustomize @@ -4796,13 +5020,13 @@ storage: region: us-east-1 ``` -### MinIO +### RustFS (S3-compatible) ```yaml storage: backend: !s3 bucket: warehouse - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 region: us-east-1 ``` @@ -4810,7 +5034,7 @@ storage: ### Failure Modes -{{product_name}} is designed for resilience through stateless compute and durable object storage: +IceGate is designed for resilience through stateless compute and durable object storage: | Component | Failure Impact | Recovery | |-----------|---------------|----------| @@ -4818,11 +5042,11 @@ storage: | Query replica fails | Reduced query capacity | Load balancer routes to healthy replicas | | Maintain/Shift | WAL segments accumulate | Restarts and resumes from last committed snapshot | | Object storage (S3) | Service outage | WAL writes fail with 503; clients should retry | -| Catalog (Nessie) | Cannot commit new data or read metadata | Queries fail; data in WAL is preserved | +| Catalog | Cannot commit new data or read metadata | Queries fail; data in WAL is preserved | ### Durability Guarantees -- **WAL persistence**: All ingested data is written to object storage (S3/MinIO) before acknowledgment. Data survives node failures. +- **WAL persistence**: All ingested data is written to object storage (S3 or RustFS) before acknowledgment. Data survives node failures. - **Exactly-once delivery**: The ingest service acknowledges only after WAL write completes. - **Immutable segments**: WAL segments are append-only Parquet files. Once written, they cannot be corrupted by subsequent operations. - **Iceberg snapshots**: Each shift operation creates an atomic Iceberg snapshot. Failed shifts do not corrupt existing data. @@ -4953,7 +5177,7 @@ environment: ### Network Security - Use TLS for all external connections -- Restrict access to MinIO/Nessie from internal network only +- Restrict access to object storage and any external catalog from internal network only - Use network policies in Kubernetes ### Authentication @@ -5024,8 +5248,9 @@ The Ingest service automatically shifts WAL data into optimized Iceberg tables v 3. Reads WAL Parquet files in parallel 4. Merges and re-partitions data 5. Writes optimized Iceberg data files -6. Commits new snapshot to catalog -7. Deletes processed WAL segments +6. Commits a new snapshot to the catalog, recording the last committed WAL offset in the snapshot summary + +Shift does not delete WAL segments. An object lifecycle rule on the queue bucket reclaims them, and the offset in the snapshot summary is what lets shift resume where it left off. ### Tuning Shift Performance @@ -5116,7 +5341,15 @@ curl http://localhost:4318/health ### Catalog Backup -Nessie stores catalog metadata. Back up the RocksDB data: +On the default S3 catalog the metadata is `root.json` plus the table metadata files in the warehouse bucket, so a backup is a copy of that prefix — there is no service to stop: + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup/ +``` + +`sync` is not an atomic snapshot: it lists, then copies, and commits landing in between can leave the copy mixing catalog generations. For a point-in-time copy, use bucket versioning (below) and read a single version, or take the copy while writes are quiesced. Verify any backup by restoring it to a scratch prefix and listing the tables before relying on it. + +If you run the REST catalog backend instead, back up Nessie's RocksDB data: ```bash # Stop Nessie @@ -5150,7 +5383,7 @@ Enable versioning on your S3 bucket for point-in-time recovery: ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` @@ -5241,15 +5474,15 @@ docker compose logs -f maintain **Symptoms:** -- "Connection refused" to MinIO +- "Connection refused" to the object store - S3 authentication errors **Solutions:** -1. Verify MinIO is running: +1. On a local RustFS deployment, verify the object store is running. The readiness path is RustFS's own — on AWS S3 or another provider, skip to step 3 instead: ```bash - curl http://localhost:9000/minio/health/ready + curl http://localhost:9000/health/ready ``` 2. Check credentials: @@ -5259,7 +5492,7 @@ docker compose logs -f maintain echo $AWS_SECRET_ACCESS_KEY ``` -3. Test S3 connection: +3. Test the S3 connection. Drop `--endpoint-url` when the backend is real AWS S3: ```bash aws s3 ls --endpoint-url http://localhost:9000 @@ -5274,19 +5507,31 @@ docker compose logs -f maintain **Solutions:** -1. Verify Nessie is running: +1. On the default S3 catalog, confirm the catalog state object is readable — there is no catalog service to check: ```bash - curl http://localhost:19120/api/v1/trees + aws --endpoint-url http://localhost:9000 s3 ls s3://warehouse/catalog/root.json ``` + A missing `root.json` means migration never ran. Run `maintain migrate create` before anything else. + 2. Check catalog configuration: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 + ``` + +3. On the REST backend only, verify Nessie is running: + + ```bash + curl http://localhost:19120/api/v1/trees ``` ## Query Issues @@ -5523,7 +5768,7 @@ You need a local Kubernetes cluster. Options: ### Run with Skaffold ```bash -# Default profile (local k8s with MinIO + Nessie) +# Default profile (local k8s with RustFS + the built-in S3 catalog) skaffold dev # OrbStack profile @@ -5552,8 +5797,7 @@ Skaffold uses Kustomize overlays that compose multiple Helm charts: | Component | Description | |-----------|-------------| -| MinIO | S3-compatible storage with buckets: `warehouse`, `queue`, `jobs` | -| Nessie | Iceberg REST catalog with RocksDB persistence | +| RustFS | S3-compatible storage with buckets: `warehouse`, `queue`, `jobs` | **Observability namespace (`observability`):** @@ -5567,7 +5811,7 @@ Skaffold uses Kustomize overlays that compose multiple Helm charts: | Profile | Overlay | Use Case | |---------|---------|----------| -| (default) | `skaffold` | Local development with MinIO + Nessie | +| (default) | `skaffold` | Local development with RustFS + the built-in S3 catalog | | `orbstack` | `orbstack` | OrbStack Kubernetes (macOS) | | `aws-glue` | `aws-glue` | AWS Glue catalog (pushes images) | | `k3s-external-s3` | `external-s3` | External S3 + Nessie (pushes images) | @@ -5620,10 +5864,9 @@ make down | Service | Port | Description | |---------|------|-------------| -| MinIO | 9000, 9001 | S3-compatible storage + console | -| Nessie | 19120 | Iceberg REST catalog | +| RustFS | 9000, 9001 | S3-compatible storage + console | | Ingest | 4317, 4318 | OTLP gRPC and HTTP receivers | -| Query | 3100, 9090, 3200 | Loki, Prometheus, Tempo APIs | +| Query | 3100, 9090, 3200, 8815 | Loki, Tempo, Arrow Flight SQL APIs; Prometheus routes return 501 except `/-/ready` | | Grafana | 3000 | Dashboards | Docker Compose profiles add optional services: @@ -5632,7 +5875,7 @@ Docker Compose profiles add optional services: |---------|----------| | `load` | otelgen (log load generator) | | `monitoring` | Jaeger (16686), Prometheus (9092), node-exporter, cAdvisor | -| `analytics` | Trino SQL engine (8082) | +| `analytics` | Nessie (19120) and Trino SQL engine (8082) | ### Docker Build @@ -5653,11 +5896,11 @@ docker build -t icegate/query:dev \ ## Environment Variables -For local development with MinIO: +For local development with RustFS: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` @@ -5767,11 +6010,11 @@ IceGate uses a Cargo workspace: Cargo.toml (workspace) ├── crates/ │ ├── icegate-common/Cargo.toml +│ ├── icegate-catalog-s3/Cargo.toml │ ├── icegate-queue/Cargo.toml │ ├── icegate-query/Cargo.toml │ ├── icegate-ingest/Cargo.toml -│ ├── icegate-maintain/Cargo.toml -│ └── icegate-jobmanager/Cargo.toml +│ └── icegate-maintain/Cargo.toml ``` Build individual crates: @@ -6335,13 +6578,15 @@ This runs: ``` crates/ ├── icegate-common/ # Shared infrastructure (catalog, storage, metrics, tracing) +├── icegate-catalog-s3/ # S3-backed Iceberg catalog (default) and its REST server ├── icegate-queue/ # Write-ahead log (Parquet on object storage) -├── icegate-query/ # Query service (Loki/Prometheus/Tempo APIs) -├── icegate-ingest/ # Ingest service (OTLP HTTP/gRPC) -├── icegate-maintain/ # Maintenance operations (schema migration) -└── icegate-jobmanager/ # Shift job state management +├── icegate-query/ # Query service (Loki/Tempo/Flight SQL; Prometheus routes 501) +├── icegate-ingest/ # Ingest service (OTLP HTTP/gRPC, WAL, shift) +└── icegate-maintain/ # Migration, compaction, orphan GC, pricing crawler ``` +The job/task framework is not a workspace crate: it lives in `icegatetech/jobmanager` and is consumed as a git-pinned dependency. + See [Architecture](../architecture/overview.md) for details. ## Pull Request Guidelines @@ -6465,7 +6710,7 @@ IceGate is an observability data lake engine that stores logs, traces, metrics, ### What makes IceGate different? - **Open Standards**: Built entirely on Apache Iceberg, Arrow, Parquet, and OpenTelemetry -- **Cost-Effective**: Uses object storage (S3/MinIO) instead of expensive databases +- **Cost-Effective**: Uses object storage (S3 or RustFS) instead of expensive databases - **ACID Transactions**: Full transaction support without a dedicated OLTP database - **Compute-Storage Separation**: Scale processing and storage independently @@ -6659,3 +6904,11 @@ See [Contributing Guide](development/contributing.md). We welcome: ### Where do I report issues? GitHub Issues: [https://github.com/icegatetech/icegate/issues](https://github.com/icegatetech/icegate/issues) + +--- + +## Trademarks + +Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL and associated project logos are either registered trademarks or trademarks of The Apache Software Foundation in the United States and/or other countries. OpenTelemetry® and Prometheus® are registered trademarks of The Linux Foundation. Grafana®, Loki® and Tempo® are registered trademarks of Raintank, Inc. dba Grafana Labs. IceGate is not affiliated with, endorsed by, or sponsored by any of these organizations. All other trademarks are the property of their respective owners. + +Where this documentation describes an API as Loki- or Tempo-compatible, IceGate implements a subset of that project's HTTP read API sufficient for the documented endpoints — not a complete reimplementation. The Prometheus-compatible API is planned but NOT implemented: every route returns 501 Not Implemented except /-/ready, which responds, and its reference page documents an intended surface rather than a working one. The API reference pages state what is supported today. diff --git a/llms.txt b/llms.txt index 3da2892..156c565 100644 --- a/llms.txt +++ b/llms.txt @@ -1,6 +1,6 @@ # IceGate Documentation -> IceGate is an Observability Data Lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables. Data is ingested via OpenTelemetry Protocol (OTLP) and queried via Loki, Prometheus, and Tempo-compatible APIs. +> IceGate is an Observability Data Lake engine that stores logs, traces, metrics, and events in Apache Iceberg tables. Data is ingested via OpenTelemetry Protocol (OTLP) and queried via Loki®- and Tempo®-compatible APIs plus Arrow Flight SQL. A Prometheus®-compatible API is planned but NOT implemented — every route returns 501 Not Implemented except /-/ready, which responds. ## Key Links @@ -20,7 +20,7 @@ | OTLP gRPC (Ingest) | 4317 | gRPC | OpenTelemetry Collector | | Loki (Query) | 3100 | HTTP | Grafana Loki | | Prometheus (Query) | 9090 | HTTP | Grafana Prometheus (under development) | -| Tempo (Query) | 3200 | HTTP | Grafana Tempo (basic retrieval, TraceQL planned) | +| Tempo (Query) | 3200 | HTTP | Grafana Tempo (retrieval and search; TraceQL supported, unimplemented features return 501) | | Metrics | 9091 | HTTP | Prometheus scrape target | ## Installation (Kubernetes with Helm) @@ -33,7 +33,7 @@ helm install icegate oci://ghcr.io/icegatetech/charts/icegate \ -f values.yaml ``` -Supported catalog backends: REST (Nessie), AWS S3 Tables, AWS Glue. +Supported catalog backends: S3 (default — catalog state is a `root.json` object in object storage, no external catalog service required), REST (Nessie), AWS S3 Tables, AWS Glue. Kustomize overlays available for: skaffold, orbstack, aws-glue, aws-s3tables, external-s3. ## Configuration @@ -43,12 +43,23 @@ IceGate uses YAML or TOML config files (auto-detected by extension). ### Services - **Ingest** (`ingest run -c config.yaml`): OTLP HTTP (4318), OTLP gRPC (4317), WAL queue, shift (WAL→Iceberg) -- **Query** (`query run -c config.yaml`): Loki (3100), Prometheus (9090), Tempo (3200), DataFusion engine +- **Query** (`query run -c config.yaml`): Loki (3100), Prometheus (9090), Tempo (3200), Arrow Flight SQL (8815), DataFusion engine - **Maintain** (`maintain migrate create/upgrade -c config.yaml`): Schema migration +- **Maintain** (`maintain run -c config.yaml`): Long-running data compaction, manifest compaction, orphan-file GC, and LLM pricing crawler +- **Catalog** (`catalog serve -c config.yaml`, optional): Standalone Iceberg REST server (8181) over the S3 catalog ### Catalog Backends ```yaml +# S3 — the default; catalog state lives in object storage, no external service +catalog: + backend: !s3 + warehouse: catalog + warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + # REST (Nessie) catalog: backend: !rest @@ -75,7 +86,7 @@ storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 # optional, for S3-compatible + endpoint: http://rustfs:9000 # optional, for S3-compatible ``` ### Environment Variables @@ -287,7 +298,7 @@ curl http://localhost:3100/loki/api/v1/label/service_name/values \ ## Query Traces (Tempo API — Port 3200) -The Tempo API is under development. Basic trace retrieval by ID and search by tags are available. TraceQL query language is planned for future releases. +The Tempo API implements a subset of Tempo's HTTP read API. Trace retrieval by ID, search by tags and /api/search are available, and TraceQL is supported for search; TraceQL features that are not yet implemented return 501 Not Implemented. ### Tempo API Endpoints @@ -325,7 +336,7 @@ curl -G http://localhost:3200/api/search \ ## Prometheus API (Port 9090) — Under Development -The Prometheus query API is under development. Metadata endpoints (labels, series) are available. PromQL queries are not yet supported. Use LogQL metric queries (rate, count_over_time, sum by) as an alternative for log-based metrics. +The Prometheus query API is NOT implemented. Every route returns 501 Not Implemented, including the metadata endpoints; only /-/ready responds. PromQL is not parsed at all yet. Use LogQL metric queries (rate, count_over_time, sum by) as an alternative for log-based metrics. ## Multi-Tenancy @@ -471,7 +482,7 @@ EXECUTE remove_orphan_files(retention_threshold => '1d'); ## Fault Tolerance -- **WAL durability**: All data written to S3/MinIO before acknowledgment +- **WAL durability**: All data written to S3 or RustFS before acknowledgment - **Stateless query**: Any replica can serve any query; no coordination needed - **Immutable WAL segments**: Once written, cannot be corrupted - **Iceberg snapshots**: Atomic commits; failed shifts don't corrupt data @@ -529,3 +540,8 @@ curl http://localhost:9090/-/ready - Data Model: https://docs.icegate.tech/en/architecture/data-model - Deployment: https://docs.icegate.tech/en/operations/deployment - Development Setup: https://docs.icegate.tech/en/development/setup +- Trademarks: https://docs.icegate.tech/en/trademarks + +## Trademarks + +Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL and associated project logos are either registered trademarks or trademarks of The Apache Software Foundation in the United States and/or other countries. OpenTelemetry® and Prometheus® are registered trademarks of The Linux Foundation. Grafana®, Loki® and Tempo® are registered trademarks of Raintank, Inc. dba Grafana Labs. IceGate is not affiliated with, endorsed by, or sponsored by any of these organizations. All other trademarks are the property of their respective owners. diff --git a/package.json b/package.json index 292d5ba..b0f3407 100644 --- a/package.json +++ b/package.json @@ -4,10 +4,10 @@ "description": "IceGate Documentation", "private": true, "scripts": { - "build": "yfm -i . -o ./build && cp llms.txt llms-full.txt context7.json ./build/", - "build:en": "yfm -i ./en -o ./build/en", - "build:fr": "yfm -i ./fr -o ./build/fr", - "build:ru": "yfm -i ./ru -o ./build/ru", + "build": "yfm -i . -o ./build --static-content && cp llms.txt llms-full.txt context7.json robots.txt ./build/", + "build:en": "yfm -i ./en -o ./build/en --static-content -c ./.yfm", + "build:fr": "yfm -i ./fr -o ./build/fr --static-content -c ./.yfm", + "build:ru": "yfm -i ./ru -o ./build/ru --static-content -c ./.yfm", "serve": "npm run build && npx serve ./build -p 8080", "lint": "yfm -i . -o ./.lint-build --strict && rm -rf ./.lint-build", "clean": "rm -rf ./build" diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..e46a5ad --- /dev/null +++ b/robots.txt @@ -0,0 +1,17 @@ +# https://docs.icegate.tech/robots.txt +# +# The documentation is meant to be crawled — by search engines and by AI +# crawlers alike. llms.txt and llms-full.txt exist for the latter and are +# deliberately left open, so there is no user-agent blocklist here. Add one only +# with a concrete reason; blocking an AI crawler also removes IceGate from the +# answers those tools give. + +User-agent: * +Allow: / + +# Repository files that end up in the published site because the build input is +# the repo root (`yfm -i .`), not a docs-only subdirectory. They are not +# documentation and are worth nothing in an index. +Disallow: /c4/ +Disallow: /package.json +Disallow: /package-lock.json diff --git a/ru/api-reference/loki.md b/ru/api-reference/loki.md index d894e37..60a7742 100644 --- a/ru/api-reference/loki.md +++ b/ru/api-reference/loki.md @@ -1,11 +1,14 @@ --- title: Справочник Loki API -description: HTTP API эндпоинты совместимые с Loki +description: HTTP API эндпоинты, совместимые с Loki, которые обслуживает {{product_name}} --- # Справочник Loki API -IceGate предоставляет HTTP API совместимый с Loki для запросов к логам. +{{product_name}} предоставляет HTTP API, совместимый с Loki®, для запросов к логам — на порту 3100. Ниже +описаны реализованные эндпоинты: это подмножество API Loki, а не полная реализация, поэтому всё, +что здесь не перечислено, следует считать нереализованным. Атрибуцию см. в разделе +[Товарные знаки](../trademarks.md). ## Базовый URL @@ -220,7 +223,7 @@ curl -G http://localhost:3100/loki/api/v1/series \ ### Explain -Получение плана выполнения запроса (расширение IceGate). +Получение плана выполнения запроса (расширение {{product_name}}). **Эндпоинт:** `GET /loki/api/v1/explain` @@ -269,5 +272,5 @@ curl -G http://localhost:3100/loki/api/v1/explain \ ## Следующие Шаги - Изучите [Запросы LogQL](../guides/querying.md) -- Изучите [Prometheus API](prometheus.md) +- Изучите [Prometheus API](prometheus.md) — запланирован, пока не реализован - Смотрите [Tempo API](tempo.md) для трейсов diff --git a/ru/api-reference/otlp.md b/ru/api-reference/otlp.md index 75bd0b0..0100bbe 100644 --- a/ru/api-reference/otlp.md +++ b/ru/api-reference/otlp.md @@ -5,7 +5,7 @@ description: Точки доступа OpenTelemetry Protocol для загру # API Загрузки OTLP -IceGate принимает данные наблюдаемости через протокол OpenTelemetry (OTLP). Поддерживаются транспорты HTTP и gRPC. +{{product_name}} принимает данные наблюдаемости через протокол OpenTelemetry (OTLP). Поддерживаются транспорты HTTP и gRPC. ## Протоколы diff --git a/ru/api-reference/prometheus.md b/ru/api-reference/prometheus.md index 3aed23b..7e38bef 100644 --- a/ru/api-reference/prometheus.md +++ b/ru/api-reference/prometheus.md @@ -1,6 +1,6 @@ --- title: Справочник Prometheus API -description: HTTP API эндпоинты совместимые с Prometheus +description: Планируемый API, совместимый с Prometheus, — пока не реализован --- # Справочник Prometheus API @@ -11,7 +11,19 @@ description: HTTP API эндпоинты совместимые с Prometheus {% endnote %} -IceGate предоставляет HTTP API совместимый с Prometheus для запросов к метрикам. +{% note warning %} + +**Пока не реализовано.** Маршруты ниже смонтированы на порту 9090, но каждый из них возвращает +`501 Not Implemented`; отвечает только `/-/ready`. Эта страница описывает *планируемую* +поверхность, чтобы интеграторы видели направление, — пока не стройте на ней интеграции. + +Для запросов к метрикам сегодня используйте [Arrow Flight SQL](../guides/querying.md) поверх тех же +данных. + +{% endnote %} + +Ниже — планируемая форма HTTP API {{product_name}}, совместимого с Prometheus®, для запросов к метрикам. +Атрибуцию см. в разделе [Товарные знаки](../trademarks.md). ## Базовый URL @@ -21,7 +33,7 @@ http://localhost:9090 ## Статус Реализации -Prometheus API в настоящее время в разработке. +Ни один из этих эндпоинтов не реализован: все они возвращают `501 Not Implemented`. Отвечает только `/-/ready`. ## Следующие Шаги diff --git a/ru/api-reference/tempo.md b/ru/api-reference/tempo.md index 5557824..ad386b7 100644 --- a/ru/api-reference/tempo.md +++ b/ru/api-reference/tempo.md @@ -1,6 +1,6 @@ --- title: Справочник Tempo API -description: HTTP API эндпоинты совместимые с Tempo +description: HTTP API эндпоинты, совместимые с Tempo, которые обслуживает {{product_name}} --- # Справочник Tempo API @@ -11,7 +11,12 @@ description: HTTP API эндпоинты совместимые с Tempo {% endnote %} -IceGate предоставляет HTTP API совместимый с Tempo для запросов к распределённым трейсам. +{{product_name}} предоставляет HTTP API, совместимый с Tempo®, для запросов к распределённым трейсам — на +порту 3200. Ниже описаны реализованные эндпоинты: это подмножество API Tempo, а не полная +реализация, поэтому всё, что здесь не перечислено, следует считать нереализованным. TraceQL +поддерживается для `/api/search`; ещё не реализованные возможности TraceQL возвращают +`501 Not Implemented`, а не молча искажённый результат. Атрибуцию см. в разделе +[Товарные знаки](../trademarks.md). ## Базовый URL diff --git a/ru/architecture/data-model.md b/ru/architecture/data-model.md index c392cd3..507320a 100644 --- a/ru/architecture/data-model.md +++ b/ru/architecture/data-model.md @@ -1,6 +1,6 @@ --- title: Модель Данных -description: Схемы таблиц Iceberg IceGate для данных наблюдаемости +description: Схемы таблиц Iceberg {{product_name}} для данных наблюдаемости --- # Модель Данных @@ -11,7 +11,7 @@ description: Схемы таблиц Iceberg IceGate для данных наб {% endnote %} -IceGate хранит данные наблюдаемости в четырёх таблицах Apache Iceberg. +{{product_name}} хранит данные наблюдаемости в пяти таблицах Apache Iceberg с разделением по тенантам — logs, spans, events, metrics и operations — плюс одна глобальная справочная таблица prices. ## Обзор Таблиц @@ -21,12 +21,14 @@ IceGate хранит данные наблюдаемости в четырёх | `spans` | Спаны распределённых трейсов | Трассировка запросов | | `events` | Семантические события | Бизнес-события | | `metrics` | Все типы метрик | Мониторинг производительности | +| `operations` | Операции LLM и агентов | Расход токенов, стоимость, сохранение промптов и ответов | +| `prices` | Глобальная тарифная таблица LLM (без `tenant_id`) | Справочные тарифы для расчёта стоимости `operations` | ## Общие Паттерны ### Мультитенантность -Все таблицы используют партиционирование по `tenant_id`. +Пять таблиц с разделением по тенантам используют identity-партиционирование по `tenant_id`. `prices` — справочные данные, общие для всех тенантов: в ней нет `tenant_id`, и партиционируется она иначе. ### Хранение Атрибутов diff --git a/ru/architecture/overview.md b/ru/architecture/overview.md index 74af0db..af90547 100644 --- a/ru/architecture/overview.md +++ b/ru/architecture/overview.md @@ -1,11 +1,11 @@ --- title: Обзор Архитектуры -description: Системная архитектура и компоненты IceGate +description: Системная архитектура и компоненты {{product_name}} --- # Обзор Архитектуры -IceGate - движок озера данных наблюдаемости, который хранит логи, трейсы, метрики и события в таблицах Apache Iceberg с DataFusion в качестве движка запросов. +{{product_name}} - движок озера данных наблюдаемости, который хранит логи, трейсы, метрики, события и LLM-операции в таблицах Apache Iceberg с DataFusion в качестве движка запросов. ## Принципы Проектирования @@ -43,13 +43,18 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet **Назначение:** Выполнение запросов к логам, трейсам, метрикам и событиям - **Движок:** Apache DataFusion + Apache Arrow -- **API:** Loki (3100), Prometheus (9090), Tempo (3200) -- **Языки Запросов:** LogQL, PromQL (планируется), TraceQL (планируется) +- **API:** Loki (3100), Tempo (3200), Arrow Flight SQL (8815); Prometheus (9090) отдаёт маршруты, но его обработчики пока возвращают `501 Not Implemented` +- **Языки Запросов:** LogQL, TraceQL, SQL; PromQL планируется +- **Мультитенантность:** Тенант берётся из заголовка `X-Scope-OrgID` либо из gRPC-метаданных `x-scope-orgid` для Flight SQL + +Arrow Flight SQL работает строго на чтение — DDL и DML отклоняются — и применяет `tenant_id` на уровне строк при каждом сканировании, поэтому клиенты JDBC, ODBC и ADBC обращаются к `iceberg.icegate.<table>` без специфичного для {{product_name}} клиентского кода. Сервис запросов читает из двух источников: - **WAL**: Данные в реальном времени (возрастом в секунды) -- **Таблицы Iceberg**: Исторические данные (компактированные) +- **Таблицы Iceberg**: Исторические данные (перенесённые shift'ом и компактированные) + +Границей между ними служит WAL-офсет, записанный в сводку снапшота Iceberg, поэтому строка читается ровно с одной стороны и никогда не учитывается дважды. ### Сервис Maintain @@ -57,10 +62,25 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet **Назначение:** Операции жизненного цикла и оптимизации данных -- **Компакция:** Слияние мелких WAL-файлов в оптимизированные таблицы Iceberg -- **TTL:** Истечение срока и удаление старых данных на основе политик хранения -- **Оптимизация:** Перезапись файлов данных для лучшей производительности запросов -- **Очистка:** Удаление осиротевших файлов и просроченных снапшотов +- **Миграция схемы:** Создание таблиц Iceberg (`maintain migrate create`) +- **Компакция данных:** Перезапись мелких Parquet-файлов в меньшее количество более крупных отсортированных файлов +- **Компакция манифестов:** Переупаковка фрагментированных манифестов Iceberg +- **GC осиротевших объектов:** Удаление объектов, на которые текущие метаданные таблицы больше не ссылаются, по истечении льготного периода +- **Краулер цен:** Сбор тарифов LLM из внешних источников в глобальную таблицу `icegate.prices` + +Компакция, GC и краулер цен выполняются как задачи, состояние которых хранится в объектном хранилище под собственным префиксом состояния задач. + +### Каталог + +![Компоненты Каталога](../../assets/c4/structurizr-CatalogComponents.png) + +**Назначение:** Организация озера данных с ACID-транзакциями без выделенной OLTP базы данных + +- **Бэкенд по умолчанию:** Собственный S3-каталог {{product_name}} — состояние каталога представляет собой объект `root.json`, обновляемый через compare-and-swap +- **Альтернативные бэкенды:** REST (Nessie), AWS S3 Tables, AWS Glue +- **Развёртывание:** По умолчанию встраивается в Ingest, Query и Maintain; опционально разворачивается отдельно как REST-сервер Iceberg на порту 8181 + +Условное чтение поддерживает актуальность закэшированного корня каталога; метаданные таблиц неизменяемы для каждого расположения, поэтому кэшируются безусловно в LRU. ### Сервис Alert (Планируется) @@ -79,15 +99,21 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet | Формат в памяти | Apache Arrow 57.0 | Обработка данных без копирования | | Формат хранения | Apache Parquet 57.0 | Колоночное хранение с ZSTD сжатием | | Загрузка | OpenTelemetry 0.31 | Стандартный протокол наблюдаемости (gRPC + HTTP) | -| Каталог | Nessie, AWS S3 Tables, AWS Glue | REST бэкенды каталога Iceberg | -| Job Manager | icegate-jobmanager | Управление состоянием задач shift на основе S3 | +| SQL-интерфейс | Arrow Flight SQL 57.0 | SQL только на чтение для клиентов JDBC, ODBC и ADBC | +| Каталог | S3-каталог (по умолчанию), Nessie, AWS S3 Tables, AWS Glue | Бэкенды каталога Iceberg; вариант по умолчанию хранит состояние в объектном хранилище | +| Объектное хранилище | RustFS или любое S3-совместимое хранилище | Сегменты WAL, данные Iceberg, состояние каталога, состояние задач | +| Job Manager | jobmanager (отдельный репозиторий) | Состояние задач shift, компакции, GC и цен на основе S3 | | Кэширование | foyer 0.22 | Гибридный кэш память + диск для чтения из S3 | -| Язык | Rust 1.92+ (2024 edition) | Безопасная по памяти, высокопроизводительная среда выполнения | +| Язык | Rust {{rust_version}}+ (2024 edition) | Безопасная по памяти, высокопроизводительная среда выполнения | ## Поток Данных ### Поток Загрузки +![Последовательность Загрузки](../../assets/c4/structurizr-IngestionFlow.png) + +Шаги 1-7 — это путь записи, подтверждаемый сразу после записи сегмента WAL. Шаги 8-14 — это shift, который выполняется независимо от запроса. + 1. Клиент отправляет данные OTLP в сервис Ingest 2. Ingest валидирует и трансформирует данные 3. Данные записываются в WAL как файлы Parquet @@ -95,6 +121,8 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet ### Поток Запросов +![Последовательность Запроса](../../assets/c4/structurizr-QueryFlow.png) + 1. Клиент отправляет запрос в сервис Query 2. Запрос парсится и планируется DataFusion 3. Данные читаются из таблиц Iceberg и/или WAL @@ -106,8 +134,17 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet 2. Группирует сегменты в задачи shift 3. Параллельно читает файлы WAL, объединяет и перепартиционирует данные 4. Записывает оптимизированные файлы данных Iceberg -5. Фиксирует новый снапшот в каталоге -6. Удаляет обработанные сегменты WAL +5. Фиксирует новый снапшот в каталоге, записывая последний зафиксированный WAL-офсет в сводку снапшота + +Shift никогда не удаляет сегменты WAL. Их освобождает правило жизненного цикла объектов на бакете очереди, а офсет в сводке снапшота — это то, что позволяет shift'у продолжить с места остановки. + +Поэтому срок жизни объектов — это параметр надёжности, а не просто уборка: сегмент обязан пережить коммит, который его покрывает. Если в момент срабатывания правила shift задержан или падает, сегменты с незафиксированными офсетами будут удалены, а данные потеряны. О выборе срока см. [Хранение Данных](../guides/data-retention.md). + +### Поток Обслуживания + +![Последовательность Обслуживания](../../assets/c4/structurizr-MaintenanceFlow.png) + +Миграция — разовая задача. Компакция, GC осиротевших объектов и краулер цен — независимые циклы со своими расписаниями: номера шагов упорядочивают каждый цикл, но не циклы между собой. Каждый резервирует работу под собственным префиксом состояния задач, поэтому циклы не спорят за владение задачами. При этом они всё равно работают с одними и теми же таблицами: компакция фиксирует снапшоты перезаписи, а GC удаляет неcсылочные объекты — именно поэтому GC удаляет только файлы старше своего льготного периода, а коммиты используют оптимистичную конкуренцию с повтором при конфликте. ## Масштабируемость @@ -115,7 +152,7 @@ Write-Ahead Log (WAL) хранит данные в виде файлов Parquet - **Ingest:** Масштабирование реплик для увеличения пропускной способности - **Query:** Масштабирование реплик для параллельных запросов -- **Maintain:** Один экземпляр (выбор лидера) +- **Maintain:** Масштабирование реплик для большей пропускной способности перезаписи — воркеры разделяют состояние задач в объектном хранилище через compare-and-swap и фиксируют изменения с оптимистичной конкуренцией, поэтому параллельные экземпляры безопасны. Сначала предпочтительнее увеличить число воркеров внутри процесса; отдача от реплик снижается по мере их роста, так как все воркеры одной таблицы конкурируют за единственный объект состояния задач. ### Масштабирование Хранилища diff --git a/ru/cookbooks/centralized-logging.md b/ru/cookbooks/centralized-logging.md index 9398f05..fce6372 100644 --- a/ru/cookbooks/centralized-logging.md +++ b/ru/cookbooks/centralized-logging.md @@ -1,6 +1,6 @@ --- title: Централизованное Логирование для Микросервисов -description: Настройка централизованного логирования с OpenTelemetry Collector и IceGate +description: Настройка централизованного логирования с OpenTelemetry Collector и {{product_name}} --- # Централизованное Логирование для Микросервисов diff --git a/ru/cookbooks/observability-correlation.md b/ru/cookbooks/observability-correlation.md index a09fc91..21f0118 100644 --- a/ru/cookbooks/observability-correlation.md +++ b/ru/cookbooks/observability-correlation.md @@ -1,6 +1,6 @@ --- title: Корреляция Сигналов Наблюдаемости -description: Корреляция логов и трейсов в IceGate для эффективной диагностики +description: Корреляция логов и трейсов в {{product_name}} для эффективной диагностики --- # Корреляция Сигналов Наблюдаемости diff --git a/ru/cookbooks/traces-end-to-end.md b/ru/cookbooks/traces-end-to-end.md index b4568e3..7b1bbef 100644 --- a/ru/cookbooks/traces-end-to-end.md +++ b/ru/cookbooks/traces-end-to-end.md @@ -1,6 +1,6 @@ --- title: Распределённая Трассировка -description: Инструментирование сервисов и запросы трейсов через Tempo API в IceGate +description: Инструментирование сервисов и запросы трейсов через Tempo API в {{product_name}} --- # Распределённая Трассировка diff --git a/ru/development/building.md b/ru/development/building.md index e23546e..6fbb0c1 100644 --- a/ru/development/building.md +++ b/ru/development/building.md @@ -1,17 +1,17 @@ --- title: Сборка -description: Сборка IceGate из исходного кода +description: Сборка {{product_name}} из исходного кода --- # Сборка из Исходного Кода -Это руководство охватывает сборку IceGate из исходного кода для разработки и продакшена. +Это руководство охватывает сборку {{product_name}} из исходного кода для разработки и продакшена. ## Предварительные Требования ### Обязательные -- **Rust** >= 1.92.0 (для поддержки Rust 2024 edition) +- **Rust** >= {{rust_version}} (для поддержки Rust 2024 edition) - **Cargo** (входит в Rust) - **Git** @@ -96,17 +96,17 @@ debug = true ## Структура Рабочего Пространства -IceGate использует Cargo workspace: +{{product_name}} использует Cargo workspace: ```text Cargo.toml (workspace) ├── crates/ │ ├── icegate-common/Cargo.toml +│ ├── icegate-catalog-s3/Cargo.toml │ ├── icegate-queue/Cargo.toml │ ├── icegate-query/Cargo.toml │ ├── icegate-ingest/Cargo.toml -│ ├── icegate-maintain/Cargo.toml -│ └── icegate-jobmanager/Cargo.toml +│ └── icegate-maintain/Cargo.toml ``` Сборка отдельных крейтов: @@ -194,7 +194,7 @@ make ci ### Ошибки Компиляции -1. Убедитесь, что версия Rust >= 1.92.0: +1. Убедитесь, что версия Rust >= {{rust_version}}: ```bash rustup update diff --git a/ru/development/contributing.md b/ru/development/contributing.md index 4b4eaec..05d81fc 100644 --- a/ru/development/contributing.md +++ b/ru/development/contributing.md @@ -1,6 +1,6 @@ --- title: Участие в Проекте -description: Как участвовать в разработке IceGate +description: Как участвовать в разработке {{product_name}} --- # Участие в Проекте @@ -11,7 +11,7 @@ description: Как участвовать в разработке IceGate {% endnote %} -Мы приветствуем участие в IceGate! Это руководство объясняет как начать. +Мы приветствуем участие в {{product_name}}! Это руководство объясняет как начать. ## Способы Участия diff --git a/ru/development/patterns.md b/ru/development/patterns.md index 73e89e5..1c3dc4a 100644 --- a/ru/development/patterns.md +++ b/ru/development/patterns.md @@ -1,6 +1,6 @@ --- title: Паттерны Разработки -description: Стандартные паттерны используемые в кодовой базе IceGate +description: Стандартные паттерны используемые в кодовой базе {{product_name}} --- # Паттерны Разработки @@ -11,7 +11,7 @@ description: Стандартные паттерны используемые в {% endnote %} -Этот документ определяет стандартные паттерны используемые в кодовой базе IceGate для конфигурации, ошибок, HTTP маршрутов, обработчиков и сервисов. +Этот документ определяет стандартные паттерны используемые в кодовой базе {{product_name}} для конфигурации, ошибок, HTTP маршрутов, обработчиков и сервисов. ## Следующие Шаги diff --git a/ru/development/setup.md b/ru/development/setup.md index 86281ab..ba97417 100644 --- a/ru/development/setup.md +++ b/ru/development/setup.md @@ -1,15 +1,15 @@ --- title: Окружение для Разработки -description: Настройка локального окружения для разработки IceGate +description: Настройка локального окружения для разработки {{product_name}} --- # Окружение для Разработки -Это руководство описывает настройку локального окружения для разработки IceGate: написания кода, запуска тестов и отладки. +Это руководство описывает настройку локального окружения для разработки {{product_name}}: написания кода, запуска тестов и отладки. ## Предварительные Требования -- **Rust** >= 1.92.0 (Rust 2024 edition) +- **Rust** >= {{rust_version}} (Rust 2024 edition) - **Docker** (для сборки контейнерных образов) - **Git** - Локальный кластер Kubernetes (для Skaffold) @@ -58,7 +58,7 @@ chmod +x skaffold && sudo mv skaffold /usr/local/bin/ ### Запуск со Skaffold ```bash -# Профиль по умолчанию (локальный k8s с MinIO + Nessie) +# Профиль по умолчанию (локальный k8s с RustFS + встроенным S3-каталогом) skaffold dev # Профиль OrbStack @@ -75,7 +75,7 @@ skaffold dev -p k3s-external-s3 Skaffold использует оверлеи Kustomize, которые компонуют несколько Helm charts: -**Пространство имён IceGate (`icegate`):** +**Пространство имён {{product_name}} (`icegate`):** | Компонент | Описание | |-----------|----------| @@ -87,22 +87,21 @@ Skaffold использует оверлеи Kustomize, которые комп | Компонент | Описание | |-----------|----------| -| MinIO | S3-совместимое хранилище с бакетами: `warehouse`, `queue`, `jobs` | -| Nessie | REST-каталог Iceberg с персистентностью RocksDB | +| RustFS | S3-совместимое хранилище с бакетами: `warehouse`, `queue`, `jobs` | **Пространство имён наблюдаемости (`observability`):** | Компонент | Описание | |-----------|----------| | Prometheus | Сбор метрик (kube-prometheus-stack) | -| Grafana | Дашборды с готовыми панелями IceGate Ingest и Query | -| Jaeger | Распределённая трассировка для сервисов IceGate | +| Grafana | Дашборды с готовыми панелями {{product_name}} Ingest и Query | +| Jaeger | Распределённая трассировка для сервисов {{product_name}} | ### Профили Skaffold | Профиль | Оверлей | Назначение | |---------|---------|------------| -| (по умолчанию) | `skaffold` | Локальная разработка с MinIO + Nessie | +| (по умолчанию) | `skaffold` | Локальная разработка с RustFS + встроенным S3-каталогом | | `orbstack` | `orbstack` | OrbStack Kubernetes (macOS) | | `aws-glue` | `aws-glue` | Каталог AWS Glue (отправляет образы) | | `k3s-external-s3` | `external-s3` | Внешний S3 + Nessie (отправляет образы) | @@ -155,10 +154,9 @@ make down | Сервис | Порт | Описание | |--------|------|----------| -| MinIO | 9000, 9001 | S3-совместимое хранилище + консоль | -| Nessie | 19120 | REST-каталог Iceberg | +| RustFS | 9000, 9001 | S3-совместимое хранилище + консоль | | Ingest | 4317, 4318 | Приёмники OTLP gRPC и HTTP | -| Query | 3100, 9090, 3200 | API Loki, Prometheus, Tempo | +| Query | 3100, 9090, 3200, 8815 | API Loki, Tempo, Arrow Flight SQL; маршруты Prometheus возвращают 501, кроме `/-/ready` | | Grafana | 3000 | Дашборды | Профили Docker Compose добавляют дополнительные сервисы: @@ -167,7 +165,7 @@ make down |---------|---------| | `load` | otelgen (генератор нагрузки логов) | | `monitoring` | Jaeger (16686), Prometheus (9092), node-exporter, cAdvisor | -| `analytics` | SQL-движок Trino (8082) | +| `analytics` | Nessie (19120) и SQL-движок Trino (8082) | ### Сборка Docker @@ -188,11 +186,11 @@ docker build -t icegate/query:dev \ ## Переменные Окружения -Для локальной разработки с MinIO: +Для локальной разработки с RustFS: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/ru/faq.md b/ru/faq.md index f55642a..986b4dd 100644 --- a/ru/faq.md +++ b/ru/faq.md @@ -1,6 +1,6 @@ --- title: FAQ -description: Часто задаваемые вопросы об IceGate +description: Часто задаваемые вопросы об {{product_name}} --- # Часто Задаваемые Вопросы @@ -13,20 +13,20 @@ description: Часто задаваемые вопросы об IceGate ## Общие Вопросы -### Что такое IceGate? +### Что такое {{product_name}}? -IceGate - движок озера данных наблюдаемости, который хранит логи, трейсы, метрики и события в таблицах Apache Iceberg. +{{product_name}} - движок озера данных наблюдаемости, который хранит логи, трейсы, метрики и события в таблицах Apache Iceberg. -### Чем IceGate отличается? +### Чем {{product_name}} отличается? - **Открытые Стандарты**: Построен на Apache Iceberg, Arrow, Parquet и OpenTelemetry -- **Экономичность**: Использует объектное хранилище (S3/MinIO) +- **Экономичность**: Использует объектное хранилище (S3 или RustFS) - **ACID Транзакции**: Полная поддержка транзакций - **Разделение Вычислений и Хранения**: Независимое масштабирование ### Каков текущий статус? -IceGate находится в **альфа** разработке. +{{product_name}} находится в **альфа** разработке. ## Начало Работы diff --git a/ru/getting-started/configuration.md b/ru/getting-started/configuration.md index f718eb5..e8a6917 100644 --- a/ru/getting-started/configuration.md +++ b/ru/getting-started/configuration.md @@ -1,6 +1,6 @@ --- title: Конфигурация -description: Настройка компонентов IceGate +description: Настройка компонентов {{product_name}} --- # Конфигурация @@ -42,24 +42,48 @@ query version ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 ``` ### Параметры Каталога | Параметр | Тип | Обязательный | По умолчанию | Описание | |----------|-----|--------------|--------------|----------| -| `backend` | enum | Да | `memory` | Тип бэкенда каталога (см. ниже) | +| `backend` | enum | Да | — | Тип бэкенда каталога (см. ниже). Значения по умолчанию нет — поле обязательное | | `warehouse` | string | Да | — | Расположение хранилища (например, `s3://warehouse/`) | | `properties` | map | Нет | `{}` | Дополнительные свойства каталога | | `cache` | object | Нет | — | Конфигурация IO-кэша (см. [Конфигурация Кэша](#конфигурация-кэша)) | ### Бэкенды Каталога +#### S3-каталог (по умолчанию) + +Собственный каталог {{product_name}}. Состояние каталога — объект `root.json` в объектном хранилище, обновляемый через compare-and-swap, поэтому внешний сервис каталога не требуется. + +```yaml +catalog: + backend: !s3 + warehouse: catalog + warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 +``` + +| Параметр | Тип | Обязательный | Описание | +|----------|-----|--------------|----------| +| `warehouse` (внутри `!s3`) | string | Да | Префикс ключей объектного хранилища с состоянием каталога | +| `properties.bucket` | string | Да | Бакет с состоянием каталога | +| `properties.region` | string | Да | Регион S3-клиента каталога | +| `properties.endpoint` | string | Нет | Пользовательский эндпоинт для S3-совместимого хранилища. Опустить для настоящего AWS S3 | + #### REST Каталог (Nessie) ```yaml @@ -115,9 +139,13 @@ catalog: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -141,21 +169,21 @@ catalog: Секция `storage` настраивает бэкенд объектного хранилища. Является общей для всех сервисов. -### S3 / S3-Совместимое (MinIO) +### S3 / S3-Совместимое (RustFS) ```yaml storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` | Параметр | Тип | Обязательный | По умолчанию | Описание | |----------|-----|--------------|--------------|----------| | `bucket` | string | Да | — | Имя бакета S3 | | `region` | string | Да | — | Регион AWS | -| `endpoint` | string | Нет | — | URL кастомного эндпоинта для S3-совместимого хранилища (MinIO и др.) | +| `endpoint` | string | Нет | — | URL кастомного эндпоинта для S3-совместимого хранилища (RustFS и др.) | ### Локальная Файловая Система @@ -184,17 +212,19 @@ storage: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 queue: common: @@ -223,7 +253,7 @@ shift: poll_interval_ms: 1000 iteration_interval_millisecs: 30000 storage: - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 bucket: jobs prefix: shifter region: us-east-1 @@ -322,11 +352,13 @@ Job manager хранит состояние задач shift в отдельно ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 cache: memory_size_mb: 1024 disk_dir: /tmp/icegate/cache @@ -336,7 +368,7 @@ storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 engine: batch_size: 8192 @@ -406,7 +438,7 @@ tracing: | `loki.enabled` | bool | `true` | Включить Loki-совместимый API запросов логов | | `loki.host` | string | `0.0.0.0` | Адрес привязки | | `loki.port` | integer | `3100` | Порт Loki API | -| `prometheus.enabled` | bool | `true` | Включить Prometheus-совместимый API метрик | +| `prometheus.enabled` | bool | `true` | Отдавать API запросов Prometheus. Маршруты зарегистрированы, но все обработчики, кроме `/-/ready`, возвращают `501 Not Implemented` — PromQL пока не реализован. Это не эндпоинт метрик: им является блок `metrics` на порту 9091 | | `prometheus.host` | string | `0.0.0.0` | Адрес привязки | | `prometheus.port` | integer | `9090` | Порт Prometheus API | | `tempo.enabled` | bool | `true` | Включить Tempo-совместимый API трейсов | @@ -419,17 +451,19 @@ tracing: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ properties: - prefix: main + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 storage: backend: !s3 bucket: warehouse region: us-east-1 - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 ``` ### CLI Maintain @@ -499,8 +533,8 @@ make run-analytics-release Переменные окружения для локальной разработки: ```bash -export AWS_ACCESS_KEY_ID=minioadmin -export AWS_SECRET_ACCESS_KEY=minioadmin +export AWS_ACCESS_KEY_ID=rustfsadmin +export AWS_SECRET_ACCESS_KEY=rustfsadmin export AWS_REGION=us-east-1 ``` diff --git a/ru/getting-started/installation.md b/ru/getting-started/installation.md index 9d36ec1..0286ebd 100644 --- a/ru/getting-started/installation.md +++ b/ru/getting-started/installation.md @@ -1,21 +1,21 @@ --- title: Установка -description: Установка IceGate в Kubernetes с помощью Helm +description: Установка {{product_name}} в Kubernetes с помощью Helm --- # Установка -IceGate разворачивается в Kubernetes с помощью Helm charts и оверлеев Kustomize для настройки под конкретное окружение. +{{product_name}} разворачивается в Kubernetes с помощью Helm charts и оверлеев Kustomize для настройки под конкретное окружение. ## Предварительные Требования - **Kubernetes** >= 1.28 с **Helm 3** -- **Объектное хранилище:** AWS S3 или S3-совместимое (MinIO) -- **Каталог Iceberg:** Nessie (REST), AWS S3 Tables или AWS Glue +- **Объектное хранилище:** AWS S3 или S3-совместимое (RustFS) +- **Каталог Iceberg:** встроенный S3-каталог (по умолчанию, без внешнего сервиса), либо Nessie (REST), AWS S3 Tables или AWS Glue ## Helm Chart -Helm chart разворачивает все компоненты IceGate: Ingest, Query и задачу Migrate (создание схемы в виде хука pre-install/pre-upgrade). +Helm chart разворачивает все компоненты {{product_name}}: Ingest, Query и задачу Migrate (создание схемы в виде хука pre-install/pre-upgrade). ### Установка из реестра OCI @@ -41,24 +41,24 @@ helm install icegate ./icegate/config/helm/icegate \ {% note info %} -Значения Helm используют camelCase и плоские ключи (например, `backend: rest` + `rest.uri`). Chart транслирует их в нативный формат конфигурации serde tagged enum (`backend: !rest`), который ожидают бинарные файлы IceGate. См. [Конфигурацию](configuration.md) для справочника по нативному формату конфигурации. +Значения Helm используют camelCase и плоские ключи (например, `backend: s3` + `s3.warehouse`). Chart транслирует их в нативный формат конфигурации serde tagged enum (`backend: !s3`), который ожидают бинарные файлы {{product_name}}. См. [Конфигурацию](configuration.md) для справочника по нативному формату конфигурации. {% endnote %} -Минимальный файл `values.yaml` для REST-каталога (Nessie) с S3-совместимым хранилищем: +Минимальный файл `values.yaml` со встроенным S3-каталогом по умолчанию и S3-совместимым хранилищем. Внешний сервис каталога не задействован — состояние каталога это объект `root.json` в бакете warehouse: ```yaml catalog: - backend: rest - rest: - uri: http://nessie:19120/iceberg + backend: s3 + s3: + warehouse: catalog warehouse: "s3://warehouse/" storage: s3: bucket: warehouse region: us-east-1 - endpoint: "http://minio:9000" + endpoint: "http://rustfs:9000" queue: common: @@ -69,6 +69,28 @@ aws: region: us-east-1 ``` +### REST-каталог (Nessie) + +Используйте только если у вас уже развёрнут Nessie или другой REST-каталог Iceberg — это добавляет внешний сервис, который не нужен развёртыванию по умолчанию: + +```yaml +catalog: + backend: rest + rest: + uri: http://nessie:19120/iceberg + warehouse: "s3://warehouse/" + +storage: + s3: + bucket: warehouse + region: us-east-1 + endpoint: "http://rustfs:9000" + +aws: + existingSecret: icegate-aws-credentials + region: us-east-1 +``` + ### Каталог AWS Glue ```yaml @@ -109,9 +131,9 @@ aws: | Значение | По умолчанию | Описание | |----------|--------------|----------| -| `catalog.backend` | `rest` | Тип каталога: `rest`, `s3tables` или `glue` | +| `catalog.backend` | `s3` | Тип каталога: `s3`, `rest`, `s3tables` или `glue` | | `storage.s3.bucket` | `warehouse` | Имя S3-бакета | -| `storage.s3.endpoint` | `""` | Пользовательский S3-эндпоинт (MinIO). Опустить для реального AWS S3 | +| `storage.s3.endpoint` | `""` | Пользовательский S3-эндпоинт (RustFS). Опустить для реального AWS S3 | | `aws.existingSecret` | `""` | Secret с ключами `aws-access-key-id` и `aws-secret-access-key` | | `query.replicaCount` | `1` | Количество реплик сервиса Query | | `ingest.replicaCount` | `1` | Количество реплик сервиса Ingest | @@ -130,19 +152,19 @@ aws: ## Оверлеи Kustomize -Для настройки под конкретное окружение IceGate предоставляет оверлеи Kustomize, которые компонуют Helm chart с зависимостями инфраструктуры. +Для настройки под конкретное окружение {{product_name}} предоставляет оверлеи Kustomize, которые компонуют Helm chart с зависимостями инфраструктуры. ### Доступные оверлеи | Оверлей | Описание | Инфраструктура | |---------|----------|----------------| -| `skaffold` | Локальная разработка со Skaffold | MinIO, Nessie, стек наблюдаемости | -| `orbstack` | Среда выполнения контейнеров OrbStack | MinIO, Nessie, стек наблюдаемости | -| `aws-glue` | Каталог AWS Glue | Стек наблюдаемости (без MinIO/Nessie) | -| `aws-s3tables` | Каталог AWS S3 Tables | Стек наблюдаемости (без MinIO/Nessie) | -| `external-s3` | Внешний S3 + каталог Nessie | Nessie, стек наблюдаемости (без MinIO) | +| `skaffold` | Локальная разработка со Skaffold | RustFS, стек наблюдаемости | +| `orbstack` | Среда выполнения контейнеров OrbStack | RustFS, стек наблюдаемости | +| `aws-glue` | Каталог AWS Glue | Стек наблюдаемости (внешний S3) | +| `aws-s3tables` | Каталог AWS S3 Tables | Стек наблюдаемости (внешний S3) | +| `external-s3` | Внешний S3 + каталог Nessie | Nessie, стек наблюдаемости | -Все оверлеи используют общую базу (`config/kustomize/base/`), которая разворачивает стек наблюдаемости: Prometheus (kube-prometheus-stack), Grafana с готовыми дашбордами IceGate и Jaeger для распределённой трассировки. +Все оверлеи используют общую базу (`config/kustomize/base/`), которая разворачивает стек наблюдаемости: Prometheus (kube-prometheus-stack), Grafana с готовыми дашбордами {{product_name}} и Jaeger для распределённой трассировки. ### Использование @@ -159,7 +181,7 @@ skaffold dev Каждый оверлей содержит: - `kustomization.yaml` — объявляет Helm charts и патчи -- `values-icegate.yaml` — значения Helm IceGate для данного окружения +- `values-icegate.yaml` — значения Helm {{product_name}} для данного окружения - `secret-aws.yaml` — Secret с учётными данными AWS (отредактировать перед применением) Для создания пользовательского оверлея: diff --git a/ru/getting-started/quickstart.md b/ru/getting-started/quickstart.md index a3ce429..a89d8d5 100644 --- a/ru/getting-started/quickstart.md +++ b/ru/getting-started/quickstart.md @@ -1,21 +1,21 @@ --- title: Быстрый Старт -description: Загрузка и запрос первых данных наблюдаемости в IceGate +description: Загрузка и запрос первых данных наблюдаемости в {{product_name}} --- # Быстрый Старт -Это руководство проведёт вас через процесс загрузки логов, трейсов и метрик в IceGate, а также их запрос через API и Grafana. +Это руководство проведёт вас через процесс загрузки логов, трейсов и метрик в {{product_name}}, а также их запрос через API и Grafana. {% note info %} -Данное руководство предполагает, что IceGate уже запущен. См. [Установка](installation.md) для развёртывания через Helm или [Настройка среды разработки](../development/setup.md) для локального окружения. +Данное руководство предполагает, что {{product_name}} уже запущен. См. [Установка](installation.md) для развёртывания через Helm или [Настройка среды разработки](../development/setup.md) для локального окружения. {% endnote %} ## Загрузка Логов -IceGate принимает данные по протоколу OpenTelemetry (OTLP) через сервис приёма данных. +{{product_name}} принимает данные по протоколу OpenTelemetry (OTLP) через сервис приёма данных. ### Отправка Логов через OTLP HTTP @@ -140,7 +140,8 @@ curl -X POST http://localhost:4318/v1/metrics \ ## Запрос Логов с помощью LogQL -IceGate предоставляет API, совместимый с Loki, через сервис запросов (порт 3100). +{{product_name}} предоставляет API, совместимый с Loki, через сервис запросов (порт 3100) — подмножество API +Loki, перечисленное в [справочнике API](../api-reference/loki.md). ### Базовый Запрос Логов @@ -219,9 +220,9 @@ curl -G http://localhost:3100/loki/api/v1/series \ ## Использование Grafana -IceGate совместим с источником данных Loki в Grafana для визуализации логов и создания дашбордов. +{{product_name}} совместим с источником данных Loki в Grafana для визуализации логов и создания дашбордов. -### Добавление IceGate как Источника Данных +### Добавление {{product_name}} как Источника Данных 1. Откройте Grafana (по умолчанию: [http://localhost:3000](http://localhost:3000)) 2. Перейдите в **Connections** > **Data sources** > **Add data source** @@ -255,11 +256,11 @@ IceGate совместим с источником данных Loki в Grafana ### Готовые Дашборды -При развёртывании с overlay-конфигурациями Kustomize или Docker Compose, Grafana поставляется с предварительно настроенными дашбордами IceGate для метрик сервисов приёма данных и запросов. +При развёртывании с overlay-конфигурациями Kustomize или Docker Compose, Grafana поставляется с предварительно настроенными дашбордами {{product_name}} для метрик сервисов приёма данных и запросов. ## Использование OpenTelemetry Collector -Для производственных нагрузок используйте [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) для пересылки данных из ваших приложений в IceGate: +Для производственных нагрузок используйте [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) для пересылки данных из ваших приложений в {{product_name}}: ```yaml # otel-collector-config.yaml @@ -286,7 +287,7 @@ service: ## Мультитенантность -IceGate изолирует данные по тенантам с помощью заголовка `X-Scope-OrgID`. Данные каждого тенанта физически разделены. +{{product_name}} изолирует данные по тенантам с помощью заголовка `X-Scope-OrgID`. Данные каждого тенанта физически разделены. ```bash # Загрузка данных для тенанта "team-a" diff --git a/ru/guides/data-retention.md b/ru/guides/data-retention.md index ed286ed..37ee58a 100644 --- a/ru/guides/data-retention.md +++ b/ru/guides/data-retention.md @@ -1,6 +1,6 @@ --- title: Хранение и Ротация Данных -description: Настройка жизненного цикла данных, политик хранения и управления хранилищем в IceGate +description: Настройка жизненного цикла данных, политик хранения и управления хранилищем в {{product_name}} --- # Хранение и Ротация Данных diff --git a/ru/guides/grafana-integration.md b/ru/guides/grafana-integration.md index 87518d4..c844f3b 100644 --- a/ru/guides/grafana-integration.md +++ b/ru/guides/grafana-integration.md @@ -1,6 +1,6 @@ --- title: Интеграция с Grafana -description: Настройка Grafana для запросов логов, трейсов и метрик из IceGate +description: Настройка Grafana для запросов логов, трейсов и метрик из {{product_name}} --- # Интеграция с Grafana @@ -11,7 +11,7 @@ description: Настройка Grafana для запросов логов, тр {% endnote %} -Это руководство описывает подключение Grafana ко всем трём API запросов {{product_name}}: Loki для логов (порт 3100), Tempo для трейсов (порт 3200) и Prometheus для метрик (порт 9090). Вы узнаете, как настроить каждый источник данных и проверить подключение. +Это руководство описывает подключение Grafana к API запросов {{product_name}}: Loki для логов (порт 3100) и Tempo для трейсов (порт 3200) — оба реализованы, — а также Prometheus для метрик (порт 9090), который запланирован, но пока не работает: все его маршруты возвращают `501 Not Implemented`, кроме `/-/ready`, который отвечает. Вы узнаете, как настроить каждый источник данных и проверить подключение. ## Следующие Шаги diff --git a/ru/guides/ingestion.md b/ru/guides/ingestion.md index da3c0d9..a2e04ce 100644 --- a/ru/guides/ingestion.md +++ b/ru/guides/ingestion.md @@ -1,6 +1,6 @@ --- title: Загрузка Данных -description: Загрузка логов, трейсов и метрик в IceGate +description: Загрузка логов, трейсов и метрик в {{product_name}} --- # Загрузка Данных @@ -11,7 +11,7 @@ description: Загрузка логов, трейсов и метрик в IceG {% endnote %} -IceGate принимает данные наблюдаемости через протокол OpenTelemetry (OTLP). +{{product_name}} принимает данные наблюдаемости через протокол OpenTelemetry (OTLP). ## Поддерживаемые Протоколы @@ -22,7 +22,7 @@ IceGate принимает данные наблюдаемости через п ## Идентификация Тенанта -IceGate мультитенантный. Укажите тенанта через заголовок `X-Scope-OrgID`: +{{product_name}} мультитенантный. Укажите тенанта через заголовок `X-Scope-OrgID`: ```bash curl -X POST http://localhost:4318/v1/logs \ diff --git a/ru/guides/multi-tenancy.md b/ru/guides/multi-tenancy.md index aee9bd6..5727f91 100644 --- a/ru/guides/multi-tenancy.md +++ b/ru/guides/multi-tenancy.md @@ -1,6 +1,6 @@ --- title: Мультитенантность -description: Настройка и использование изоляции мультитенантности в IceGate +description: Настройка и использование изоляции мультитенантности в {{product_name}} --- # Мультитенантность @@ -11,7 +11,7 @@ description: Настройка и использование изоляции {% endnote %} -IceGate разработан как мультитенантная система, обеспечивающая изоляцию данных между разными организациями или командами. +{{product_name}} разработан как мультитенантная система, обеспечивающая изоляцию данных между разными организациями или командами. ## Идентификация Тенанта diff --git a/ru/guides/performance-tuning.md b/ru/guides/performance-tuning.md index 4c22a12..9bbcecb 100644 --- a/ru/guides/performance-tuning.md +++ b/ru/guides/performance-tuning.md @@ -1,6 +1,6 @@ --- title: Оптимизация Производительности -description: Оптимизация пропускной способности загрузки, производительности запросов и компакции в IceGate +description: Оптимизация пропускной способности загрузки, производительности запросов и компакции в {{product_name}} --- # Оптимизация Производительности diff --git a/ru/guides/querying.md b/ru/guides/querying.md index 627a7fd..f84bc1f 100644 --- a/ru/guides/querying.md +++ b/ru/guides/querying.md @@ -5,7 +5,10 @@ description: Запросы к логам, трейсам и метрикам с # Запросы к Данным -IceGate предоставляет API совместимые с Loki, Prometheus и Tempo для запросов к данным наблюдаемости. +{{product_name}} предоставляет API, совместимые с Loki и Tempo, для запросов к данным наблюдаемости, а также +Arrow Flight SQL для универсального SQL. [API, совместимый с Prometheus](../api-reference/prometheus.md), +запланирован, но пока не реализован — до тех пор запрашивайте метрики через Flight SQL. Источником +истины о том, какие эндпоинты обслуживаются сегодня, являются страницы справочника API. ## LogQL для Логов diff --git a/ru/index.yaml b/ru/index.yaml index 49d9afb..8f694e6 100644 --- a/ru/index.yaml +++ b/ru/index.yaml @@ -2,7 +2,10 @@ title: Документация IceGate description: | Движок озера данных для наблюдаемости, разработанный быть быстрым, простым в использовании, экономичным, масштабируемым и отказоустойчивым. meta: - title: IceGate - Движок Озера Данных для Наблюдаемости + # See en/index.yaml for why the product name is dropped here and why `description` is needed. + title: Движок Озера Данных для Наблюдаемости + # Quoted: a bare `: ` inside a YAML scalar is parsed as a mapping and fails the build. + description: "Документация IceGate — движка озера данных для наблюдаемости с открытым исходным кодом. Установка, запросы и эксплуатация на Apache Iceberg, Arrow и Parquet." links: - title: Начало Работы description: Установите IceGate и выполните первые запросы за несколько минут @@ -11,7 +14,7 @@ links: description: Узнайте как загружать данные, делать запросы к логам и настраивать мультитенантность href: guides/ingestion.md - title: Справочник API - description: API совместимые с Loki, Prometheus и Tempo + description: API, совместимые с Loki® и Tempo® (Prometheus® планируется) href: api-reference/loki.md - title: Архитектура description: Понимание архитектуры разделения вычислений и хранения IceGate diff --git a/ru/operations/deployment.md b/ru/operations/deployment.md index 0a95786..b6f3c98 100644 --- a/ru/operations/deployment.md +++ b/ru/operations/deployment.md @@ -1,16 +1,16 @@ --- title: Развёртывание -description: Развёртывание IceGate в продакшен окружениях +description: Развёртывание {{product_name}} в продакшен окружениях --- # Развёртывание -Это руководство охватывает развёртывание IceGate в продакшен окружениях. +Это руководство охватывает развёртывание {{product_name}} в продакшен окружениях. ## Предварительные Требования -- **Объектное Хранилище:** S3, MinIO или S3-совместимое хранилище -- **Каталог Iceberg:** Nessie (REST), AWS S3 Tables или AWS Glue +- **Объектное Хранилище:** S3, RustFS или S3-совместимое хранилище +- **Каталог Iceberg:** встроенный S3-каталог (по умолчанию), либо Nessie (REST), AWS S3 Tables или AWS Glue - **Docker/Kubernetes:** Для оркестрации контейнеров ## Архитектурные Решения @@ -21,7 +21,7 @@ description: Развёртывание IceGate в продакшен окруж |-----------|-----------------|------------| | Ingest | Горизонтальное | Масштабируйте для увеличения пропускной способности записи | | Query | Горизонтальное | Масштабируйте для увеличения параллелизма запросов | -| Maintain | Один лидер | Координирует компакцию | +| Maintain | Горизонтальное | Воркеры координируются через состояние задач в объектном хранилище (compare-and-swap) | ### Требования к Ресурсам @@ -50,7 +50,7 @@ description: Развёртывание IceGate в продакшен окруж Проект включает профили Docker Compose для различных сценариев развёртывания: ```bash -# Основные сервисы: MinIO, Nessie, Ingest, Query, Maintain +# Основные сервисы: RustFS, Ingest, Query, Maintain make run-core-release # Основные + генератор нагрузки для тестирования @@ -66,26 +66,19 @@ make run-analytics-release ```yaml # docker-compose.yml services: - minio: - image: minio/minio:latest - command: server /data --console-address ":9001" + rustfs: + image: rustfs/rustfs:1.0.0-beta.8 environment: - MINIO_ROOT_USER: ${S3_ACCESS_KEY} - MINIO_ROOT_PASSWORD: ${S3_SECRET_KEY} + RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${S3_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_CONSOLE_ENABLE: "true" + RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001" volumes: - - minio-data:/data + - rustfs-data:/data ports: - - "9000:9000" - - "9001:9001" - - nessie: - image: projectnessie/nessie:latest - environment: - NESSIE_VERSION_STORE_TYPE: ROCKSDB - volumes: - - nessie-data:/data - ports: - - "19120:19120" + - "9000:9000" # S3 API + - "9001:9001" # Console ingest: image: icegate/ingest:latest @@ -100,8 +93,7 @@ services: - "4318:4318" # OTLP HTTP - "9091:9091" # Prometheus metrics depends_on: - - minio - - nessie + - rustfs query: image: icegate/query:latest @@ -116,9 +108,9 @@ services: - "3100:3100" # Loki API - "9090:9090" # Prometheus API - "3200:3200" # Tempo API + - "8815:8815" # Arrow Flight SQL depends_on: - - minio - - nessie + - rustfs maintain: image: icegate/maintain:latest @@ -128,12 +120,10 @@ services: volumes: - ./config/maintain.yaml:/etc/icegate/maintain.yaml:ro depends_on: - - minio - - nessie + - rustfs volumes: - minio-data: - nessie-data: + rustfs-data: query-cache: ``` @@ -165,7 +155,7 @@ docker build -t icegate/maintain:latest \ ### Helm Charts -IceGate включает Helm charts для развёртывания в Kubernetes: +{{product_name}} включает Helm charts для развёртывания в Kubernetes: ```bash # Установка из локальных charts @@ -187,7 +177,7 @@ helm install icegate ./config/helm/icegate \ | `orbstack` | Среда выполнения контейнеров OrbStack | | `aws-glue` | Интеграция с каталогом AWS Glue | | `aws-s3tables` | Интеграция каталога AWS S3 Tables | -| `external-s3` | Внешнее хранилище S3 (не MinIO) | +| `external-s3` | Внешнее хранилище S3 с каталогом Nessie | ```bash # Применение с kustomize @@ -205,13 +195,13 @@ storage: region: us-east-1 ``` -### MinIO +### RustFS (S3-совместимое) ```yaml storage: backend: !s3 bucket: warehouse - endpoint: http://minio:9000 + endpoint: http://rustfs:9000 region: us-east-1 ``` @@ -242,7 +232,7 @@ services: ### Метрики -Сервисы IceGate предоставляют метрики Prometheus на выделенном порту (по умолчанию: 9091): +Сервисы {{product_name}} предоставляют метрики Prometheus на выделенном порту (по умолчанию: 9091): - Метрики Ingest: `http://ingest:9091/metrics` - Метрики Query: `http://query:9091/metrics` @@ -259,7 +249,7 @@ metrics: ### Самонаблюдаемость с Трейсингом -IceGate может экспортировать собственные трейсы через OTLP для отладки: +{{product_name}} может экспортировать собственные трейсы через OTLP для отладки: ```yaml tracing: @@ -283,7 +273,7 @@ environment: ### Сетевая Безопасность - Используйте TLS для всех внешних подключений -- Ограничьте доступ к MinIO/Nessie только внутренней сетью +- Ограничьте доступ к объектному хранилищу и любому внешнему каталогу только внутренней сетью - Используйте сетевые политики в Kubernetes ### Аутентификация diff --git a/ru/operations/maintenance.md b/ru/operations/maintenance.md index 05fc292..48effce 100644 --- a/ru/operations/maintenance.md +++ b/ru/operations/maintenance.md @@ -1,6 +1,6 @@ --- title: Обслуживание -description: Обслуживание IceGate для оптимальной производительности +description: Обслуживание {{product_name}} для оптимальной производительности --- # Обслуживание @@ -19,7 +19,7 @@ maintain migrate create -c maintain.yaml ### Обновление Схемы -Обновление схем существующих таблиц при обновлении IceGate: +Обновление схем существующих таблиц при обновлении {{product_name}}: ```bash maintain migrate upgrade -c maintain.yaml @@ -52,8 +52,9 @@ maintain migrate upgrade -c maintain.yaml --dry-run 3. Параллельно читает Parquet файлы WAL 4. Объединяет и перепартиционирует данные 5. Записывает оптимизированные файлы данных Iceberg -6. Фиксирует новый снапшот в каталоге -7. Удаляет обработанные сегменты WAL +6. Фиксирует новый снапшот в каталоге, записывая последний зафиксированный WAL-офсет в сводку снапшота + +Shift не удаляет сегменты WAL. Их освобождает правило жизненного цикла объектов на бакете очереди, а офсет в сводке снапшота — это то, что позволяет shift'у продолжить с места остановки. ### Настройка Производительности Shift @@ -144,7 +145,15 @@ curl http://localhost:4318/health ### Резервное Копирование Каталога -Nessie хранит метаданные каталога. Создайте резервную копию данных RocksDB: +В S3-каталоге по умолчанию метаданные — это `root.json` и файлы метаданных таблиц в бакете warehouse, поэтому резервная копия сводится к копированию этого префикса, без остановки какого-либо сервиса: + +```bash +aws s3 sync s3://warehouse/catalog/ ./catalog-backup/ +``` + +`sync` не является атомарным снимком: он сначала получает список, затем копирует, и коммиты, попавшие в этот промежуток, могут оставить копию со смешанными поколениями каталога. Для копии на момент времени используйте версионирование бакета (см. ниже), читая одну версию, либо делайте копию при остановленных записях. Проверяйте резервную копию, восстановив её в тестовый префикс и получив список таблиц, прежде чем на неё полагаться. + +Если вы используете REST-бэкенд каталога, создайте резервную копию данных RocksDB Nessie: ```bash # Остановка Nessie @@ -178,7 +187,7 @@ CALL icegate.system.rollback_to_snapshot('logs', 123456789); ```bash aws s3api put-bucket-versioning \ - --bucket icegate-warehouse \ + --bucket warehouse \ --versioning-configuration Status=Enabled ``` diff --git a/ru/operations/troubleshooting.md b/ru/operations/troubleshooting.md index 73c3fc5..059daae 100644 --- a/ru/operations/troubleshooting.md +++ b/ru/operations/troubleshooting.md @@ -1,6 +1,6 @@ --- title: Устранение Неполадок -description: Диагностика и решение распространённых проблем IceGate +description: Диагностика и решение распространённых проблем {{product_name}} --- # Устранение Неполадок @@ -60,15 +60,15 @@ docker compose logs -f ingest **Симптомы:** -- "Connection refused" к MinIO +- "Connection refused" к объектному хранилищу - Ошибки аутентификации S3 **Решения:** -1. Проверьте, что MinIO запущен: +1. В локальном развёртывании RustFS проверьте, что объектное хранилище работает. Путь готовности специфичен для RustFS — на AWS S3 или у другого провайдера сразу переходите к шагу 3: ```bash - curl http://localhost:9000/minio/health/ready + curl http://localhost:9000/health/ready ``` 2. Проверьте учётные данные: @@ -78,7 +78,7 @@ docker compose logs -f ingest echo $AWS_SECRET_ACCESS_KEY ``` -3. Протестируйте подключение к S3: +3. Протестируйте подключение к S3. Уберите `--endpoint-url`, если бэкенд — настоящий AWS S3: ```bash aws s3 ls --endpoint-url http://localhost:9000 @@ -93,19 +93,31 @@ docker compose logs -f ingest **Решения:** -1. Проверьте, что Nessie запущен: +1. В S3-каталоге по умолчанию убедитесь, что объект состояния каталога читается, — отдельного сервиса каталога здесь нет: ```bash - curl http://localhost:19120/api/v1/trees + aws --endpoint-url http://localhost:9000 s3 ls s3://warehouse/catalog/root.json ``` + Отсутствие `root.json` означает, что миграция не выполнялась. Сначала запустите `maintain migrate create`. + 2. Проверьте конфигурацию каталога: ```yaml catalog: - backend: !rest - uri: http://nessie:19120/iceberg + backend: !s3 + warehouse: catalog warehouse: s3://warehouse/ + properties: + bucket: warehouse + region: us-east-1 + endpoint: http://rustfs:9000 + ``` + +3. Только для REST-бэкенда проверьте, что Nessie запущен: + + ```bash + curl http://localhost:19120/api/v1/trees ``` ## Проблемы с Запросами @@ -272,10 +284,10 @@ docker compose logs -f ingest docker stats > stats.txt ``` -2. Обратитесь к [GitHub Issues](https://github.com/icegatetech/icegate/issues) +2. Обратитесь к [GitHub Issues]({{repo_url}}/issues) 3. Включите: - - Версию IceGate + - Версию {{product_name}} - Конфигурацию (очищенную от секретов) - Сообщения об ошибках - Шаги для воспроизведения diff --git a/ru/toc.yaml b/ru/toc.yaml index df8e3c7..95cd5bf 100644 --- a/ru/toc.yaml +++ b/ru/toc.yaml @@ -96,3 +96,6 @@ items: - name: FAQ href: faq.md + + - name: Товарные знаки + href: trademarks.md diff --git a/ru/trademarks.md b/ru/trademarks.md new file mode 100644 index 0000000..52c97c4 --- /dev/null +++ b/ru/trademarks.md @@ -0,0 +1,38 @@ +--- +title: Товарные знаки +description: Атрибуция товарных знаков третьих лиц, упоминаемых в документации {{product_name}} +--- + +# Товарные знаки + +{{product_name}} разрабатывается компанией TripleCloud и распространяется по лицензии {{license}}. + +Эта документация упоминает сторонние проекты, чтобы фактически описать, какие форматы записывает +{{product_name}} и какие протоколы реализуют его API. Такое номинативное использование не +подразумевает аффилированности с правообладателями этих знаков, равно как их одобрения или +спонсорства. + +Apache®, Apache Iceberg, Apache Arrow, Apache Parquet, Apache DataFusion, Apache Arrow Flight SQL +и логотипы соответствующих проектов являются зарегистрированными товарными знаками или товарными +знаками The Apache Software Foundation в США и/или других странах. + +OpenTelemetry® и Prometheus® — зарегистрированные товарные знаки The Linux Foundation. + +Grafana®, Loki® и Tempo® — зарегистрированные товарные знаки Raintank, Inc. dba Grafana Labs. + +{{product_name}} не аффилирован с этими организациями, не одобрен и не спонсируется ими. Все +прочие товарные знаки принадлежат их правообладателям. + +## Что здесь означает «совместимый» + +Когда эта документация описывает API как совместимый с Loki или Tempo, это означает, что +{{product_name}} реализует подмножество HTTP API чтения соответствующего проекта — достаточное для +эндпоинтов, описанных в справочниках [Loki](api-reference/loki.md) и [Tempo](api-reference/tempo.md), но не полную реализацию. + +API, совместимый с Prometheus, **запланирован, но не реализован**: все его маршруты возвращают +`501 Not Implemented`, кроме `/-/ready`, который отвечает. Его +[страница справочника](api-reference/prometheus.md) описывает предполагаемую поверхность, а не +работающую. + +Страницы справочника API — источник истины о том, что работает сегодня: если эндпоинт или параметр +там не указан, считайте, что он ещё не реализован.