Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ but `hoptimator-api` is the actual extension point.
- **Multi-hop, declarative.** You don't write Flink jobs and you don't request
topics. The planner figures out the topology from a query.
- **Kubernetes out of the box, not as a hard requirement.** The bundled
deployers target Kubernetes, so pipelines show up as first-class CRDs and
deployers target Kubernetes, so pipelines show up as first-class custom resources and
`kubectl get pipelines` Just Works. The `Deployer` interface is the actual
extension point — anything that knows how to materialize a spec can take
the place of the defaults.
Expand Down
10 changes: 5 additions & 5 deletions docs/extending/data-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Adding a new external system to Hoptimator usually means three things:

1. A **JDBC adapter** that exposes the system's tables and schemas through a
JDBC connection — Hoptimator reads metadata through this.
2. A **`Database` CRD** that registers the adapter (a JDBC URL plus a
2. A **`Database` custom resource** that registers the adapter (a JDBC URL plus a
schema name) so the catalog includes it.
3. A **`TableTemplate`** (and possibly a **`JobTemplate`**) that tells
Hoptimator how to deploy resources for the system — Kafka topics, Venice
Expand Down Expand Up @@ -53,7 +53,7 @@ com.example.hoptimator.mysystem.MySystemDriver

## Registering with the catalog

Once your driver is on the classpath, a `Database` CRD makes it visible to
Once your driver is on the classpath, a `Database` custom resource makes it visible to
Hoptimator:

```yaml
Expand All @@ -72,7 +72,7 @@ your driver expects. See the
[Database CRD reference](../kubernetes/crd-reference.md#database) for all
fields.

After applying the CRD, Hoptimator's catalog picks it up on the next
After applying the custom resource, Hoptimator's catalog picks it up on the next
connection — `!tables` in the SQL CLI should show your system's tables.

## Telling Hoptimator how to deploy resources
Expand Down Expand Up @@ -107,13 +107,13 @@ spec:
```

No Java. The bundled Kafka deployment uses this pattern — Hoptimator emits
a `KafkaTopic` CRD; Strimzi creates the topic. See
a `KafkaTopic` custom resource; Strimzi creates the topic. See
[Templates and configuration](../kubernetes/templates.md) for the placeholder syntax and
matching rules.

### Path B: imperative, via a custom Deployer

If your system needs an admin API call rather than a YAML CRD apply (e.g.
If your system needs an admin API call rather than a YAML custom resource apply (e.g.
calling a Venice controller's REST endpoint), you'll need a custom
`Deployer`. This is the path the bundled `hoptimator-venice` and
`hoptimator-kafka` modules take in addition to (or instead of) templates.
Expand Down
4 changes: 2 additions & 2 deletions docs/extending/deployers.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ with a spec" can plug in.
You'll need a deployer when:

- You're integrating a system whose resources are created via an admin API
(REST, gRPC, command-line) rather than a CRD apply. Templates aren't
(REST, gRPC, command-line) rather than a custom resource apply. Templates aren't
enough — you need imperative code.
- You want pipelines to deploy to something other than Kubernetes — a
Nomad cluster, an external service registry, your own internal control
Expand Down Expand Up @@ -116,7 +116,7 @@ The bundled Kafka path is a good shape to copy:
- [`KafkaDeployerProvider`](https://github.com/linkedin/Hoptimator/blob/main/hoptimator-kafka/src/main/java/com/linkedin/hoptimator/kafka/KafkaDeployerProvider.java)
— type-checks the `Deployable`, resolves the `Database`'s connection
config via `context.databaseProperties(catalog, schema, "jdbc:kafka://")`
(the JDBC URL the `Database` CRD points at), and constructs the deployer.
(the JDBC URL the `Database` custom resource points at), and constructs the deployer.
- [`KafkaDeployer`](https://github.com/linkedin/Hoptimator/blob/main/hoptimator-kafka/src/main/java/com/linkedin/hoptimator/kafka/KafkaDeployer.java)
— `create()` calls Kafka's AdminClient API to create the topic;
`restore()` walks back and deletes any topic the current operation
Expand Down
6 changes: 3 additions & 3 deletions docs/extending/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Extending Hoptimator

Hoptimator's behavior is driven by Java SPI plug-ins (`ServiceLoader`-based)
and by the `TableTemplate` / `JobTemplate` CRDs. Most extensions don't need
and by the `TableTemplate` / `JobTemplate` custom resources. Most extensions don't need
both — pick the layer that matches what you're doing.

## Pick the right surface
Expand Down Expand Up @@ -47,15 +47,15 @@ For surfaces that produce multiple values for the same input — `Validator`,

### "I just want to add my system to the catalog"

The lowest-friction path is **a JDBC driver + a `Database` CRD**. Hoptimator
The lowest-friction path is **a JDBC driver + a `Database` custom resource**. Hoptimator
treats anything that responds to a JDBC URL as a potential catalog source.
You point a `Database` at it, and Hoptimator pulls schemas and tables from
that connection. See [Data sources → Adapter](data-sources.md#the-jdbc-adapter).

### "I need Hoptimator to actually deploy my system's resources"

After the adapter, ship a `TableTemplate` (or `JobTemplate`) that emits the
YAML for your storage system's CRD or operator. Templates are a CRD, so
YAML for your storage system's custom resource or operator. Templates are a custom resource, so
this is YAML-only — no Java needed. See
[Templates and configuration](../kubernetes/templates.md).

Expand Down
4 changes: 2 additions & 2 deletions docs/extending/validators.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Validators

A `Validator` inspects a SQL statement, a CRD, or a planned pipeline element
A `Validator` inspects a SQL statement, a custom resource, or a planned pipeline element
*before* anything is deployed and rejects it if it doesn't meet your
constraints. Validators are the right place for environment-specific policy:
naming conventions, schema compatibility checks, ACL enforcement, anything
Expand All @@ -20,7 +20,7 @@ DDL path:

If any validator emits an `Issues.error(...)` at any of those points, the
whole operation aborts and the error message surfaces to the user (or to
the operator's status field for CRD-driven changes).
the operator's status field for custom-resource-driven changes).

## The interfaces

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ deploy the operator standalone and feed it Subscriptions from CI.
```

The same flow applies whether you start from SQL (CLI, JDBC, MCP) or from a
`Subscription` CRD applied with `kubectl apply -f`.
`Subscription` custom resource applied with `kubectl apply -f`.

## Step 1 — Parse and resolve

Expand Down Expand Up @@ -164,7 +164,7 @@ supplies a Calcite-backed `DeploymentContext` (row type read from the catalog);
the direct path supplies a `DirectDeploymentContext` that carries the caller's
Avro schema (deriving the row type on demand) and resolves `Database` config
registry-natively via a `DatabaseConfigResolver`
(the K8s implementation reads `Database` CRDs directly — see
(the K8s implementation reads `Database` custom resources directly — see
`DatabaseConfigResolvers`). The direct path opens no connection and touches no
Calcite catalog; only the SQL engine's read/plan path still uses the JDBC
driver layer.
Expand Down Expand Up @@ -205,7 +205,7 @@ contributions should not target them.
the runtime's operator already understands (e.g. a Beam `FlinkSessionJob`,
a Spark Operator `SparkApplication`). Hoptimator generates the spec; the
target operator runs the job. No `Engine` registration is required for
this path — the `Engine` CRD is only needed if Hoptimator should submit
this path — the `Engine` custom resource is only needed if Hoptimator should submit
*queries* (not pipelines) directly to the runtime.
- **A new deployment target**: implement `Deployer` and register it via
`DeployerProvider`. Kubernetes is the default but not a hard requirement;
Expand Down
40 changes: 20 additions & 20 deletions docs/getting-started/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,23 @@ the documentation will read more naturally.

## At a glance

| Concept | What it is |
| ------------------- | ------------------------------------------------------------------------------------------------ |
| **Database** | A connection to an external system that exposes tables (Kafka, Venice, MySQL, etc.). |
| **Catalog** | The unified namespace that lets a single SQL statement reference tables across many databases. |
| **View** | A named SQL query, evaluated lazily. |
| **Materialized view** | A view backed by a running data pipeline that continuously writes results to a sink. |
| **Pipeline** | The set of sources, sink, and job that together implement a materialized view. |
| **Engine** | A runtime Hoptimator can submit *queries* to (e.g. a Flink SQL gateway). Optional. Pipeline materialization does *not* require one. |
| **Connector** | Configuration that tells a runtime how to read from or write to a database. Used by the planner and embedded in template output. |
| **Deployer** | The component that turns a planned pipeline element into real infrastructure. |
| **Validator** | Pre-deploy check that rejects SQL, CRDs, or planned pipelines that violate environment policy. |
| **TableTemplate** | Declarative recipe for materializing a source/sink in a particular database. |
| **JobTemplate** | Declarative recipe for materializing a job on a particular engine. |
| **TableTrigger** | Fires a Kubernetes job when an upstream table changes (or on a schedule). |
| **LogicalTable** | An abstraction model: one named entity that physically lives in many backends (nearline / online / offline). Auto-syncs and auto-backfills its tiers. |
| **Subscription** | YAML-native way to declare a materialized view; equivalent to `CREATE MATERIALIZED VIEW ... AS`. |
| **Hint** | Key/value passed at runtime that templates and connectors can pick up. |
| Concept | What it is |
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Database** | A connection to an external system that exposes tables (Kafka, Venice, MySQL, etc.). |
| **Catalog** | The unified namespace that lets a single SQL statement reference tables across many databases. |
| **View** | A named SQL query, evaluated lazily. |
| **Materialized view** | A view backed by a running data pipeline that continuously writes results to a sink. |
| **Pipeline** | The set of sources, sink, and job that together implement a materialized view. |
| **Engine** | A runtime Hoptimator can submit *queries* to (e.g. a Flink SQL gateway). Optional. Pipeline materialization does *not* require one. |
| **Connector** | Configuration that tells a runtime how to read from or write to a database. Used by the planner and embedded in template output. |
| **Deployer** | The component that turns a planned pipeline element into real infrastructure. |
| **Validator** | Pre-deploy check that rejects SQL, custom resources, or planned pipelines that violate environment policy. |
| **TableTemplate** | Declarative recipe for materializing a source/sink in a particular database. |
| **JobTemplate** | Declarative recipe for materializing a job on a particular engine. |
| **TableTrigger** | Fires a Kubernetes job when an upstream table changes (or on a schedule). |
| **LogicalTable** | An abstraction model: one named entity that physically lives in many backends (nearline / online / offline). Auto-syncs and auto-backfills its tiers. |
| **Subscription** | YAML-native way to declare a materialized view; equivalent to `CREATE MATERIALIZED VIEW ... AS`. |
| **Hint** | Key/value passed at runtime that templates and connectors can pick up. |

## Databases, schemas, and tables

Expand Down Expand Up @@ -108,7 +108,7 @@ Operator — picks it up and runs the job. Hoptimator is not in the data path.

## Engines (optional)

An **Engine** CRD registers a runtime Hoptimator can submit **queries** to —
An **Engine** custom resource registers a runtime Hoptimator can submit **queries** to —
typically a Flink SQL gateway behind a JDBC URL. This is the path used when
Hoptimator needs to *execute* SQL itself, e.g. for interactive `SELECT`
against tables that aren't in-process.
Expand Down Expand Up @@ -136,7 +136,7 @@ See [Extending Hoptimator](../extending/index.md) when those docs land.

## Validators

A **Validator** inspects a SQL statement, a CRD, or a planned pipeline
A **Validator** inspects a SQL statement, a custom resource, or a planned pipeline
element *before* it deploys and rejects it if it doesn't meet your
constraints. Where `Deployer` is "make this real," `Validator` is "check
this is allowed."
Expand Down Expand Up @@ -256,7 +256,7 @@ infrastructure for each binding:
the normal Deployer SPI to create whatever the storage system needs (a
Kafka topic, a Venice store, an HDFS dataset).
- **Implicit inter-tier pipelines.** Hoptimator auto-deploys
`nearline → online` and `nearline → offline` Pipeline CRDs to keep the
`nearline → online` and `nearline → offline` Pipeline custom resources to keep the
tiers consistent. You don't write the Kafka-to-Venice job; it appears
because the LogicalTable says it should.
- **Auto-backfill triggers.** When an offline tier is present, a
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ See the **[Kubernetes guide](kubernetes/index.md)**:
See **[Extending Hoptimator](extending/index.md)**:

- [Adding a new data source](extending/data-sources.md) — JDBC adapter,
`Database` CRD, `TableTemplate` authoring.
`Database` custom resource, `TableTemplate` authoring.
- [Deployers](extending/deployers.md) — implementing `Deployer` for a new
deployment target, `DeployerProvider` registration, lifecycle.
- [Validators](extending/validators.md) — pre-deploy policy enforcement
Expand Down
14 changes: 7 additions & 7 deletions docs/kubernetes/crd-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,16 +360,16 @@ spec:

### Spec fields

| Field | Type | Description |
| ----------- | ------ | -------------------------------------------------------------------------------------- |
| `tableName` | string | Original table name as declared in `CREATE TABLE` (e.g. `audience`). |
| `tiers` | object | Map of tier name (`nearline`, `online`, `offline`) to a tier binding. |
| Field | Type | Description |
|-------------|--------|-----------------------------------------------------------------------|
| `tableName` | string | Original table name as declared in `CREATE TABLE` (e.g. `audience`). |
| `tiers` | object | Map of tier name (`nearline`, `online`, `offline`) to a tier binding. |

Each tier binding has one field:

| Field | Type | Required | Description |
| ---------- | ------ | :------: | ------------------------------------------------- |
| `database` | string | yes | Name of the `Database` CRD backing this tier. |
| Field | Type | Required | Description |
|------------|--------|:--------:|-----------------------------------------------------------|
| `database` | string | yes | Name of the `Database` custom resource backing this tier. |

The `LogicalTableDeployer` runs at create time to deploy physical tier
resources, the implicit inter-tier sync pipelines, and the offline-tier
Expand Down
2 changes: 1 addition & 1 deletion docs/kubernetes/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,4 @@ operator reconciles them.
- [Architecture](../getting-started/architecture.md) — what the operator
is doing in the bigger picture.
- [DDL reference](../user-guide/ddl-reference.md) — SQL DDL that has YAML
CRD equivalents.
custom resource equivalents.
8 changes: 4 additions & 4 deletions docs/kubernetes/operator.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# The operator

`hoptimator-operator` is the long-running Kubernetes controller that
reconciles Hoptimator's CRDs. It uses the same Deployer machinery as the
reconciles Hoptimator's custom resources. It uses the same Deployer machinery as the
SQL path — when a `Subscription`, `Pipeline`, `View`, or `TableTrigger`
changes, it asks the deployers to bring the cluster state in line with the
spec.
Expand Down Expand Up @@ -47,7 +47,7 @@ command:

## Namespace scoping

By default, the operator watches **all namespaces** for the CRDs it owns.
By default, the operator watches **all namespaces** for the custom resources it owns.
To restrict it to one namespace, pass `--watch <namespace>`:

```yaml
Expand Down Expand Up @@ -143,8 +143,8 @@ Logs are the primary debugging surface today.
## When *not* to run the operator

The operator is only required when you want continuous reconciliation —
typically when applying CRDs via `kubectl` rather than driving everything
typically when applying custom resources via `kubectl` rather than driving everything
through the JDBC path. If your workflow is "developer runs `./hoptimator`
to create materialized views" and nothing else applies CRDs, you don't
to create materialized views" and nothing else applies custom resources, you don't
strictly need the operator running. The CLI deploys synchronously and waits
for success.
Loading
Loading