diff --git a/INTEGRATION-TESTS.md b/INTEGRATION-TESTS.md index 1d7d39406..2567d7c54 100644 --- a/INTEGRATION-TESTS.md +++ b/INTEGRATION-TESTS.md @@ -54,29 +54,23 @@ local runs; skip them: ## Presto -The test is self-contained: it creates and seeds its own `memory.wayang_it` -tables in Presto's built-in **in-memory connector** (scaled to 120k rows so the -optimizer elects SQL pushdown) and drops them afterwards — no Hive metastore or -object storage required. +The test creates and seeds its own `memory.wayang_it` tables in a user-managed +Presto deployment with the memory connector enabled. It drops the fixtures +afterward. ```bash -# 1. start a single PrestoDB node with the in-memory connector -cd presto-setup && docker compose up -d --wait && cd .. - -# 2. run the operator tests (JDK 17) +# Run the operator tests against the configured deployment (JDK 17) +PRESTO_HOST=presto.example.com PRESTO_PORT=8080 PRESTO_USER=wayang \ JAVA_HOME=/path/to/jdk-17 \ mvn -o test -pl wayang-platforms/wayang-presto \ - -Dtest=AllOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ + -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ -Drat.skip=true -Dlicense.skip=true -Dmaven.javadoc.skip=true -Pskip-prerequisite-check - -# 3. tear down when done -cd presto-setup && docker compose down -v && cd .. ``` Expected: `Tests run: 4, Failures: 0, Errors: 0, Skipped: 0`. -`docker compose up -d --wait` blocks on the container healthcheck, so Presto is -query-ready when it returns. Presto listens on host port **8081** (container 8080). +Connection settings are supplied through `PRESTO_HOST`, `PRESTO_PORT`, and +`PRESTO_USER`. --- diff --git a/bigquery-setup/README.md b/bigquery-setup/README.md deleted file mode 100644 index bffd6433d..000000000 --- a/bigquery-setup/README.md +++ /dev/null @@ -1,321 +0,0 @@ -# BigQuery Local Setup - -Local BigQuery emulator and validation instructions for the Wayang BigQuery -platform. - -The local validation path has two parts and does not require a GCP account: - -1. Build the Wayang BigQuery platform and run the shared JDBC SQL-generation tests. -2. Run BigQuery-compatible SQL tests against the local emulator. - -There is also an optional real-BigQuery validation path: - -3. Run the Wayang BigQuery operator tests through JDBC against real BigQuery. - -Run the commands below from the repository root. Java 17 and Docker with Docker -Compose are required for the emulator tests. A GCP project and service-account -key, plus the `gcloud` SDK, are required only for the optional real-BigQuery -operator tests. Maven is provided by the repository wrapper. - -## Command Conventions - -Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on -Windows PowerShell from the repository root. Docker Compose commands are the -same on both platforms. The `gcloud` commands also work on Windows; either run -each command on one line or replace Bash line-continuation backslashes with -PowerShell backticks. - -## Stack - -| Component | Image | Port | Role | -|-----------|-------|------|------| -| **BigQuery Emulator** | `ghcr.io/goccy/bigquery-emulator:0.6.6` | 9050 (HTTP) / 9060 (gRPC) | BigQuery-compatible SQL engine | - -Single container. Data is seeded from `data.yaml` on startup and lives in memory. - -## Directory Layout - -``` -bigquery-setup/ -|-- docker-compose.yml # Emulator container -|-- data.yaml # Seed data (test-project.sales.orders) -|-- pom.xml # Standalone Maven project -`-- src/test/java/.../ - `-- BigQueryEmulatorIT.java # JUnit 5 integration tests - -wayang-platforms/wayang-bigquery/src/test/java/.../ -`-- BigQueryOperatorsIT.java # Wayang operator tests against real BigQuery -``` - -## 1. Test the Wayang BigQuery Platform - -Build the BigQuery platform and its required modules: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -DskipTests -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -DskipTests -Drat.skip=true test -``` - -Then run the shared JDBC SQL-generation tests: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -Expected result: - -```text -Wayang Platform BigQuery ... SUCCESS -Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -``` - -## 2. Test the Local BigQuery Emulator - -### 1. Start the emulator - -```bash -docker compose -f bigquery-setup/docker-compose.yml up -d -``` - -The emulator starts in ~2 seconds. Data from `data.yaml` is loaded automatically. - -### 2. Run integration tests - -```bash -./mvnw -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test -``` - -The successful result must show that no tests were skipped: - -```text -Tests run: 7, Failures: 0, Errors: 0, Skipped: 0 -``` - -If the emulator is unavailable, Maven can still print `BUILD SUCCESS` while -showing `Skipped: 7`. That does not count as a successful emulator test. - -### 3. Manual exploration - -Query via curl: - -```bash -curl -s -X POST \ - "http://localhost:9050/bigquery/v2/projects/test-project/queries" \ - -H "Content-Type: application/json" \ - -d '{"query": "SELECT * FROM sales.orders LIMIT 5", "useLegacySql": false}' \ - | python3 -m json.tool -``` - -### 4. Tear down - -```bash -docker compose -f bigquery-setup/docker-compose.yml down -``` - -## 3. Optional: Test the Wayang Operators Against Real BigQuery - -`BigQueryOperatorsIT` uses the BigQuery JDBC driver and cannot run against the -local emulator. It requires a real GCP project and a service-account JSON key. - -The test setup creates its own fixture tables in a configurable dataset -(`wayang_it` by default): `orders`, `regions`, and `operator_result`. The tests -issue `SELECT`, `CREATE TABLE AS`, and `DROP` statements, then remove those -tables during cleanup. - -### 1. Enable BigQuery and create a service account - -Replace `YOUR_PROJECT_ID` in the following commands: - -```bash -gcloud auth login -gcloud config set project YOUR_PROJECT_ID -gcloud services enable bigquery.googleapis.com - -gcloud iam service-accounts create wayang-bq \ - --display-name="Wayang BigQuery IT" - -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ - --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/bigquery.jobUser" - -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ - --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/bigquery.dataEditor" - -gcloud iam service-accounts keys create "$HOME/wayang-bq-key.json" \ - --iam-account="wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" -``` - -On Windows PowerShell, the same setup can be run as: - -```powershell -gcloud auth login -gcloud config set project YOUR_PROJECT_ID -gcloud services enable bigquery.googleapis.com -gcloud iam service-accounts create wayang-bq --display-name="Wayang BigQuery IT" -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.jobUser" -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID --member="serviceAccount:wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.dataEditor" -gcloud iam service-accounts keys create "$HOME\wayang-bq-key.json" --iam-account="wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com" -``` - -The service account needs `jobUser` to run queries and `dataEditor` to create, -read, and drop the test dataset tables. - -### 2. Choose the test dataset - -The test creates the dataset if it does not exist. By default it uses -`wayang_it`; override it with `-Dbigquery.dataset=DATASET_ID` or -`BIGQUERY_DATASET` if the project needs a different dataset name. - -### 3. Run the operator tests - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am \ - -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false \ - -Dbigquery.project=YOUR_PROJECT_ID \ - -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com \ - -Dbigquery.keyPath="$HOME/wayang-bq-key.json" \ - -Dbigquery.location=US \ - -Dbigquery.dataset=wayang_it \ - -Drat.skip=true -Dlicense.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Dbigquery.project=YOUR_PROJECT_ID -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com -Dbigquery.keyPath=C:\path\to\wayang-bq-key.json -Dbigquery.location=US -Dbigquery.dataset=wayang_it -Drat.skip=true -Dlicense.skip=true test -``` - -System properties take precedence over the equivalent environment variables: - -| System property | Environment variable | Default | -|-----------------|----------------------|---------| -| `bigquery.project` | `BIGQUERY_PROJECT` | `your-project` | -| `bigquery.saEmail` | `BIGQUERY_SA_EMAIL` | `wayang-bq@.iam.gserviceaccount.com` | -| `bigquery.keyPath` | `BIGQUERY_KEY_PATH` | `$HOME/wayang-bq-key.json` | -| `bigquery.location` | `BIGQUERY_LOCATION` | `US` | -| `bigquery.dataset` | `BIGQUERY_DATASET` | `wayang_it` | - -Successful real-BigQuery validation must show: - -```text -Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 -``` - -If the browser uses a local proxy, pass the same proxy to both CLI tools and -the Maven test JVM. For example, with a proxy at `127.0.0.1:7890`, set -`HTTP_PROXY`/`HTTPS_PROXY` and use `JAVA_TOOL_OPTIONS` with -`-Dhttp.proxyHost`, `-Dhttp.proxyPort`, `-Dhttps.proxyHost`, and -`-Dhttps.proxyPort`. - -On PowerShell: - -```powershell -$env:HTTP_PROXY="http://127.0.0.1:7890" -$env:HTTPS_PROXY="http://127.0.0.1:7890" -$env:JAVA_TOOL_OPTIONS="-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=7890 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=7890" -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Dbigquery.project=YOUR_PROJECT_ID -Dbigquery.saEmail=wayang-bq@YOUR_PROJECT_ID.iam.gserviceaccount.com -Dbigquery.keyPath=C:\path\to\wayang-bq-key.json -Dbigquery.location=US -Dbigquery.dataset=wayang_it -Drat.skip=true -Dlicense.skip=true test -Remove-Item Env:HTTP_PROXY, Env:HTTPS_PROXY, Env:JAVA_TOOL_OPTIONS -``` - -If credentials or the project configuration are missing, Maven can still print -`BUILD SUCCESS` with `Skipped: 13`. That does not count as successful -real-BigQuery validation. - -## 4. Optional: Re-run Cost Profiling - -Follow the shared cost-profiling guide in -[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide -only covers the BigQuery emulator and real BigQuery validation setup. BigQuery -cost profiling uses `BigQueryCostPilotIT`, so it needs the same real-BigQuery -credentials as the optional operator tests above. - -BigQuery-specific profiling values: - -| Item | Value | -|------|-------| -| Maven module | `wayang-platforms/wayang-bigquery` | -| Profiling test | `BigQueryCostPilotIT` | -| Property prefix | `bigquery.profile.*` | -| Profiling dataset property | `bigquery.profile.dataset` | -| Default output directory | `target/cost-profiling/bigquery` | -| Learned parameters file | `wayang-platforms/wayang-bigquery/src/main/resources/wayang-bigquery-defaults.properties` | - -## Test Coverage - -### Local emulator tests - -| Test | What it checks | -|------|----------------| -| `testDatasetVisible` | `sales` dataset exists | -| `testFullScan` | Full table scan, 10 rows | -| `testFilterByRegion` | `WHERE region = 'APAC'` | -| `testFilterByAmount` | `WHERE amount > 1000` | -| `testAggregation` | `GROUP BY region` + `SUM(amount)` | -| `testProjection` | `SELECT region, product LIMIT 5` | -| `testCount` | `SELECT count(*)`, used by Wayang for cardinality estimation | - -### Real BigQuery operator tests - -| Test | What it checks | -|------|----------------| -| `tableSource` | Full table scan through Wayang into a BigQuery sink table | -| `filter` | String filter pushdown | -| `projection` | Multi-column projection pushdown | -| `join` | Full Wayang join plan with normalization before the sink table | -| `globalReduce` | Global `SUM(amount)` | -| `reduceBy` | `SUM(amount) GROUP BY region` | -| `sort` | BigQuery sort operator SQL-clause contract | -| `tableSink` | `CREATE TABLE AS SELECT` and cleanup | -| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> writeTable` | -| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> writeTable` | -| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> writeTable` | -| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` | -| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> writeTable` | - -The combination tests use `.withTargetPlatform(BigQuery.platform())` so the -small 10-row fixture still exercises BigQuery SQL pushdown. The join test creates -and cleans up a temporary distinct-region lookup table. - -## Emulator Environment Variable - -```bash -BIGQUERY_HOST=http://localhost:9050 ./mvnw -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test -``` - -On PowerShell: - -```powershell -$env:BIGQUERY_HOST="http://localhost:9050" -.\mvnw.cmd --% -f bigquery-setup/pom.xml -Dtest=BigQueryEmulatorIT test -Remove-Item Env:BIGQUERY_HOST -``` - -## Notes - -- Emulator tests use the `google-cloud-bigquery` client library (REST-based, no - JDBC). -- The emulator client connects with `NoCredentials`; no GCP account is needed. -- The BigQuery JDBC driver (`google-cloud-bigquery-jdbc`) requires OAuth even - against the emulator, so `BigQueryOperatorsIT` runs only against real - BigQuery. -- Emulator tests validate SQL compatibility, but only `BigQueryOperatorsIT` - validates end-to-end Wayang-to-BigQuery JDBC execution. diff --git a/bigquery-setup/data.yaml b/bigquery-setup/data.yaml deleted file mode 100644 index c1a283285..000000000 --- a/bigquery-setup/data.yaml +++ /dev/null @@ -1,72 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -projects: -- id: test-project - datasets: - - id: sales - tables: - - id: orders - columns: - - name: order_id - type: INTEGER - - name: region - type: STRING - - name: product - type: STRING - - name: amount - type: FLOAT - data: - - order_id: 1 - region: APAC - product: Widget A - amount: 1500.0 - - order_id: 2 - region: EMEA - product: Widget B - amount: 800.5 - - order_id: 3 - region: AMER - product: Widget A - amount: 2200.0 - - order_id: 4 - region: APAC - product: Widget C - amount: 350.75 - - order_id: 5 - region: EMEA - product: Widget A - amount: 1100.0 - - order_id: 6 - region: AMER - product: Widget B - amount: 950.25 - - order_id: 7 - region: APAC - product: Widget B - amount: 1750.0 - - order_id: 8 - region: EMEA - product: Widget C - amount: 420.0 - - order_id: 9 - region: AMER - product: Widget C - amount: 680.5 - - order_id: 10 - region: APAC - product: Widget A - amount: 3000.0 diff --git a/bigquery-setup/demo.sh b/bigquery-setup/demo.sh deleted file mode 100644 index 270ab869a..000000000 --- a/bigquery-setup/demo.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -WAYANG_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -exec "$WAYANG_ROOT/demo-bigquery.sh" "$@" diff --git a/bigquery-setup/docker-compose.yml b/bigquery-setup/docker-compose.yml deleted file mode 100644 index 4f3dd69e0..000000000 --- a/bigquery-setup/docker-compose.yml +++ /dev/null @@ -1,43 +0,0 @@ ---- -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Stack: BigQuery Emulator (goccy/bigquery-emulator) -# -# Single container — no metastore, no object storage needed. -# Data is seeded from data.yaml on startup and lives in memory. -# -# Ports: -# HTTP (REST API): http://localhost:9050 -# gRPC (Storage API): localhost:9060 - -services: - - bigquery: - image: ghcr.io/goccy/bigquery-emulator:0.6.6 - platform: linux/amd64 - container_name: bigquery-emulator - ports: - - "9050:9050" - - "9060:9060" - volumes: - - ./data.yaml:/data.yaml - command: --project=test-project --data-from-yaml=/data.yaml - healthcheck: - test: ["CMD-SHELL", "bash -c ': >/dev/tcp/localhost/9050'"] - interval: 10s - timeout: 5s - retries: 5 diff --git a/bigquery-setup/pom.xml b/bigquery-setup/pom.xml deleted file mode 100644 index 9ffd14533..000000000 --- a/bigquery-setup/pom.xml +++ /dev/null @@ -1,79 +0,0 @@ - - - - 4.0.0 - - org.apache.wayang - bigquery-setup - 1.0-SNAPSHOT - jar - - BigQuery Local Setup — Integration Tests - - Standalone integration tests for a local BigQuery emulator. - Independent of the Wayang codebase. - - - - 11 - 11 - UTF-8 - 5.10.2 - 2.49.0 - - - - - - com.google.cloud - google-cloud-bigquery - ${bigquery.version} - test - - - - - org.junit.jupiter - junit-jupiter - ${junit.version} - test - - - - - org.slf4j - slf4j-simple - 2.0.12 - test - - - - - - - org.apache.maven.plugins - maven-surefire-plugin - 3.2.5 - - - - diff --git a/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java b/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java deleted file mode 100644 index 6c03f7843..000000000 --- a/bigquery-setup/src/test/java/org/apache/wayang/bigquery/BigQueryEmulatorIT.java +++ /dev/null @@ -1,202 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.wayang.bigquery; - -import com.google.auth.oauth2.GoogleCredentials; -import com.google.cloud.NoCredentials; -import com.google.cloud.bigquery.*; -import org.junit.jupiter.api.*; - -import java.util.ArrayList; -import java.util.List; - -import static org.junit.jupiter.api.Assertions.*; - -/** - * Integration tests for the local BigQuery emulator. - * - * Prerequisites: run `docker-compose up -d` first. - * - * Run tests: - * mvn test -Pintegration - */ -@TestMethodOrder(MethodOrderer.OrderAnnotation.class) -class BigQueryEmulatorIT { - - private static final String EMULATOR_HOST = System.getenv().getOrDefault("BIGQUERY_HOST", "http://localhost:9050"); - private static final String PROJECT_ID = "test-project"; - private static final String DATASET = "sales"; - - private static BigQuery bigquery; - private static boolean emulatorAvailable = false; - - @BeforeAll - static void setupClient() { - try { - bigquery = BigQueryOptions.newBuilder() - .setHost(EMULATOR_HOST) - .setLocation("US") - .setProjectId(PROJECT_ID) - .setCredentials(NoCredentials.getInstance()) - .build() - .getService(); - - // Quick connectivity check - bigquery.getDataset(DatasetId.of(PROJECT_ID, DATASET)); - emulatorAvailable = true; - System.out.printf("Connected to BigQuery emulator at %s%n", EMULATOR_HOST); - } catch (Exception e) { - System.err.println("BigQuery emulator not available: " + e.getMessage()); - } - } - - private List> runQuery(String sql) throws InterruptedException { - QueryJobConfiguration config = QueryJobConfiguration.newBuilder(sql) - .setUseLegacySql(false) - .build(); - TableResult result = bigquery.query(config); - List> rows = new ArrayList<>(); - for (FieldValueList row : result.iterateAll()) { - List r = new ArrayList<>(); - for (FieldValue val : row) { - r.add(val.isNull() ? null : val.getValue()); - } - rows.add(r); - } - return rows; - } - - // ── Test 1: Dataset visible ────────────────────────────────────────── - - @Test - @Order(1) - @DisplayName("BigQuery emulator: dataset 'sales' is visible") - void testDatasetVisible() { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - Dataset ds = bigquery.getDataset(DatasetId.of(PROJECT_ID, DATASET)); - assertNotNull(ds, "Dataset 'sales' should exist"); - System.out.println("[PASS] Dataset 'sales' is visible"); - } - - // ── Test 2: Full table scan ────────────────────────────────────────── - - @Test - @Order(2) - @DisplayName("BigQuery emulator: full table scan on orders") - void testFullScan() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT * FROM `test-project.sales.orders` ORDER BY order_id" - ); - assertEquals(10, rows.size(), "Expected 10 rows"); - System.out.println("[PASS] Full scan: " + rows.size() + " rows"); - rows.forEach(r -> System.out.println(" " + r)); - } - - // ── Test 3: Filter by region ───────────────────────────────────────── - - @Test - @Order(3) - @DisplayName("BigQuery emulator: filter by region = APAC") - void testFilterByRegion() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT order_id, region, amount FROM `test-project.sales.orders` WHERE region = 'APAC' ORDER BY order_id" - ); - assertFalse(rows.isEmpty(), "Should have APAC rows"); - rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows should be APAC")); - System.out.printf("[PASS] Filter: %d APAC rows%n", rows.size()); - } - - // ── Test 4: Filter by amount ───────────────────────────────────────── - - @Test - @Order(4) - @DisplayName("BigQuery emulator: filter by amount > 1000") - void testFilterByAmount() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT order_id, amount FROM `test-project.sales.orders` WHERE amount > 1000 ORDER BY amount DESC" - ); - assertFalse(rows.isEmpty()); - rows.forEach(r -> assertTrue( - Double.parseDouble(r.get(1).toString()) > 1000.0, - "All amounts should be > 1000" - )); - System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size()); - } - - // ── Test 5: Aggregation ────────────────────────────────────────────── - - @Test - @Order(5) - @DisplayName("BigQuery emulator: aggregate by region") - void testAggregation() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT region, COUNT(*) AS cnt, SUM(amount) AS total " + - "FROM `test-project.sales.orders` GROUP BY region ORDER BY total DESC" - ); - assertFalse(rows.isEmpty()); - System.out.println("[PASS] Aggregation by region:"); - rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%s%n", - r.get(0), r.get(1), r.get(2))); - } - - // ── Test 6: Projection ─────────────────────────────────────────────── - - @Test - @Order(6) - @DisplayName("BigQuery emulator: projection (region, product)") - void testProjection() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT region, product FROM `test-project.sales.orders` LIMIT 5" - ); - assertEquals(5, rows.size()); - rows.forEach(r -> { - assertNotNull(r.get(0), "region should not be null"); - assertNotNull(r.get(1), "product should not be null"); - }); - System.out.println("[PASS] Projection (region, product): 5 rows"); - } - - // ── Test 7: COUNT(*) ───────────────────────────────────────────────── - - @Test - @Order(7) - @DisplayName("BigQuery emulator: SELECT count(*)") - void testCount() throws Exception { - Assumptions.assumeTrue(emulatorAvailable, "Emulator not available"); - - List> rows = runQuery( - "SELECT count(*) FROM `test-project.sales.orders`" - ); - assertEquals(1, rows.size()); - long count = Long.parseLong(rows.get(0).get(0).toString()); - assertEquals(10, count, "Should have 10 rows"); - System.out.println("[PASS] COUNT(*) = " + count); - } -} diff --git a/demo-bigquery.sh b/demo-bigquery.sh deleted file mode 100644 index dcfe42b40..000000000 --- a/demo-bigquery.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -WAYANG_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -LIVE_MODE=false -[[ "${1:-}" == "--live" ]] && LIVE_MODE=true - -BQ_PROJECT="${BQ_PROJECT:-my-project}" -BQ_URL="${BQ_URL:-}" -MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true" - -banner() { - echo - echo "============================================================" - printf " %s\n" "$*" - echo "============================================================" - echo -} - -step() { - echo - echo "-- $*" - echo -} - -pause() { - if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then - echo - read -rp "Press ENTER to continue..." _ || true - echo - fi -} - -run_demo_class() { - local main_class="$1" - shift - cd "$WAYANG_ROOT" - "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-bigquery \ - -Dexec.mainClass="$main_class" \ - "$@" \ - ${MAVEN_FLAGS} -q 2>/dev/null || true -} - -banner "ACT 1: BigQuery cost model" -step "Read cost settings from wayang-bigquery-defaults.properties" -run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \ - "-Dbigquery.mode=cost" \ - "-Dbigquery.project=${BQ_PROJECT}" - -pause - -banner "ACT 2: BigQuery filter operator" -if [[ "$LIVE_MODE" == true && -n "$BQ_URL" ]]; then - run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \ - "-Dbigquery.mode=filter" \ - "-Dbigquery.url=${BQ_URL}" \ - "-Dbigquery.project=${BQ_PROJECT}" -else - run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \ - "-Dbigquery.mode=filter" \ - "-Dbigquery.project=${BQ_PROJECT}" -fi - -pause - -banner "ACT 3: BigQuery projection operator" -if [[ "$LIVE_MODE" == true && -n "$BQ_URL" ]]; then - run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \ - "-Dbigquery.mode=projection" \ - "-Dbigquery.url=${BQ_URL}" \ - "-Dbigquery.project=${BQ_PROJECT}" -else - run_demo_class "org.apache.wayang.bigquery.BigQueryDemo" \ - "-Dbigquery.mode=projection" \ - "-Dbigquery.project=${BQ_PROJECT}" -fi - -banner "Demo complete" diff --git a/demo-trino.sh b/demo-trino.sh deleted file mode 100644 index 45b1fad7a..000000000 --- a/demo-trino.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -WAYANG_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -exec "$WAYANG_ROOT/trino-setup/demo.sh" "$@" diff --git a/env_template_osx.sh b/env_template_osx.sh deleted file mode 100755 index 6ec3cc021..000000000 --- a/env_template_osx.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -export BOOTSTRAP_SERVER= ... -export CLUSTER_API_KEY= ... -export CLUSTER_API_SECRET= ... -export SR_ENDPOINT= ... -export SR_API_KEY= ... -export SR_API_SECRET= ... -export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... " -export SCHEMA_REGISTRY_URL="https://.... " - -export SPARK_HOME= ... -export HADOOP_HOME= ... -export PATH=$PATH:$HADOOP_HOME/bin -export WAYANG_VERSION= ... -export WAYANG_HOME= ... -export WAYANG_APP_HOME= ... - -echo "Hadoop home : $HADOOP_HOME" -echo "Spark home : $SPARK_HOME" -echo "Wayang home : $WAYANG_HOME" -echo "Wayang app : $WAYANG_APP_HOME" -echo "Wayang version : $WAYANG_VERSION" - - diff --git a/guides/cost-profiling.md b/guides/cost-profiling.md index 8dc5295bf..ce0d8a97c 100644 --- a/guides/cost-profiling.md +++ b/guides/cost-profiling.md @@ -257,7 +257,7 @@ model should predict: - If Wayang should predict user-visible runtime, fit wall-clock elapsed time. - If the platform reports CPU time and the model uses CPU load, make sure the conversion to Wayang cost is consistent with the resource model. -- Parameters learned on a local Docker setup should be treated as local +- Parameters learned on one deployment should be treated as environment-specific reference values, not universal defaults for every deployment. ## 6. Running a Profiling Experiment @@ -265,11 +265,11 @@ model should predict: The exact command depends on the platform module, test class, and property prefix. -| Platform | Setup guide | Maven module | Test class | Property prefix | Default output directory | +| Platform | Platform guide | Maven module | Test class | Property prefix | Default output directory | |----------|-------------|--------------|------------|-----------------|--------------------------| -| Trino | `trino-setup/README.md` | `wayang-platforms/wayang-trino` | `TrinoCostPilotIT` | `trino.profile.*` | `target/cost-profiling/trino` | -| Presto | `presto-setup/README.md` | `wayang-platforms/wayang-presto` | `PrestoCostPilotIT` | `presto.profile.*` | `target/cost-profiling/presto` | -| BigQuery | `bigquery-setup/README.md` | `wayang-platforms/wayang-bigquery` | `BigQueryCostPilotIT` | `bigquery.profile.*` | `target/cost-profiling/bigquery` | +| Trino | `wayang-platforms/wayang-trino/README.md` | `wayang-platforms/wayang-trino` | `TrinoCostPilotIT` | `trino.profile.*` | `target/cost-profiling/trino` | +| Presto | `wayang-platforms/wayang-presto/README.md` | `wayang-platforms/wayang-presto` | `PrestoCostPilotIT` | `presto.profile.*` | `target/cost-profiling/presto` | +| BigQuery | `wayang-platforms/wayang-bigquery/README.md` | `wayang-platforms/wayang-bigquery` | `BigQueryCostPilotIT` | `bigquery.profile.*` | `target/cost-profiling/bigquery` | The commands below use PowerShell. On macOS/Linux, use `./mvnw` instead of `.\mvnw.cmd` and replace PowerShell backticks with Bash line-continuation diff --git a/improvement.md b/improvement.md index 5485fd929..46086e695 100644 --- a/improvement.md +++ b/improvement.md @@ -106,13 +106,10 @@ The high-level tests also rely on the `withSqlUdf` / `withSqlUdfs` additions to sort builders can carry SQL implementations. ```bash -docker compose -f presto-setup/docker-compose.yml up -d - +PRESTO_HOST=presto.example.com PRESTO_PORT=8080 PRESTO_USER=wayang \ JAVA_HOME= mvn test -pl wayang-platforms/wayang-presto -am \ -Dtest=PrestoOperatorsIT -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \ -Drat.skip=true -Dlicense.skip=true -Pskip-prerequisite-check - -docker compose -f presto-setup/docker-compose.yml down ``` Expected: `Tests run: 13, Failures: 0, Errors: 0, Skipped: 0`. The suite scales diff --git a/platforms-setup-guides/trino-setup/README.md b/platforms-setup-guides/trino-setup/README.md deleted file mode 100644 index 8be4f7f3c..000000000 --- a/platforms-setup-guides/trino-setup/README.md +++ /dev/null @@ -1,282 +0,0 @@ -# Trino Local Setup - -Local Trino environment backed by an **Iceberg** data lake, completely containerised. - -The current validation has three parts: - -1. Build the Wayang Trino platform and run the shared JDBC SQL-generation tests. -2. Run the Wayang Trino operator tests against the live local stack. -3. Run standalone JDBC integration tests against the local Trino, Iceberg, and MinIO stack. - -Run the commands below from the repository root. Java 17 and Docker with -Docker Compose are required; Maven is provided by the repository wrapper. - -The pure Trino platform branch is named `wayang-trino`: - -```bash -git checkout wayang-trino -``` - -## Command Conventions - -Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on -Windows PowerShell from the repository root. Docker Compose commands are the -same on both platforms. - -## Stack - -| Component | Image | Port | Role | -|-----------|-------|------|------| -| **Trino** | `trinodb/trino:435` | 8080 | SQL query engine | -| **Hive Metastore** | `naushadh/hive-metastore:latest` | 9083 | Iceberg table catalog (Thrift) | -| **PostgreSQL** | `postgres:15-alpine` | 5432 | HMS metadata backing store | -| **MinIO** | `minio/minio:latest` | 9000 / 9001 | S3-compatible object storage | - -HMS is the battle-tested Iceberg catalog for Trino. Parquet data files are written by Trino directly to MinIO; HMS only stores schema/table metadata. - -## Directory Layout - -``` -platforms-setup-guides/ -`-- trino-setup/ - |-- docker-compose.yml # Full stack definition - |-- demo.sh # End-to-end Trino + Wayang walkthrough - |-- trino/ - | |-- config.properties # Trino node config - | `-- catalog/ - | |-- iceberg.properties # Iceberg via HMS + MinIO - | `-- tpch.properties # Built-in TPC-H (no storage needed) - |-- scripts/ - | |-- init.sql # Creates iceberg.sales.orders + sample rows - | `-- run-init.sh # Helper: waits for Trino then runs init.sql - |-- pom.xml # Standalone Maven project (Java 17) - `-- src/test/java/.../ - `-- TrinoIntegrationTest.java # JUnit 5 integration tests -``` - -## 1. Test the Wayang Trino Platform - -Build the Trino platform and its required modules: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test -``` - -Then run the shared JDBC SQL-generation tests: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -Expected result: - -```text -Wayang Platform Trino ... SUCCESS -Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -``` - -## 2. Test Against the Local Trino Stack - -### 1. Start the stack - -```bash -docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml up -d -``` - -Wait ~30 seconds for all services to become healthy. Check with: - -```bash -docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml ps -# or watch the Trino UI at http://localhost:8080 -``` - -### 2. Run the Wayang Trino operator tests - -`TrinoOperatorsIT` exercises the Wayang Trino implementation against the live -Trino stack. It checks `TableSource`, `Filter`, `Projection`, `Join`, -`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the -expected SQL reached Trino. The standalone join test now runs a full Wayang -plan and normalizes both possible join result shapes before collecting records: -logical joins can produce `Tuple2`, while pushed-down JDBC joins -can return a flat `Record`. - -The suite is self-contained: it creates `iceberg.wayang_it`, scales its test -data to 120,000 rows so the optimizer selects SQL pushdown, and drops its test -tables afterward. It does not require `scripts/init.sql`. The suite also -contains five JavaPlanBuilder `readTable` combination tests that cover filter, -projection, global reduce, reduce-by plus sort, table sink, and join through -the public API. - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \ - -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -Expected result: - -```text -Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 -``` - -Verified on June 18, 2026 against the local Docker stack with the full-plan -join test and all five JavaPlanBuilder combination tests enabled. - -If Trino is unreachable, these tests are skipped instead of failed. A result -with skipped tests does not confirm that the operators work. - -### 3. Load sample Iceberg data - -```bash -bash platforms-setup-guides/trino-setup/scripts/run-init.sh -``` - -On PowerShell: - -```powershell -Get-Content -Raw platforms-setup-guides/trino-setup/scripts/init.sql | docker exec -i trino trino --server http://localhost:8080 --user admin -``` - -This creates the schema `iceberg.sales` and inserts 20 sample orders into -`iceberg.sales.orders` (Parquet files on MinIO). - -### 4. Run the walkthrough demo - -The optional demo script starts the local Trino/Iceberg stack, seeds -`iceberg.sales.orders`, shows direct Trino CLI queries, and then runs -`org.apache.wayang.trino.TrinoDemo` to demonstrate Wayang filter and projection -pushdown through the Trino platform. - -```bash -bash platforms-setup-guides/trino-setup/demo.sh -``` - -Set `WAYANG_DEMO_AUTO=true` to skip the interactive pauses. - -### 5. Run the standalone stack integration tests - -```bash -./mvnw -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -Tests are skipped by default (no `-Pintegration`) to avoid requiring Docker in CI. -These tests validate the stack and direct JDBC queries independently of the -Wayang operator implementation. - -Expected result: - -```text -Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 -BUILD SUCCESS -``` - -### 6. Manual exploration - -Open the **Trino UI**: http://localhost:8080 - -Or connect via the Trino CLI inside the container: - -```bash -docker exec -it trino trino --catalog iceberg --schema sales -``` - -```sql --- TPC-H built-in data (no init.sql needed) -SELECT * FROM tpch.tiny.orders LIMIT 5; - --- Iceberg table -SELECT region, SUM(amount) FROM iceberg.sales.orders GROUP BY region; - --- Iceberg file metadata -SELECT * FROM iceberg.sales."orders$files"; - --- Iceberg history -SELECT * FROM iceberg.sales."orders$history"; -``` - -**MinIO console**: http://localhost:9001 (login: `minioadmin` / `minioadmin`) -Look for Parquet files under `warehouse/sales/orders/`. - -### 7. Tear down - -```bash -docker compose -f platforms-setup-guides/trino-setup/docker-compose.yml down -v -``` - -The `-v` option removes volumes and clears the local MinIO and PostgreSQL data. - -## Test Coverage - -### Wayang operator integration tests - -| Test | What it checks | -|------|----------------| -| `tableSource` | Full table scan through `TrinoTableSource` | -| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown | -| `projection` | Column projection pushed into the Trino query | -| `join` | Full Wayang join plan with normalization before the collecting sink | -| `globalReduce` | Global aggregation such as `SUM` | -| `reduceBy` | Grouped aggregation and SQL `GROUP BY` | -| `sort` | Wayang sort and SQL `ORDER BY` | -| `tableSink` | Filtered result written with `CREATE TABLE AS` | -| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> collect` | -| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> collect` | -| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> collect` | -| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` | -| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> collect` | - -### Standalone stack integration tests - -| Test | What it checks | -|------|----------------| -| `testConnectivity` | `SELECT 1`, JDBC connection works | -| `testTpchConnector` | TPC-H built-in connector, no storage needed | -| `testTpchTopOrders` | ORDER BY + LIMIT on TPC-H | -| `testIcebergSchemaVisible` | Schema created by `init.sql` is visible | -| `testIcebergSelectAll` | Full table scan, 20 rows | -| `testIcebergFilterByRegion` | WHERE pushdown on string column | -| `testIcebergAggregate` | GROUP BY + SUM aggregation | -| `testIcebergFilterByAmount` | WHERE pushdown on double column | -| `testIcebergProjection` | SELECT subset of columns | -| `testIcebergFilesMetadata` | `$files` system table, confirms Parquet on MinIO | - -## Environment Variables - -Override defaults if running Trino on a different host/port: - -```bash -TRINO_HOST=my-trino-host TRINO_PORT=8080 ./mvnw -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -On PowerShell: - -```powershell -$env:TRINO_HOST="my-trino-host" -$env:TRINO_PORT="8080" -.\mvnw.cmd --% -f platforms-setup-guides/trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -Remove-Item Env:TRINO_HOST, Env:TRINO_PORT -``` diff --git a/platforms-setup-guides/trino-setup/demo.sh b/platforms-setup-guides/trino-setup/demo.sh deleted file mode 100644 index 50b2f0018..000000000 --- a/platforms-setup-guides/trino-setup/demo.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -WAYANG_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -TRINO_SETUP="$SCRIPT_DIR" -TRINO_CONTAINER="trino" -MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true" - -banner() { - echo - echo "============================================================" - printf " %s\n" "$*" - echo "============================================================" - echo -} - -step() { - echo - echo "-- $*" - echo -} - -pause() { - if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then - echo - read -rp "Press ENTER to continue..." _ || true - echo - fi -} - -run_wayang_demo() { - "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-trino \ - -Dexec.mainClass="org.apache.wayang.trino.TrinoDemo" \ - ${MAVEN_FLAGS} -} - -banner "ACT 1: Start Trino + Iceberg via Docker" - -step "1a. Starting the stack" -cd "$TRINO_SETUP" -docker compose up -d - -step "1b. Containers running" -docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" \ - | grep -E "NAMES|trino|minio|metastore|postgres" - -step "1c. Waiting for Trino to be ready" -MAX_WAIT=90 -ELAPSED=0 -until docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT 1" --output-format ALIGNED >/dev/null 2>&1; do - if [[ "$ELAPSED" -ge "$MAX_WAIT" ]]; then - echo "Timed out waiting for Trino after ${MAX_WAIT}s" - exit 1 - fi - printf ". waiting (%ds elapsed)\r" "$ELAPSED" - sleep 3 - ELAPSED=$((ELAPSED + 3)) -done -echo "Trino is ready at http://localhost:8080" - -step "1d. Initialising Iceberg tables" -docker exec -i "$TRINO_CONTAINER" trino < "$TRINO_SETUP/scripts/init.sql" 2>&1 \ - | grep -v "^WARNING\|jline\|org.jline" || true -echo "iceberg.sales.orders seeded" - -step "1e. Table schema" -docker exec "$TRINO_CONTAINER" \ - trino --execute "DESCRIBE iceberg.sales.orders" \ - --output-format ALIGNED 2>/dev/null - -pause - -banner "ACT 2: Query Iceberg directly via Trino CLI" - -step "2a. Full table scan" -echo "SQL: SELECT * FROM iceberg.sales.orders" -docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT * FROM iceberg.sales.orders ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -step "2b. Filter: region = 'AMER'" -echo "SQL: SELECT * FROM iceberg.sales.orders WHERE region = 'AMER'" -docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT * FROM iceberg.sales.orders WHERE region = 'AMER' ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -step "2c. Projection with filter" -echo "SQL: SELECT region, product, amount FROM iceberg.sales.orders WHERE region = 'AMER'" -docker exec "$TRINO_CONTAINER" \ - trino --execute \ - "SELECT region, product, amount - FROM iceberg.sales.orders - WHERE region = 'AMER' - ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -pause - -banner "ACT 3: Wayang API filter + projection pushdown" -cd "$WAYANG_ROOT" -run_wayang_demo - -banner "Demo complete" -echo "Trino UI: http://localhost:8080" -echo "MinIO UI: http://localhost:9001 (minioadmin / minioadmin)" -echo -echo "To stop the stack:" -echo " cd platforms-setup-guides/trino-setup && docker compose down" diff --git a/platforms-setup-guides/trino-setup/docker-compose.yml b/platforms-setup-guides/trino-setup/docker-compose.yml deleted file mode 100644 index cf582184e..000000000 --- a/platforms-setup-guides/trino-setup/docker-compose.yml +++ /dev/null @@ -1,142 +0,0 @@ ---- -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Stack: Trino + Hive Metastore + MinIO (S3 storage) -# -# This is the battle-tested Trino + Iceberg local setup. -# Hive Metastore (HMS) stores Iceberg table metadata over Thrift on port 9083. -# MinIO provides S3-compatible object storage for Parquet data files. -# Trino's Iceberg connector uses HMS as catalog and writes Parquet to MinIO. -# -# Ports: -# Trino: http://localhost:8080 (UI + JDBC) -# MinIO S3: http://localhost:9000 -# MinIO UI: http://localhost:9001 (minioadmin / minioadmin) -# HMS: localhost:9083 (Thrift, internal) -# Postgres: localhost:5432 (HMS backing store) - -services: - - # PostgreSQL (Hive Metastore backing database) - postgres: - image: postgres:15-alpine - container_name: trino-postgres - environment: - POSTGRES_DB: metastore - POSTGRES_USER: hive - POSTGRES_PASSWORD: hive - ports: - - "5432:5432" - volumes: - - postgres-data:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U hive -d metastore"] - interval: 10s - timeout: 5s - retries: 5 - - # MinIO (S3-compatible object storage) - minio: - image: minio/minio:latest - container_name: trino-minio - environment: - MINIO_ROOT_USER: minioadmin - MINIO_ROOT_PASSWORD: minioadmin - ports: - - "9000:9000" - - "9001:9001" - command: server /data --console-address ":9001" - volumes: - - minio-data:/data - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 10s - timeout: 5s - retries: 5 - - # Create the warehouse bucket before HMS starts - minio-init: - image: minio/mc:latest - container_name: trino-minio-init - depends_on: - minio: - condition: service_healthy - entrypoint: > - /bin/sh -c " - mc alias set local http://minio:9000 minioadmin minioadmin; - mc mb local/warehouse --ignore-existing; - echo 'bucket warehouse ready'; - exit 0; - " - - # Hive Metastore - # naushadh/hive-metastore is a minimal, pre-configured HMS image - # that supports S3-compatible storage via env vars. - metastore: - image: naushadh/hive-metastore:latest - container_name: trino-metastore - depends_on: - postgres: - condition: service_healthy - minio: - condition: service_healthy - minio-init: - condition: service_completed_successfully - ports: - - "9083:9083" - environment: - DATABASE_HOST: postgres - DATABASE_DB: metastore - DATABASE_USER: hive - DATABASE_PASSWORD: hive - # S3 / MinIO - S3_ENDPOINT_URL: http://minio:9000 - S3_BUCKET: warehouse - S3_PREFIX: / - AWS_ACCESS_KEY_ID: minioadmin - AWS_SECRET_ACCESS_KEY: minioadmin - REGION: us-east-1 - # No nc/curl in this image; use bash's /dev/tcp built-in - healthcheck: - test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/localhost/9083 2>/dev/null && exit 0 || exit 1"] - interval: 15s - timeout: 10s - retries: 15 - - # Trino - trino: - image: trinodb/trino:435 - container_name: trino - depends_on: - metastore: - condition: service_healthy - minio: - condition: service_healthy - ports: - - "8080:8080" - volumes: - - ./trino/catalog:/etc/trino/catalog - - ./trino/config.properties:/etc/trino/config.properties - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"] - interval: 15s - timeout: 10s - retries: 10 - -volumes: - postgres-data: - minio-data: diff --git a/platforms-setup-guides/trino-setup/pom.xml b/platforms-setup-guides/trino-setup/pom.xml deleted file mode 100644 index c72cc7751..000000000 --- a/platforms-setup-guides/trino-setup/pom.xml +++ /dev/null @@ -1,95 +0,0 @@ - - - - 4.0.0 - - org.apache.wayang - trino-setup - 1.0-SNAPSHOT - jar - - Trino Local Setup - Integration Tests - - Standalone integration tests for a local Trino stack - (Trino + Nessie Iceberg catalog + MinIO S3 storage). - Independent of the Wayang codebase. - - - - 17 - 17 - UTF-8 - 435 - 5.10.2 - true - - - - - - io.trino - trino-jdbc - ${trino.version} - test - - - - - org.junit.jupiter - junit-jupiter - ${junit.version} - test - - - - - org.slf4j - slf4j-simple - 2.0.12 - test - - - - - - - org.apache.maven.plugins - maven-surefire-plugin - 3.2.5 - - - ${skipIntegrationTests} - - - - - - - - - integration - - false - - - - diff --git a/platforms-setup-guides/trino-setup/scripts/init.sql b/platforms-setup-guides/trino-setup/scripts/init.sql deleted file mode 100644 index 3ce74bcf8..000000000 --- a/platforms-setup-guides/trino-setup/scripts/init.sql +++ /dev/null @@ -1,66 +0,0 @@ --- Licensed to the Apache Software Foundation (ASF) under one --- or more contributor license agreements. See the NOTICE file --- distributed with this work for additional information --- regarding copyright ownership. The ASF licenses this file --- to you under the Apache License, Version 2.0 (the --- "License"); you may not use this file except in compliance --- with the License. You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, software --- distributed under the License is distributed on an "AS IS" BASIS, --- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --- See the License for the specific language governing permissions and --- limitations under the License. - --- Run this after the stack is up to create sample Iceberg tables. --- Usage: ./scripts/run-init.sh --- Or manually: docker exec -it trino trino < /scripts/init.sql - --- ── Schema ──────────────────────────────────────────────────────────────── -CREATE SCHEMA IF NOT EXISTS iceberg.sales; - --- ── Orders table (Iceberg / Parquet on MinIO) ───────────────────────────── -CREATE TABLE IF NOT EXISTS iceberg.sales.orders ( - order_id BIGINT, - region VARCHAR, - product VARCHAR, - amount DOUBLE, - order_date DATE -) -WITH (format = 'PARQUET'); - --- ── Idempotent seed: clear before inserting so re-runs don't duplicate rows ─ -DELETE FROM iceberg.sales.orders; - --- ── Sample data: 20 rows, 4 regions (AMER/APAC/EMEA/LATAM), 5 products ──── --- AMER rows: 3, 6, 9, 12, 16 → 5 rows for filter demo --- Projection demo selects only: region, product, amount -INSERT INTO iceberg.sales.orders VALUES - (1, 'APAC', 'Widget A', 1500.00, DATE '2024-01-15'), - (2, 'EMEA', 'Widget B', 800.50, DATE '2024-01-16'), - (3, 'AMER', 'Widget A', 2200.00, DATE '2024-01-17'), - (4, 'APAC', 'Widget C', 350.75, DATE '2024-01-18'), - (5, 'EMEA', 'Widget A', 1100.00, DATE '2024-01-19'), - (6, 'AMER', 'Widget B', 950.25, DATE '2024-01-20'), - (7, 'APAC', 'Widget B', 1750.00, DATE '2024-01-21'), - (8, 'EMEA', 'Widget C', 420.00, DATE '2024-01-22'), - (9, 'AMER', 'Widget C', 680.50, DATE '2024-01-23'), - (10, 'APAC', 'Widget A', 3000.00, DATE '2024-01-24'), - (11, 'LATAM', 'Widget D', 560.00, DATE '2024-01-25'), - (12, 'AMER', 'Widget D', 1320.75, DATE '2024-01-26'), - (13, 'EMEA', 'Widget D', 990.00, DATE '2024-01-27'), - (14, 'LATAM', 'Widget E', 2100.50, DATE '2024-01-28'), - (15, 'APAC', 'Widget E', 4500.00, DATE '2024-01-29'), - (16, 'AMER', 'Widget E', 3750.00, DATE '2024-01-30'), - (17, 'EMEA', 'Widget E', 1250.00, DATE '2024-01-31'), - (18, 'LATAM', 'Widget A', 870.25, DATE '2024-02-01'), - (19, 'APAC', 'Widget D', 1680.00, DATE '2024-02-02'), - (20, 'LATAM', 'Widget B', 440.50, DATE '2024-02-03'); - --- ── Verify ──────────────────────────────────────────────────────────────── -SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount -FROM iceberg.sales.orders -GROUP BY region -ORDER BY total_amount DESC; diff --git a/platforms-setup-guides/trino-setup/scripts/run-init.sh b/platforms-setup-guides/trino-setup/scripts/run-init.sh deleted file mode 100644 index ebaeb337c..000000000 --- a/platforms-setup-guides/trino-setup/scripts/run-init.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Runs init.sql against the local Trino instance. -# The stack must be fully up before running this. - -set -e - -TRINO_HOST=${TRINO_HOST:-localhost} -TRINO_PORT=${TRINO_PORT:-8080} - -echo "Waiting for Trino to be ready..." -until curl -sf "http://${TRINO_HOST}:${TRINO_PORT}/v1/info" | grep -q '"starting":false'; do - echo " Trino not ready yet, retrying in 5s..." - sleep 5 -done -echo "Trino is ready." - -echo "Running init.sql..." -docker exec -i trino trino \ - --server "http://${TRINO_HOST}:${TRINO_PORT}" \ - --user admin \ - < "$(dirname "$0")/init.sql" - -echo "Done. Sample Iceberg data loaded into iceberg.sales.orders" diff --git a/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java b/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java deleted file mode 100644 index 3c2735b82..000000000 --- a/platforms-setup-guides/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java +++ /dev/null @@ -1,233 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.wayang.trino; - -import org.junit.jupiter.api.*; - -import java.sql.*; -import java.util.ArrayList; -import java.util.List; -import java.util.Properties; - -import static org.junit.jupiter.api.Assertions.*; - -/** - * Integration tests for the local Trino stack. - * - * Prerequisites: run `docker-compose up -d` and `./scripts/run-init.sh` first. - * - * Run tests: - * mvn test -Pintegration - * - * Or skip infrastructure setup and run with a custom host: - * TRINO_HOST=localhost TRINO_PORT=8080 mvn test -Pintegration - */ -@TestMethodOrder(MethodOrderer.OrderAnnotation.class) -class TrinoIntegrationTest { - - private static final String TRINO_HOST = System.getenv().getOrDefault("TRINO_HOST", "localhost"); - private static final int TRINO_PORT = Integer.parseInt(System.getenv().getOrDefault("TRINO_PORT", "8080")); - private static final String JDBC_URL = String.format("jdbc:trino://%s:%d", TRINO_HOST, TRINO_PORT); - - private static Connection connection; - - // ── Lifecycle ───────────────────────────────────────────────────────── - - @BeforeAll - static void openConnection() throws Exception { - Properties props = new Properties(); - props.setProperty("user", "admin"); // Trino requires a non-empty user - connection = DriverManager.getConnection(JDBC_URL, props); - System.out.printf("Connected to Trino at %s%n", JDBC_URL); - } - - @AfterAll - static void closeConnection() throws Exception { - if (connection != null && !connection.isClosed()) { - connection.close(); - } - } - - // ── Helper ──────────────────────────────────────────────────────────── - - private List> query(String sql) throws SQLException { - List> rows = new ArrayList<>(); - try (Statement stmt = connection.createStatement(); - ResultSet rs = stmt.executeQuery(sql)) { - int cols = rs.getMetaData().getColumnCount(); - while (rs.next()) { - List row = new ArrayList<>(); - for (int i = 1; i <= cols; i++) row.add(rs.getObject(i)); - rows.add(row); - } - } - return rows; - } - - // ── Test 1: Basic connectivity ──────────────────────────────────────── - - @Test - @Order(1) - @DisplayName("Trino responds to a simple SELECT 1") - void testConnectivity() throws SQLException { - List> rows = query("SELECT 1"); - assertEquals(1, rows.size()); - assertEquals(1L, ((Number) rows.get(0).get(0)).longValue()); - System.out.println("[PASS] Basic connectivity OK"); - } - - // ── Test 2: TPC-H built-in connector ───────────────────────────────── - - @Test - @Order(2) - @DisplayName("TPC-H tiny catalog: count orders") - void testTpchConnector() throws SQLException { - List> rows = query("SELECT COUNT(*) FROM tpch.tiny.orders"); - long count = ((Number) rows.get(0).get(0)).longValue(); - assertTrue(count > 0, "tpch.tiny.orders should have rows"); - System.out.printf("[PASS] TPC-H tiny.orders has %,d rows%n", count); - } - - @Test - @Order(3) - @DisplayName("TPC-H tiny catalog: top 5 orders by total price") - void testTpchTopOrders() throws SQLException { - List> rows = query(""" - SELECT orderkey, totalprice - FROM tpch.tiny.orders - ORDER BY totalprice DESC - LIMIT 5 - """); - assertEquals(5, rows.size(), "Expected exactly 5 rows"); - System.out.println("[PASS] TPC-H top 5 orders:"); - rows.forEach(r -> System.out.printf(" orderkey=%s totalprice=%s%n", r.get(0), r.get(1))); - } - - // ── Test 4: Iceberg — schema exists ────────────────────────────────── - - @Test - @Order(4) - @DisplayName("Iceberg catalog: schema 'sales' is visible") - void testIcebergSchemaVisible() throws SQLException { - List> rows = query("SHOW SCHEMAS IN iceberg LIKE 'sales'"); - assertFalse(rows.isEmpty(), "Schema 'sales' should exist in iceberg catalog. " + - "Did you run scripts/run-init.sh?"); - System.out.println("[PASS] Iceberg schema 'sales' is visible"); - } - - // ── Test 5: Iceberg — full table scan ──────────────────────────────── - - @Test - @Order(5) - @DisplayName("Iceberg table: select all orders") - void testIcebergSelectAll() throws SQLException { - List> rows = query("SELECT * FROM iceberg.sales.orders ORDER BY order_id"); - assertEquals(20, rows.size(), "Expected 20 rows inserted by init.sql"); - System.out.println("[PASS] Iceberg full scan: 20 rows"); - rows.forEach(r -> System.out.printf(" %s%n", r)); - } - - // ── Test 6: Iceberg — pushdown filter ──────────────────────────────── - - @Test - @Order(6) - @DisplayName("Iceberg table: filter by region = APAC") - void testIcebergFilterByRegion() throws SQLException { - List> rows = query(""" - SELECT order_id, region, amount - FROM iceberg.sales.orders - WHERE region = 'APAC' - ORDER BY order_id - """); - assertFalse(rows.isEmpty(), "Should have APAC orders"); - rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows must be APAC")); - System.out.printf("[PASS] Filter pushdown: %d APAC rows%n", rows.size()); - } - - // ── Test 7: Iceberg — aggregation ──────────────────────────────────── - - @Test - @Order(7) - @DisplayName("Iceberg table: aggregate total_amount by region") - void testIcebergAggregate() throws SQLException { - List> rows = query(""" - SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount - FROM iceberg.sales.orders - GROUP BY region - ORDER BY total_amount DESC - """); - assertFalse(rows.isEmpty(), "Aggregation should return rows"); - System.out.println("[PASS] Aggregation by region:"); - rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%.2f%n", - r.get(0), r.get(1), ((Number) r.get(2)).doubleValue())); - } - - // ── Test 8: Iceberg — amount threshold filter ───────────────────────── - - @Test - @Order(8) - @DisplayName("Iceberg table: filter orders with amount > 1000") - void testIcebergFilterByAmount() throws SQLException { - List> rows = query(""" - SELECT order_id, amount - FROM iceberg.sales.orders - WHERE amount > 1000.0 - ORDER BY amount DESC - """); - rows.forEach(r -> assertTrue( - ((Number) r.get(1)).doubleValue() > 1000.0, - "All rows must have amount > 1000" - )); - System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size()); - } - - // ── Test 9: Iceberg — projection (select subset of columns) ─────────── - - @Test - @Order(9) - @DisplayName("Iceberg table: project only region and product columns") - void testIcebergProjection() throws SQLException { - List> rows = query(""" - SELECT region, product - FROM iceberg.sales.orders - LIMIT 5 - """); - assertEquals(5, rows.size()); - rows.forEach(r -> { - assertNotNull(r.get(0), "region should not be null"); - assertNotNull(r.get(1), "product should not be null"); - }); - System.out.println("[PASS] Projection (region, product): 5 rows returned"); - } - - // ── Test 10: Iceberg metadata — table files ────────────────────────── - - @Test - @Order(10) - @DisplayName("Iceberg metadata: $files table lists at least one Parquet file") - void testIcebergFilesMetadata() throws SQLException { - List> rows = query(""" - SELECT file_path, record_count - FROM iceberg.sales."orders$files" - """); - assertFalse(rows.isEmpty(), "There should be at least one data file after inserts"); - System.out.println("[PASS] Iceberg $files metadata:"); - rows.forEach(r -> System.out.printf(" %s (records=%s)%n", r.get(0), r.get(1))); - } -} diff --git a/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties b/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties deleted file mode 100644 index ed05c180a..000000000 --- a/platforms-setup-guides/trino-setup/trino/catalog/iceberg.properties +++ /dev/null @@ -1,32 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Iceberg catalog backed by Hive Metastore (HMS) + MinIO (S3 storage). -# HMS stores Iceberg table metadata; Trino writes Parquet files to MinIO via S3. -connector.name=iceberg -iceberg.catalog.type=hive_metastore -hive.metastore.uri=thrift://metastore:9083 - -# Native S3 filesystem — handles both s3:// and s3a:// (which HMS uses internally). -# This avoids the "No FileSystem for scheme s3" error when HMS assigns locations. -fs.native-s3.enabled=true -s3.endpoint=http://minio:9000 -s3.path-style-access=true -s3.aws-access-key=minioadmin -s3.aws-secret-key=minioadmin -s3.region=us-east-1 - -iceberg.file-format=PARQUET diff --git a/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties b/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties deleted file mode 100644 index 064863fc7..000000000 --- a/platforms-setup-guides/trino-setup/trino/catalog/tpch.properties +++ /dev/null @@ -1,20 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Built-in TPC-H connector – no external dependencies. -# Useful for testing basic Trino connectivity without any storage setup. -# Usage: SELECT * FROM tpch.tiny.orders LIMIT 10; -connector.name=tpch diff --git a/platforms-setup-guides/trino-setup/trino/config.properties b/platforms-setup-guides/trino-setup/trino/config.properties deleted file mode 100644 index 8584e4aea..000000000 --- a/platforms-setup-guides/trino-setup/trino/config.properties +++ /dev/null @@ -1,20 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -coordinator=true -node-scheduler.include-coordinator=true -http-server.http.port=8080 -discovery.uri=http://localhost:8080 diff --git a/presto-setup/README.md b/presto-setup/README.md deleted file mode 100644 index b5e4bf4dd..000000000 --- a/presto-setup/README.md +++ /dev/null @@ -1,248 +0,0 @@ -# Presto Local Setup - -Local PrestoDB environment using the built-in **memory** connector, completely -containerised. - -The current validation has two parts: - -1. Build the Wayang Presto platform and run the shared JDBC SQL-generation tests. -2. Run the Wayang Presto operator tests against the live local PrestoDB. - -Run the commands below from the repository root. Java 17 and Docker with Docker -Compose are required; Maven is provided by the repository wrapper. - -## Command Conventions - -Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on -Windows PowerShell from the repository root. Docker Compose commands are the -same on both platforms. - -## Stack - -| Component | Image | Port | Role | -|-----------|-------|------|------| -| **PrestoDB** | `prestodb/presto:0.289` | 8081 | SQL engine and in-memory test storage | - -The container listens on port `8080`; Docker exposes it as `8081` to avoid -clashing with the Trino setup. The `memory` connector needs no metastore, -database, or object storage. All tables disappear when the container stops. - -## Directory Layout - -```text -presto-setup/ -|-- docker-compose.yml -|-- README.md -`-- etc/ - `-- catalog/ - `-- memory.properties - -wayang-platforms/wayang-presto/src/test/java/.../ -|-- PrestoOperatorsIT.java -`-- PrestoCostPilotIT.java -``` - -## 1. Test the Wayang Presto Platform - -Build the Presto platform and its required modules: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \ - -DskipTests -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -DskipTests -Drat.skip=true test -``` - -Then run the shared JDBC SQL-generation tests: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am \ - -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -Expected result: - -```text -Wayang Platform Presto ... SUCCESS -Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -``` - -## 2. Test Against the Local Presto Stack - -### 1. Start Presto - -```bash -docker compose -f presto-setup/docker-compose.yml up -d --wait -``` - -Presto can take 20-60 seconds to accept queries. Confirm that it is healthy: - -```bash -docker compose -f presto-setup/docker-compose.yml ps -``` - -The Presto web UI is available at . - -### 2. Run the Wayang Presto operator tests - -`PrestoOperatorsIT` exercises the Wayang Presto implementation against the live -container. It checks `TableSource`, `Filter`, `Projection`, `Join`, -`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the -expected SQL reached Presto through `system.runtime.queries`. - -The standalone join test now runs as a full Wayang plan: -`PrestoTableSource + PrestoTableSource -> JoinOperator -> MapOperator -> sink`. -The normalization map accepts both logical `Tuple2` output and -pushed-down JDBC flat `Record` output. The suite also includes five -`JavaPlanBuilder.readTable` combination plans. Together, they cover every -supported Presto operator through the public API. - -The suite is self-contained. It creates `memory.wayang_it`, generates 120,000 -rows so the optimizer selects SQL pushdown, runs the tests, and drops its tables -afterward. - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \ - -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -Successful validation must show: - -```text -Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 -BUILD SUCCESS -``` - -If Presto is unreachable, the tests are skipped instead of failed. A result -with skipped tests does not confirm that the operators work. Errors while -creating the test schema or tables are treated as real failures. - -### Verified Result - -On June 18, 2026, the suite completed successfully against the local PrestoDB -0.289 container, including the full-plan join validation: - -```text -[PrestoOperatorsIT] Connected to Presto at jdbc:presto://localhost:8081/memory -Executed sql sink: CREATE TABLE memory.wayang_it.amer_orders AS SELECT ... -Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 -BUILD SUCCESS -``` - -This verified the complete `Wayang -> Presto JDBC -> live PrestoDB` path, -including reads, SQL pushdown, join normalization, aggregation, sorting, and -`CREATE TABLE AS SELECT`. - -### 3. Tear down - -```bash -docker compose -f presto-setup/docker-compose.yml down -``` - -## Test Coverage - -| Test | What it checks | -|------|----------------| -| `tableSource` | Full table scan through `PrestoTableSource` | -| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown | -| `projection` | Column projection pushed into the Presto query | -| `join` | Full Wayang join plan with normalization before the sink table | -| `globalReduce` | Global aggregation such as `SUM` | -| `reduceBy` | Grouped aggregation and SQL `GROUP BY` | -| `sort` | Wayang sort and SQL `ORDER BY` | -| `tableSink` | Filtered result written with `CREATE TABLE AS` | -| `javaPlanBuilderReadTableFilterProjection` | Public API filter and projection combination | -| `javaPlanBuilderReadTableFilterGlobalReduce` | Public API filter and global aggregation combination | -| `javaPlanBuilderReadTableReduceBySort` | Public API grouped aggregation and sort combination | -| `javaPlanBuilderReadTableFilterProjectionTableSink` | Public API filtered projection written to a table | -| `javaPlanBuilderReadTableJoin` | Public API two-table join with pushed-down record output | - -## Environment Variables - -Override the default endpoint when running against another PrestoDB: - -| Variable | Default | -|----------|---------| -| `PRESTO_HOST` | `localhost` | -| `PRESTO_PORT` | `8081` | -| `PRESTO_USER` | `test` | - -Example: - -```bash -PRESTO_HOST=my-presto PRESTO_PORT=8080 PRESTO_USER=wayang \ - ./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \ - -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -On PowerShell: - -```powershell -$env:PRESTO_HOST="my-presto" -$env:PRESTO_PORT="8080" -$env:PRESTO_USER="wayang" -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -Remove-Item Env:PRESTO_HOST, Env:PRESTO_PORT, Env:PRESTO_USER -``` - -## Cost Profiling - -Follow the shared cost-profiling guide in -[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide -only covers the Presto stack itself. - -Presto-specific profiling values: - -| Item | Value | -|------|-------| -| Maven module | `wayang-platforms/wayang-presto` | -| Profiling test | `PrestoCostPilotIT` | -| Property prefix | `presto.profile.*` | -| Profiling schema | `memory.wayang_profile` | -| Default output directory | `target/cost-profiling/presto` | -| Learned parameters file | `wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties` | - -`PrestoCostPilotIT` uses the same `PRESTO_HOST`, `PRESTO_PORT`, and -`PRESTO_USER` endpoint variables as `PrestoOperatorsIT`. - -## Troubleshooting - -### `Catalog does not exist: memory` - -Check that `presto-setup/etc/catalog/memory.properties` is a regular file -before starting the container. If Docker first created the container while the -source file was absent, it may have mounted a directory at the catalog path. -Recreate the container after confirming the catalog file is present: - -```bash -docker compose -f presto-setup/docker-compose.yml down -docker compose -f presto-setup/docker-compose.yml up -d --force-recreate --wait -``` - -Confirm the mounted path inside the container is a file: - -```bash -docker exec presto sh -c \ - "ls -l /opt/presto-server/etc/catalog/memory.properties" -``` - -Then rerun `PrestoOperatorsIT`. diff --git a/presto-setup/docker-compose.yml b/presto-setup/docker-compose.yml deleted file mode 100644 index 99d8e4fe1..000000000 --- a/presto-setup/docker-compose.yml +++ /dev/null @@ -1,47 +0,0 @@ ---- -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Stack: a single PrestoDB coordinator/worker with the in-memory connector. -# -# The `memory` connector supports CREATE SCHEMA / CREATE TABLE / INSERT / SELECT -# entirely in-memory, so the integration test is fully self-contained — no Hive -# metastore, object storage, or external catalog required. -# -# Ports: -# Presto: http://localhost:8081 (UI + JDBC; container listens on 8080) -# -# The host port is 8081 to avoid clashing with the Trino stack (which uses 8080). - -services: - - presto: - image: prestodb/presto:0.289 - container_name: presto - ports: - - "8081:8080" - volumes: - # Enable the in-memory connector by adding a catalog properties file. - - ./etc/catalog/memory.properties:/opt/presto-server/etc/catalog/memory.properties - # Presto needs ~20-60s before it accepts queries. Gate on /v1/info so callers - # (and `docker compose up -d --wait`) can wait for readiness; the container - # reports "Up" long before the coordinator is ready. - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"] - interval: 10s - timeout: 5s - retries: 15 - start_period: 30s diff --git a/presto-setup/etc/catalog/memory.properties b/presto-setup/etc/catalog/memory.properties deleted file mode 100644 index 2accfc274..000000000 --- a/presto-setup/etc/catalog/memory.properties +++ /dev/null @@ -1,26 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# PrestoDB in-memory connector. -# Backs the `memory` catalog used by the Wayang Presto integration tests: -# tables created here (CREATE TABLE memory..) live in worker -# memory and are dropped at teardown. -connector.name=memory -# Cap on-heap connector data per node. The image's default heap is -Xmx1G with -# -XX:+ExitOnOutOfMemoryError, so keep this well under the heap; 256MB is ample -# for the small, transient test tables. Raise the heap (custom jvm.config) before -# increasing this if you reuse the stack for larger inserts. -memory.max-data-per-node=256MB diff --git a/trino-setup/README.md b/trino-setup/README.md deleted file mode 100644 index 56dcca59b..000000000 --- a/trino-setup/README.md +++ /dev/null @@ -1,283 +0,0 @@ -# Trino Local Setup - -Local Trino environment backed by an **Iceberg** data lake, completely containerised. - -The current validation has three parts: - -1. Build the Wayang Trino platform and run the shared JDBC SQL-generation tests. -2. Run the Wayang Trino operator tests against the live local stack. -3. Run standalone JDBC integration tests against the local Trino, Iceberg, and MinIO stack. - -Run the commands below from the repository root. Java 17 and Docker with -Docker Compose are required; Maven is provided by the repository wrapper. - -The Trino cost-profiling branch is named `feature/trino-cost-profiling`: - -```bash -git checkout feature/trino-cost-profiling -``` - -## Command Conventions - -Use the `bash` blocks on macOS/Linux terminals. Use the `powershell` blocks on -Windows PowerShell from the repository root. Docker Compose commands are the -same on both platforms. - -## Stack - -| Component | Image | Port | Role | -|-----------|-------|------|------| -| **Trino** | `trinodb/trino:435` | 8080 | SQL query engine | -| **Hive Metastore** | `naushadh/hive-metastore:latest` | 9083 | Iceberg table catalog (Thrift) | -| **PostgreSQL** | `postgres:15-alpine` | 5432 | HMS metadata backing store | -| **MinIO** | `minio/minio:latest` | 9000 / 9001 | S3-compatible object storage | - -HMS is the battle-tested Iceberg catalog for Trino. Parquet data files are written by Trino directly to MinIO; HMS only stores schema/table metadata. - -## Directory Layout - -``` -trino-setup/ -|-- docker-compose.yml # Full stack definition -|-- trino/ -| |-- config.properties # Trino node config -| `-- catalog/ -| |-- iceberg.properties # Iceberg via HMS + MinIO -| `-- tpch.properties # Built-in TPC-H (no storage needed) -|-- scripts/ -| |-- init.sql # Creates iceberg.sales.orders + sample rows -| `-- run-init.sh # Helper: waits for Trino then runs init.sql -|-- pom.xml # Standalone Maven project (Java 17) -`-- src/test/java/.../ - `-- TrinoIntegrationTest.java # JUnit 5 integration tests -``` - -## 1. Test the Wayang Trino Platform - -Build the Trino platform and its required modules: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -DskipTests -Drat.skip=true test -``` - -Then run the shared JDBC SQL-generation tests: - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-jdbc-template -am -Dtest=JdbcExecutorTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true test -``` - -Expected result: - -```text -Wayang Platform Trino ... SUCCESS -Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 -``` - -## 2. Test Against the Local Trino Stack - -### 1. Start the stack - -```bash -docker compose -f trino-setup/docker-compose.yml up -d -``` - -Wait ~30 seconds for all services to become healthy. Check with: - -```bash -docker compose -f trino-setup/docker-compose.yml ps -# or watch the Trino UI at http://localhost:8080 -``` - -### 2. Run the Wayang Trino operator tests - -`TrinoOperatorsIT` exercises the Wayang Trino implementation against the live -Trino stack. It checks `TableSource`, `Filter`, `Projection`, `Join`, -`GlobalReduce`, `ReduceBy`, `Sort`, and `TableSink`, and confirms that the -expected SQL reached Trino. The standalone join test now runs a full Wayang -plan and normalizes both possible join result shapes before collecting records: -logical joins can produce `Tuple2`, while pushed-down JDBC joins -can return a flat `Record`. - -The suite is self-contained: it creates `iceberg.wayang_it`, scales its test -data to 120,000 rows so the optimizer selects SQL pushdown, and drops its test -tables afterward. It does not require `scripts/init.sql`. The suite also -contains five JavaPlanBuilder `readTable` combination tests that cover filter, -projection, global reduce, reduce-by plus sort, table sink, and join through -the public API. - -```bash -./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \ - -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ - -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -Drat.skip=true -Dlicense.skip=true test -``` - -Expected result: - -```text -Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 -``` - -Verified on June 18, 2026 against the local Docker stack with the full-plan -join test and all five JavaPlanBuilder combination tests enabled. - -If Trino is unreachable, these tests are skipped instead of failed. A result -with skipped tests does not confirm that the operators work. - -### 3. Load sample Iceberg data - -```bash -bash trino-setup/scripts/run-init.sh -``` - -On PowerShell: - -```powershell -Get-Content -Raw trino-setup/scripts/init.sql | docker exec -i trino trino --server http://localhost:8080 --user admin -``` - -This creates the schema `iceberg.sales` and inserts 20 sample orders into -`iceberg.sales.orders` (Parquet files on MinIO). - -### 4. Run the standalone stack integration tests - -```bash -./mvnw -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -On PowerShell: - -```powershell -.\mvnw.cmd --% -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -Tests are skipped by default (no `-Pintegration`) to avoid requiring Docker in CI. -These tests validate the stack and direct JDBC queries independently of the -Wayang operator implementation. - -Expected result: - -```text -Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 -BUILD SUCCESS -``` - -### 5. Manual exploration - -Open the **Trino UI**: http://localhost:8080 - -Or connect via the Trino CLI inside the container: - -```bash -docker exec -it trino trino --catalog iceberg --schema sales -``` - -```sql --- TPC-H built-in data (no init.sql needed) -SELECT * FROM tpch.tiny.orders LIMIT 5; - --- Iceberg table -SELECT region, SUM(amount) FROM iceberg.sales.orders GROUP BY region; - --- Iceberg file metadata -SELECT * FROM iceberg.sales."orders$files"; - --- Iceberg history -SELECT * FROM iceberg.sales."orders$history"; -``` - -**MinIO console**: http://localhost:9001 (login: `minioadmin` / `minioadmin`) -Look for Parquet files under `warehouse/sales/orders/`. - -### 6. Tear down - -```bash -docker compose -f trino-setup/docker-compose.yml down -v -``` - -The `-v` option removes volumes and clears the local MinIO and PostgreSQL data. - -## Test Coverage - -### Wayang operator integration tests - -| Test | What it checks | -|------|----------------| -| `tableSource` | Full table scan through `TrinoTableSource` | -| `filter` | Wayang `FilterOperator` and SQL `WHERE` pushdown | -| `projection` | Column projection pushed into the Trino query | -| `join` | Full Wayang join plan with normalization before the collecting sink | -| `globalReduce` | Global aggregation such as `SUM` | -| `reduceBy` | Grouped aggregation and SQL `GROUP BY` | -| `sort` | Wayang sort and SQL `ORDER BY` | -| `tableSink` | Filtered result written with `CREATE TABLE AS` | -| `javaPlanBuilderReadTableFilterProjection` | `readTable -> filter -> projection -> collect` | -| `javaPlanBuilderReadTableFilterGlobalReduce` | `readTable -> filter -> globalReduce -> collect` | -| `javaPlanBuilderReadTableReduceBySort` | `readTable -> reduceByKey -> sort -> collect` | -| `javaPlanBuilderReadTableFilterProjectionTableSink` | `readTable -> filter -> projection -> writeTable` | -| `javaPlanBuilderReadTableJoin` | `readTable + readTable -> join -> collect` | - -### Standalone stack integration tests - -| Test | What it checks | -|------|----------------| -| `testConnectivity` | `SELECT 1`, JDBC connection works | -| `testTpchConnector` | TPC-H built-in connector, no storage needed | -| `testTpchTopOrders` | ORDER BY + LIMIT on TPC-H | -| `testIcebergSchemaVisible` | Schema created by `init.sql` is visible | -| `testIcebergSelectAll` | Full table scan, 20 rows | -| `testIcebergFilterByRegion` | WHERE pushdown on string column | -| `testIcebergAggregate` | GROUP BY + SUM aggregation | -| `testIcebergFilterByAmount` | WHERE pushdown on double column | -| `testIcebergProjection` | SELECT subset of columns | -| `testIcebergFilesMetadata` | `$files` system table, confirms Parquet on MinIO | - -## Cost Profiling - -Follow the shared cost-profiling guide in -[`guides/cost-profiling.md`](../guides/cost-profiling.md). This setup guide -only covers the Trino stack itself. - -Trino-specific profiling values: - -| Item | Value | -|------|-------| -| Maven module | `wayang-platforms/wayang-trino` | -| Profiling test | `TrinoCostPilotIT` | -| Property prefix | `trino.profile.*` | -| Default output directory | `target/cost-profiling/trino` | -| Learned parameters file | `wayang-platforms/wayang-trino/src/main/resources/wayang-trino-defaults.properties` | - -## Environment Variables - -Override defaults if running Trino on a different host/port: - -```bash -TRINO_HOST=my-trino-host TRINO_PORT=8080 ./mvnw -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -``` - -On PowerShell: - -```powershell -$env:TRINO_HOST="my-trino-host" -$env:TRINO_PORT="8080" -.\mvnw.cmd --% -f trino-setup/pom.xml -Pintegration -Dtest=TrinoIntegrationTest test -Remove-Item Env:TRINO_HOST, Env:TRINO_PORT -``` diff --git a/trino-setup/demo.sh b/trino-setup/demo.sh deleted file mode 100644 index ea3df7892..000000000 --- a/trino-setup/demo.sh +++ /dev/null @@ -1,126 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -WAYANG_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -TRINO_SETUP="$SCRIPT_DIR" -TRINO_CONTAINER="trino" -MAVEN_FLAGS="-Pskip-prerequisite-check -Drat.skip=true -Dmaven.javadoc.skip=true" - -banner() { - echo - echo "============================================================" - printf " %s\n" "$*" - echo "============================================================" - echo -} - -step() { - echo - echo "-- $*" - echo -} - -pause() { - if [[ "${WAYANG_DEMO_AUTO:-false}" != "true" ]]; then - echo - read -rp "Press ENTER to continue..." _ || true - echo - fi -} - -run_wayang_demo() { - "$WAYANG_ROOT/mvnw" exec:java -pl wayang-platforms/wayang-trino \ - -Dexec.mainClass="org.apache.wayang.trino.TrinoDemo" \ - ${MAVEN_FLAGS} -} - -banner "ACT 1: Start Trino + Iceberg via Docker" - -step "1a. Starting the stack" -cd "$TRINO_SETUP" -docker compose up -d - -step "1b. Containers running" -docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" \ - | grep -E "NAMES|trino|minio|metastore|postgres" - -step "1c. Waiting for Trino to be ready" -MAX_WAIT=90 -ELAPSED=0 -until docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT 1" --output-format ALIGNED >/dev/null 2>&1; do - if [[ "$ELAPSED" -ge "$MAX_WAIT" ]]; then - echo "Timed out waiting for Trino after ${MAX_WAIT}s" - exit 1 - fi - printf ". waiting (%ds elapsed)\r" "$ELAPSED" - sleep 3 - ELAPSED=$((ELAPSED + 3)) -done -echo "Trino is ready at http://localhost:8080" - -step "1d. Initialising Iceberg tables" -docker exec -i "$TRINO_CONTAINER" trino < "$TRINO_SETUP/scripts/init.sql" 2>&1 \ - | grep -v "^WARNING\|jline\|org.jline" || true -echo "iceberg.sales.orders seeded" - -step "1e. Table schema" -docker exec "$TRINO_CONTAINER" \ - trino --execute "DESCRIBE iceberg.sales.orders" \ - --output-format ALIGNED 2>/dev/null - -pause - -banner "ACT 2: Query Iceberg directly via Trino CLI" - -step "2a. Full table scan" -echo "SQL: SELECT * FROM iceberg.sales.orders" -docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT * FROM iceberg.sales.orders ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -step "2b. Filter: region = 'AMER'" -echo "SQL: SELECT * FROM iceberg.sales.orders WHERE region = 'AMER'" -docker exec "$TRINO_CONTAINER" \ - trino --execute "SELECT * FROM iceberg.sales.orders WHERE region = 'AMER' ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -step "2c. Projection with filter" -echo "SQL: SELECT region, product, amount FROM iceberg.sales.orders WHERE region = 'AMER'" -docker exec "$TRINO_CONTAINER" \ - trino --execute \ - "SELECT region, product, amount - FROM iceberg.sales.orders - WHERE region = 'AMER' - ORDER BY order_id" \ - --output-format ALIGNED 2>/dev/null - -pause - -banner "ACT 3: Wayang API filter + projection pushdown" -cd "$WAYANG_ROOT" -run_wayang_demo - -banner "Demo complete" -echo "Trino UI: http://localhost:8080" -echo "MinIO UI: http://localhost:9001 (minioadmin / minioadmin)" -echo -echo "To stop the stack:" -echo " cd trino-setup && docker compose down" diff --git a/trino-setup/docker-compose.yml b/trino-setup/docker-compose.yml deleted file mode 100644 index 0c057540e..000000000 --- a/trino-setup/docker-compose.yml +++ /dev/null @@ -1,141 +0,0 @@ ---- -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Stack: Trino + Hive Metastore + MinIO (S3 storage) -# -# This is the battle-tested Trino + Iceberg local setup. -# Hive Metastore (HMS) stores Iceberg table metadata over Thrift on port 9083. -# MinIO provides S3-compatible object storage for Parquet data files. -# Trino's Iceberg connector uses HMS as catalog and writes Parquet to MinIO. -# -# Ports: -# Trino: http://localhost:8080 (UI + JDBC) -# MinIO S3: http://localhost:9000 -# MinIO UI: http://localhost:9001 (minioadmin / minioadmin) -# HMS: localhost:9083 (Thrift, internal) -# Postgres: localhost:5432 (HMS backing store) - -services: - - # ── PostgreSQL (Hive Metastore backing database) ─────────────────────────── - postgres: - image: postgres:15-alpine - container_name: trino-postgres - environment: - POSTGRES_DB: metastore - POSTGRES_USER: hive - POSTGRES_PASSWORD: hive - ports: - - "5432:5432" - volumes: - - postgres-data:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U hive -d metastore"] - interval: 10s - timeout: 5s - retries: 5 - - # ── MinIO (S3-compatible object storage) ────────────────────────────────── - minio: - image: minio/minio:latest - container_name: trino-minio - environment: - MINIO_ROOT_USER: minioadmin - MINIO_ROOT_PASSWORD: minioadmin - ports: - - "9000:9000" - - "9001:9001" - command: server /data --console-address ":9001" - volumes: - - minio-data:/data - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 10s - timeout: 5s - retries: 5 - - # Create the warehouse bucket before HMS starts - minio-init: - image: minio/mc:latest - container_name: trino-minio-init - depends_on: - minio: - condition: service_healthy - entrypoint: > - /bin/sh -c " - mc alias set local http://minio:9000 minioadmin minioadmin; - mc mb local/warehouse --ignore-existing; - echo 'bucket warehouse ready'; - exit 0; - " - - # ── Hive Metastore ──────────────────────────────────────────────────────── - # naushadh/hive-metastore is a minimal, pre-configured HMS image - # that supports S3-compatible storage via env vars. - metastore: - image: naushadh/hive-metastore:latest - container_name: trino-metastore - depends_on: - postgres: - condition: service_healthy - minio: - condition: service_healthy - minio-init: - condition: service_completed_successfully - ports: - - "9083:9083" - environment: - DATABASE_HOST: postgres - DATABASE_DB: metastore - DATABASE_USER: hive - DATABASE_PASSWORD: hive - # S3 / MinIO - S3_ENDPOINT_URL: http://minio:9000 - S3_BUCKET: warehouse - S3_PREFIX: / - AWS_ACCESS_KEY_ID: minioadmin - AWS_SECRET_ACCESS_KEY: minioadmin - REGION: us-east-1 - # No nc/curl in this image; use bash's /dev/tcp built-in - healthcheck: - test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/localhost/9083 2>/dev/null && exit 0 || exit 1"] - interval: 15s - timeout: 10s - retries: 15 - - # ── Trino ───────────────────────────────────────────────────────────────── - trino: - image: trinodb/trino:435 - container_name: trino - depends_on: - metastore: - condition: service_healthy - minio: - condition: service_healthy - ports: - - "8080:8080" - volumes: - - ./trino/catalog:/etc/trino/catalog - - ./trino/config.properties:/etc/trino/config.properties - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/v1/info"] - interval: 15s - timeout: 10s - retries: 10 - -volumes: - postgres-data: - minio-data: diff --git a/trino-setup/pom.xml b/trino-setup/pom.xml deleted file mode 100644 index f04955203..000000000 --- a/trino-setup/pom.xml +++ /dev/null @@ -1,93 +0,0 @@ - - - - 4.0.0 - - org.apache.wayang - trino-setup - 1.0-SNAPSHOT - jar - - Trino Local Setup — Integration Tests - - Standalone integration tests for a local Trino stack - (Trino + Nessie Iceberg catalog + MinIO S3 storage). - Independent of the Wayang codebase. - - - - 17 - 17 - UTF-8 - 435 - 5.10.2 - - - - - - io.trino - trino-jdbc - ${trino.version} - test - - - - - org.junit.jupiter - junit-jupiter - ${junit.version} - test - - - - - org.slf4j - slf4j-simple - 2.0.12 - test - - - - - - - org.apache.maven.plugins - maven-surefire-plugin - 3.2.5 - - - ${skipIntegrationTests} - - - - - - - - - integration - - false - - - - diff --git a/trino-setup/scripts/init.sql b/trino-setup/scripts/init.sql deleted file mode 100644 index 245ffbce3..000000000 --- a/trino-setup/scripts/init.sql +++ /dev/null @@ -1,66 +0,0 @@ --- Licensed to the Apache Software Foundation (ASF) under one --- or more contributor license agreements. See the NOTICE file --- distributed with this work for additional information --- regarding copyright ownership. The ASF licenses this file --- to you under the Apache License, Version 2.0 (the --- "License"); you may not use this file except in compliance --- with the License. You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, software --- distributed under the License is distributed on an "AS IS" BASIS, --- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --- See the License for the specific language governing permissions and --- limitations under the License. - --- Run this after the stack is up to create sample Iceberg tables. --- Usage: ./scripts/run-init.sh --- Or manually: docker exec -it trino trino < /scripts/init.sql - --- Schema -CREATE SCHEMA IF NOT EXISTS iceberg.sales; - --- Orders table (Iceberg / Parquet on MinIO) -CREATE TABLE IF NOT EXISTS iceberg.sales.orders ( - order_id BIGINT, - region VARCHAR, - product VARCHAR, - amount DOUBLE, - order_date DATE -) -WITH (format = 'PARQUET'); - --- Idempotent seed: clear before inserting so re-runs do not duplicate rows. -DELETE FROM iceberg.sales.orders; - --- Sample data: 20 rows, 4 regions (AMER/APAC/EMEA/LATAM), 5 products. --- AMER rows: 3, 6, 9, 12, 16 -> 5 rows for filter demo --- Projection demo selects only: region, product, amount -INSERT INTO iceberg.sales.orders VALUES - (1, 'APAC', 'Widget A', 1500.00, DATE '2024-01-15'), - (2, 'EMEA', 'Widget B', 800.50, DATE '2024-01-16'), - (3, 'AMER', 'Widget A', 2200.00, DATE '2024-01-17'), - (4, 'APAC', 'Widget C', 350.75, DATE '2024-01-18'), - (5, 'EMEA', 'Widget A', 1100.00, DATE '2024-01-19'), - (6, 'AMER', 'Widget B', 950.25, DATE '2024-01-20'), - (7, 'APAC', 'Widget B', 1750.00, DATE '2024-01-21'), - (8, 'EMEA', 'Widget C', 420.00, DATE '2024-01-22'), - (9, 'AMER', 'Widget C', 680.50, DATE '2024-01-23'), - (10, 'APAC', 'Widget A', 3000.00, DATE '2024-01-24'), - (11, 'LATAM', 'Widget D', 560.00, DATE '2024-01-25'), - (12, 'AMER', 'Widget D', 1320.75, DATE '2024-01-26'), - (13, 'EMEA', 'Widget D', 990.00, DATE '2024-01-27'), - (14, 'LATAM', 'Widget E', 2100.50, DATE '2024-01-28'), - (15, 'APAC', 'Widget E', 4500.00, DATE '2024-01-29'), - (16, 'AMER', 'Widget E', 3750.00, DATE '2024-01-30'), - (17, 'EMEA', 'Widget E', 1250.00, DATE '2024-01-31'), - (18, 'LATAM', 'Widget A', 870.25, DATE '2024-02-01'), - (19, 'APAC', 'Widget D', 1680.00, DATE '2024-02-02'), - (20, 'LATAM', 'Widget B', 440.50, DATE '2024-02-03'); - --- Verify -SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount -FROM iceberg.sales.orders -GROUP BY region -ORDER BY total_amount DESC; diff --git a/trino-setup/scripts/run-init.sh b/trino-setup/scripts/run-init.sh deleted file mode 100644 index 91d279192..000000000 --- a/trino-setup/scripts/run-init.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Runs init.sql against the local Trino instance. -# The stack must be fully up before running this. - -set -e - -TRINO_HOST=${TRINO_HOST:-localhost} -TRINO_PORT=${TRINO_PORT:-8080} - -echo "Waiting for Trino to be ready..." -until curl -sf "http://${TRINO_HOST}:${TRINO_PORT}/v1/info" | grep -q '"starting":false'; do - echo " Trino not ready yet, retrying in 5s..." - sleep 5 -done -echo "Trino is ready." - -echo "Running init.sql..." -docker exec -i trino trino \ - --server "http://${TRINO_HOST}:${TRINO_PORT}" \ - --user admin \ - < "$(dirname "$0")/init.sql" - -echo "Done. Sample Iceberg data loaded into iceberg.sales.orders" diff --git a/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java b/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java deleted file mode 100644 index 081beea99..000000000 --- a/trino-setup/src/test/java/org/apache/wayang/trino/TrinoIntegrationTest.java +++ /dev/null @@ -1,232 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.wayang.trino; - -import org.junit.jupiter.api.*; - -import java.sql.*; -import java.util.ArrayList; -import java.util.List; -import java.util.Properties; - -import static org.junit.jupiter.api.Assertions.*; - -/** - * Integration tests for the local Trino stack. - * - * Prerequisites: run `docker-compose up -d` and `./scripts/run-init.sh` first. - * - * Run tests: - * mvn test -Pintegration - * - * Or skip infrastructure setup and run with a custom host: - * TRINO_HOST=localhost TRINO_PORT=8080 mvn test -Pintegration - */ -@TestMethodOrder(MethodOrderer.OrderAnnotation.class) -class TrinoIntegrationTest { - - private static final String TRINO_HOST = System.getenv().getOrDefault("TRINO_HOST", "localhost"); - private static final int TRINO_PORT = Integer.parseInt(System.getenv().getOrDefault("TRINO_PORT", "8080")); - private static final String JDBC_URL = String.format("jdbc:trino://%s:%d", TRINO_HOST, TRINO_PORT); - - private static Connection connection; - - // ── Lifecycle ───────────────────────────────────────────────────────── - - @BeforeAll - static void openConnection() throws Exception { - Properties props = new Properties(); - props.setProperty("user", "admin"); // Trino requires a non-empty user - connection = DriverManager.getConnection(JDBC_URL, props); - System.out.printf("Connected to Trino at %s%n", JDBC_URL); - } - - @AfterAll - static void closeConnection() throws Exception { - if (connection != null && !connection.isClosed()) { - connection.close(); - } - } - - // ── Helper ──────────────────────────────────────────────────────────── - - private List> query(String sql) throws SQLException { - List> rows = new ArrayList<>(); - try (Statement stmt = connection.createStatement(); - ResultSet rs = stmt.executeQuery(sql)) { - int cols = rs.getMetaData().getColumnCount(); - while (rs.next()) { - List row = new ArrayList<>(); - for (int i = 1; i <= cols; i++) row.add(rs.getObject(i)); - rows.add(row); - } - } - return rows; - } - - // ── Test 1: Basic connectivity ──────────────────────────────────────── - - @Test - @Order(1) - @DisplayName("Trino responds to a simple SELECT 1") - void testConnectivity() throws SQLException { - List> rows = query("SELECT 1"); - assertEquals(1, rows.size()); - assertEquals(1L, ((Number) rows.get(0).get(0)).longValue()); - System.out.println("[PASS] Basic connectivity OK"); - } - - // ── Test 2: TPC-H built-in connector ───────────────────────────────── - - @Test - @Order(2) - @DisplayName("TPC-H tiny catalog: count orders") - void testTpchConnector() throws SQLException { - List> rows = query("SELECT COUNT(*) FROM tpch.tiny.orders"); - long count = ((Number) rows.get(0).get(0)).longValue(); - assertTrue(count > 0, "tpch.tiny.orders should have rows"); - System.out.printf("[PASS] TPC-H tiny.orders has %,d rows%n", count); - } - - @Test - @Order(3) - @DisplayName("TPC-H tiny catalog: top 5 orders by total price") - void testTpchTopOrders() throws SQLException { - List> rows = query(""" - SELECT orderkey, totalprice - FROM tpch.tiny.orders - ORDER BY totalprice DESC - LIMIT 5 - """); - assertEquals(5, rows.size(), "Expected exactly 5 rows"); - System.out.println("[PASS] TPC-H top 5 orders:"); - rows.forEach(r -> System.out.printf(" orderkey=%s totalprice=%s%n", r.get(0), r.get(1))); - } - - // ── Test 4: Iceberg — schema exists ────────────────────────────────── - - @Test - @Order(4) - @DisplayName("Iceberg catalog: schema 'sales' is visible") - void testIcebergSchemaVisible() throws SQLException { - List> rows = query("SHOW SCHEMAS IN iceberg LIKE 'sales'"); - assertFalse(rows.isEmpty(), "Schema 'sales' should exist in iceberg catalog. " + - "Did you run scripts/run-init.sh?"); - System.out.println("[PASS] Iceberg schema 'sales' is visible"); - } - - // ── Test 5: Iceberg — full table scan ──────────────────────────────── - - @Test - @Order(5) - @DisplayName("Iceberg table: select all orders") - void testIcebergSelectAll() throws SQLException { - List> rows = query("SELECT * FROM iceberg.sales.orders ORDER BY order_id"); - assertEquals(20, rows.size(), "Expected 20 rows inserted by init.sql"); - System.out.println("[PASS] Iceberg full scan: 20 rows"); - rows.forEach(r -> System.out.printf(" %s%n", r)); - } - - // ── Test 6: Iceberg — pushdown filter ──────────────────────────────── - - @Test - @Order(6) - @DisplayName("Iceberg table: filter by region = APAC") - void testIcebergFilterByRegion() throws SQLException { - List> rows = query(""" - SELECT order_id, region, amount - FROM iceberg.sales.orders - WHERE region = 'APAC' - ORDER BY order_id - """); - assertFalse(rows.isEmpty(), "Should have APAC orders"); - rows.forEach(r -> assertEquals("APAC", r.get(1), "All rows must be APAC")); - System.out.printf("[PASS] Filter pushdown: %d APAC rows%n", rows.size()); - } - - // ── Test 7: Iceberg — aggregation ──────────────────────────────────── - - @Test - @Order(7) - @DisplayName("Iceberg table: aggregate total_amount by region") - void testIcebergAggregate() throws SQLException { - List> rows = query(""" - SELECT region, COUNT(*) AS order_count, SUM(amount) AS total_amount - FROM iceberg.sales.orders - GROUP BY region - ORDER BY total_amount DESC - """); - assertFalse(rows.isEmpty(), "Aggregation should return rows"); - System.out.println("[PASS] Aggregation by region:"); - rows.forEach(r -> System.out.printf(" region=%-5s count=%s total=%.2f%n", - r.get(0), r.get(1), ((Number) r.get(2)).doubleValue())); - } - - // ── Test 8: Iceberg — amount threshold filter ───────────────────────── - - @Test - @Order(8) - @DisplayName("Iceberg table: filter orders with amount > 1000") - void testIcebergFilterByAmount() throws SQLException { - List> rows = query(""" - SELECT order_id, amount - FROM iceberg.sales.orders - WHERE amount > 1000.0 - ORDER BY amount DESC - """); - rows.forEach(r -> assertTrue( - ((Number) r.get(1)).doubleValue() > 1000.0, - "All rows must have amount > 1000" - )); - System.out.printf("[PASS] Amount filter: %d rows with amount > 1000%n", rows.size()); - } - - // ── Test 9: Iceberg — projection (select subset of columns) ─────────── - - @Test - @Order(9) - @DisplayName("Iceberg table: project only region and product columns") - void testIcebergProjection() throws SQLException { - List> rows = query(""" - SELECT region, product - FROM iceberg.sales.orders - LIMIT 5 - """); - assertEquals(5, rows.size()); - rows.forEach(r -> { - assertNotNull(r.get(0), "region should not be null"); - assertNotNull(r.get(1), "product should not be null"); - }); - System.out.println("[PASS] Projection (region, product): 5 rows returned"); - } - - // ── Test 10: Iceberg metadata — table files ────────────────────────── - - @Test - @Order(10) - @DisplayName("Iceberg metadata: $files table lists at least one Parquet file") - void testIcebergFilesMetadata() throws SQLException { - List> rows = query(""" - SELECT file_path, record_count - FROM iceberg.sales."orders$files" - """); - assertFalse(rows.isEmpty(), "There should be at least one data file after inserts"); - System.out.println("[PASS] Iceberg $files metadata:"); - rows.forEach(r -> System.out.printf(" %s (records=%s)%n", r.get(0), r.get(1))); - } -} diff --git a/trino-setup/trino/catalog/iceberg.properties b/trino-setup/trino/catalog/iceberg.properties deleted file mode 100644 index 5aabe5275..000000000 --- a/trino-setup/trino/catalog/iceberg.properties +++ /dev/null @@ -1,32 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Iceberg catalog backed by Hive Metastore (HMS) and MinIO (S3 storage). -# HMS stores Iceberg table metadata; Trino writes Parquet files to MinIO via S3. -connector.name=iceberg -iceberg.catalog.type=hive_metastore -hive.metastore.uri=thrift://metastore:9083 - -# Native S3 filesystem handles both s3:// and s3a:// (which HMS uses internally). -# This avoids the "No FileSystem for scheme s3" error when HMS assigns locations. -fs.native-s3.enabled=true -s3.endpoint=http://minio:9000 -s3.path-style-access=true -s3.aws-access-key=minioadmin -s3.aws-secret-key=minioadmin -s3.region=us-east-1 - -iceberg.file-format=PARQUET diff --git a/trino-setup/trino/catalog/tpch.properties b/trino-setup/trino/catalog/tpch.properties deleted file mode 100644 index ee4f23783..000000000 --- a/trino-setup/trino/catalog/tpch.properties +++ /dev/null @@ -1,20 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Built-in TPC-H connector with no external dependencies. -# Useful for testing basic Trino connectivity without any storage setup. -# Usage: SELECT * FROM tpch.tiny.orders LIMIT 10; -connector.name=tpch diff --git a/trino-setup/trino/config.properties b/trino-setup/trino/config.properties deleted file mode 100644 index 8584e4aea..000000000 --- a/trino-setup/trino/config.properties +++ /dev/null @@ -1,20 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -coordinator=true -node-scheduler.include-coordinator=true -http-server.http.port=8080 -discovery.uri=http://localhost:8080 diff --git a/wayang-applications/README.md b/wayang-applications/README.md index 3332b3caa..fa61185b2 100644 --- a/wayang-applications/README.md +++ b/wayang-applications/README.md @@ -76,4 +76,8 @@ The file _env.demo1.sh_ contains additional properties, need in a particlar demo In this file we will never see cluster or user specific details, only properties which are specific to the particular application are listed here. +## SQL Platform Examples +- [BigQuery](bigquery.md) +- [Presto](presto.md) +- [Trino](trino.md) diff --git a/wayang-applications/bigquery.md b/wayang-applications/bigquery.md new file mode 100644 index 000000000..a63327c54 --- /dev/null +++ b/wayang-applications/bigquery.md @@ -0,0 +1,103 @@ + + +# BigQuery Example + +This example demonstrates the BigQuery cost model and executes filter/projection +pushdown against an existing BigQuery table. + +## Prerequisites + +- Java 17 +- A Google Cloud project with the BigQuery API enabled +- A dataset and a service account that can run queries and read the example table +- A service-account JSON key, or another authentication mode supported by the + bundled Google BigQuery JDBC driver + +Wayang does not provision a BigQuery emulator or Google Cloud resources. + +## Prepare the table + +Run this query in the BigQuery console or with your preferred BigQuery client, +after replacing `my-project` with your project ID: + +```sql +CREATE SCHEMA IF NOT EXISTS `my-project.sales`; + +CREATE OR REPLACE TABLE `my-project.sales.orders` AS +SELECT 1 AS order_id, 'AMER' AS region, 'book' AS product, + 25.50 AS amount, '2026-01-10' AS order_date +UNION ALL +SELECT 2, 'EMEA', 'desk', 300.00, '2026-01-11' +UNION ALL +SELECT 3, 'AMER', 'chair', 85.25, '2026-01-12'; +``` + +You can instead use an existing table with `order_id`, `region`, `product`, +`amount`, and `order_date` columns in that order. + +## Configure Wayang + +Create a properties file such as `/tmp/bigquery-example.properties`: + +```properties +wayang.bigquery.jdbc.url = jdbc:bigquery://https://www.googleapis.com/bigquery/v2;ProjectId=my-project;OAuthType=0;OAuthServiceAcctEmail=service-account@example.com;OAuthPvtKeyPath=/path/to/key.json +wayang.bigquery.jdbc.user = +wayang.bigquery.jdbc.password = +wayang.bigquery.demo.table = `my-project.sales.orders` +``` + +Authentication is configured through the JDBC URL supported by the BigQuery +JDBC driver. Keep the backticks around the fully qualified table name because +BigQuery project IDs can contain hyphens. Do not commit the service account key +or a properties file containing credentials. + +## Build and run + +Build the application and its dependencies from the repository root: + +```bash +./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \ + -DskipTests -Dpython.worker.tests.skip=true install +``` + +Run all segments against the configured table: + +```bash +./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \ + -Dexec.mainClass=org.apache.wayang.applications.BigQueryDemo \ + "-Dexec.args=file:///tmp/bigquery-example.properties all" +``` + +The optional second argument is `cost`, `filter`, `projection`, or `all`. +`filter`, `projection`, and `all` execute against the configured BigQuery table +and require the JDBC URL. The `cost` mode does not connect to BigQuery, so the +JDBC URL can be omitted when running that mode alone. + +On Windows, replace `./mvnw` with `.\mvnw.cmd` and use a file URL such as +`file:///C:/Temp/bigquery-example.properties`. + +## Troubleshooting + +- **Project or dataset not found:** check `ProjectId` in the JDBC URL and the + fully qualified table name. +- **Permission denied:** grant the identity permission to create query jobs and + read the table data. +- **Private key error:** use an absolute path in `OAuthPvtKeyPath` and confirm + that the Wayang process can read the JSON file. +- **Only testing configuration:** run the `cost` mode; it does not contact + BigQuery or require credentials. diff --git a/wayang-applications/pom.xml b/wayang-applications/pom.xml index 28fba545d..0afa1d7af 100644 --- a/wayang-applications/pom.xml +++ b/wayang-applications/pom.xml @@ -56,6 +56,26 @@ + + org.apache.wayang + wayang-bigquery + ${project.version} + + + org.apache.wayang + wayang-presto + ${project.version} + + + org.apache.wayang + wayang-trino + ${project.version} + + + org.antlr + antlr4-runtime + 4.13.1 + org.apache.wayang wayang-core @@ -104,6 +124,11 @@ 3.9.2 + + com.fasterxml.jackson.core + jackson-core + 2.18.8 + com.fasterxml.jackson.core jackson-databind diff --git a/wayang-applications/presto.md b/wayang-applications/presto.md new file mode 100644 index 000000000..c418131f0 --- /dev/null +++ b/wayang-applications/presto.md @@ -0,0 +1,97 @@ + + +# Presto Example + +This example reads an existing PrestoDB table, filters rows whose `region` is +`AMER`, and returns only the `region`, `product`, and `amount` columns. Wayang +pushes the filter and projection into the SQL query executed by Presto. + +## Prerequisites + +- Java 17 +- A PrestoDB deployment reachable from the machine running Wayang +- A catalog and schema in which you can create or read the example table +- A Presto user with `SELECT` permission on that table + +Wayang does not start or configure PrestoDB. The endpoint can be a local, +shared, or hosted deployment. + +## Prepare the table + +Run the following with your Presto CLI or SQL client. Replace +`memory.default.orders` with a table name supported by your catalog when the +memory connector is unavailable. + +```sql +CREATE TABLE memory.default.orders ( + order_id BIGINT, + region VARCHAR, + product VARCHAR, + amount DOUBLE, + order_date VARCHAR +); + +INSERT INTO memory.default.orders VALUES + (1, 'AMER', 'book', 25.50, '2026-01-10'), + (2, 'EMEA', 'desk', 300.00, '2026-01-11'), + (3, 'AMER', 'chair', 85.25, '2026-01-12'); +``` + +The example accepts any table with these five columns in this order: +`order_id`, `region`, `product`, `amount`, and `order_date`. + +## Configure Wayang + +Create `/tmp/presto-example.properties`: + +```properties +wayang.presto.jdbc.url = jdbc:presto://presto.example.com:8080/memory/default +wayang.presto.jdbc.user = wayang +wayang.presto.demo.table = memory.default.orders +``` + +Change the URL, credentials, and fully qualified table name for your deployment. +Add `wayang.presto.jdbc.password` only when the deployment requires it; the +Presto driver rejects an empty password on a non-TLS connection. TLS and other +driver options can be included in the JDBC URL. + +## Build and run + +From the repository root: + +```bash +./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \ + -DskipTests -Dpython.worker.tests.skip=true install + +./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \ + -Dexec.mainClass=org.apache.wayang.applications.PrestoDemo \ + "-Dexec.args=file:///tmp/presto-example.properties" +``` + +The output should contain the two `AMER` rows and only three columns. On +Windows, use `.\mvnw.cmd` and a URL such as +`file:///C:/Temp/presto-example.properties`. + +## Troubleshooting + +- **Connection refused:** check the host, port, and network access from the + Wayang machine. +- **Catalog, schema, or table not found:** use a fully qualified table name and + confirm it with `SELECT * FROM catalog.schema.table` in a Presto client. +- **Authentication failed:** set the user, password, and any TLS or access-token + options required by your Presto JDBC endpoint. diff --git a/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java new file mode 100644 index 000000000..c709fc1b5 --- /dev/null +++ b/wayang-applications/src/main/java/org/apache/wayang/applications/BigQueryDemo.java @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.wayang.applications; + +import org.apache.wayang.basic.data.Record; +import org.apache.wayang.basic.function.ProjectionDescriptor; +import org.apache.wayang.basic.operators.FilterOperator; +import org.apache.wayang.basic.operators.LocalCallbackSink; +import org.apache.wayang.basic.operators.MapOperator; +import org.apache.wayang.basic.types.RecordType; +import org.apache.wayang.bigquery.BigQuery; +import org.apache.wayang.bigquery.operators.BigQueryTableSource; +import org.apache.wayang.bigquery.platform.BigQueryPlatform; +import org.apache.wayang.core.api.Configuration; +import org.apache.wayang.core.api.WayangContext; +import org.apache.wayang.core.function.PredicateDescriptor; +import org.apache.wayang.core.plan.wayangplan.WayangPlan; +import org.apache.wayang.core.types.DataSetType; +import org.apache.wayang.java.Java; + +import java.util.ArrayList; +import java.util.List; + +/** + * Configurable example for the Wayang BigQuery connector. + * + *

The optional second argument selects a mode: + *

    + *
  • {@code cost}: cost model output (no credentials needed)
    • + *
  • {@code filter}: filter operator pushdown
    • + *
  • {@code projection}: projection and filter operator pushdown
    • + *
+ * + *

See {@code wayang-applications/bigquery.md} for configuration and usage. + */ +public class BigQueryDemo { + + private static Configuration configuration; + private static String jdbcUrl; + private static String sourceTable; + + public static void main(String[] args) { + if (args.length < 1 || args.length > 2) { + throw new IllegalArgumentException( + "Usage: BigQueryDemo [cost|filter|projection|all]" + ); + } + configuration = new Configuration(args[0]); + jdbcUrl = configuration.getStringProperty("wayang.bigquery.jdbc.url", ""); + sourceTable = configuration.getStringProperty( + "wayang.bigquery.demo.table", "`my-project.sales.orders`" + ); + String mode = args.length == 2 ? args[1] : "all"; + switch (mode) { + case "cost": costModel(); break; + case "filter": filterDemo(); break; + case "projection": projectionDemo(); break; + case "all": + costModel(); + filterDemo(); + projectionDemo(); + break; + default: + throw new IllegalArgumentException("Unknown BigQuery demo mode: " + mode); + } + } + + static void costModel() { + BigQueryPlatform.getInstance().configureDefaults(configuration); + + long mhz = configuration.getLongProperty("wayang.bigquery.cpu.mhz", 0); + long cores = configuration.getLongProperty("wayang.bigquery.cores", 0); + double fix = configuration.getDoubleProperty("wayang.bigquery.costs.fix", 0); + double perMs = configuration.getDoubleProperty("wayang.bigquery.costs.per-ms", 1); + + long rows = 10; + long alpha = 5; + long beta = 2_000_000; + long cpuCycles = alpha * rows + beta; + double timeMs = cpuCycles / (cores * mhz * 1000.0); + double cost = fix + perMs * timeMs; + + System.out.println(); + System.out.println("BigQuery cost model"); + System.out.println(); + System.out.println(" Layer 1: cost formula (wayang-bigquery-defaults.properties)"); + System.out.printf(" tablesource : %s%n", configuration.getStringProperty("wayang.bigquery.tablesource.load", null)); + System.out.printf(" filter : %s%n", configuration.getStringProperty("wayang.bigquery.filter.load", null)); + System.out.println(); + System.out.println(" Layer 2: hardware profile (CPU cycles to wall-clock time)"); + System.out.printf(" cpu.mhz = %d cores = %d%n", mhz, cores); + System.out.println(); + System.out.println(" Layer 3: time to abstract cost"); + System.out.printf(" costs.fix = %.1f costs.per-ms = %.1f%n", fix, perMs); + System.out.println(); + System.out.println(" -- Worked example: 10-row table scan --"); + System.out.printf(" alpha = %d (per-row, serverless columnar)%n", alpha); + System.out.printf(" beta = %,d (cold-start / slot reservation)%n", beta); + System.out.printf(" cpu cycles = %d * %d + %,d = %,d%n", alpha, rows, beta, cpuCycles); + System.out.printf(" time = %,d / (%d * %d * 1000) = %.4f ms%n", cpuCycles, cores, mhz, timeMs); + System.out.printf(" cost = %.1f + %.1f * %.4f = %.4f%n", fix, perMs, timeMs, cost); + System.out.println(); + System.out.println(); + } + + static void filterDemo() { + requireJdbcUrl(); + String table = sourceTable; + + System.out.println(); + System.out.println("BigQuery filter pushdown"); + System.out.println(); + System.out.println(" Operator: FilterOperator -> BigQueryFilterOperator"); + System.out.printf(" SQL sent: SELECT * FROM %s%n", table); + System.out.println(" WHERE region = 'AMER'"); + System.out.println(); + + runLiveFilter(table); + + System.out.println(); + } + + private static void runLiveFilter(String table) { + WayangContext wayang = buildWayang(); + List results = new ArrayList<>(); + + BigQueryTableSource source = new BigQueryTableSource( + table, "order_id", "region", "product", "amount", "order_date" + ); + FilterOperator filter = new FilterOperator<>( + new PredicateDescriptor<>( + r -> "AMER".equals(r.getField(1)), Record.class + ).withSqlImplementation("region = 'AMER'") + ); + LocalCallbackSink sink = LocalCallbackSink.createCollectingSink(results, Record.class); + source.connectTo(0, filter, 0); + filter.connectTo(0, sink, 0); + wayang.execute("BigQuery-Filter-Demo", new WayangPlan(sink)); + + System.out.println(" Results returned by Wayang:"); + System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", + "order_id", "region", "product", "amount", "order_date"); + System.out.println(" " + repeat('-', 54)); + for (Record r : results) { + System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", + r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4)); + } + System.out.println(); + System.out.printf(" %d AMER rows returned.%n%n", results.size()); + } + + static void projectionDemo() { + requireJdbcUrl(); + String table = sourceTable; + + System.out.println(); + System.out.println("BigQuery projection pushdown"); + System.out.println(); + System.out.println(" Operators: FilterOperator -> BigQueryFilterOperator"); + System.out.println(" MapOperator -> BigQueryProjectionOperator"); + System.out.printf(" SQL sent: SELECT region, product, amount%n"); + System.out.printf(" FROM %s%n", table); + System.out.println(" WHERE region = 'AMER'"); + System.out.println(); + System.out.println(" Both operators are combined in one SQL query; only 3 of 5"); + System.out.println(" columns transferred; order_id + order_date never leave BQ."); + System.out.println(); + + runLiveProjection(table); + + System.out.println(); + } + + private static void runLiveProjection(String table) { + WayangContext wayang = buildWayang(); + List results = new ArrayList<>(); + + BigQueryTableSource source = new BigQueryTableSource( + table, "order_id", "region", "product", "amount", "order_date" + ); + FilterOperator filter = new FilterOperator<>( + new PredicateDescriptor<>( + r -> "AMER".equals(r.getField(1)), Record.class + ).withSqlImplementation("region = 'AMER'") + ); + // Use the Record-specific descriptor for a projection with multiple fields. + MapOperator projection = new MapOperator<>( + ProjectionDescriptor.createForRecords( + new RecordType("order_id", "region", "product", "amount", "order_date"), + "region", "product", "amount"), + DataSetType.createDefault(Record.class), + DataSetType.createDefault(Record.class) + ); + LocalCallbackSink sink = LocalCallbackSink.createCollectingSink(results, Record.class); + source.connectTo(0, filter, 0); + filter.connectTo(0, projection, 0); + projection.connectTo(0, sink, 0); + wayang.execute("BigQuery-Projection-Demo", new WayangPlan(sink)); + + System.out.println(" Results returned by Wayang (projected columns only):"); + System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount"); + System.out.println(" " + repeat('-', 30)); + for (Record r : results) { + System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2)); + } + System.out.println(); + System.out.printf(" %d AMER rows returned with 3 columns.%n%n", + results.size()); + } + + private static WayangContext buildWayang() { + return new WayangContext(configuration) + .withPlugin(Java.basicPlugin()) + .withPlugin(BigQuery.plugin()); + } + + private static void requireJdbcUrl() { + if (jdbcUrl.isEmpty()) { + throw new IllegalArgumentException( + "wayang.bigquery.jdbc.url is required for filter and projection modes." + ); + } + } + + private static String repeat(char c, int n) { + StringBuilder sb = new StringBuilder(n); + for (int i = 0; i < n; i++) sb.append(c); + return sb.toString(); + } +} diff --git a/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java new file mode 100644 index 000000000..c9590df54 --- /dev/null +++ b/wayang-applications/src/main/java/org/apache/wayang/applications/PrestoDemo.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.wayang.applications; + +import org.apache.wayang.basic.data.Record; +import org.apache.wayang.basic.function.ProjectionDescriptor; +import org.apache.wayang.basic.operators.FilterOperator; +import org.apache.wayang.basic.operators.LocalCallbackSink; +import org.apache.wayang.basic.operators.MapOperator; +import org.apache.wayang.basic.types.RecordType; +import org.apache.wayang.core.api.Configuration; +import org.apache.wayang.core.api.WayangContext; +import org.apache.wayang.core.function.PredicateDescriptor; +import org.apache.wayang.core.plan.wayangplan.WayangPlan; +import org.apache.wayang.core.types.DataSetType; +import org.apache.wayang.java.Java; +import org.apache.wayang.presto.Presto; +import org.apache.wayang.presto.operators.PrestoTableSource; + +import java.util.ArrayList; +import java.util.List; + +/** Configurable filter and projection example for the Wayang Presto connector. */ +public class PrestoDemo { + + public static void main(String[] args) { + if (args.length != 1) { + throw new IllegalArgumentException("Usage: PrestoDemo "); + } + + Configuration configuration = new Configuration(args[0]); + configuration.getStringProperty("wayang.presto.jdbc.url"); + String table = configuration.getStringProperty( + "wayang.presto.demo.table", "memory.default.orders" + ); + + WayangContext wayang = new WayangContext(configuration) + .withPlugin(Java.basicPlugin()) + .withPlugin(Presto.plugin()); + List results = new ArrayList<>(); + + PrestoTableSource source = new PrestoTableSource( + table, "order_id", "region", "product", "amount", "order_date" + ); + FilterOperator filter = new FilterOperator<>( + new PredicateDescriptor<>( + record -> "AMER".equals(record.getField(1)), Record.class + ).withSqlImplementation("region = 'AMER'") + ); + MapOperator projection = new MapOperator<>( + ProjectionDescriptor.createForRecords( + new RecordType("order_id", "region", "product", "amount", "order_date"), + "region", "product", "amount" + ), + DataSetType.createDefault(Record.class), + DataSetType.createDefault(Record.class) + ); + LocalCallbackSink sink = LocalCallbackSink.createCollectingSink(results, Record.class); + + source.connectTo(0, filter, 0); + filter.connectTo(0, projection, 0); + projection.connectTo(0, sink, 0); + wayang.execute("Presto-Filter-Projection-Demo", new WayangPlan(sink)); + + System.out.printf("%-8s %-16s %10s%n", "region", "product", "amount"); + for (Record result : results) { + System.out.printf("%-8s %-16s %10s%n", + result.getField(0), result.getField(1), result.getField(2)); + } + System.out.printf("%n%d AMER rows returned from %s.%n", results.size(), table); + } +} diff --git a/wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java b/wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java similarity index 50% rename from wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java rename to wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java index 9becd061d..c0017e9e3 100644 --- a/wayang-platforms/wayang-trino/src/main/java/org/apache/wayang/trino/TrinoDemo.java +++ b/wayang-applications/src/main/java/org/apache/wayang/applications/TrinoDemo.java @@ -16,7 +16,7 @@ * limitations under the License. */ -package org.apache.wayang.trino; +package org.apache.wayang.applications; import org.apache.wayang.basic.data.Record; import org.apache.wayang.basic.function.ProjectionDescriptor; @@ -30,62 +30,55 @@ import org.apache.wayang.core.plan.wayangplan.WayangPlan; import org.apache.wayang.core.types.DataSetType; import org.apache.wayang.java.Java; +import org.apache.wayang.trino.Trino; import org.apache.wayang.trino.operators.TrinoTableSource; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.ResultSet; import java.util.ArrayList; import java.util.List; -import java.util.Properties; /** - * Standalone demo for the Wayang Trino connector. + * Configurable example for the Wayang Trino connector. * *

Demonstrates two Trino operator types: *

    - *
  1. Seg 3 — Filter pushdown: WHERE region = 'AMER'.
    2. - *
  2. Seg 4 — Projection + Filter pushdown: + *
  3. Filter pushdown: WHERE region = 'AMER'.
    4. + *
  4. Projection and filter pushdown: * SELECT region, product, amount ... WHERE region = 'AMER'.
    5. *
* - *

Run with: - * 

- *   cd /path/to/wayang
- *   mvn exec:java -pl wayang-platforms/wayang-trino \
- *     -Dexec.mainClass=org.apache.wayang.trino.TrinoDemo \
- *     -Pskip-prerequisite-check -Drat.skip=true \
- *     [-Dtrino.url=jdbc:trino://localhost:8080] [-Dtrino.user=admin]
- *
+ *

See {@code wayang-applications/trino.md} for configuration and usage. */ public class TrinoDemo { - private static final String JDBC_URL = System.getProperty("trino.url", "jdbc:trino://localhost:8080"); - private static final String JDBC_USER = System.getProperty("trino.user", "admin"); + private static Configuration configuration; + private static String sourceTable; public static void main(String[] args) throws Exception { - seg3Filter(); - seg4Projection(); + if (args.length != 1) { + throw new IllegalArgumentException("Usage: TrinoDemo "); + } + configuration = new Configuration(args[0]); + configuration.getStringProperty("wayang.trino.jdbc.url"); + sourceTable = configuration.getStringProperty( + "wayang.trino.demo.table", "iceberg.sales.orders" + ); + filterDemo(); + projectionDemo(); } - // ── Seg 3 — Filter pushdown ─────────────────────────────────────────────── - - static void seg3Filter() throws Exception { - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(" Seg 3 — Filter Operator Pushdown"); - System.out.println("══════════════════════════════════════════════════════"); + static void filterDemo() throws Exception { + System.out.println("Trino filter pushdown"); System.out.println(); System.out.println(" Operator: FilterOperator -> TrinoFilterOperator"); - System.out.println(" SQL sent: SELECT * FROM iceberg.sales.orders"); + System.out.println(" SQL sent: SELECT * FROM " + sourceTable); System.out.println(" WHERE region = 'AMER'"); System.out.println(); - Configuration config = buildConfig(); - WayangContext wayang = buildWayang(config); + WayangContext wayang = buildWayang(); List results = new ArrayList<>(); TrinoTableSource source = new TrinoTableSource( - "iceberg.sales.orders", "order_id", "region", "product", "amount", "order_date" + sourceTable, "order_id", "region", "product", "amount", "order_date" ); FilterOperator filter = new FilterOperator<>( new PredicateDescriptor<>( @@ -107,48 +100,35 @@ static void seg3Filter() throws Exception { r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4)); } System.out.println(); - System.out.printf(" ✓ %d AMER rows — filter pushed to Trino as SQL WHERE clause%n", results.size()); - - verifyInQueryHistory("iceberg.sales.orders"); - - System.out.println("══════════════════════════════════════════════════════"); + System.out.printf(" %d AMER rows returned.%n", results.size()); System.out.println(); } - // ── Seg 4 — Projection + Filter pushdown ───────────────────────────────── - - static void seg4Projection() throws Exception { - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(" Seg 4 — Projection Operator Pushdown"); - System.out.println("══════════════════════════════════════════════════════"); + static void projectionDemo() throws Exception { + System.out.println("Trino projection pushdown"); System.out.println(); System.out.println(" Operators: FilterOperator -> TrinoFilterOperator"); System.out.println(" MapOperator -> TrinoProjectionOperator"); System.out.println(" SQL sent: SELECT region, product, amount"); - System.out.println(" FROM iceberg.sales.orders"); + System.out.println(" FROM " + sourceTable); System.out.println(" WHERE region = 'AMER'"); System.out.println(); - System.out.println(" Both operators get pushed into a single SQL query —"); + System.out.println(" Both operators get pushed into a single SQL query;"); System.out.println(" no unnecessary columns are transferred over the network."); System.out.println(); - Configuration config = buildConfig(); - WayangContext wayang = buildWayang(config); + WayangContext wayang = buildWayang(); List results = new ArrayList<>(); TrinoTableSource source = new TrinoTableSource( - "iceberg.sales.orders", "order_id", "region", "product", "amount", "order_date" + sourceTable, "order_id", "region", "product", "amount", "order_date" ); FilterOperator filter = new FilterOperator<>( new PredicateDescriptor<>( r -> "AMER".equals(r.getField(1)), Record.class ).withSqlImplementation("region = 'AMER'") ); - // Use the Record-aware projection (multi-field). The plain - // ProjectionDescriptor(Class, Class, fields...) builds a POJO projection - // whose Java implementation only supports a single field; createForRecords - // yields a multi-field Record implementation that also works if Wayang - // executes the projection on the Java side instead of pushing it to Trino. + // Use the Record-specific descriptor for a projection with multiple fields. MapOperator projection = new MapOperator<>( ProjectionDescriptor.createForRecords( new RecordType("order_id", "region", "product", "amount", "order_date"), @@ -170,52 +150,17 @@ static void seg4Projection() throws Exception { System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2)); } System.out.println(); - System.out.printf(" ✓ %d AMER rows — only 3 of 5 columns fetched (projection pushed to SQL)%n", + System.out.printf(" %d AMER rows returned with 3 columns.%n", results.size()); - - verifyInQueryHistory("iceberg.sales.orders"); - - System.out.println("══════════════════════════════════════════════════════"); System.out.println(); } - // ── Shared helpers ──────────────────────────────────────────────────────── - - private static Configuration buildConfig() { - Configuration config = new Configuration(); - config.setProperty("wayang.trino.jdbc.url", JDBC_URL); - config.setProperty("wayang.trino.jdbc.user", JDBC_USER); - config.setProperty("wayang.trino.jdbc.password", ""); - return config; - } - - private static WayangContext buildWayang(Configuration config) { - return new WayangContext(config) + private static WayangContext buildWayang() { + return new WayangContext(configuration) .withPlugin(Java.basicPlugin()) .withPlugin(Trino.plugin()); } - private static void verifyInQueryHistory(String tableHint) throws Exception { - System.out.println(); - System.out.println(" Checking Trino's system.runtime.queries for proof..."); - Properties props = new Properties(); - props.setProperty("user", JDBC_USER); - try (Connection conn = DriverManager.getConnection(JDBC_URL, props)) { - ResultSet rs = conn.createStatement().executeQuery( - "SELECT query FROM system.runtime.queries " + - "WHERE state = 'FINISHED' AND query LIKE '%" + tableHint + "%' " + - "ORDER BY created DESC LIMIT 2" - ); - System.out.println(); - System.out.println(" Last SQL Trino executed:"); - while (rs.next()) { - System.out.println(" > " + rs.getString(1).replaceAll("\\s+", " ")); - } - } - System.out.println(); - System.out.println(" ✓ Wayang-assembled SQL confirmed in Trino query history."); - } - private static String repeat(char c, int n) { StringBuilder sb = new StringBuilder(n); for (int i = 0; i < n; i++) sb.append(c); diff --git a/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java b/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java new file mode 100644 index 000000000..3394639f1 --- /dev/null +++ b/wayang-applications/src/test/java/org/apache/wayang/applications/SqlPlatformDemoTest.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.wayang.applications; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.nio.file.Path; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class SqlPlatformDemoTest { + + @TempDir + Path directory; + + @Test + void bigQueryCostModeLoadsWayangConfiguration() throws Exception { + Path configuration = directory.resolve("bigquery.properties"); + Files.writeString(configuration, "wayang.bigquery.demo.table = `project.dataset.orders`\n"); + + assertDoesNotThrow(() -> BigQueryDemo.main( + new String[]{configuration.toUri().toString(), "cost"} + )); + } + + @Test + void demosRequireConfigurationUrls() { + assertThrows(IllegalArgumentException.class, () -> BigQueryDemo.main(new String[0])); + assertThrows(IllegalArgumentException.class, () -> PrestoDemo.main(new String[0])); + assertThrows(IllegalArgumentException.class, () -> TrinoDemo.main(new String[0])); + } + + @Test + void bigQueryOperatorModesRequireJdbcConfiguration() throws Exception { + Path configuration = directory.resolve("bigquery-no-jdbc.properties"); + Files.writeString(configuration, "wayang.bigquery.demo.table = `project.dataset.orders`\n"); + + assertThrows(IllegalArgumentException.class, () -> BigQueryDemo.main( + new String[]{configuration.toUri().toString(), "filter"} + )); + } +} diff --git a/wayang-applications/trino.md b/wayang-applications/trino.md new file mode 100644 index 000000000..7da14376a --- /dev/null +++ b/wayang-applications/trino.md @@ -0,0 +1,101 @@ + + +# Trino Example + +This example runs filter and projection plans against an existing Trino table. +Wayang pushes `region = 'AMER'` and the selected columns into SQL executed by +Trino. + +## Prerequisites + +- Java 17 +- A Trino deployment reachable from the machine running Wayang +- A writable catalog for preparing the sample data, or an existing readable table +- A Trino user with `SELECT` permission on the example table + +Wayang does not start Trino, its catalog, or its storage services. + +## Prepare the table + +Using the Trino CLI or another SQL client, replace `iceberg.sales` with a catalog +and schema available in your deployment, then run: + +```sql +CREATE SCHEMA IF NOT EXISTS iceberg.sales; + +CREATE TABLE iceberg.sales.orders ( + order_id BIGINT, + region VARCHAR, + product VARCHAR, + amount DOUBLE, + order_date VARCHAR +); + +INSERT INTO iceberg.sales.orders VALUES + (1, 'AMER', 'book', 25.50, '2026-01-10'), + (2, 'EMEA', 'desk', 300.00, '2026-01-11'), + (3, 'AMER', 'chair', 85.25, '2026-01-12'); +``` + +If the catalog is read-only, point the example at an existing table with these +five columns in this order. + +## Configure Wayang + +Create a properties file such as `/tmp/trino-example.properties`: + +```properties +wayang.trino.jdbc.url = jdbc:trino://trino.example.com:8080/iceberg/sales +wayang.trino.jdbc.user = wayang +wayang.trino.jdbc.password = +wayang.trino.demo.table = iceberg.sales.orders +``` + +Change the endpoint, credentials, catalog, schema, and table for your deployment. +Trino JDBC options such as SSL can be added to the JDBC URL. + +## Build and run + +Build the application and its dependencies from the repository root: + +```bash +./mvnw -Pskip-prerequisite-check -pl wayang-applications -am \ + -DskipTests -Dpython.worker.tests.skip=true install +``` + +Run the example: + +```bash +./mvnw -Pskip-prerequisite-check -pl wayang-applications exec:java \ + -Dexec.mainClass=org.apache.wayang.applications.TrinoDemo \ + "-Dexec.args=file:///tmp/trino-example.properties" +``` + +Both sections should return the two `AMER` rows. The projection section returns +only `region`, `product`, and `amount`. On Windows, replace `./mvnw` with +`.\mvnw.cmd` and use a URL such as `file:///C:/Temp/trino-example.properties`. + +## Troubleshooting + +- **Connection refused:** verify the host, port, and network access from Wayang. +- **Catalog, schema, or table not found:** check the fully qualified table name + with the Trino CLI. +- **Authentication or TLS failure:** add the credentials and JDBC URL options + required by the deployment. +- **Create table is unsupported:** prepare the five columns in a catalog that + supports writes, or use an existing table. diff --git a/wayang-platforms/wayang-bigquery/README.md b/wayang-platforms/wayang-bigquery/README.md new file mode 100644 index 000000000..9e6d479c6 --- /dev/null +++ b/wayang-platforms/wayang-bigquery/README.md @@ -0,0 +1,51 @@ + + +# Wayang Platform BigQuery + +This module connects Wayang to a user-managed Google BigQuery project through +JDBC. Supply the project and authentication settings in the JDBC URL; the +example format is documented in `wayang-bigquery-defaults.properties`. + +```properties +wayang.bigquery.jdbc.url = jdbc:bigquery://https://www.googleapis.com/bigquery/v2;ProjectId=my-project;OAuthType=0;OAuthServiceAcctEmail=service-account@example.com;OAuthPvtKeyPath=/path/to/key.json +wayang.bigquery.jdbc.user = +wayang.bigquery.jdbc.password = +``` + +The runnable filter/projection example and its table requirements are documented +in [`wayang-applications/bigquery.md`](../../wayang-applications/bigquery.md). + +## Integration tests + +`BigQueryOperatorsIT` is intended for connector development. The service +account must be able to create query jobs and create, read, and delete tables in +the selected dataset. The test creates its fixtures and removes their tables +afterward; it leaves the dataset in place. + +Configure the test with environment variables and run it from the repository +root: + +```bash +BIGQUERY_PROJECT=my-project \ +BIGQUERY_SA_EMAIL=wayang-bq@my-project.iam.gserviceaccount.com \ +BIGQUERY_KEY_PATH=/absolute/path/to/key.json \ +BIGQUERY_DATASET=wayang_it \ +BIGQUERY_LOCATION=US \ +./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-bigquery -am \ + -Dtest=BigQueryOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ + -DfailIfNoTests=false test +``` + +The equivalent system properties are `bigquery.project`, `bigquery.saEmail`, +`bigquery.keyPath`, `bigquery.dataset`, and `bigquery.location`. These settings +configure the integration test only; applications use the +`wayang.bigquery.jdbc.*` properties described above. + +Cost calibration uses `BigQueryCostPilotIT`; see +[`guides/cost-profiling.md`](../../guides/cost-profiling.md). diff --git a/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java b/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java deleted file mode 100644 index 18349d3db..000000000 --- a/wayang-platforms/wayang-bigquery/src/main/java/org/apache/wayang/bigquery/BigQueryDemo.java +++ /dev/null @@ -1,307 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.wayang.bigquery; - -import org.apache.wayang.basic.data.Record; -import org.apache.wayang.basic.function.ProjectionDescriptor; -import org.apache.wayang.basic.operators.FilterOperator; -import org.apache.wayang.basic.operators.LocalCallbackSink; -import org.apache.wayang.basic.operators.MapOperator; -import org.apache.wayang.basic.types.RecordType; -import org.apache.wayang.bigquery.operators.BigQueryTableSource; -import org.apache.wayang.bigquery.platform.BigQueryPlatform; -import org.apache.wayang.core.api.Configuration; -import org.apache.wayang.core.api.WayangContext; -import org.apache.wayang.core.function.PredicateDescriptor; -import org.apache.wayang.core.plan.wayangplan.WayangPlan; -import org.apache.wayang.core.types.DataSetType; -import org.apache.wayang.java.Java; - -import java.util.ArrayList; -import java.util.List; - -/** - * Standalone demo for the Wayang BigQuery connector. - * - *

Controlled by {@code -Dbigquery.mode}: - *

    - *
  • {@code cost} — three-layer cost model (no credentials needed)
    • - *
  • {@code filter} — filter operator pushdown demo
    • - *
  • {@code projection} — projection + filter operator pushdown demo
    • - *
- * - *

Run with: - * 

- *   mvn exec:java -pl wayang-platforms/wayang-bigquery \
- *     -Dexec.mainClass=org.apache.wayang.bigquery.BigQueryDemo \
- *     -Dbigquery.mode=cost \
- *     -Pskip-prerequisite-check -Drat.skip=true
- *
- */ -public class BigQueryDemo { - - private static final String MODE = System.getProperty("bigquery.mode", "cost"); - private static final String JDBC_URL = System.getProperty("bigquery.url", ""); - private static final String PROJECT = System.getProperty("bigquery.project", "my-project"); - - // 20-row dataset: 4 regions (AMER/APAC/EMEA/LATAM), 5 products (Widget A-E) - // AMER rows: 3, 6, 9, 12, 16 → 5 rows for filter demo - private static final String[][] SAMPLE_DATA = { - {"1", "APAC", "Widget A", "1500.00", "2024-01-15"}, - {"2", "EMEA", "Widget B", "800.50", "2024-01-16"}, - {"3", "AMER", "Widget A", "2200.00", "2024-01-17"}, - {"4", "APAC", "Widget C", "350.75", "2024-01-18"}, - {"5", "EMEA", "Widget A", "1100.00", "2024-01-19"}, - {"6", "AMER", "Widget B", "950.25", "2024-01-20"}, - {"7", "APAC", "Widget B", "1750.00", "2024-01-21"}, - {"8", "EMEA", "Widget C", "420.00", "2024-01-22"}, - {"9", "AMER", "Widget C", "680.50", "2024-01-23"}, - {"10", "APAC", "Widget A", "3000.00", "2024-01-24"}, - {"11", "LATAM", "Widget D", "560.00", "2024-01-25"}, - {"12", "AMER", "Widget D", "1320.75", "2024-01-26"}, - {"13", "EMEA", "Widget D", "990.00", "2024-01-27"}, - {"14", "LATAM", "Widget E", "2100.50", "2024-01-28"}, - {"15", "APAC", "Widget E", "4500.00", "2024-01-29"}, - {"16", "AMER", "Widget E", "3750.00", "2024-01-30"}, - {"17", "EMEA", "Widget E", "1250.00", "2024-01-31"}, - {"18", "LATAM", "Widget A", "870.25", "2024-02-01"}, - {"19", "APAC", "Widget D", "1680.00", "2024-02-02"}, - {"20", "LATAM", "Widget B", "440.50", "2024-02-03"}, - }; - - public static void main(String[] args) { - switch (MODE) { - case "cost": costModel(); break; - case "filter": filterDemo(); break; - case "projection": projectionDemo(); break; - default: - costModel(); - filterDemo(); - projectionDemo(); - } - } - - // ── Cost model ──────────────────────────────────────────────────────────── - - static void costModel() { - Configuration config = new Configuration(); - BigQueryPlatform.getInstance().configureDefaults(config); - - long mhz = config.getLongProperty("wayang.bigquery.cpu.mhz", 0); - long cores = config.getLongProperty("wayang.bigquery.cores", 0); - double fix = config.getDoubleProperty("wayang.bigquery.costs.fix", 0); - double perMs = config.getDoubleProperty("wayang.bigquery.costs.per-ms", 1); - - long rows = 10; - long alpha = 5; - long beta = 2_000_000; - long cpuCycles = alpha * rows + beta; - double timeMs = cpuCycles / (cores * mhz * 1000.0); - double cost = fix + perMs * timeMs; - - System.out.println(); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(" BigQuery — Cost Model Integration"); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - System.out.println(" LAYER 1 — Cost formula (wayang-bigquery-defaults.properties)"); - System.out.printf(" tablesource : %s%n", config.getStringProperty("wayang.bigquery.tablesource.load", null)); - System.out.printf(" filter : %s%n", config.getStringProperty("wayang.bigquery.filter.load", null)); - System.out.println(); - System.out.println(" LAYER 2 — Hardware profile (cpu cycles -> wall-clock ms)"); - System.out.printf(" cpu.mhz = %d cores = %d%n", mhz, cores); - System.out.println(); - System.out.println(" LAYER 3 — Time -> abstract cost"); - System.out.printf(" costs.fix = %.1f costs.per-ms = %.1f%n", fix, perMs); - System.out.println(); - System.out.println(" -- Worked example: 10-row table scan --"); - System.out.printf(" alpha = %d (per-row, serverless columnar)%n", alpha); - System.out.printf(" beta = %,d (cold-start / slot reservation)%n", beta); - System.out.printf(" cpu cycles = %d * %d + %,d = %,d%n", alpha, rows, beta, cpuCycles); - System.out.printf(" time = %,d / (%d * %d * 1000) = %.4f ms%n", cpuCycles, cores, mhz, timeMs); - System.out.printf(" cost = %.1f + %.1f * %.4f = %.4f%n", fix, perMs, timeMs, cost); - System.out.println(); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - } - - // ── Filter pushdown ─────────────────────────────────────────────────────── - - static void filterDemo() { - String table = String.format("`%s.sales.orders`", PROJECT); - - System.out.println(); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(" BigQuery — Filter Operator Pushdown"); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - System.out.println(" Operator: FilterOperator -> BigQueryFilterOperator"); - System.out.printf(" SQL sent: SELECT * FROM %s%n", table); - System.out.println(" WHERE region = 'AMER'"); - System.out.println(); - - if (!JDBC_URL.isEmpty()) { - runLiveFilter(table); - } else { - System.out.println(" Results (20-row dataset, AMER rows only):"); - System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", - "order_id", "region", "product", "amount", "order_date"); - System.out.println(" " + repeat('-', 54)); - int count = 0; - for (String[] row : SAMPLE_DATA) { - if ("AMER".equals(row[1])) { - System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", - row[0], row[1], row[2], row[3], row[4]); - count++; - } - } - System.out.println(); - System.out.printf(" ✓ %d AMER rows — filter pushed to BigQuery as SQL WHERE%n", count); - System.out.println(" (pass -Dbigquery.url=... for live execution)"); - } - - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - } - - private static void runLiveFilter(String table) { - WayangContext wayang = buildWayang(); - List results = new ArrayList<>(); - - BigQueryTableSource source = new BigQueryTableSource( - table, "order_id", "region", "product", "amount", "order_date" - ); - FilterOperator filter = new FilterOperator<>( - new PredicateDescriptor<>( - r -> "AMER".equals(r.getField(1)), Record.class - ).withSqlImplementation("region = 'AMER'") - ); - LocalCallbackSink sink = LocalCallbackSink.createCollectingSink(results, Record.class); - source.connectTo(0, filter, 0); - filter.connectTo(0, sink, 0); - wayang.execute("BigQuery-Filter-Demo", new WayangPlan(sink)); - - System.out.println(" Results returned by Wayang:"); - System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", - "order_id", "region", "product", "amount", "order_date"); - System.out.println(" " + repeat('-', 54)); - for (Record r : results) { - System.out.printf(" %-10s %-6s %-10s %10s %-12s%n", - r.getField(0), r.getField(1), r.getField(2), r.getField(3), r.getField(4)); - } - System.out.println(); - System.out.printf(" ✓ %d AMER rows via Wayang -> BigQuery SQL pushdown%n%n", results.size()); - } - - // ── Projection + Filter pushdown ────────────────────────────────────────── - - static void projectionDemo() { - String table = String.format("`%s.sales.orders`", PROJECT); - - System.out.println(); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(" BigQuery — Projection Operator Pushdown"); - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - System.out.println(" Operators: FilterOperator -> BigQueryFilterOperator"); - System.out.println(" MapOperator -> BigQueryProjectionOperator"); - System.out.printf(" SQL sent: SELECT region, product, amount%n"); - System.out.printf(" FROM %s%n", table); - System.out.println(" WHERE region = 'AMER'"); - System.out.println(); - System.out.println(" Both operators collapsed into one SQL — only 3 of 5"); - System.out.println(" columns transferred; order_id + order_date never leave BQ."); - System.out.println(); - - if (!JDBC_URL.isEmpty()) { - runLiveProjection(table); - } else { - System.out.println(" Results (projected: region, product, amount — AMER only):"); - System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount"); - System.out.println(" " + repeat('-', 30)); - int count = 0; - for (String[] row : SAMPLE_DATA) { - if ("AMER".equals(row[1])) { - System.out.printf(" %-6s %-10s %10s%n", row[1], row[2], row[3]); - count++; - } - } - System.out.println(); - System.out.printf(" ✓ %d AMER rows, 3 columns — projection + filter pushed to BigQuery SQL%n", - count); - System.out.println(" (pass -Dbigquery.url=... for live execution)"); - } - - System.out.println("══════════════════════════════════════════════════════"); - System.out.println(); - } - - private static void runLiveProjection(String table) { - WayangContext wayang = buildWayang(); - List results = new ArrayList<>(); - - BigQueryTableSource source = new BigQueryTableSource( - table, "order_id", "region", "product", "amount", "order_date" - ); - FilterOperator filter = new FilterOperator<>( - new PredicateDescriptor<>( - r -> "AMER".equals(r.getField(1)), Record.class - ).withSqlImplementation("region = 'AMER'") - ); - // Record-aware multi-field projection (see TrinoDemo for rationale). - MapOperator projection = new MapOperator<>( - ProjectionDescriptor.createForRecords( - new RecordType("order_id", "region", "product", "amount", "order_date"), - "region", "product", "amount"), - DataSetType.createDefault(Record.class), - DataSetType.createDefault(Record.class) - ); - LocalCallbackSink sink = LocalCallbackSink.createCollectingSink(results, Record.class); - source.connectTo(0, filter, 0); - filter.connectTo(0, projection, 0); - projection.connectTo(0, sink, 0); - wayang.execute("BigQuery-Projection-Demo", new WayangPlan(sink)); - - System.out.println(" Results returned by Wayang (projected columns only):"); - System.out.printf(" %-6s %-10s %10s%n", "region", "product", "amount"); - System.out.println(" " + repeat('-', 30)); - for (Record r : results) { - System.out.printf(" %-6s %-10s %10s%n", r.getField(0), r.getField(1), r.getField(2)); - } - System.out.println(); - System.out.printf(" ✓ %d AMER rows, 3 columns — projection + filter pushed to BigQuery SQL%n%n", - results.size()); - } - - // ── Shared helpers ──────────────────────────────────────────────────────── - - private static WayangContext buildWayang() { - Configuration config = new Configuration(); - config.setProperty("wayang.bigquery.jdbc.url", JDBC_URL); - return new WayangContext(config) - .withPlugin(Java.basicPlugin()) - .withPlugin(BigQuery.plugin()); - } - - private static String repeat(char c, int n) { - StringBuilder sb = new StringBuilder(n); - for (int i = 0; i < n; i++) sb.append(c); - return sb.toString(); - } -} diff --git a/wayang-platforms/wayang-presto/README.md b/wayang-platforms/wayang-presto/README.md new file mode 100644 index 000000000..268108928 --- /dev/null +++ b/wayang-platforms/wayang-presto/README.md @@ -0,0 +1,47 @@ + + +# Wayang Platform Presto + +This module connects Wayang to a user-managed PrestoDB deployment through JDBC. +It does not require Docker or a repository-provided Presto environment. + +Configure the connection in a Wayang properties file: + +```properties +wayang.presto.jdbc.url = jdbc:presto://presto.example.com:8080/catalog/schema +wayang.presto.jdbc.user = wayang +``` + +Add `wayang.presto.jdbc.password` only when the deployment requires one. + +For a complete table definition, sample data, and runnable Wayang plan, see +[`wayang-applications/presto.md`](../../wayang-applications/presto.md). + +## Integration tests + +`PrestoOperatorsIT` is intended for connector development. It requires a +writable `memory` catalog and permission to read `system.runtime.queries`. The +test creates the `memory.wayang_it` schema and its fixture tables, exercises the +supported JDBC operators, and removes the fixtures afterward. + +Set the endpoint when it differs from the defaults shown below, then run the +test from the repository root: + +```bash +PRESTO_HOST=localhost PRESTO_PORT=8080 PRESTO_USER=test \ +./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-presto -am \ + -Dtest=PrestoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \ + -DfailIfNoTests=false test +``` + +These environment variables configure the integration test only. Applications +use the `wayang.presto.jdbc.*` properties described above. + +Cost calibration uses `PrestoCostPilotIT`; see +[`guides/cost-profiling.md`](../../guides/cost-profiling.md). diff --git a/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties b/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties index d614fa087..3c63f1621 100644 --- a/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties +++ b/wayang-platforms/wayang-presto/src/main/resources/wayang-presto-defaults.properties @@ -18,7 +18,7 @@ # Connection (override per deployment in wayang.properties) # wayang.presto.jdbc.url = jdbc:presto://localhost:8080 # wayang.presto.jdbc.user = test -# wayang.presto.jdbc.password = +# wayang.presto.jdbc.password = (optional; omit when no password is required) wayang.presto.jdbc.driverName = com.facebook.presto.jdbc.PrestoDriver # Hardware profile used by LoadProfileToTimeConverter. diff --git a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java index 0020a77c2..1ab4fb7c9 100644 --- a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java +++ b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoCostPilotIT.java @@ -71,7 +71,7 @@ class PrestoCostPilotIT { private static final String HOST = System.getenv().getOrDefault("PRESTO_HOST", "localhost"); - private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8081")); + private static final int PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8080")); private static final String USER = System.getenv().getOrDefault("PRESTO_USER", "test"); private static final String JDBC_URL = String.format("jdbc:presto://%s:%d/memory", HOST, PORT); diff --git a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java index d92f30cf6..2ab989a2a 100644 --- a/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java +++ b/wayang-platforms/wayang-presto/src/test/java/org/apache/wayang/presto/PrestoOperatorsIT.java @@ -85,8 +85,8 @@ * {@code CREATE TABLE ... AS SELECT} ran entirely inside Presto. * *

Prerequisites: a Presto reachable at {@code PRESTO_HOST:PRESTO_PORT} - * (defaults {@code localhost:8081}) with the {@code memory} connector enabled — - * e.g. {@code cd presto-setup && docker compose up -d}. If Presto is not reachable + * (defaults {@code localhost:8080}) with the {@code memory} connector enabled. + * If Presto is not reachable * the whole class is skipped (not failed). * * 

@@ -99,7 +99,7 @@
 class PrestoOperatorsIT {
 
     private static final String HOST = System.getenv().getOrDefault("PRESTO_HOST", "localhost");
-    private static final int    PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8081"));
+    private static final int    PORT = Integer.parseInt(System.getenv().getOrDefault("PRESTO_PORT", "8080"));
     private static final String USER = System.getenv().getOrDefault("PRESTO_USER", "test");
     private static final String JDBC_URL = String.format("jdbc:presto://%s:%d/memory", HOST, PORT);
 
diff --git a/wayang-platforms/wayang-trino/README.md b/wayang-platforms/wayang-trino/README.md
new file mode 100644
index 000000000..1376531e1
--- /dev/null
+++ b/wayang-platforms/wayang-trino/README.md
@@ -0,0 +1,44 @@
+
+
+# Wayang Platform Trino
+
+This module connects Wayang to a user-managed Trino deployment through JDBC.
+Configure the endpoint and credentials in a Wayang properties file:
+
+```properties
+wayang.trino.jdbc.url = jdbc:trino://trino.example.com:8080/catalog/schema
+wayang.trino.jdbc.user = wayang
+wayang.trino.jdbc.password =
+```
+
+The runnable filter/projection example and its table requirements are documented
+in [`wayang-applications/trino.md`](../../wayang-applications/trino.md).
+
+## Integration tests
+
+`TrinoOperatorsIT` is intended for connector development. It requires a
+writable catalog named `iceberg`, support for Parquet tables, permission to
+create the `iceberg.wayang_it` schema, and permission to read
+`system.runtime.queries`. It creates and removes its own fixture tables.
+
+Set the endpoint when it differs from the defaults shown below, then run the
+test from the repository root:
+
+```bash
+TRINO_HOST=localhost TRINO_PORT=8080 TRINO_USER=admin \
+./mvnw -Pskip-prerequisite-check -pl wayang-platforms/wayang-trino -am \
+  -Dtest=TrinoOperatorsIT -Dsurefire.failIfNoSpecifiedTests=false \
+  -DfailIfNoTests=false test
+```
+
+These environment variables configure the integration test only. Applications
+use the `wayang.trino.jdbc.*` properties described above.
+
+Cost calibration uses `TrinoCostPilotIT`; see
+[`guides/cost-profiling.md`](../../guides/cost-profiling.md).
diff --git a/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java b/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
index 283fefe8d..fcdb2bdca 100644
--- a/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
+++ b/wayang-platforms/wayang-trino/src/test/java/org/apache/wayang/trino/TrinoOperatorsIT.java
@@ -80,8 +80,8 @@
  * execution itself does not require the Java plugin. Result assertions use
  * plain JDBC only after the Wayang execution has completed.
  *
- * 
Prerequisites: a Trino reachable at {@code TRINO_HOST:TRINO_PORT}
- * (defaults {@code localhost:8080}); e.g. {@code cd platforms-setup-guides/trino-setup && docker compose up -d}.
+ * 
Prerequisites: a user-managed Trino reachable at
+ * {@code TRINO_HOST:TRINO_PORT} (defaults {@code localhost:8080}).
  * If Trino is not reachable the whole class is skipped (not failed).
  *
  * 
Run: