diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml index f5f89f9..732baed 100644 --- a/.github/workflows/maven.yml +++ b/.github/workflows/maven.yml @@ -34,12 +34,11 @@ jobs: mobilitydb-ref: master build-libmeos: "true" - - name: Derive the streaming-relevance baseline from the catalog - # The facade generator reads tools/baseline/streaming-relevance-baseline.json; derive - # it from the master catalog with the deterministic classifier so the generator input - # tracks master too. The baseline is gitignored — it only ever exists as a build artifact. + - name: Stage the derived MEOS catalog + # codegen_jvm.py reads tools/meos-idl.json; place the derived catalog there so its + # wiring is unchanged. The catalog is gitignored — it only ever exists as a build artifact. run: | - tools/regen_baseline.sh "${{ steps.provision.outputs.catalog-path }}" + cp "${{ steps.provision.outputs.catalog-path }}" tools/meos-idl.json echo "LD_LIBRARY_PATH=/usr/local/lib" >> "$GITHUB_ENV" - name: Check out JMEOS at master @@ -71,7 +70,7 @@ jobs: -DgroupId=org.jmeos -DartifactId=meos -Dversion=1.0 -Dpackaging=jar - name: Build + generate + smoke tests - # generate-sources runs the facade generator over the installed jar's surface ∩ baseline; + # generate-sources runs codegen_jvm.py --engine flink over the installed jar's full surface; # the smoke tests exercise the facades against the freshly built libmeos from /usr/local/lib. working-directory: flink-processor run: mvn -B -Dmeos.lib.dir=/usr/local/lib -Dmeos.enabled=true clean test diff --git a/.gitignore b/.gitignore index 96d7870..7ed2aac 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,6 @@ __pycache__/ # Maven build output flink-processor/target/ -# Derived generator input — the streaming-relevance baseline is regenerated in CI -# from the master MEOS catalog (tools/regen_baseline.sh), never committed. -tools/baseline/streaming-relevance-baseline.json +# Derived generator input — the MEOS catalog is staged in CI from upstream +# MobilityDB master (provision-meos), never committed. +tools/meos-idl.json diff --git a/GENERATION.md b/GENERATION.md index 00889ec..684c7d4 100644 --- a/GENERATION.md +++ b/GENERATION.md @@ -19,35 +19,35 @@ equivalence probe). ## MobilityFlink scope: generated MEOS facades over the JMEOS surface MobilityFlink is a **consumer** binding: it binds the **JMEOS jar** (the JVM FFI projection -of the catalog), not MEOS-API directly. Its generator **`tools/codegen_facades.py`** reads -the bundled JMEOS raw-FFI surface (intersected with the streaming-relevance baseline) and -emits the `org.mobilitydb.meos.MeosOps*` 1:1 forwarder facades the Flink processor consumes. -The facades are a *consumer* projection (they live here, not in JMEOS, so the JMEOS FFI line -and the facade line do not diverge). - -## The streaming-relevance baseline (generator input) - -`tools/codegen_facades.py` emits a facade only for functions in the -**streaming-relevant** tiers, read from `tools/baseline/streaming-relevance-baseline.json`. -That baseline is itself **generated and reproducible** — it is not a hand-maintained -list. It is produced by **`tools/classify_streaming_relevance.py`**, a deterministic -classifier: the tier of a function is decided purely by its name, its object-model role, -and its number of temporal parameters (zero per-function judgement), so the same MEOS -catalog always yields the same baseline. +of the catalog), not MEOS-API directly. Its generator is the shared +**`tools/codegen_jvm.py --engine flink`**, the single generator vendored identically by every +JVM binding (MobilitySpark, MobilityFlink, MobilityKafka); the `flink` and `kafka` engines +emit the `org.mobilitydb.meos.MeosOps*` 1:1 forwarder facades the streaming processors +consume. The facades are a *consumer* projection (they live here, not in JMEOS, so the JMEOS +FFI line and the facade line do not diverge). + +## Full surface, grouped by the catalog object model + +`codegen_jvm.py --engine flink` emits a facade for **every** function on the bundled JMEOS +`functions.GeneratedFunctions` surface, grouped by the MEOS-API catalog object model: one +`MeosOps` per object-model class plus one `MeosOpsFree
` per source header for +the free functions, with a shared `MeosOpsRuntime` that probes libmeos once per JVM. Each +forwarder carries a runtime guard: functions whose catalog return type is sequence-typed +(build a whole `TSequence`/`SeqSet`, inherently non-streamable) throw +`UnsupportedOperationException`; all others forward to `GeneratedFunctions` behind the +`MEOS_AVAILABLE` probe. The class/role/header are read straight from the catalog's +`objectModel`, and the sequence check from `returnType.canonical` — no separate classifier. The full input chain, from upstream MobilityDB master: ``` MobilityDB @ master → provision-meos (MEOS-API/run.py over the MEOS headers) → meos-idl.json (the catalog) - → tools/classify_streaming_relevance.py → tools/baseline/streaming-relevance-baseline.json - → tools/codegen_facades.py (jar-surface ∩ baseline) → org.mobilitydb.meos.MeosOps* facades + → tools/codegen_jvm.py --engine flink (full jar surface) → org.mobilitydb.meos.MeosOps* facades ``` -CI derives the baseline from the master catalog with `tools/regen_baseline.sh -` before the build; the baseline is gitignored, not committed. -Because the classifier is deterministic, the same catalog regenerates it byte-for-byte, so -a local refresh is: build the catalog from master with MEOS-API, then run the same script. +CI stages the master-derived catalog to `tools/meos-idl.json` before the build; the catalog +is gitignored, not committed. ## Generate-then-retire — the green-CI version is the probe @@ -56,10 +56,10 @@ never wipe-first**: regenerate, build green, **prove generated ⊇ hand** agains green-CI version** (the test suite + the streaming benchmark), then retire the hand path. The `MeosOps*` facades are emitted at build time and are **not committed**: Maven -`generate-sources` runs `tools/codegen_facades.py` into `target/generated-facades`, and -`build-helper` adds it as a source root. Only the generator and its `tools/` classifier are -tracked; the catalog, the baseline and the JMEOS jar are all derived in CI, and the sole -hand-written class under `org.mobilitydb.meos` is `MeosSetSetJoin`. +`generate-sources` runs `tools/codegen_jvm.py --engine flink` into `target/generated-facades`, +and `build-helper` adds it as a source root. Only the generator is tracked; the catalog and +the JMEOS jar are all derived in CI, and the sole hand-written class under +`org.mobilitydb.meos` is `MeosSetSetJoin`. ## Surface match diff --git a/flink-processor/pom.xml b/flink-processor/pom.xml index 2245525..92d8079 100644 --- a/flink-processor/pom.xml +++ b/flink-processor/pom.xml @@ -221,13 +221,15 @@ python3 - ${project.basedir}/../tools/codegen_facades.py + ${project.basedir}/../tools/codegen_jvm.py + --engine + flink + --catalog + ${project.basedir}/../tools/meos-idl.json --jar ${settings.localRepository}/org/jmeos/meos/1.0/meos-1.0.jar --out ${project.build.directory}/generated-facades - --engine - flink diff --git a/tools/classify_streaming_relevance.py b/tools/classify_streaming_relevance.py deleted file mode 100755 index 3690fff..0000000 --- a/tools/classify_streaming_relevance.py +++ /dev/null @@ -1,321 +0,0 @@ -#!/usr/bin/env python3 -"""Classify the MEOS public API into streaming tiers — the reproducible -producer of ``tools/baseline/streaming-relevance-baseline.json``. - -The baseline is the generator input consumed by ``codegen_facades.py`` -(which emits a facade only for the streaming-relevant tiers). This script -is deterministic: the tier of a function is decided purely by its name, -its object-model role, and the number of temporal parameters — there is -zero per-function judgement, so the same MEOS catalog always yields the -same baseline. - -Usage: - classify_streaming_relevance.py [-o ] - [--source-ref