[jackylee-ch]

Problem

The Glue/S3 Test job has been failing on essentially every PR with a wall of
TABLE_OR_VIEW_ALREADY_EXISTS (TableAlreadyExistsException from
GlueNamespace.declareTable), e.g.
Table already exists: default.test_table. It is not any individual PR's
fault — the shared test catalog is being poisoned.

Root cause: every run shares one Glue database, default, with hardcoded
table names (default.test_table, default.employees, …); only the S3 root
is unique per run. Two things follow. First, concurrent PR runs live in
different concurrency groups, so they execute at the same time and collide
directly on the shared table names → TABLE_OR_VIEW_ALREADY_EXISTS on
CREATE TABLE. Second, PR runs use cancel-in-progress: true, so pushing a new
commit cancels an in-flight job mid-test and a test's teardown DROP TABLE
never runs, leaving its entry in default; the cleanup step only ran
aws s3 rm and never deleted the Glue entry, so those orphaned entries
accumulate in the shared database for later runs to trip over.

Fix

Give every CI run its own Glue database, the same way the S3 root is already
per-run, and reap it afterwards:

• integration-tests/conftest.py and integration-tests/test_lance_spark.py:
  introduce LANCE_NAMESPACE = os.environ.get("LANCE_TEST_NAMESPACE", "default")
  and route every namespace-qualified reference through it. The default is
  "default", so local runs and every non-Glue backend are byte-for-byte
  unchanged.
• .github/workflows/spark-aws.yml: the Glue job sets
  LANCE_TEST_NAMESPACE=lancetest_<run_id>_<run_attempt> (a unique Glue
  database per run), and the if: always() cleanup step now runs
  aws glue delete-database (which drops the database and every table it
  still holds, reaping any orphan regardless of whether its data survives)
  before the existing aws s3 rm.

For the Glue backend singleLevelNs is false
(BaseLanceNamespaceSparkCatalog.java), so the Spark namespace name maps
directly to the Glue database name — a per-run namespace is a per-run Glue
database. This removes both failure modes: concurrent runs no longer share a
namespace, and an interrupted/cancelled run can only ever orphan entries inside
its own unique database, which is dropped wholesale by cleanup.

Testing

• Local dir backend, TEST_BACKENDS=local (Spark 4.1 / Scala 2.13):
  • LANCE_TEST_NAMESPACE unset → 107 passed, 8 skipped, 1 xpassed (no
    regression; identical SQL to before).
  • LANCE_TEST_NAMESPACE=ns_probe → same result, and the on-disk layout shows
    tables under ns_probe$… instead of default$…, confirming the namespace
    actually takes effect and isolates.
• The rewrite was applied by a tokenizer-based transform with a lockstep
  verifier proving zero changes outside SQL string literals and that each
  changed literal differs only by default → {LANCE_NAMESPACE}.

Notes for maintainers

• The Glue/S3 Test job only runs for same-repo PRs (forks can't read the AWS
  secrets), so it will not run on this PR if it comes from a fork. It can be
  validated via workflow_dispatch on a branch in this repo. The new path needs
  AWS permission to create and delete Glue databases; the first same-repo run
  should confirm both the create and the delete-database cleanup.
• The already-poisoned default database may still contain orphaned entries
  from before this change. New runs no longer touch default, so this is
  harmless, but a one-time aws glue delete-database --name default (or
  per-table cleanup) will tidy it up.
• If the cleanup step itself is cancelled, a per-run lancetest_* database can
  linger. It never collides with anything, but a periodic sweep of stale
  lancetest_* databases would be a reasonable follow-up.

[jackylee-ch]

Thanks — you're right, I couldn't substantiate the "DROP TABLE IF EXISTS becomes a no-op" claim either (Spark reads the Glue metadata, and the pinned Glue implementation deletes the entry before dataset cleanup), so I've removed it from the description. The rationale now rests on the two mechanisms that do hold: concurrent runs colliding on the shared default table names, and cancelled runs leaving Glue entries that the s3-only cleanup never reaped — both resolved by the per-run database.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The updated base includes #875, which fixes skipped Glue drops caused by the existence-method mismatch. Per-run databases still address concurrent jobs colliding on shared table names and confine entries left by interrupted runs to an isolated namespace. Leaving the namespace override unset preserves the base test behavior.

The remaining risk is operational: the AWS role's database creation and deletion permissions are unverified, and this fork cannot exercise the Glue job. A same-repo run can validate creation and cleanup. Failed or interrupted cleanup can leave isolated lancetest_* databases for later removal.

[jackylee-ch]

Closing in favor of #875, thanks for that fix: DROP TABLE IF EXISTS had been skipping Glue tables and leaving them for later runs, and the Glue/S3 job has passed on every push to main since. Two same-repo runs at once could still collide on the shared default tables, and a cancelled run's tables are now dropped by the next run's cleanup; I have not seen a collision fail a run, and this change rewrites about 650 lines of test_lance_spark.py, which PRs editing that file would keep conflicting with. If collisions do show up, a smaller fix can go in its own PR.