fix: recreating an existing namespace with OPTIMIZED_SIBLING_CHECK is a conflict

[andybradshaw]

Split out of #5520 as a separate PR, addressing the HTTP status code portion of #5521.

Checklist

• 🛡️ Don't disclose security issues! (contact security@apache.org)
• 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
• 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
• 💡 Added comments for complex logic
• 🧾 Updated CHANGELOG.md (if needed)
• 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

[dimas-b]

LGTM 👍 Thanks, @andybradshaw !

[ayushtkn]

Thanx @andybradshaw for the fix, I think the title is wrong. It says recreating an existing table but the fix is recreating the namespace, the CHANGELOG is correct.

The table path may still have a gap. The same-name exclusion

polaris/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java

Lines 1596 to 1607 in 65f9413

	siblingTables.forEach(
	tbl -> {
	if (!tbl.name().equals(name)) {
	pathsToResolve.add(ResolvedPathKey.ofTableLike(tbl));
	}
	});
	siblingNamespaces.forEach(
	ns -> {
	if (!ns.level(ns.length() - 1).equals(name)) {
	pathsToResolve.add(ResolvedPathKey.ofNamespace(ns));
	}
	});

exists only in the fallback — the optimized path has no equivalent. And for tables, validateNoLocationOverlap

polaris/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java

Lines 1425 to 1439 in 65f9413

	// Create a fake IcebergTableLikeEntity to check for overlap, since no real entity
	// has been created yet.
	var lastNamespace = resolvedNamespace.getLast();
	IcebergTableLikeEntity virtualEntity =
	IcebergTableLikeEntity.of(
	new PolarisEntity.Builder()
	.setName(identifier.name())
	.setType(PolarisEntityType.TABLE_LIKE)
	.setSubType(PolarisEntitySubType.ICEBERG_TABLE)
	.setParentId(lastNamespace.getId())
	.setCatalogId(lastNamespace.getCatalogId())
	.setProperties(Map.of(PolarisEntityConstants.ENTITY_BASE_LOCATION, location))
	.build());
	validateNoLocationOverlap(virtualEntity, resolvedNamespace);

builds a virtual IcebergTableLikeEntity with no persisted id, so nothing could exclude the real one even with #5520's ancestor filtering. A sequential recreate won't show this — Iceberg's create() checks ops.current() and 409s first — but where a create reaches doCommit(null, …) with the entity already present, mainly concurrent same-name creates, the optimized path can surface 403 instead of 409. That also skips the AlreadyExistsException catch used for idempotency replay.

I think we should update the title to say it is namespace & maybe chase the table path as a followup.

[dimas-b]

Good point about follow-up from @ayushtkn

Are we ok to merge this PR "as is" and do further improvements on main later?

[ayushtkn]

Looks good to me