Retry processor runs a dynamic query that creates the auto index Auto/RetryBatches/ByStatus #5960

Describe the bug

The retry processor runs a dynamic query. RavenDB creates an auto index for it. ServiceControl must only use its own static indexes, so this auto index must never exist.

RetryStagingStore.GetStagingBatch runs session.Query<RetryBatch>() with a filter on Status and no static index. RavenDB serves the query with the auto index Auto/RetryBatches/ByStatus. The retry processor calls GetStagingBatch on every processing cycle, so the auto index appears seconds after the error database is created, also when the RetryBatches collection is empty. Every error database has it.

The query has been dynamic since group retries were added in July 2015 (ServiceControl 1.6.0). The static index RetryBatches_ByStatusAndSession already maps Status, so the auto index is not needed.

Why this is a bug:

• ServiceControl deploys and manages its index definitions at start-up. An auto index is outside that control. It is created at run time with the database defaults of that moment, and RavenDB can mark it idle and delete it. This gives an index set that differs per installation and over time.
• Every index costs disk space, memory and indexing work on each write to the RetryBatches collection.
• Side effect since 6.20.0: the Error Database Search Engine custom check and the start-up warning list all indexes that use Corax. An auto index has no per-index search engine setting, so the per-index migration procedure cannot change it. The check keeps failing after all static indexes use Lucene.

Steps to reproduce

1. Create an error database with ServiceControl, any version with RavenDB persistence.
2. Start the primary instance.
3. Open the index list in RavenDB Studio. Auto/RetryBatches/ByStatus is present next to the static indexes.

Relevant log output

Side effect on 6.20.0 and 6.21.0 after all static indexes were migrated to Lucene:

Warn|ServiceControl.RavenDB.StartupChecks|The following RavenDB index(es) use the Corax search engine: Auto/RetryBatches/ByStatus. ...

Additional Information

• Affected versions: all versions with RavenDB persistence since 1.6.0.
• Found in the smoke test of 6.21.1 for Migrated Lucene indexes are reset to Corax at start-up when the index is not locked #5948.
• Fixed on master by Query the staging retry batch through the static index #5953, backport to 6.21.1 in Query the staging retry batch through the static index #5959. The fix queries through the static index RetryBatches_ByStatusAndSession.
• The fix does not remove an existing auto index. RavenDB marks it idle after 30 minutes without queries and deletes it 72 hours later. It can also be deleted in RavenDB Studio.