feat: resolve tiles-deletion storage locators from task params (MAPCO-11295) - #37
Open
almog8k wants to merge 7 commits into
Open
feat: resolve tiles-deletion storage locators from task params (MAPCO-11295)#37almog8k wants to merge 7 commits into
almog8k wants to merge 7 commits into
Conversation
Temporary pointer to a locally packed tarball while the reshaped deletion schemas are unreleased. Must be swapped for a published 8.3.0-alpha before merge: the tarball reports the same version as the published alpha, so a plain install on CI would resolve the older package instead.
…ms (MAPCO-11295)
Tiles deletion read its storage locator from
strategies.tilesDeletion.{s3Bucket,fsSubPath}, which pinned every task in a
deployment to one bucket and one sub path. The reshaped raster-shared deletion
schemas carry the locator in the task params, so each task now brings its own.
- storageProvider replaces sourceProvider and tilesRelativePath replaces
tilesPath, following the shared schemas
- an FS storage target is now a sub path of the configured base path.
FsStorageProvider joins its own base path and rejects a sub path that falls
outside the configured deletion sub paths, so the strategy holds no
filesystem knowledge at all and reads only its batching knobs from config
- REDIS joined both shared unions but has no provider here yet, so tiles
deletion rejects it as unrecoverable (MAPCO-11261)
Producers of tiles-deletion and artifacts-deletion tasks must now send the
bucket or subPath in the task parameters.
isPathWithinAllowedSubPaths holds the rule that a path must sit strictly under one of the configured sub paths and still resolve inside the base path, which makes it unit testable on its own rather than only through the provider. FsStorageProvider's private check becomes an assertion instead of a boolean, so a caller cannot forget to act on the result, and the thrown message now names the offending paths, the base path and the allowed sub paths.
delete(storageTarget, paths) now reads the same way as targetExists(storageTarget, relativePath).
strategies.tilesDeletion.s3Bucket and .fsSubPath are no longer read now that the locator travels with the task, so they go along with their configmap entries (TILES_DELETION_S3_BUCKET, TILES_DELETION_FS_SUB_PATH) and the s3.tilesBucket value that fed the former. storage.fs.subPaths stays: it is the allowlist FS deletion targets are checked against. cleanupStorageProviders now defaults to empty so a deployment states its providers explicitly.
|
🎫 Related Jira Issue: MAPCO-11295 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues: MAPCO-11295 (epic MAPCO-11261)
Further information:
Adapts the cleaner to the reshaped
raster-shareddeletion schemas and moves the tiles-deletion storage locator out of deployment config into the task parameters.What changed
TilesDeletionStrategytakes the S3 bucket and the FS sub path from the task params instead ofstrategies.tilesDeletion.{s3Bucket,fsSubPath}, so a single deployment is no longer pinned to one bucket and one sub path.storageProviderreplacessourceProviderandtilesRelativePathreplacestilesPath, following the shared schemas.FsStorageProviderjoins its own base path and rejects a sub path that falls outside the configured deletion sub paths, so the strategy holds no filesystem knowledge and reads onlybatchSize/concurrencyfrom config.isPathWithinAllowedSubPathsincleaner/utils/path.ts, used by bothtargetExistsanddeleteResources, and is now an assertion rather than a boolean whose result could be ignored. The thrown message names the offending paths, the base path and the allowed sub paths.IStorageProvider.deletetakes the storage target first, matchingtargetExists.strategies.tilesDeletion.s3Bucketand.fsSubPath, their configmap entries (TILES_DELETION_S3_BUCKET,TILES_DELETION_FS_SUB_PATH) and thes3.tilesBucketvalue that fed the former.storage.fs.subPathsstays — it is the allowlist FS deletion targets are checked against.cleanupStorageProvidersnow defaults to empty so a deployment states its providers explicitly.The five commits are meant to be reviewed in order: dependency pointer, the feature, the validator extraction, the parameter reorder, then the config cleanup.
Breaking change
Producers of
tiles-deletionandartifacts-deletiontasks must now sendbucket(S3) orsubPath(FS) in the task parameters, and the twostrategies.tilesDeletionconfig keys above no longer exist. Thefeatcommit is not marked!, so release-please will cut a minor — say the word if this should be a major instead.