Update snapshot expiration to reclaim orphan files that are part of snapshots being expired by maluchari · Pull Request #447 · linkedin/openhouse

maluchari · 2026-02-07T19:48:45Z

Previously, snapshot expiration only removed metadata while leaving orphaned data files on storage until a separate Orphan File Deletion (OFD) job ran. This caused delayed storage reclamation (days) and also a need for OFD to process a lot of files.

This change enables immediate file deletion during snapshot expiration, providing faster storage reclamation and cost savings.

Note:

OFD is still required to clean up orphan files from failed jobs or other edge cases. This optimization addresses the common case of normal snapshot expiration.
SE without deleteFiles uses metadata-only expiration avoiding filesystem walks, while SE with deleteFiles uses deleteWith(consumer) which triggers Iceberg to scan and identify expired files; using a single code path with a conditional no-op would incur unnecessary filesystem overhead when deletion is disabled. Hence implementation is different for both.

Added --deleteFiles command-line flag to JobsScheduler
Updated OperationTaskFactory and TableSnapshotsExpirationTask to propagate the deleteFiles parameter
Modified Operations.expireSnapshots() to conditionally delete files based on the flag (default: metadata-only for backward compatibility)
Added metrics tracking for snapshot expiration with deleteFiles flag
Updated tests with deleteFiles parameter and added test coverage for both deletion modes

Metadata-only expiration (default):

--type SNAPSHOTS_EXPIRATION --cluster local \
    --tablesURL http://openhouse-tables:8080 \
    --jobsURL http://openhouse-jobs:8080

With file deletion:

--type SNAPSHOTS_EXPIRATION --cluster local \
    --tablesURL http://openhouse-tables:8080 \
    --jobsURL http://openhouse-jobs:8080 \
    --deleteFiles

Verified with local Docker environment (oh-hadoop-spark):

Created table with versions=2 policy, generated 5 snapshots via overwrites
Without --deleteFiles: 5→2 snapshots, 5→5 files (no deletion)
With --deleteFiles: 5→2 snapshots, 5→1 file (3 files deleted)
Job logs and HDFS verification confirmed expected behavior

// HDFS before and after with deletefiles set

Adding test results for backup dir:

Summary

Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.

Changes

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

Manually Tested on local docker setup. Please include commands ran, and their output.
Added new tests for the changes made.
Updated existing tests to reflect the changes made.
No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

Breaking Changes
Deprecations
Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

sumedhsakdeo

Any risks with concurrent runs of snapshot expiration with --deleteFiles set to true and orphan file deletion job?

maluchari · 2026-02-14T21:19:06Z

Any risks with concurrent runs of snapshot expiration with --deleteFiles set to true and orphan file deletion job?

There should not be any corruption issues. It is possible for both these jobs to delete the same files that could waste compute and cause FNFEs , but these exceptions are handled in the jobs and jobs can proceed with rest of the files

dushyantk1509

Overall looks good.

dushyantk1509 · 2026-02-17T14:32:09Z

-          .expireOlderThan(System.currentTimeMillis())
-          .retainLast(versions)
-          .commit();
+      result =


Result from prev operation will be lost... we should combine to get the correct metrics.

dushyantk1509 · 2026-02-17T14:33:14Z

+   * Process a single expired file: backup if needed, otherwise delete. This method runs on Spark
+   * executors in a distributed manner.
+   */
+  private void processExpiredFileWithBackup(


We can leverage this in OFD.

dushyantk1509 · 2026-02-17T14:36:03Z

+        result.deletedManifestListsCount());
+
+    // Emit metrics
+    recordMetrics(duration);


Should we also record result metrics?

dushyantk1509 · 2026-02-17T14:43:07Z

  public void expireSnapshots(Table table, int maxAge, String granularity, int versions) {
-    ExpireSnapshots expireSnapshotsCommand = table.expireSnapshots().cleanExpiredFiles(false);
+    // Call the Result-returning version but ignore the result for backward compatibility
+    expireSnapshots(table, maxAge, granularity, versions, false);


[may be in separate PR] There are now 6 overloads of expireSnapshots. Should we use consider using parameter object something like below for better readability and maintenance?

ExpireSnapshotsConfig config = ExpireSnapshotsConfig.builder()
.maxAge(maxAge).granularity(granularity).versions(versions)
.deleteFiles(true).backupEnabled(true).backupDir(".backup")
.build();
ops.expireSnapshots(fqtn, config);

Do we need all the overloaded methods? Is it possible to use builder pattern instead? Alternately, we can define one additional overloaded method with all the parameters and pass null or false when values are not supplied.

teamurko

Thank you @maluchari. Should we merge delete func used in OFD and SE so that this parallel code doesn't diverge?

Previously, snapshot expiration only removed metadata while leaving orphaned data files on storage until a separate Orphan File Deletion (OFD) job ran. This caused: - Delayed storage reclamation (days/weeks) - Increased storage costs - Operational overhead of coordinating two jobs This change enables immediate file deletion during snapshot expiration, providing faster storage reclamation and cost savings. **Note:** OFD is still required to clean up orphan files from failed jobs or other edge cases. This optimization addresses the common case of normal snapshot expiration. - Added --deleteFiles command-line flag to JobsScheduler - Updated OperationTaskFactory and TableSnapshotsExpirationTask to propagate the deleteFiles parameter - Modified Operations.expireSnapshots() to conditionally delete files based on the flag (default: metadata-only for backward compatibility) - Added metrics tracking for snapshot expiration with deleteFiles flag - Fixed jobs-scheduler.Dockerfile ENTRYPOINT for proper argument passing - Updated tests with deleteFiles parameter and added test coverage for both deletion modes Metadata-only expiration (default): ```bash --type SNAPSHOTS_EXPIRATION --cluster local \ --tablesURL http://openhouse-tables:8080 \ --jobsURL http://openhouse-jobs:8080 ``` With file deletion: ```bash --type SNAPSHOTS_EXPIRATION --cluster local \ --tablesURL http://openhouse-tables:8080 \ --jobsURL http://openhouse-jobs:8080 \ --deleteFiles ``` Verified with local Docker environment (oh-hadoop-spark): - Created table with versions=2 policy, generated 5 snapshots via overwrites - Without --deleteFiles: 5→2 snapshots, 5→5 files (no deletion) - With --deleteFiles: 5→2 snapshots, 5→1 file (3 files deleted) - Job logs and HDFS verification confirmed expected behavior [Screenshots to be added] Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Address PR linkedin#447 review comment (teamurko): merge the delete function used by Orphan File Deletion (OFD) and Snapshot Expiration (SE) so the parallel implementations do not diverge. - Extract Operations#buildFileDeleteHandler, a shared deleteWith Consumer<String> encapsulating the metadata.json skip, backup-dir skip, and backup-or-delete logic for data files. - Both deleteOrphanFiles (OFD) and createExpireSnapshotsActionWithBackup (SE) now delegate to this single handler instead of maintaining separate copies. - Add tests exercising the shared handler from the SE side: testSnapshotsExpirationWithBackupMovesDataFilesToBackupDir and testSnapshotsExpirationWithoutBackupDeletesDataFilesDirectly, using RTAS to orphan data files so snapshot expiration actually reclaims them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…Dockerfile change - Operations.expireSnapshots(): the versions-based expiration phase was overwriting the maxAge-based phase's ExpireSnapshots.Result instead of accumulating it, silently dropping the first phase's deleted-file/manifest counts from returned metrics and logs (deletion itself was unaffected). Added combineResults() to sum both phases' Result fields. - Corrected javadoc on expireSnapshotsWithFiles/createExpireSnapshotsActionWithBackup: only manifest/snapshot planning is distributed via Spark; the actual delete/backup callback runs driver-side via Iceberg's internal executor pool. - Reverted jobs-scheduler.Dockerfile ENTRYPOINT change: while a legitimate fix for the classic `sh -c "... $@"` first-arg-dropping bug, it is unrelated to this PR's snapshot-expiration deleteFiles work and had no test coverage here. - Added unit test for combineResults() and an integration-level regression test (RTAS + real snapshot aging) verifying both phases' deleted-file counts are reflected in the final Result. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

abhisheknath2011 · 2026-08-17T23:27:37Z

-                }
-              }
-            });
+            buildFileDeleteHandler(table, backupEnabled, backupDir, dataManifestsCache));


Can we also capture the total number of files that are deleted so that it helpful to know the number of files SE is deleting in every execution?

abhisheknath2011 · 2026-08-17T23:37:48Z

+  static ExpireSnapshots.Result combineResults(
+      ExpireSnapshots.Result first, ExpireSnapshots.Result second) {
+    return ImmutableExpireSnapshots.Result.builder()
+        .deletedDataFilesCount(first.deletedDataFilesCount() + second.deletedDataFilesCount())


This seem to have the complete results. Can we print all the deleted files with file type? We could emit metrics as well.

abhisheknath2011 · 2026-08-17T23:41:38Z

+            io.opentelemetry.api.common.AttributeKey.stringKey(AppConstants.TABLE_NAME),
+            fqtn,
+            io.opentelemetry.api.common.AttributeKey.booleanKey(AppConstants.DELETE_FILES_ENABLED),
+            deleteFiles);


Can we emit total files deleted as well?

maluchari changed the title ~~Update snapshot expiration to reclaim orphan files that are part of snapshots being expired~~ [WIP] Update snapshot expiration to reclaim orphan files that are part of snapshots being expired Feb 7, 2026

sumedhsakdeo reviewed Feb 12, 2026

View reviewed changes

maluchari force-pushed the malini/optimize_se_with_files_deletion branch from 8285eef to 3476d02 Compare February 15, 2026 00:04

dushyantk1509 reviewed Feb 17, 2026

View reviewed changes

teamurko requested changes Feb 20, 2026

View reviewed changes

teamurko assigned jiang95-dev Feb 20, 2026

maluchari and others added 3 commits August 17, 2026 11:15

Fix SE with deleteFiles disabled - continue as optimized job

ed27583

maluchari force-pushed the malini/optimize_se_with_files_deletion branch from 3476d02 to c048533 Compare August 17, 2026 19:47

maluchari changed the title ~~[WIP] Update snapshot expiration to reclaim orphan files that are part of snapshots being expired~~ Update snapshot expiration to reclaim orphan files that are part of snapshots being expired Aug 17, 2026

maluchari requested a review from teamurko August 17, 2026 22:19

abhisheknath2011 reviewed Aug 17, 2026

View reviewed changes

Merge branch 'main' into malini/optimize_se_with_files_deletion

44d342d

mkuchenbecker mentioned this pull request Sep 2, 2026

Delete files synchronously during snapshot expiration #710

Closed

2 tasks

Navigation Menu

Uh oh!

Update snapshot expiration to reclaim orphan files that are part of snapshots being expired - #447

Update snapshot expiration to reclaim orphan files that are part of snapshots being expired#447
maluchari wants to merge 5 commits into
linkedin:mainfrom
maluchari:malini/optimize_se_with_files_deletion

maluchari commented Feb 7, 2026 •

edited

Loading

Uh oh!

sumedhsakdeo left a comment

Uh oh!

maluchari commented Feb 14, 2026

Uh oh!

dushyantk1509 left a comment

Uh oh!

dushyantk1509 Feb 17, 2026

Uh oh!

dushyantk1509 Feb 17, 2026

Uh oh!

dushyantk1509 Feb 17, 2026

Uh oh!

dushyantk1509 Feb 17, 2026

Uh oh!

abhisheknath2011 Aug 17, 2026 •

edited

Loading

Uh oh!

teamurko left a comment

Uh oh!

abhisheknath2011 Aug 17, 2026 •

edited

Loading

Uh oh!

abhisheknath2011 Aug 17, 2026

Uh oh!

abhisheknath2011 Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

maluchari commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing Done

Additional Information

Uh oh!

sumedhsakdeo left a comment

Choose a reason for hiding this comment

Uh oh!

maluchari commented Feb 14, 2026

Uh oh!

dushyantk1509 left a comment

Choose a reason for hiding this comment

Uh oh!

dushyantk1509 Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

dushyantk1509 Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

dushyantk1509 Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

dushyantk1509 Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

abhisheknath2011 Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teamurko left a comment

Choose a reason for hiding this comment

Uh oh!

abhisheknath2011 Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhisheknath2011 Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

abhisheknath2011 Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

maluchari commented Feb 7, 2026 •

edited

Loading

abhisheknath2011 Aug 17, 2026 •

edited

Loading

abhisheknath2011 Aug 17, 2026 •

edited

Loading