Skip to content

[AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin#4209

Open
zhangwl9 wants to merge 1 commit intoapache:masterfrom
zhangwl9:AMORO-optimize-OrphanFileDeleteExecutor-dev
Open

[AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin#4209
zhangwl9 wants to merge 1 commit intoapache:masterfrom
zhangwl9:AMORO-optimize-OrphanFileDeleteExecutor-dev

Conversation

@zhangwl9
Copy link
Copy Markdown
Contributor

@zhangwl9 zhangwl9 commented May 7, 2026

Why are the changes needed?

Close #4208.

Brief change log

Refactor Iceberg orphanFiles cleaning from the inline scheduler into a pluggable process model (ProcessFactory + ExecuteEngine) refer to #4107 :

Implementation Plan

  1. Create OrphanFilesCleaningProcess

    • Implement TableProcess and LocalProcess interfaces
    • Add to IcebergProcessFactory with proper trigger strategy
    • Support configuration: clean-orphan-files.enabled and clean-orphan-files.interval
  2. Remove standalone scheduler

    • Delete OrphanFilesCleaningExecutor from InlineTableExecutors
    • Remove registration from AmoroServiceContainer
  3. Add configuration options

    • clean-orphan-files.enabled (default: true)
    • clean-orphan-files.interval (default: 1 day)
    • Configure via process-factories.yaml plugin config
  4. Enhance state tracking

    • Use TableRuntimeCleanupState to track last orphan files clean time
  5. Update execution engine config

    • Add pool.orphan-files-cleaning.thread-count to execute-engines.yaml

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

# Conflicts:
#	amoro-common/src/test/java/org/apache/amoro/process/TestLocalExecutionEngine.java
@zhangwl9 zhangwl9 force-pushed the AMORO-optimize-OrphanFileDeleteExecutor-dev branch from 2677e62 to ded0027 Compare May 7, 2026 10:54
@zhangwl9
Copy link
Copy Markdown
Contributor Author

zhangwl9 commented May 8, 2026

@baiyangtx @xxubai Could you help me review it when you are free? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: Refactor orphan-files-cleaning via ProcessFactory plugin

1 participant