Repository navigation
NIFI-16271 Allow Stateless Process Groups to buffer FlowFile content in memory - #11676
Merged
pvillard31 merged 5 commits intoOct 2, 2026
Merged
Conversation
markap14
force-pushed
the
NIFI-16271-stateless-group-in-memory
branch
from
September 27, 2026 21:38
5db294c to
07ed86b
Compare
markap14
marked this pull request as ready for review
September 27, 2026 21:39
…in memory Adds a Stateless Content Storage Location (INHERITED, CONTENT_REPOSITORY, IN_MEMORY) to Process Groups so that a Process Group running on the Stateless Execution Engine can buffer FlowFile content in memory instead of writing to the Content Repository. - Bump nifi-api to 2.12.0-SNAPSHOT (StatelessContentStorageLocation enum and VersionedProcessGroup.statelessContentStorageLocation). - Thread the value through the ProcessGroup interface, StandardProcessGroup (get/set/resolve/verify), DTO, DtoFactory, DAO, flow mapping and flow synchronization, mirroring executionEngine. - Defer the Stateless engine's ContentRepository selection until the group starts via DeferredStatelessContentRepository, so the configured location is honored (the group node is created before configuration is applied). When resolved to IN_MEMORY, a ByteArrayContentRepository buffers content in memory; otherwise the NiFi Content Repository is used. - Restrict IN_MEMORY to Process Groups that are disconnected from all other components so content never crosses repositories: reject setting IN_MEMORY when the group has incoming/outgoing connections, and reject connecting into or out of a group configured to buffer content in memory. - Add the FlowFile Content Storage selector to the edit-process-group dialog, shown when the Execution Engine is Stateless. - Add StandardProcessGroup unit tests and StatelessInMemoryContentIT system tests (in-memory processing with an on-disk Content Repository scan proving nothing is written to disk, plus the connection and inheritance constraints). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Account for allocated buffer capacity, support content beyond 2 GiB, and release zero-count claims when frozen. Reduce runtime overhead with cached limits, 64 KiB segments, and read fast paths while consolidating regression coverage. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
markap14
force-pushed
the
NIFI-16271-stateless-group-in-memory
branch
from
September 28, 2026 13:24
07ed86b to
b3c7ef6
Compare
pvillard31
approved these changes
Oct 2, 2026
pvillard31
left a comment
pvillard31
left a comment
Contributor
There was a problem hiding this comment.
Changes LGTM and have been playing with it over the last few days, everything worked as expected. Thanks for this improvement @markap14 !
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NIFI-16271
Allows a Stateless Process Group to keep FlowFile content in memory up to a configured data-size limit, heap-percentage limit, or the smaller of both limits. Content above the limit spills to the backing Content Repository. The default leaves in-memory content disabled.
Content leaving a Stateless Process Group is transferred to a backing claim before the FlowFile Repository is updated. Failed repository updates roll back claim counts so timed-out and failed runs do not retain content.
The REST API, user interface, flow mapping, and documentation include both settings. The limits cannot be changed while the Stateless Process Group is running. This change uses the released NiFi API 2.13.0 fields.
Verification
StatelessInMemoryContentITsystem tests passed in 13.62 seconds, including NiFi startupgit diff --checkpassed./mvnw clean install -P contrib-checkon JDK 21 and JDK 25No new dependencies were added.