While, Do-while loop implementation for fetching paginated data from an API

@ricardozanini

What would you like to be added?

Support for while or do-while loop constructs to simplify workflows that need to fetch paginated data from APIs until a condition is met (e.g., no more pages to fetch).

Proposal(s):

Currently, implementing workflows that require repetitive API calls until a condition is met (e.g., while nextPageToken is present) is cumbersome and verbose using the existing specification. There is no native support for loop constructs that check a condition after each iteration (i.e., a do-while pattern).

Proposal:

Introduce a native while or do-while loop construct within the Serverless Workflow specification.
Support a conditional expression (JQ) that evaluates whether the loop should continue.
Allow a loopBody or array of steps to be executed within the loop.
Optional: support max iterations to prevent infinite loops.

Example syntax (pseudo-DSL):

document:
  dsl: '1.0.0'
  namespace: test
  name: paginated-fetch-example
  version: '0.1.0'

do:
  - fetchPaginatedData:
      while: .hasNextPage == true
      postConditionCheck: false // Enables do-while 
      maxIterations: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Alternative(s):

Currently, the same can be achieved by:

Manually chaining states with transition and using a switch to re-enter a state based on a condition.
Using recursion with workflow calls, which increases complexity and reduces readability.
Implementing a custom function or orchestrator outside of the workflow itself.

These approaches are harder to read, maintain, and error-prone when retry logic and timeouts are introduced.

Additional info:

This feature would greatly improve:

Workflows that deal with pagination (API data fetching, batch processing).
Polling use cases where the data becomes available over time.
Any retry-until-success or loop-until-condition patterns.

Let me know if you'd like me to open a PR to help explore this idea! Or do you think this can be achieved with current For loop? @ricardozanini @cdavernas

Community Notes

Please vote by adding a 👍 reaction to the feature to help us prioritize.
If you are interested to work on this feature, please leave a comment.

However, It appears that the current for task can be utilized as a loop, allowing a For task to operate as a traditional while loop in Serverless Workflow.

By setting the for.in property of a For task to an empty collection, the while property takes over as the primary control mechanism for the loop, facilitating the use of both while and do-while loop patterns within Serverless Workflows.

Example:

 - name: WhileLoopExample
    type: while
    for: 
      in: ${ [] }
    while: ${ .counter < 5 }
    maxIterations: 100
    do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);

Example:

- name: DoWhileLoopExample
   type: do-while
   for: 
     in: ${ [] }
   while: ${ .counter < 5 } # false initially
   maxIterations: 100
   do:
     - name: IncrementCounter
       type: set
       data:
         counter: ${ .counter + 1 }
     - name: LogMessage
       type: run
       run:
         script:
           lang: javascript
           code: |
             console.log("This runs even though .counter < 5 is false!");

May I know if for is mandatory or can we get it of

for: 
     in: ${ [] }

According to typescript SDK it's optional, but its no-where that's mentioned in the specs, can you clarify please?

https://github.com/serverlessworkflow/sdk-typescript/blob/main/src/lib/generated/definitions/specification.ts#L252

@nyamathshaik

@nyamathshaik Yes, for is mandatory, and also serves as a discriminator. In addition, your suggestion would not work because the enumeration would stop before evaluating the while condition, as the array is empty. while in that context is an exit condition, not a continuation one, in which case it would create many unpleasant side effects (one being a null ref for enumerated item).

Therefore, you initial proposal looks to me to be the most adequate, and would also restore a construct we had in previous versions.

@cdavernas

@cdavernas Thank you for the feedback. I was considering something along the lines of the following, but I am still open to suggestions. Please review it and let me know if it looks good.

export type LoopTask = TaskBase &{
    /**
     * The type of loop to use.
     */
    loop?: 'for' | 'while' | 'do-while';
    /**
     * The configuration for the for loop.
     */
    for?: ForTaskConfiguration;
    /**
     * The configuration or conditions for the while loop.
     */
    while?: WhileTaskConfiguration | string;
    /**
     * The tasks to execute if the loop is a for, while, do-while loop.
     */
    do?: TaskList;
   
    
    [k: string]: unknown;
};

export interface WhileTaskConfiguration {
    /**
     * The name of the variable used to store the current item being enumerated.
     */
    condition?: string;
    /**
     * A runtime expression used to get the collection to enumerate.
     */
    maxIterations?: number;
    /**
     * The name of the variable used to store the index of the current item being enumerated.
     */
    at?: string;
}

Benefits of Proposed Approach

Backward Compatibility: Existing ForTask implementation remains unchanged.
Enhanced Clarity: Clear differentiation between for, while, and do-while loops.
Safety Mechanism: The addition of maxIterations prevents infinite loops.
Extensibility: Future improvements can be made without affecting existing task definitions.

This proposed solution provides a structured approach to implementing loop tasks while maintaining compatibility with Serverless Workflow specifications.

@nyamathshaik

@nyamathshaik First of all, thanks for your awesome work!

Second, I have a couple of remarks regarding the proposal you made in related PR:

do:
  - fetchPaginatedData:
      while: .hasNextPage == true
      postConditionCheck: false # Enables do-while behavior
      maxIterations: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

The while keyword will need to change, either in the task you propose to add, or in the for task, or discrimination will no longer be (easily) possible (i.e. check for presence of while keyword, which as a primary identifier must not be use by any other task)
I'm personnaly not a fan of the postConditionCheck and maxIterations terminology, which goes against the design guidelines. I'm convinced we can come up with imperative, actionable, single words names instead!

@cdavernas

@cdavernas Thanks a lot for the feedback and kind words! 🙌 Really appreciate you taking the time to review the proposal.

Totally hear you on the concerns around the while keyword clash and the naming of postConditionCheck and maxIterations.

For the keyword conflict: I’m happy to revise the structure to avoid any ambiguity with for tasks. Would something like below look good?

do:
  - fetchPaginatedData:
      loop: while | do-while
      condition: .hasNextPage == true
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Open to suggestions! I'm happy to iterate further on the proposal based on whatever naming conventions or structural guidance the maintainers feel would fit best.

Looking forward to your thoughts!

or how about just simply renaming the initial proposal to below :

do:
  - fetchPaginatedData:
      condition: .hasNextPage == true
      isDoWhile: true # Enables do-while behavior
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Why not:

 - name: WhileLoopExample
   while: ${ .counter < 5 }
   do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);

And

 - name: WhileLoopExample
   do:
      - name: IncrementCounter
        type: set
        data:
          counter: ${ .counter + 1 }
      - name: LogMessage
        type: run
        run:
          script:
            lang: javascript
            code: |
              console.log("Counter is: " + context.counter);
    while: ${ .counter < 5 }

The challenge is to interpret this in the JSON Schema.

Essentially, you want a for without in, correct?

We can add the limit keyword, no problem. But yes, we must avoid camel-cased and composition words.

The first is a do task with an optional while parameter. The other is a while task. I think we can accommodate this in the JSON schema with backwards comp for 1.1.0.

@ricardozanini

Thanks for reviewing @ricardozanini JSON Schema doesn’t support positional logic. So I don't think comment actually works. (i.e., if while appears before, interpret it one way; if after, interpret differently).

Also, this creates more Ambiguity in Semantics (while appearing before or after do: changes the behavior).

@ricardozanini

That said, I am still aligned with below :

Approach 1:

do:
  - fetchPaginatedData:
      loop: while | do-while
      condition: .hasNextPage == true
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Approach 2:

do:
  - fetchPaginatedData:
      condition: .hasNextPage == true
      isDoWhile: true # Enables do-while behavior
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

Please let me know if you have any further suggestions, or an entire new suggestion is fine too. So that we can agree upon the final solution and I can work on raising a PR.

@ricardozanini @cdavernas

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@geomagilles

After reviewing this discussion, I think the existing if + then mechanism already covers both while and do-while patterns without requiring new task types. Here's how:

While loop (condition checked before each iteration)

document:
  dsl: '1.0.0'
  namespace: examples
  name: while-loop-pattern
  version: '0.1.0'
do:
  - loop:
      if: ${ .counter < 5 }
      set:
        counter: ${ .counter + 1 }
      then: loop
  - afterLoop:
      set:
        result: done

The if guards execution — when .counter < 5 is false, the task is skipped and flow continues to afterLoop. When true, set runs and then: loop jumps back to itself.

Do-while loop (body runs at least once, condition checked after)

document:
  dsl: '1.0.0'
  namespace: examples
  name: do-while-loop-pattern
  version: '0.1.0'
do:
  - loop:
      set:
        counter: ${ (.counter // 0) + 1 }
      then: check
  - check:
      if: ${ .counter < 5 }
      set:
        continuing: true
      then: loop
  - afterLoop:
      set:
        result: done

Here loop always executes (no if guard), then check evaluates the condition. If true, it jumps back to loop. If false, check is skipped and flow falls through to afterLoop.

Given that both patterns work today with the current spec, I'd suggest we document these as "how-to" patterns (as @geomagilles also suggested) rather than introduce new task types. Adding while/repeat tasks is a permanent commitment that would impact all SDKs, and the benefit over the if+then approach doesn't seem to justify the complexity — especially since introducing new discriminator keywords risks collisions with the existing for.while property.

If the community feels strongly about dedicated loop constructs in the future, we can revisit, but for now documenting these patterns should address the original use case.

@nyamathshaik would you be interested in contributing a "how-to" guide or examples for these patterns instead?

@ricardozanini

@ricardozanini I agree with the rational but if-then pattern currently has a side effect in task json pointer. In a for, every executed task has a different jsopointer, while if-then repeats the same jsonpointer several times (jsonpointer is part of the life cycle cloud event, so this has to be adressed somehow in the spec, either mentioning that when using this pattern the jsonpointer of the task is repeated or by introducing a way to generate different json pointers)

@fjtirado

@fjtirado great catch! Yes, indeed, in the lifecycle event, a monitoring actor will see repeatable tasks without an index we have in for.

Alternatively, I was thinking of a workaround where you add a single task before for, so you guarantee it executes once. I think we mentioned this alternative in this thread already.

Nevertheless, I think it's not worth a new task for this use case given we already have alternatives to tackle this problem.

Navigation Menu

Repository navigation

While, Do-while loop implementation for fetching paginated data from an API #1096

What would you like to be added?

Proposal(s):

Alternative(s):

Additional info:

Community Notes

62 remaining items

While loop (condition checked before each iteration)

Do-while loop (body runs at least once, condition checked after)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

While, Do-while loop implementation for fetching paginated data from an API #1096

Description

What would you like to be added?

Proposal(s):

Alternative(s):

Additional info:

Community Notes

Activity

nyamathshaik commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

cdavernas commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

cdavernas commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

ricardozanini commented on Apr 14, 2025

ricardozanini commented on Apr 14, 2025

ricardozanini commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

nyamathshaik commented on Apr 14, 2025

62 remaining items

github-actions commented on May 16, 2026

ricardozanini commented on Jul 23, 2026

While loop (condition checked before each iteration)

Do-while loop (body runs at least once, condition checked after)

fjtirado commented on Jul 24, 2026

ricardozanini commented on Jul 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions