Skip to content

smoke: wait until grcli events has subscribed - #753

Merged
christophefontaine merged 1 commit into
DPDK:mainfrom
rjarry:arp-ndp-smoke-fix
Sep 18, 2026
Merged

christophefontaine merged 1 commit into
DPDK:mainfrom
rjarry:arp-ndp-smoke-fix

Conversation

@rjarry

@rjarry rjarry commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

The event stream is started in the background and races with interface creation. If the subscription lands after an interface is created, its add event is missed and grcli prints the interface by its numeric id, which breaks wait_event pattern matching. This happens regularly on slow CI runners and never locally.

Redirect the grout logs to files and tail them, so the harness can watch them. When starting the event stream, temporarily raise the grout.api log level to debug and block until the daemon logs the successful GR_EVENT_SUBSCRIBE before returning. Interfaces are then only created once the stream is guaranteed to receive their events.

Comment thread smoke/_init.sh Outdated
Comment on lines +489 to +490
local end=$(wc -l < $tmp/grout.log)
local level=$(grcli -j log level show | jq -re '.[] | select(.name == "grout.api") .level')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Smoke setup fails without local logs 🐞 Bug ☼ Reliability

_start_events_stream reads $tmp/grout.log even when _start_grout was not used to create that
file. With the default event-following mode, tests connected to an existing daemon and tests
launched under the debugger exit at this command before running their test body.
Agent Prompt
## Issue description
Event startup assumes `$tmp/grout.log` exists, but existing-daemon and debugger modes do not call `_start_grout` and therefore never create or populate it.

## Fix Focus Areas
- smoke/_init.sh[444-490]

## Recommended Fix
Make every supported startup mode expose daemon logs through `$tmp/grout.log`, or use a subscription readiness mechanism that does not depend on that local file. Preserve support for both `run_grout=false` and `GDB=true`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment thread smoke/_init.sh
Comment on lines +512 to +514
{ tail -f -n +$((end + 1)) $tmp/grout.log || : ; } |
timeout 10 grep -qE 'GR_EVENT_SUBSCRIBE.*Success' ||
fail "timeout after 10s waiting for grcli events"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Event startup can hang indefinitely 🐞 Bug ☼ Reliability

The readiness pipeline waits for the tail -f producer after grep -q exits, while the shell's
ignored pipe signal means the producer only notices the closed reader on a later write. If the
subscription message is the final log record, the function never reaches the level restoration, and
the stated ten-second timeout does not bound the whole pipeline.
Agent Prompt
## Issue description
The timeout covers only `grep`, while the shell still waits for the persistent `tail -f` process after a successful match.

## Fix Focus Areas
- smoke/_init.sh[512-516]

## Recommended Fix
Run the log follower under explicit lifecycle management: capture its PID, wait for the bounded matcher, and terminate and reap the follower immediately on either success or timeout. Preserve the matcher status before cleanup so timeout still calls `fail`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment thread smoke/_init.sh
Comment on lines +512 to +514
{ tail -f -n +$((end + 1)) $tmp/grout.log || : ; } |
timeout 10 grep -qE 'GR_EVENT_SUBSCRIBE.*Success' ||
fail "timeout after 10s waiting for grcli events"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

3. Another subscriber can bypass the wait 🐞 Bug ≡ Correctness

The readiness check accepts any successful GR_EVENT_SUBSCRIBE log record without verifying the
logged client PID belongs to the newly launched event command. During daemon restart, the surviving
routing process reconnects and issues the same request, so its subscription can satisfy the check
while the event command is still racing with subsequent interface creation.
Agent Prompt
## Issue description
The readiness expression cannot distinguish the smoke event client from other processes subscribing concurrently, particularly the routing process reconnecting after restart.

## Fix Focus Areas
- smoke/_init.sh[498-514]
- main/api.c[335-345]

## Recommended Fix
Launch `grcli events` so its actual process PID is captured, then require both that PID and the successful subscription request in the daemon log. Ensure pipelines or process substitutions do not replace the captured PID with that of `tee` or `awk`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

The event stream is started in the background and races with interface
creation. If the subscription lands after an interface is created, its
add event is missed and grcli prints the interface by its numeric id,
which breaks wait_event pattern matching. This happens regularly on slow
CI runners and never locally.

Redirect the grout logs to files and tail them, so the harness can watch
them. When starting the event stream, temporarily raise the grout.api
log level to debug and block until the daemon logs the successful
GR_EVENT_SUBSCRIBE before returning. Interfaces are then only created
once the stream is guaranteed to receive their events.

Signed-off-by: Robin Jarry <rjarry@redhat.com>
@christophefontaine
christophefontaine merged commit bef6fe5 into DPDK:main Sep 18, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants