perf(workflow): batch step persistence after execution

[xiami762]

Summary

• keep workflow step callbacks free of SQLite calls and event-loop future waits by collecting compacted steps in memory
• atomically persist the final execution summary and its complete step batch through a dedicated completion connection
• preserve the initial queued execution row for manual runs and high-frequency Kafka, Syslog, Poller, and custom trigger paths
• update workflow stats and enforce retention after terminal persistence as separate best-effort operations
• coalesce manual-run progress updates without blocking the workflow runner thread
• allow Kafka workflows that do not use session tools to opt out of ToolContext creation with tool_context_required: false
• reinitialize workflow database connections safely after fork() without waiting on inherited aiosqlite worker threads

Persistence behavior and guarantees

The normal high-frequency trigger write sequence is:

1. initial queued execution row: one commit
2. final execution summary plus all execution steps: one atomic completion commit
3. workflow stats UPSERT: one separate commit
4. retention cleanup: one separate commit
5. audit history: best-effort background persistence

Only the final summary and its step batch share the completion transaction. If that transaction fails, both are rolled back. Stats and retention run after the terminal transaction; failures are logged and do not roll back the already committed summary and steps.

Step callbacks do not call record_step, record_steps, or run_coroutine_threadsafe. Completed step details are buffered until terminal persistence, so running executions expose current-node and progress summary fields but detailed step rows become queryable after completion.

Performance validation

Real SQLite benchmark for the completion transaction only, with four concurrent executions and seven steps each:

• callback SQL calls: 0
• callback time per worker: 0.015-0.037 ms
• completion wall time for all four executions: 1.363 ms
• completion commits: 4
• persisted steps: 28
• persisted executions: 4

These completion numbers intentionally exclude the queued-row, stats, retention, progress, and audit writes described above. End-to-end trigger persistence normally performs at least four SQLite commits per execution.

Production validation also confirmed that the Kafka ToolContext opt-out removes the multi-second temporary Session/Message setup for workflows that do not require session tools.

Safety bounds and trade-offs

• workflow execution is capped by the engine at 10,000 steps
• Kafka execution concurrency is capped at 8 workers per workflow
• step inputs and outputs are compacted before buffering and persistence
• running step details are not persisted incrementally; this avoids restoring the SQLite contention that caused the original queue backlog

Validation

• tests/server/routes/test_workflow_run_route.py: 16 passed
• tests/workflow/test_workflow_store.py: 7 passed
• tests/ingest/test_kafka_manager.py: 18 passed
• Ruff and git diff --check: passed