Repository navigation
perf(aws): GetLogEvents を時間範囲チャンキングで並列化 (#26) - #27
Merged
Merged
Conversation
Issue #26 対応。単一ストリームで NextForwardToken による逐次 ページネーションが大規模ログ取得のボトルネックだったのを、 DescribeLogStreams で得た first/last event 時刻範囲を timeChunks 等分し、各窓を StartTime/EndTime 指定で並列に GetLogEvents する 方式に置き換える。 - LogStream に FirstEventTimestamp 追加 + toLogStream ヘルパー - DescribeLogStream / GetLogEventsByTimeRange を追加 - GetMultiStreamLogEvents を時間チャンク並列に再実装 - グローバルセマフォ maxConcurrent=8 で AWS スロットルを回避 - range 不明 / 極小ストリームは逐次 GetLogEvents へフォールバック Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GetLogEventsをNextForwardToken逐次ページネーションから時間範囲チャンキング並列フェッチに変更し、300k 行クラスのログ取得を大幅高速化(Issue feat: It takes a long time to retrieve the logs #26)DescribeLogStreamsでfirstEventTimestamp/lastEventTimestampを取得 →planTimeChunksで[first, last+1)を最大 8 等分 → 各窓をStartTime/EndTime指定で並列GetLogEventsmaxConcurrent = 8で全 API 呼び出しの並列度を抑制(CloudWatch Logs 25 TPS 制限への配慮)。describe失敗 / range 不正 / 極小ストリームは逐次GetLogEventsへフォールバックWhy this works
GetLogEventsのNextForwardTokenはトークン依存のため単一ストリームでは並列化不能。一方StartTime包含・EndTime排他の時間窓指定をサポートするため、[first, last+1)を隣接窓に分割すれば重複もギャップもなく並列に投げられる。同一ミリ秒の境界イベントは右側の窓に入る。想定効果: 300,000行 × ~30 直列呼び出し ≒ 90s → 8 並列で律速チャンク数回 ≈ ~10x 短縮見込み(密度の偏りに依存)。
Changes
LogStreamにFirstEventTimestampを追加 +toLogStreamヘルパーへ整理DescribeLogStream(group, stream)— exact-match で単一ストリーム metadata を取得GetLogEventsByTimeRange(group, stream, startMs, endMs)—StartTime/EndTime指定 + ページネーションGetMultiStreamLogEventsを時間チャンク並列に再実装(チャンクは時間順・非重複なので concat で timestamp 昇順を維持)TestFetchMultiLogEvents_ConcurrencyLimitをセマフォ仕様に追従Test plan
go test ./...全テスト Greengofmt -l差分なし /go vet ./...クリーンCloses #26
🤖 Generated with Claude Code