Skip to content

Fix duplicate load job recovery when the table is missing - #233

Merged
Watson1978 merged 2 commits into
fluent-plugins-nursery:masterfrom
lbarrosby:fix-duplicate-load-job-missing-table
Nov 18, 2026
Merged

Watson1978 merged 2 commits into
fluent-plugins-nursery:masterfrom
lbarrosby:fix-duplicate-load-job-missing-table

Conversation

@lbarrosby

@lbarrosby lbarrosby commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

With prevent_duplicate_load true, Writer#create_load_job builds a deterministic job id and handles a 409 Already Exists: Job from jobs.insert by returning a JobReference with table.location, so the existing job is polled in the right region.

table comes from the get_table call a few lines above. When auto_create_table true is set and the table does not exist yet, that call raises 404 Not Found: Table, the rescue adds the schema to the load configuration, and table stays nil. That is exactly the state in which the duplicate job is likely: the first attempt's load job is the one creating the table, and the chunk is retried (timeout, restart, flush retry) before it finishes. The 409 branch then hits

NoMethodError: undefined method 'location' for nil

Fluentd treats this NoMethodError as unrecoverable, so the chunk goes to the <secondary>, or without one to the backup directory, before the existing job is checked.

The table lives in its dataset's location, so when table is nil this reads it from datasets.get instead.

Errors from datasets.get are wrapped with Fluent::BigQuery::Error.wrap, like the other API calls. So a 503 there is retried, and a 403 or 404 raises UnRetryableError, which sends the chunk to the <secondary> if one is configured.

Test

test_write_with_prevent_duplicate_load_while_auto_creating_table (test_out_bigquery_load.rb): get_table raises 404, insert_job raises 409, get_dataset returns asia-northeast1; the test checks the job ID prefix and the location passed to fetch_load_job.

test_write_with_not_retryable_error_from_dataset_location: the same, but get_dataset raises 403; the test expects UnRetryableError with status code 403 and an immediate switch to the secondary.

  • master: the first test fails with fetch_load_job ... Called 0 times, because Fluentd handles the NoMethodError as unrecoverable before the job is polled.
  • this branch: both pass; bundle exec rake test 37 runs, 0 failures.

…issing

With prevent_duplicate_load, a 409 from jobs.insert is resolved by polling
the existing job in the table's location. When auto_create_table is on and
the table does not exist yet (the duplicate job is the one creating it),
get_table raised 404 and left table nil, so table.location raised
NoMethodError and the chunk was retried forever. Fall back to the dataset
location, which is where the table and the load job live.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. Falling back to the dataset location looks right, since a load job runs in the location of its destination dataset.

I checked the change with mocked API calls. The new test fails on master and passes on this branch. A chunk restored from a file buffer after a restart now polls the existing job too, with both use_delayed_commit true and false.

One issue is left in the error handling of the new datasets.get call. Please see the inline comment.

Comment thread lib/fluent/plugin/bigquery/writer.rb Resolved
@Watson1978 Watson1978 added the bug label Nov 17, 2026
Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@Watson1978 Watson1978 changed the title Use the dataset location for a duplicate load job when the table is missing Fix duplicate load job recovery when the table is missing Nov 18, 2026

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The issue from my last review is fixed.

Errors from datasets.get are now wrapped like the other API calls. With a <secondary> and the default retry settings, a 403 accessDenied or a 404 there now sends the chunk to the secondary on the next flush. This also works with retry_forever true. A 503 from datasets.get is still retried. The new test fails on 60b366d with the raw Google::Apis::ClientError and passes here.

@Watson1978
Watson1978 merged commit 14a9bce into fluent-plugins-nursery:master Nov 18, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants