Repository navigation
Fix duplicate load job recovery when the table is missing - #233
Watson1978 merged 2 commits into
Conversation
…issing With prevent_duplicate_load, a 409 from jobs.insert is resolved by polling the existing job in the table's location. When auto_create_table is on and the table does not exist yet (the duplicate job is the one creating it), get_table raised 404 and left table nil, so table.location raised NoMethodError and the chunk was retried forever. Fall back to the dataset location, which is where the table and the load job live. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix. Falling back to the dataset location looks right, since a load job runs in the location of its destination dataset.
I checked the change with mocked API calls. The new test fails on master and passes on this branch. A chunk restored from a file buffer after a restart now polls the existing job too, with both use_delayed_commit true and false.
One issue is left in the error handling of the new datasets.get call. Please see the inline comment.
Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. The issue from my last review is fixed.
Errors from datasets.get are now wrapped like the other API calls. With a <secondary> and the default retry settings, a 403 accessDenied or a 404 there now sends the chunk to the secondary on the next flush. This also works with retry_forever true. A 503 from datasets.get is still retried. The new test fails on 60b366d with the raw Google::Apis::ClientError and passes here.
14a9bce
into
fluent-plugins-nursery:master
With
prevent_duplicate_load true,Writer#create_load_jobbuilds a deterministic job id and handles a409 Already Exists: Jobfromjobs.insertby returning aJobReferencewithtable.location, so the existing job is polled in the right region.tablecomes from theget_tablecall a few lines above. Whenauto_create_table trueis set and the table does not exist yet, that call raises404 Not Found: Table, the rescue adds the schema to the load configuration, andtablestaysnil. That is exactly the state in which the duplicate job is likely: the first attempt's load job is the one creating the table, and the chunk is retried (timeout, restart, flush retry) before it finishes. The 409 branch then hitsFluentd treats this
NoMethodErroras unrecoverable, so the chunk goes to the<secondary>, or without one to the backup directory, before the existing job is checked.The table lives in its dataset's location, so when
tableis nil this reads it fromdatasets.getinstead.Errors from
datasets.getare wrapped withFluent::BigQuery::Error.wrap, like the other API calls. So a 503 there is retried, and a 403 or 404 raisesUnRetryableError, which sends the chunk to the<secondary>if one is configured.Test
test_write_with_prevent_duplicate_load_while_auto_creating_table(test_out_bigquery_load.rb):get_tableraises 404,insert_jobraises 409,get_datasetreturnsasia-northeast1; the test checks the job ID prefix and the location passed tofetch_load_job.test_write_with_not_retryable_error_from_dataset_location: the same, butget_datasetraises 403; the test expectsUnRetryableErrorwith status code 403 and an immediate switch to the secondary.fetch_load_job ... Called 0 times, because Fluentd handles theNoMethodErroras unrecoverable before the job is polled.bundle exec rake test37 runs, 0 failures.