Repository navigation
Send time_partitioning_expiration as an integer number of milliseconds - #231
Conversation
time_partitioning_expiration is a Fluentd time parameter, so a plain number such as 2592000 is parsed as a Float. Multiplied by 1000 it became 2592000000.0, which the client serializes as "2592000000.0", and BigQuery rejects the load job or the table creation with "Invalid value at 'job.configuration.load.time_partitioning.expiration_ms.value' (TYPE_INT64)". Values with a unit such as 30d were Integers and worked. Convert the milliseconds to an Integer, and say in the README that the parameter takes a time value and is sent in milliseconds. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix and the clear write-up.
I checked that a plain number such as 2592000 now gives "expirationMs":"2592000000" for both the load job and insert_table with auto_create_table. Values with a unit are unchanged. The new test fails on master as described.
Two requests, with details inline. The new test reaches the real BigQuery API, so it fails without network access. The README should also say that a plain number is in seconds. The NIT on the assertion is optional.
A plain time_partitioning_expiration is in seconds. The test now mocks fetch_load_job and commit_load_job so it runs without network access, and checks that expiration_ms is an Integer. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks for checking. Pushed f02fa82: The new test now mocks
|
|
The The workflows on master pin |
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the quick update. All three points are addressed.
The new test no longer reaches BigQuery, and it passes with the network blocked. With master's writer.rb, it fails at the new assert_kind_of check. The README now says that a plain number is in seconds.
6b6aee3
into
fluent-plugins-nursery:master
Fixes #220
time_partitioning_expirationis declared as a Fluentd:timeparameter, and the writer sendsvalue * 1000asexpiration_ms. A unit value (30d,720h) is parsed as an Integer, but a plain number (2592000) is parsed as a Float, soexpiration_msbecame2592000000.0. The client serializes that int64 field as"2592000000.0"and BigQuery refuses the load job (orinsert_tableforbigquery_insertwithauto_create_table):This converts the milliseconds with
to_i. The README described the parameter as "Expiration milliseconds", which is what led to the report in #220 (users entered milliseconds); it now says it takes a time value and is sent in milliseconds.Reproduction
Request body built by the client for the load job (
Job::Representation), master:{"configuration":{"load":{"timePartitioning":{"expirationMs":"2592000000.0","type":"DAY"}, ...}}}this branch:
{"configuration":{"load":{"timePartitioning":{"expirationMs":"2592000000","type":"DAY"}, ...}}}How the parameter is parsed (Fluentd 1.20.1):
Tests
test_write_with_auto_create_table_time_partitioning_expiration_in_secondsintest/plugin/test_out_bigquery_load.rb: captures the job passed toinsert_joband serializes it withGoogle::Apis::BigqueryV2::Job::Representation, asserting"expirationMs" => "2592000000". Fails on master with"2592000000.0".bundle exec rake test: 35 tests, 0 failures (Ruby 4.0.7).