Skip to content

Send time_partitioning_expiration as an integer number of milliseconds - #231

Merged
Watson1978 merged 2 commits into
fluent-plugins-nursery:masterfrom
lbarrosby:fix-partition-expiration-integer-ms
Nov 15, 2026
Merged

Watson1978 merged 2 commits into
fluent-plugins-nursery:masterfrom
lbarrosby:fix-partition-expiration-integer-ms

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

Fixes #220

time_partitioning_expiration is declared as a Fluentd :time parameter, and the writer sends value * 1000 as expiration_ms. A unit value (30d, 720h) is parsed as an Integer, but a plain number (2592000) is parsed as a Float, so expiration_ms became 2592000000.0. The client serializes that int64 field as "2592000000.0" and BigQuery refuses the load job (or insert_table for bigquery_insert with auto_create_table):

Invalid value at 'job.configuration.load.time_partitioning.expiration_ms.value' (TYPE_INT64), "2592000000.0"

This converts the milliseconds with to_i. The README described the parameter as "Expiration milliseconds", which is what led to the report in #220 (users entered milliseconds); it now says it takes a time value and is sent in milliseconds.

Reproduction

<match **>
  @type bigquery_load
  ...
  auto_create_table true
  time_partitioning_type day
  time_partitioning_expiration 2592000
</match>

Request body built by the client for the load job (Job::Representation), master:

{"configuration":{"load":{"timePartitioning":{"expirationMs":"2592000000.0","type":"DAY"}, ...}}}

this branch:

{"configuration":{"load":{"timePartitioning":{"expirationMs":"2592000000","type":"DAY"}, ...}}}

How the parameter is parsed (Fluentd 1.20.1):

2592000  -> 2592000.0 -> expiration_ms 2592000000.0
2592000s -> 2592000   -> expiration_ms 2592000000
30d      -> 2592000   -> expiration_ms 2592000000

Tests

  • New test_write_with_auto_create_table_time_partitioning_expiration_in_seconds in test/plugin/test_out_bigquery_load.rb: captures the job passed to insert_job and serializes it with Google::Apis::BigqueryV2::Job::Representation, asserting "expirationMs" => "2592000000". Fails on master with "2592000000.0".
  • bundle exec rake test: 35 tests, 0 failures (Ruby 4.0.7).

time_partitioning_expiration is a Fluentd time parameter, so a plain
number such as 2592000 is parsed as a Float. Multiplied by 1000 it
became 2592000000.0, which the client serializes as "2592000000.0",
and BigQuery rejects the load job or the table creation with
"Invalid value at 'job.configuration.load.time_partitioning.expiration_ms.value' (TYPE_INT64)".
Values with a unit such as 30d were Integers and worked.

Convert the milliseconds to an Integer, and say in the README that the
parameter takes a time value and is sent in milliseconds.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix and the clear write-up.

I checked that a plain number such as 2592000 now gives "expirationMs":"2592000000" for both the load job and insert_table with auto_create_table. Values with a unit are unchanged. The new test fails on master as described.

Two requests, with details inline. The new test reaches the real BigQuery API, so it fails without network access. The README should also say that a plain number is in seconds. The NIT on the assertion is optional.

Comment thread test/plugin/test_out_bigquery_load.rb Resolved
Comment thread test/plugin/test_out_bigquery_load.rb Resolved
Comment thread README.md Resolved
A plain time_partitioning_expiration is in seconds. The test now mocks
fetch_load_job and commit_load_job so it runs without network access,
and checks that expiration_ms is an Integer.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Contributor Author

Thanks for checking. Pushed f02fa82:

The new test now mocks fetch_load_job and commit_load_job like test_write, so it no longer reaches BigQuery. I ran it with the network blocked through a dead proxy and it passes. I also added the assert_kind_of(Integer, ...) check, and the README now says a plain number is in seconds.

bundle exec rake test: 35 tests, 0 failures.

@Watson1978

Copy link
Copy Markdown
Contributor

The Ruby 3.0 unit testing on windows-latest failure is not caused by this PR. ruby/setup-ruby fails before the tests run, while it downloads the MSYS2 mingw64 build tools for Ruby 3.0:

https://github.com/ruby/setup-msys2-gcc/releases/latest/download/msys2-mingw64-mingw-w64@r688-gcc@14-openssl@1.1.7z
Error: Unexpected HTTP response: 404

The workflows on master pin ruby/setup-ruby to v1.335.0, which takes this file from the latest release of ruby/setup-msys2-gcc. The latest release no longer has the mingw64 files. v1.338.0 points to a fixed release instead. So this needs a setup-ruby update on master. No change is needed in this PR for it.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick update. All three points are addressed.

The new test no longer reaches BigQuery, and it passes with the network blocked. With master's writer.rb, it fails at the new assert_kind_of check. The README now says that a plain number is in seconds.

@Watson1978 Watson1978 added the bug label Nov 15, 2026
@Watson1978
Watson1978 merged commit 6b6aee3 into fluent-plugins-nursery:master Nov 15, 2026
17 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

time_partitioning_expiration should be provided in days

2 participants