Skip to content

Add compression option to bigquery_load output - #232

Open
0kate wants to merge 1 commit into
fluent-plugins-nursery:masterfrom
0kate:feat/bigquery-load-gzip-compression
Open

0kate wants to merge 1 commit into
fluent-plugins-nursery:masterfrom
0kate:feat/bigquery-load-gzip-compression

Conversation

@0kate

@0kate 0kate commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add compression (none / gzip) and compression_level parameters to the bigquery_load output.

With compression gzip, each chunk is gzip-compressed into a temporary file before it is uploaded for the load job. BigQuery detects gzip-compressed input automatically, so the job configuration does not change. The default is none, so existing behavior is unchanged.

Motivation

When fluentd runs outside Google Cloud (e.g. on AWS), every load job upload is billed as internet egress. Newline-delimited JSON compresses well, so compressing the upload cuts transfer volume and cost a lot.

Changes

  • compression: none (default) or gzip.
  • compression_level: zlib level 0-9, or -1 (default, zlib's default, equivalent to 6). Other values raise Fluent::ConfigError.
  • compression gzip together with source_format avro raises Fluent::ConfigError, because BigQuery does not accept gzip-compressed Avro files.
  • The chunk is written with chunk.write_to. A buffer configured with compress gzip is therefore decompressed first and re-compressed into a single gzip stream, instead of uploading the concatenated gzip members in the buffer file.
  • At debug level, the plugin logs raw and compressed bytes and elapsed time for each chunk.
  • README and tests are updated.

Trade-offs

  • fluentd side: compression uses CPU. Zlib releases the GVL while compressing, so other threads are not blocked.
  • BigQuery side: BigQuery cannot read a gzip file in parallel, so load jobs for large chunks can take longer. This is documented in the README.

Test

  • bundle exec rake test: 40 tests, 0 failures (Ruby 3.4).

🤖 Generated with Claude Code

Add `compression` (`none` / `gzip`) and `compression_level` parameters
to the bigquery_load output. With `compression gzip`, each chunk is
gzip-compressed into a temporary file before it is uploaded for the
load job. BigQuery detects gzip-compressed input automatically, so no
change to the job configuration is needed.

This reduces network egress for newline-delimited JSON / CSV uploads,
which typically compress to 10-30% of their size. The trade-off is CPU
on the fluentd side and slower load jobs for large chunks, because
BigQuery cannot read a gzip file in parallel. `source_format avro` is
rejected because BigQuery does not accept gzip-compressed Avro files.

The chunk is written through `chunk.write_to`, so a buffer configured
with `compress gzip` is decompressed first and re-compressed into a
single gzip stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@0kate
0kate marked this pull request as ready for review November 17, 2026 05:12
@0kate

0kate commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@joker1007
Could you take a look at this PR when you have time?

It adds an opt-in compression gzip option to bigquery_load, so that load job uploads are sent gzip-compressed.

This mainly helps when fluentd runs outside Google Cloud and uploads are billed as internet egress.
The default stays none, so existing configurations are not affected.

@joker1007

Copy link
Copy Markdown
Collaborator

I think it would be fine if the buffer itself were stored using gzip compression,
but for now, I think implementing it with a simple feature would be a good idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants