Repository navigation
Conversation
Add `compression` (`none` / `gzip`) and `compression_level` parameters to the bigquery_load output. With `compression gzip`, each chunk is gzip-compressed into a temporary file before it is uploaded for the load job. BigQuery detects gzip-compressed input automatically, so no change to the job configuration is needed. This reduces network egress for newline-delimited JSON / CSV uploads, which typically compress to 10-30% of their size. The trade-off is CPU on the fluentd side and slower load jobs for large chunks, because BigQuery cannot read a gzip file in parallel. `source_format avro` is rejected because BigQuery does not accept gzip-compressed Avro files. The chunk is written through `chunk.write_to`, so a buffer configured with `compress gzip` is decompressed first and re-compressed into a single gzip stream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0kate
marked this pull request as ready for review
November 17, 2026 05:12
Contributor
Author
|
@joker1007 It adds an opt-in This mainly helps when fluentd runs outside Google Cloud and uploads are billed as internet egress. |
Collaborator
|
I think it would be fine if the buffer itself were stored using gzip compression, |
joker1007
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
compression(none/gzip) andcompression_levelparameters to thebigquery_loadoutput.With
compression gzip, each chunk is gzip-compressed into a temporary file before it is uploaded for the load job. BigQuery detects gzip-compressed input automatically, so the job configuration does not change. The default isnone, so existing behavior is unchanged.Motivation
When fluentd runs outside Google Cloud (e.g. on AWS), every load job upload is billed as internet egress. Newline-delimited JSON compresses well, so compressing the upload cuts transfer volume and cost a lot.
Changes
compression:none(default) orgzip.compression_level: zlib level0-9, or-1(default, zlib's default, equivalent to 6). Other values raiseFluent::ConfigError.compression gziptogether withsource_format avroraisesFluent::ConfigError, because BigQuery does not accept gzip-compressed Avro files.chunk.write_to. A buffer configured withcompress gzipis therefore decompressed first and re-compressed into a single gzip stream, instead of uploading the concatenated gzip members in the buffer file.Trade-offs
Test
bundle exec rake test: 40 tests, 0 failures (Ruby 3.4).🤖 Generated with Claude Code