Conversation
Store the source timezone in column metadata and reconstruct the arrow `TimestampType` with that zone instead of relying on the fixed-width globals (which are hardcoded to UTC). On read, values are returned in the schema's zone rather than forced to UTC, so the original offset survives the round trip into targets such as Snowflake `TIMESTAMP_TZ`. - Default timestamp unit is now microsecond (iceberg-compatible) - Zone-less `datetime` columns stay zone-less instead of being labelled UTC - Append path shifts zone-less wall-clock values onto UTC so digits are preserved without re-expressing the instant
…ection Call streamPart.Cancel when exiting loops early in importViaHTTP and WriteDataflowReadyViaDuckDB, so the producer doesn't stay parked on an unbuffered send and hang the process. Add stall detection to the DuckDB query watcher: if a query produces no output for 10 minutes (configurable via SLING_DUCKDB_STALL_TIMEOUT, 0 disables), the process is killed and the reader is unblocked. This handles cases where the DuckDB process is alive but wedged and the scanner can't acquire the process mutex to make progress.
When duckdb's read_arrow stops reading after it has enough data, the arrow/csv writer's flush blocks forever on the pipe with no reader. Add closeArrowWriter to abort the flush when the stream context is cancelled, and closeBatchReader to fail fast on abandoned batch pipes. Bridge the dataflow and import contexts so failures propagate across both, and bump the stall timeout from 10 to 60 minutes to avoid killing legitimate long-running direct_insert queries.
- Add metadata `sling:declaredType` to Arrow fields to retain the original iop type. - Prevents a zone-labeled `datetime` from being promoted to `timestamptz` when converting back from Arrow. - This ensures CDC rows match snapshot rows and target DDLs are generated correctly as `TIMESTAMP` rather than `TIMESTAMPTZ`. - Add test to verify the zone label is preserved without altering the column type.
The max_line_size bump added in v1.5.19 for hex-encoded binary columns only triggered when a binary column was present. Sources with large text-class columns (sqlserver text/ntext/xml/varchar(max), clob) still generated read_csv with the default max_line_size=2000000, so any row whose serialized CSV line exceeded 2 MB failed the whole stream with: Invalid Input Error: CSV Error ... Maximum line size of 2000000 bytes exceeded. Actual Size:5022477 bytes. Extend the existing condition to also bump max_line_size to 256 MB when any column is TextType, mirroring the binary/hex raise. Fixes #787 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1JYFWjSiEVsGKWjGVpqr1
Add unit test cases for the read_csv max_line_size computation in DataflowToHttpStream, covering all three branches: binary column raise (v1.5.19, previously untested), text-class column raise (#787 fix), and the 2 MB default for plain string columns. Also correct the debug message emitted when arrow streaming is disabled via SLING_DUCKDB_ARROW — it referred to "duckdb extension arrow" but the toggle disables sling's arrow streaming format, not a DuckDB extension. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1JYFWjSiEVsGKWjGVpqr1
The ErrorHelper hint for "Invalid Input Error: CSV Error on Line:" unconditionally suggested `copy_method: arrow_http`, but that property only applies to DuckDB / MotherDuck / DuckLake connections — for other sources (e.g. sqlserver) the hint is a dead end (noted in #787). - ErrorHelper now accepts the task's connection types (variadic, so the existing signature remains compatible) and only suggests arrow_http when a DuckDB-class connection is involved. - Add a specific help message for the max_line_size-exceeded failure signature, which previously fell through to the misleading arrow_http hint. - Add unit tests covering the tailored help strings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H1JYFWjSiEVsGKWjGVpqr1
Add nil checks for source and target primary key columns in GenerateMergeConfigWithStrategy to prevent nil pointer dereferences and provide a clearer error message when a specified primary key is missing.
Extract MaxLineSize into a reusable method on DuckDb and apply it consistently to CSV imports via temp files, named pipes, and HTTP streaming. The method bumps the line limit for string columns (text, binary, etc.) to 256 MB instead of DuckDB's 2 MB default, and honors a new `max_line_size` property override. This fixes the case where large strings or binaries caused failures during temp-CSV and named-pipe ingestion, which previously used the hardcoded default.
Raise duckdb read_csv max_line_size for large text/xml columns (fixes #787)
Previously, ADBC ingestion ignored the database/catalog part of a 3-part table name (database.schema.table) and ingested data into the connection's default database. This sets the `Catalog` property in `IngestStreamOptions` and updates the SQL Server metadata template to query the correct database context. Adds a test case to verify ingestion into a non-default database (issue #785).
createBuilder had no case for TIME32, TIME64, or EXTENSION (UUID) arrow types, so they fell through to a string builder while the schema declared them as time/uuid. This caused a panic when building the record due to the type mismatch, making any parquet write with a time or uuid column fail. Add explicit cases for TIME32/TIME64 builders and use the generic array.NewBuilder for EXTENSION types so arrow selects the proper builder (e.g. UUIDBuilder). Added a test that writes and reads back time, timez, and uuid columns, including null values.
Open remote reader bodies lazily via ReaderReady.Open to prevent idle connections from being reset before they are consumed. Previously, readers were opened eagerly and queued in a channel, leaving remote HTTP bodies idle until the pipe consumer reached them.
MergeReaders opened each body lazily at first read, serializing remote opens behind a single-file consumer. Add a prefetch worker pool that opens bodies ahead of the consumer while keeping concurrently open bodies bounded by the worker count plus channel buffer, rather than growing with the file count (issue #789). Bodies that a remote store resets while waiting are reopened at read time. Also skip duplicate paths in listings, since stores like Google Drive can list the same file more than once. Update tests to assert the new bounded prefetch window and cover bodies reset mid-stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
SQL Server named instances:
host\instanceandhost/instanceare parsed into host + instance. Warns when port1433is set alongside an instance name (the driver ignores the instance), and surfaces a clearer error when SQL Server Browser (UDP 1434) cannot resolve the instance.DuckDB
max_line_sizeoverride: Connection propertymax_line_sizenow overrides DuckDBread_csvline limits across temp-file, named-pipe, and HTTP import paths.Bug Fixes
DuckDB large text/XML columns:
read_csvmax_line_sizeis raised to 256 MB for text-class columns (not only binary), so largetext/ntext/xml/varchar(max)/clobvalues no longer fail at the 2 MB default (fixes read_csv max_line_size (2MB) not raised for large text/xml columns — extend the v1.5.19 hex-blob raise #787). CSV error hints now suggestcopy_method: arrow_httponly for DuckDB-class connections.ADBC 3-part table names: Ingestion now targets the catalog/database in
database.schema.tableinstead of the connection default database (fixes Error "Invalid object name 'Destschema.desttable_tmp'" during ADBC snowflake-to-mssql replication on Windows #785).Parquet TIME and UUID columns: Parquet writes with
time/timez/uuidcolumns no longer panic from an Arrow builder type mismatch.Multi-file reader hangs and duplicate listings: Remote file bodies are opened lazily and prefetched with a bounded worker pool (issue S3 Connection #789), instead of holding one idle connection per file. Duplicate listing paths (e.g. Google Drive) are skipped.
Missing merge primary key: Merge config generation no longer nil-dereferences when a specified primary key column is absent; it returns a clear error instead.
Timestamp timezone through Arrow: Source timezone is preserved through Arrow conversion (e.g. Snowflake
TIMESTAMP_TZ). Zone-labeleddatetimestaysTIMESTAMPinstead of being promoted toTIMESTAMPTZ. Iceberg appends stamp timestamps as UTC so they match Icebergtimestamptz.DuckDB producer/consumer hangs: Import cancels the producer when the consumer stops, aborts Arrow/CSV flushes on cancelled streams, and kills stalled DuckDB queries after 60 minutes (
SLING_DUCKDB_STALL_TIMEOUT;0disables).CDC snapshot logs: Per-stream CDC snapshot output is attributed to the correct stream instead of mixing into the group log.
MySQL/MariaDB CDC timestamps:
TIMESTAMPkeeps its offset (timestamptz) andDATETIMEstaysdatetime, so CDC rows match the snapshot path instead of writingTIMESTAMP_NTZto targets.