Repository navigation
fix(csv): report rows that cannot be imported instead of changing or dropping them - #237
Open
HarshMN2345 wants to merge 3 commits into
Open
HarshMN2345 wants to merge 3 commits into
HarshMN2345 wants to merge 3 commits into
Conversation
A CSV import stored NULL or a cast value for cells that do not parse as their column type, trimmed text, read a quoted "null" as NULL, kept the UTF-8 byte order mark in the first header, aborted on a row with the wrong number of columns, and let one non-UTF-8 value drop its whole batch without an error. Each such row is now rejected with its row number and counted as an error, while the rest of the file is imported. Text keeps its whitespace, only an unquoted null is NULL, and an empty cell is NULL for every column except text. The export quotes the text "null" so it reads back unchanged.
Rows were marked as succeeded when buffered, so a batch the database rejected still counted every row but one as a success. A rejected last row also skipped the write, dropping the rows buffered before it. The batch is now written even when its last row is rejected, a failed write marks every row in it as an error, and row counters are updated after the write.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CSV import (
Sources\CSV):Row 5: 'age' must be an integer.) instead of storing NULL or a cast value. This covers integers (including1.7and out-of-range values), floats, booleans, array elements and JSON columns. Whole numbers written as1.0or1e3still import. Datetime parsing is unchanged and matches the REST API.error. The first 100 are listed in the migration errors.nullis NULL, so"null"imports as text. An empty cell is NULL for every column except text, including enum, email, ip and url.nullis also NULL for array columns.CSV export (
Destinations\CSV): the textnullis written quoted, so it reads back as text.Row writes (
Destinations\Appwrite):successonly once its batch is written. A batch the database rejects marks every row in it aserror. Before, a 300-row file with one bad value in row 200 stored 200 rows and reportedsuccess: 299, error: 1.$idin the last row dropped every row buffered before it.Verification
Local Appwrite 2.4.0 on Postgres, with this branch mounted over
vendor/utopia-php/migration:POST /v1/migrations/csv/importsand was read back. The table hasvarchar,integer,float,boolean,datetime,enum,email,varchar[],integer[]andtextcolumns. Results: the 300-row file with one bad value stores 200 rows and reportssuccess: 200, error: 100. Wrong column count in one row stores 299 and reportserror: 1. The BOM file keeps its$ids. The Latin-1 file stores the valid row and reportsRow 2 is not valid UTF-8."null", NULL, padded text and arrays now gives the same values back.testCreateCSVImport*,testCreateCSVExport,testCreateJSONImport*,testAppwriteMigrationDatabasesRow,testAppwriteMigrationRowsOnDuplicateandtestAppwriteMigrationOverwriteDropsOrphanColumnpass. Themissing-row.csvassertion was updated to the new error.fgetcsv()on about 600k generated records for,;\t|. There were no mismatches.composer lintandcomposer checkare clean.Fixes appwrite/appwrite#14333