Special/Unicode Characters Not Parsing Properly  #104

Activity

1. TomeCirun commented on Jun 29, 2023

   

   TomeCirun

   on Jun 29, 2023

   Contributor

   More actions

   

   

   Working fine on the dev-v1.0 branch with QSV version 0.91.0 @sagargg

   We are currently testing and preparing the DP+ for a version 1.0 release
   more info here
2. jqnatividad commented on Jul 5, 2023

   

   jqnatividad

   on Jul 5, 2023

   Collaborator

   More actions

   Hi @sagargg ,
   Can you confirm what version of DP+ you're using? 0.15.0 should also work without issues.

   If its still a problem, what's the encoding of the file you're uploading?

   DP+ normalizes non UTF8-encoded files to their "lossy" encoding - i.e. replacing invalid UTF-8 sequences with U+FFFD REPLACEMENT CHARACTER, which is the � character you're getting instead of the euro currency symbol.
3. sagargg commented on Jul 7, 2023

   

   sagargg

   on Jul 7, 2023

   Author

   More actions

   @jqnatividad Maybe it is happening only on macOS.
   

   Even it is happening on the latest version of qsv.
4. jqnatividad commented on Jul 7, 2023

   

   jqnatividad

   on Jul 7, 2023

   Last edited by jqnatividad

   Collaborator

   More actions

   @sagargg , do you mind uploading a sample file here?

   I also have macOS (M2 Pro Mac Mini) and it works without issues.

   testeuro.csv

   $ qsvdp --version
   qsvdp 0.108.0-standard-Luau 0.581;self_update-12-12;25.60 GiB-0B-0B-32.00 GiB (aarch64-apple-darwin compiled with Rust 1.70.0) prebuilt
   
   $ qsvdp input testeuro.csv 
   Name,Age,City,Amount
   John,25,New York,$2000
   Jane,30,London,€2000
   
   $ qsvdp sniff testeuro.csv 
   Path: /Users/joelnatividad/Local/GitHub/qsv/testeuro.csv
   Sniff Timestamp: 2023-07-07T04:07:46.429694+00:00
   Last Modified: 2023-07-07T04:06:33+00:00
   Delimiter: ,
   Header Row: true
   Preamble Rows: 0
   Quote Char: none
   Flexible: false
   Is UTF8: true
   Detected Mime Type: application/csv
   Retrieved Size (bytes): 67
   File Size (bytes): 67
   Sampled Records: 2
   Estimated: false
   Num Records: 2
   Avg Record Len (bytes): 19
   Num Fields: 4
   Stats Types: false
   Fields:
       0:  Text      Name
       1:  Unsigned  Age
       2:  Text      City
       3:  Text      Amount
   
5. jqnatividad commented on Jul 7, 2023

   

   jqnatividad

   on Jul 7, 2023

   Collaborator

   More actions

   BTW, I just noticed you're on a very old version of qsv - 0.92.0.

   You should upgrade to qsv 0.108.0
6. sagargg commented on Jul 7, 2023

   

   sagargg

   on Jul 7, 2023

   Author

   More actions

   @jqnatividad yeah, it worked with the file that you've shared.
   

   Here is the file i'm using.
   csv.csv
7. jqnatividad commented on Jul 7, 2023

   

   jqnatividad

   on Jul 7, 2023

   Last edited by jqnatividad

   Collaborator

   More actions

   The encoding of the file is not utf-8, that's why qsv input did a lossy-transcoding to utf8.

   $ file -I csv.csv
   csv.csv: text/plain; charset=iso-8859-1
   
   $ file -I testeuro.csv 
   testeuro.csv: text/csv; charset=utf-8
   
8. sagargg commented on Jul 7, 2023

   

   sagargg

   on Jul 7, 2023

   Last edited by sagargg

   Author

   More actions

   okay. Now I understood, Thank you for clarifying.
9. dathere

   locked and limited conversation to collaborators on Jul 7, 2023
10. jqnatividad

    converted this issue into a discussion #110 on Jul 7, 2023