[Python][C++] pyarrow parquet writer - missing time logical type

@quanghgx

pyarrow.parquet.write_table used for write Parquet file
In parquet schema missed logical type for TIME, it’s just long type

PyArrow Schema

NUMBER: int64
DECIMAL: int64
NUMERIC: int64
INT: int64
FLOAT: double
VARCHAR: string
TEXT: string
CHAR: string
BOOLEAN: bool
ARR: string
VAR: string
OBJ: string
TIMESTAMP: timestamp[ns]
DATE: date64[ms]
TIME: time64[ns]
PK: int64
UUID: binary
UUID2: string
UUID3: string

Parquet schema

{
  "type" : "record",
  "name" : "schema",
  "fields" : [ {
    "name" : "NUMBER",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "DECIMAL",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "NUMERIC",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "INT",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "FLOAT",
    "type" : [ "null", "double" ],
    "default" : null
  }, {
    "name" : "VARCHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TEXT",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "CHAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "BOOLEAN",
    "type" : [ "null", "boolean" ],
    "default" : null
  }, {
    "name" : "ARR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "VAR",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "OBJ",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "TIMESTAMP",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-micros"
    } ],
    "default" : null
  }, {
    "name" : "DATE",
    "type" : [ "null", {
      "type" : "int",
      "logicalType" : "date"
    } ],
    "default" : null
  }, {
    "name" : "TIME",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "PK",
    "type" : [ "null", "long" ],
    "default" : null
  }, {
    "name" : "UUID",
    "type" : [ "null", "bytes" ],
    "default" : null
  }, {
    "name" : "UUID2",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "UUID3",
    "type" : [ "null", "string" ],
    "default" : null
  } ]
}

Reporter: Vadim Goy
Assignee: Quang Hoang / @quanghgx

_{Note: This issue was originally created as ARROW-17506. Please see the migration documentation for further details.}

@lidavidm

Vadim Goy:
@lidavidm , can I get any update? Maybe need something additional from my side?

@lidavidm

David Li / @lidavidm:
I'm not really familiar for the Parquet reader, so I was following this more out of interest. If you don't get a reply here, maybe try users@arrow.apache.org? I don't see anything in the docs about this, it seems the logical type should be set appropriately.

apacheGH-32766: [CI][C++] Use ARROW_CMAKE_ARGS not CMAKE_ARGS

GH-37266: [CI][C++] Use ARROW_CMAKE_ARGS not CMAKE_ARGS (#37272)

@pitrou

@pitrou was this being marked as completed by #37272 intended?

@rok

@rok Oops, no, it seems something misbehaved here. Let me undo this.

Is this still unfixed?

>>> pq.write_table(mytab, 'example.parquet')
>>> mytab
pyarrow.Table
DATE_OF_BIRTH: timestamp[ns, tz=UTC-8]

and schema of "example.parquet" is:

"Columns": [
     { "Id": "0", "Name": "DATE_OF_BIRTH", "PhysicalType": "INT64", "ConvertedType": "TIMESTAMP_MICROS", "LogicalType": {"Type": "Timestamp", "isAdjustedToUTC": true, "timeUnit": "microseconds", "is_from_converted_type": false, "force_set_converted_type": false} }
  ],

How can I reproduce this issue?

apacheGH-37266: [CI][C++] Use ARROW_CMAKE_ARGS not CMAKE_ARGS (apache…

@thisisnic

I don't think this is a bug in the writer. time64[ns] has been written with the Parquet TIME(NANOS) logical type since ARROW-6077 (pyarrow 0.15, 2019), and this is covered in arrow_schema_test.cc and arrow_reader_writer_test.cc. On pyarrow 24.0.0:

import io, pyarrow as pa, pyarrow.parquet as pq
t = pa.table({"TIME": pa.array([1, 2], type=pa.time64("ns"))})
buf = io.BytesIO()
pq.write_table(t, buf)
c = pq.ParquetFile(buf).metadata.schema.column(0)
print(c.physical_type, c.converted_type, c.logical_type)
# INT64 NONE Time(isAdjustedToUTC=false, timeUnit=nanoseconds)

What the report is showing is an Avro-style schema dump (parquet-mr / parquet-tools), and that converter works from the legacy ConvertedType, not the newer LogicalType. Parquet only has TIME_MILLIS and TIME_MICROS converted types; there is no TIME_NANOS, so a TIME(NANOS) column has no ConvertedType to fall back to and the tool prints a bare long. The other columns in the same dump confirm this: DATE shows logicalType: date and TIMESTAMP shows timestamp-micros because those do have ConvertedType equivalents.

If you need legacy readers to see the annotation, write the column as time64[us] (which maps to TIME_MICROS) rather than nanoseconds. Note that since 22.0.0 (#47316) TIME_MICROS is only emitted when write_time_adjusted_to_utc=True, because the legacy converted types imply UTC semantics.

@thisisnic, @rok I think this can be closed as not a bug (behaviour of parquet-mr's Avro schema converter, not the Arrow writer).

Note: I used Claude Code to help research this, but I read the ticket, the code and the linked commits myself and the conclusion here is mine, not the tool's.

@stackedsax

Thanks for the investigation @stackedsax, I agree this can be closed as not a bug.

Navigation Menu

Repository navigation

[Python][C++] pyarrow parquet writer - missing time logical type #32766

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python][C++] pyarrow parquet writer - missing time logical type #32766

Description

Activity

asfimport commented on Aug 26, 2022

asfimport commented on Aug 29, 2022

rok commented on Aug 22, 2023

pitrou commented on Aug 22, 2023

mapleFU commented on Aug 22, 2023

stackedsax commented on Oct 4, 2026

adamreeve commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions