[R] Update the docs to show how to avoid situations like data loss with leading zero in partition column

Describe the bug, including details regarding any error messages, version, and platform.

Arrow does not correctly interpret some partition values as character columns, resulting in data loss.

library(arrow)
library(dplyr)

# character col with leading zero
mtcars2 <- mutate(mtcars, test = paste0(0, mtcars$carb)) |> as_tibble()

tmp <- tempfile()

write_dataset(mtcars2, tmp, partitioning = "test")

mtcars3 <- open_dataset(tmp) |> collect()

unique(mtcars2$test)
#> [1] "04" "01" "02" "03" "06" "08"
unique(mtcars3$test)
#> [1] 1 2 3 4 6 8

^{Created on 2024-01-17 with reprex v2.0.2}

Component(s)

R

@paleolimbot

Looking at the generated dataset it correctly writes the leading zero: 'test=08' but the parquet reader implicitly converts to int. So the fix probably needs to happen in the C++ lib, right @paleolimbot

I think it may work to explicitly set the partitioning argument in open_dataset()...something like open_dataset(..., partitioning = hive_partitioning(test = string()).

@paleolimbot

Thanks @paleolimbot, that does work:

r$> mtcars4 = open_dataset(tmp, partitioning = hive_partition(test = string()))

r$> mtcars4
FileSystemDataset with 6 Parquet files
mpg: double
cyl: double
disp: double
hp: double
drat: double
wt: double
qsec: double
vs: double
am: double
gear: double
carb: double
test: string

r$> mtcars4 |> as_tibble()
# A tibble: 32 × 12
     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb test
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <chr>
 1  22.8     4 108      93  3.85  2.32  18.6     1     1     4     1 01
 2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1 01
 3  18.1     6 225     105  2.76  3.46  20.2     1     0     3     1 01
 4  32.4     4  78.7    66  4.08  2.2   19.5     1     1     4     1 01
 5  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1 01
 6  21.5     4 120.     97  3.7   2.46  20.0     1     0     3     1 01
 7  27.3     4  79      66  4.08  1.94  18.9     1     1     4     1 01
 8  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2 02
 9  24.4     4 147.     62  3.69  3.19  20       1     0     4     2 02
10  22.8     4 141.     95  3.92  3.15  22.9     1     0     4     2 02

This is great stuff! We should add this to the docs as it's not immediately obvious (i.e. I had no idea this was the case)

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

Anyone working on this?

apacheGH-39660: [R] Document partition value types

apacheGH-39660: refine partitioning example

Issue resolved by pull request 50857
#50857

apacheGH-39660: [R] Document partition value types (apache#50857)

Navigation Menu

Repository navigation

[R] Update the docs to show how to avoid situations like data loss with leading zero in partition column #39660

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[R] Update the docs to show how to avoid situations like data loss with leading zero in partition column #39660

Description

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Activity

assignUser commented on Jan 17, 2024

paleolimbot commented on Jan 18, 2024

assignUser commented on Jan 18, 2024

thisisnic commented on Jan 30, 2024

github-actions commented on Nov 18, 2025

nil68657 commented on Aug 10, 2026

thisisnic commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions