Skip to contents

Convert an input file to parquet format

Source: R/table_to_parquet.R
table_to_parquet.Rd

This function allows to convert an input file to parquet format.

It handles SAS, SPSS and Stata files in a same function. There is only one function to use for these 3 cases. For these 3 cases, the function guesses the data format using the extension of the input file (in the `path_to_file` argument).

Two conversions possibilities are offered :

  • Convert to a single parquet file. Argument `path_to_parquet` must then be used;

  • Convert to a partitioned parquet file. Additionnal arguments `partition` and `partitioning` must then be used;

To avoid overcharging R's RAM, the conversion can be done by chunk. One of arguments `max_memory` or `max_rows` must then be used. This is very useful for huge tables and for computers with little RAM because the conversion is then done with less memory consumption. For more information, see here.

Usage

table_to_parquet(
  path_to_file,
  path_to_parquet,
  max_memory = NULL,
  max_rows = NULL,
  chunk_size = lifecycle::deprecated(),
  chunk_memory_size = lifecycle::deprecated(),
  columns = "all",
  by_chunk = lifecycle::deprecated(),
  skip = 0,
  partition = "no",
  encoding = NULL,
  chunk_memory_sample_lines = 10000,
  compression = "snappy",
  compression_level = NULL,
  user_na = FALSE,
  ...
)

Arguments

path_to_file

String that indicates the path to the input file (don't forget the extension).

path_to_parquet

String that indicates the path to the directory where the parquet files will be stored.

max_memory

Memory size (in Mb) in which data of one parquet file should roughly fit.

max_rows

Number of lines that defines the size of the chunk. This argument can not be filled in if max_memory is used.

chunk_size

DEPRECATED use max_rows

chunk_memory_size

DEPRECATED use max_memory

columns

Character vector of columns to select from the input file (by default, all columns are selected).

by_chunk

DEPRECATED use max_memory or max_rows instead

skip

By default 0. This argument must be filled in if `by_chunk` is TRUE. Number of lines to ignore when converting.

partition

String ("yes" or "no" - by default) that indicates whether you want to create a partitioned parquet file. If "yes", `"partitioning"` argument must be filled in. In this case, a folder will be created for each modality of the variable filled in `"partitioning"`. Be careful, this argument can not be "yes" if `max_memory` or `max_rows` argument are not NULL.

encoding

String that indicates the character encoding for the input file.

chunk_memory_sample_lines

Number of lines to read to evaluate max_memory. Default to 10 000.

compression

compression algorithm. Default "snappy".

compression_level

compression level. Meaning depends on compression algorithm.

user_na

If `TRUE` variables with user defined missing will be read into [haven::labelled_spss()] objects. If `FALSE`, the default, user-defined missings will be converted to `NA`.

...

Additional format-specific arguments, see arrow::write_parquet() and arrow::write_dataset() for more informations.

Value

Parquet files, invisibly

Examples

# Conversion from a SAS file to a single parquet file :

table_to_parquet(
  path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
  path_to_parquet = tempfile(fileext = ".parquet")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpTd607Q/file18547d55772f.parquet
#> Writing data...

#> Reading data...

#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpTd607Q/file18547d55772f.parquet
#> Reading data...


# Conversion from a SPSS file to a single parquet file :

table_to_parquet(
  path_to_file = system.file("examples","iris.sav", package = "haven"),
  path_to_parquet = tempfile(fileext = ".parquet"),
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpTd607Q/file18545c0df9a4.parquet
#> Writing data...

#> Reading data...

#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sav file is available in parquet format under /tmp/RtmpTd607Q/file18545c0df9a4.parquet
#> Reading data...

# Conversion from a Stata file to a single parquet file without progress bar :

table_to_parquet(
  path_to_file = system.file("examples","iris.dta", package = "haven"),
  path_to_parquet = tempfile(fileext = ".parquet")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpTd607Q/file18541e2a4c96.parquet
#> Writing data...

#> Reading data...

#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.dta file is available in parquet format under /tmp/RtmpTd607Q/file18541e2a4c96.parquet
#> Reading data...


# Reading SPSS file by chunk (using `max_rows` argument)
# and conversion to multiple parquet files :

table_to_parquet(
  path_to_file = system.file("examples","iris.sav", package = "haven"),
  path_to_parquet = tempfile(),
  max_rows = 50,
)
#> Reading data...
#> Writing file18544e7c8928-1-50.parquet...
#> Reading data...
#> Writing file18544e7c8928-51-100.parquet...
#> Reading data...
#> Writing file18544e7c8928-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpTd607Q/file18544e7c8928/
#> Reading data...


# Reading SPSS file by chunk (using `max_memory` argument)
# and conversion to multiple parquet files of 5 Kb when loaded (5 Mb / 1024)
# (in real files, you should use bigger value that fit in memory like 3000
# or 4000) :

table_to_parquet(
  path_to_file = system.file("examples","iris.sav", package = "haven"),
  path_to_parquet = tempfile(),
  max_memory = 5 / 1024
)
#> Reading data...
#> Writing file185424130f8-1-82.parquet...
#> Reading data...
#> Writing file185424130f8-83-150.parquet...
#> ✔ Data are available in parquet dataset under /tmp/RtmpTd607Q/file185424130f8/
#> Writing file185424130f8-83-150.parquet...


# Reading SAS file by chunk of 50 lines with encoding
# and conversion to multiple files :

table_to_parquet(
  path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
  path_to_parquet = tempfile(),
  max_rows = 50,
  encoding = "utf-8"
)
#> Reading data...
#> Writing file18545277cf17-1-50.parquet...
#> Reading data...
#> Writing file18545277cf17-51-100.parquet...
#> Reading data...
#> Writing file18545277cf17-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpTd607Q/file18545277cf17/
#> Reading data...


# Conversion from a SAS file to a single parquet file and select only
# few columns  :

table_to_parquet(
  path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
  path_to_parquet = tempfile(fileext = ".parquet"),
  columns = c("Species","Petal_Length")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpTd607Q/file185441e5d08b.parquet
#> Writing data...

#> Reading data...

#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpTd607Q/file185441e5d08b.parquet
#> Reading data...


# Conversion from a SAS file to a partitioned parquet file  :

table_to_parquet(
  path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
  path_to_parquet = tempfile(),
  partition = "yes",
  partitioning =  c("Species") # vector use as partition key
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpTd607Q/file18546d4280f2
#> Writing data...

#> Reading data...

#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpTd607Q/file18546d4280f2
#> Reading data...


# Reading SAS file by chunk of 50 lines
# and conversion to multiple files with zstd, compression level 10

if (isTRUE(arrow::arrow_info()$capabilities[['zstd']])) {
  table_to_parquet(
    path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
    path_to_parquet = tempfile(),
    max_rows = 50,
    compression = "zstd",
    compression_level = 10
  )
}
#> Reading data...
#> Writing file18546cc94562-1-50.parquet...
#> Reading data...
#> Writing file18546cc94562-51-100.parquet...
#> Reading data...
#> Writing file18546cc94562-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpTd607Q/file18546cc94562/
#> Reading data...