This function allows to convert an input file to parquet format.
It handles SAS, SPSS and Stata files in a same function. There is only one function to use for these 3 cases.
For these 3 cases, the function guesses the data format using the extension of the input file (in the `path_to_file` argument).
Two conversions possibilities are offered :
Convert to a single parquet file. Argument `path_to_parquet` must then be used;
Convert to a partitioned parquet file. Additionnal arguments `partition` and `partitioning` must then be used;
To avoid overcharging R's RAM, the conversion can be done by chunk. One of arguments `max_memory` or `max_rows` must then be used. This is very useful for huge tables and for computers with little RAM because the conversion is then done with less memory consumption. For more information, see here.
Usage
table_to_parquet(
path_to_file,
path_to_parquet,
max_memory = NULL,
max_rows = NULL,
chunk_size = lifecycle::deprecated(),
chunk_memory_size = lifecycle::deprecated(),
columns = "all",
by_chunk = lifecycle::deprecated(),
skip = 0,
partition = "no",
encoding = NULL,
chunk_memory_sample_lines = 10000,
compression = "snappy",
compression_level = NULL,
user_na = FALSE,
...
)
Arguments
- path_to_file
String that indicates the path to the input file (don't forget the extension).
- path_to_parquet
String that indicates the path to the directory where the parquet files will be stored.
- max_memory
Memory size (in Mb) in which data of one parquet file should roughly fit.
- max_rows
Number of lines that defines the size of the chunk. This argument can not be filled in if max_memory is used.
- chunk_size
DEPRECATED use max_rows
- chunk_memory_size
DEPRECATED use max_memory
- columns
Character vector of columns to select from the input file (by default, all columns are selected).
- by_chunk
DEPRECATED use max_memory or max_rows instead
- skip
By default 0. This argument must be filled in if `by_chunk` is TRUE. Number of lines to ignore when converting.
- partition
String ("yes" or "no" - by default) that indicates whether you want to create a partitioned parquet file. If "yes", `"partitioning"` argument must be filled in. In this case, a folder will be created for each modality of the variable filled in `"partitioning"`. Be careful, this argument can not be "yes" if `max_memory` or `max_rows` argument are not NULL.
- encoding
String that indicates the character encoding for the input file.
- chunk_memory_sample_lines
Number of lines to read to evaluate max_memory. Default to 10 000.
- compression
compression algorithm. Default "snappy".
- compression_level
compression level. Meaning depends on compression algorithm.
- user_na
If `TRUE` variables with user defined missing will be read into [haven::labelled_spss()] objects. If `FALSE`, the default, user-defined missings will be converted to `NA`.
- ...
Additional format-specific arguments, see arrow::write_parquet() and arrow::write_dataset() for more informations.
Examples
# Conversion from a SAS file to a single parquet file :
table_to_parquet(
path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
path_to_parquet = tempfile(fileext = ".parquet")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpHNodKR/file164824ea7fda.parquet
#> Writing data...
#> Reading data...
#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpHNodKR/file164824ea7fda.parquet
#> Reading data...
# Conversion from a SPSS file to a single parquet file :
table_to_parquet(
path_to_file = system.file("examples","iris.sav", package = "haven"),
path_to_parquet = tempfile(fileext = ".parquet"),
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpHNodKR/file1648585a0e3f.parquet
#> Writing data...
#> Reading data...
#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sav file is available in parquet format under /tmp/RtmpHNodKR/file1648585a0e3f.parquet
#> Reading data...
# Conversion from a Stata file to a single parquet file without progress bar :
table_to_parquet(
path_to_file = system.file("examples","iris.dta", package = "haven"),
path_to_parquet = tempfile(fileext = ".parquet")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpHNodKR/file164856345fa6.parquet
#> Writing data...
#> Reading data...
#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.dta file is available in parquet format under /tmp/RtmpHNodKR/file164856345fa6.parquet
#> Reading data...
# Reading SPSS file by chunk (using `max_rows` argument)
# and conversion to multiple parquet files :
table_to_parquet(
path_to_file = system.file("examples","iris.sav", package = "haven"),
path_to_parquet = tempfile(),
max_rows = 50,
)
#> Reading data...
#> Writing file1648606aadd9-1-50.parquet...
#> Reading data...
#> Writing file1648606aadd9-51-100.parquet...
#> Reading data...
#> Writing file1648606aadd9-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpHNodKR/file1648606aadd9/
#> Reading data...
# Reading SPSS file by chunk (using `max_memory` argument)
# and conversion to multiple parquet files of 5 Kb when loaded (5 Mb / 1024)
# (in real files, you should use bigger value that fit in memory like 3000
# or 4000) :
table_to_parquet(
path_to_file = system.file("examples","iris.sav", package = "haven"),
path_to_parquet = tempfile(),
max_memory = 5 / 1024
)
#> Reading data...
#> Writing file16485b511f08-1-82.parquet...
#> Reading data...
#> Writing file16485b511f08-83-150.parquet...
#> ✔ Data are available in parquet dataset under /tmp/RtmpHNodKR/file16485b511f08/
#> Writing file16485b511f08-83-150.parquet...
# Reading SAS file by chunk of 50 lines with encoding
# and conversion to multiple files :
table_to_parquet(
path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
path_to_parquet = tempfile(),
max_rows = 50,
encoding = "utf-8"
)
#> Reading data...
#> Writing file164852bd11c7-1-50.parquet...
#> Reading data...
#> Writing file164852bd11c7-51-100.parquet...
#> Reading data...
#> Writing file164852bd11c7-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpHNodKR/file164852bd11c7/
#> Reading data...
# Conversion from a SAS file to a single parquet file and select only
# few columns :
table_to_parquet(
path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
path_to_parquet = tempfile(fileext = ".parquet"),
columns = c("Species","Petal_Length")
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet file under /tmp/RtmpHNodKR/file1648d9ee031.parquet
#> Writing data...
#> Reading data...
#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpHNodKR/file1648d9ee031.parquet
#> Reading data...
# Conversion from a SAS file to a partitioned parquet file :
table_to_parquet(
path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
path_to_parquet = tempfile(),
partition = "yes",
partitioning = c("Species") # vector use as partition key
)
#> Reading data...
#> Writing data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpHNodKR/file16485b102c89
#> Writing data...
#> Reading data...
#> ✔ The /home/runner/work/_temp/Library/haven/examples/iris.sas7bdat file is available in parquet format under /tmp/RtmpHNodKR/file16485b102c89
#> Reading data...
# Reading SAS file by chunk of 50 lines
# and conversion to multiple files with zstd, compression level 10
if (isTRUE(arrow::arrow_info()$capabilities[['zstd']])) {
table_to_parquet(
path_to_file = system.file("examples","iris.sas7bdat", package = "haven"),
path_to_parquet = tempfile(),
max_rows = 50,
compression = "zstd",
compression_level = 10
)
}
#> Reading data...
#> Writing file16487cce7be6-1-50.parquet...
#> Reading data...
#> Writing file16487cce7be6-51-100.parquet...
#> Reading data...
#> Writing file16487cce7be6-101-150.parquet...
#> Reading data...
#> ✔ Data are available in parquet dataset under /tmp/RtmpHNodKR/file16487cce7be6/
#> Reading data...