Skip to content

Latest commit

 

History

History
123 lines (91 loc) · 4.76 KB

README.md

File metadata and controls

123 lines (91 loc) · 4.76 KB
Raw

test codecov Crates.io Crates.io

orc-rust

A native Rust implementation of the Apache ORC file format, providing API's to read data into Apache Arrow in-memory arrays.

See the documentation for examples on how to use this crate.

Supported features

This crate currently only supports reading ORC files into Arrow arrays. Write support is planned (see Roadmap). The below features listed relate only to reading ORC files. At this time, we aim to support the ORCv1 specification only.

  • Read synchronously & asynchronously (using Tokio)
  • All compression types (Zlib, Snappy, Lzo, Lz4, Zstd)
  • All ORC data types
  • All encodings
  • Rudimentary support for retrieving statistics
  • Retrieving user metadata into Arrow schema metadata

Roadmap

The long term vision for this crate is to be feature complete enough to be donated to the arrow-rs project.

The following lists the rough roadmap for features to be implemented, from highest to lowest priority.

  • Performance enhancements
  • Predicate pushdown
  • Row indices
  • Bloom filters
  • Write from Arrow arrays
  • Encryption

A non-Arrow API interface is not planned at the moment. Feel free to raise an issue if there is such a use case.

Version compatibility

No guarantees are provided about stability across versions. We will endeavour to keep the top level API's (ArrowReader and ArrowStreamReader) as stable as we can, but other API's provided may change as we explore the interface we want the library to expose.

Versions will be released on an ad-hoc basis (with no fixed schedule).

Mapping ORC types to Arrow types

The following table lists how ORC data types are read into Arrow data types:

ORC Data Type Arrow Data Type Notes
Boolean Boolean
TinyInt Int8
SmallInt Int16
Int Int32
BigInt Int64
Float Float32
Double Float64
String Utf8
Char Utf8
VarChar Utf8
Binary Binary
Decimal Decimal128
Date Date32
Timestamp Timestamp(Nanosecond, None) ¹
Timestamp instant Timestamp(Nanosecond, UTC) ¹
Struct Struct
List List
Map Map
Union Union(_, Sparse) ²

¹: ArrowReaderBuilder::with_schema allows configuring different time units or decoding to Decimal128(38, 9) (i128 of non-leap nanoseconds since UNIX epoch). Overflows may happen while decoding to a non-Seconds time unit, and results in OrcError. Loss of precision may happen while decoding to a non-Nanosecond time unit, and results in OrcError. Decimal128(38, 9) avoids both overflows and loss of precision.

²: Currently only supports a maximum of 127 variants

Contributing

All contributions are welcome! Feel free to raise an issue if you have a feature request, bug report, or a question. Feel free to raise a Pull Request without raising an issue first, as long as the Pull Request is descriptive enough.

Some tools we use in addition to the standard cargo that require installation are:

cargo install typos-cli
cargo install taplo-cli
# Building the crate
cargo build

# Running the test suite
cargo test

# Simple benchmarks
cargo bench

# Formatting TOML files
taplo format

# Detect any typos in the codebase
typos

To regenerate/update the proto.rs file, execute the regen.sh script.

./regen.sh