Skip to content

Commit

Permalink
Merge pull request #13 from StephenTaylor-Kx/2023.03.01
Browse files Browse the repository at this point in the history
merge/move docs from code.kx.com to GitHub
  • Loading branch information
nmcdonnell-kx authored Mar 1, 2023
2 parents 73ef9fc + b0fecd2 commit a40803e
Show file tree
Hide file tree
Showing 6 changed files with 362 additions and 472 deletions.
24 changes: 13 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# arrowkdb

![Arrow](apache_arrow.png)

[![GitHub release (latest by date)](https://img.shields.io/github/v/release/kxsystems/arrowkdb?include_prereleases)](https://github.com/kxsystems/arrowkdb/releases) [![Travis (.com) branch](https://travis-ci.com/KxSystems/arrowkdb.svg?branch=main)](https://travis-ci.com/KxSystems/arrowkdb)


Expand All @@ -17,7 +19,7 @@ This is part of the [*Fusion for kdb+*](http://code.kx.com/q/interfaces/fusion/)

## New to kdb+ ?

Kdb+ is the world's fastest time-series database, optimized for ingesting, analyzing and storing massive amounts of structured data. To get started with kdb+, please visit https://code.kx.com/q/learn/ for downloads and developer information. For general information, visit https://kx.com/
Kdb+ is the world's fastest time-series database, optimized for ingesting, analyzing and storing massive amounts of structured data. To get started with kdb+, please visit https://code.kx.com/q/ for downloads and developer information. For general information, visit https://kx.com/



Expand Down Expand Up @@ -48,7 +50,7 @@ Conversely, Arrow is an in-memory format meant for direct and efficient use for



### Third-Party Library Installation
### Third-party library installation

#### Linux

Expand All @@ -61,7 +63,7 @@ sudo apt install -y -V libarrow-dev=9.0.0-1
sudo apt install -y -V libparquet-dev=9.0.0-1
```

#### MacOS
#### macOS

Follow the instructions [here](https://arrow.apache.org/install/#c-and-glib-c-packages-on-homebrew) to install `apache-arrow` using Homebrew.

Expand All @@ -76,7 +78,7 @@ C:\Git> git clone https://github.com/apache/arrow.git
C:\Git> cd arrow
```

Switch to the 9.0.0 tag:
Switch to the `9.0.0` tag:

```bash
C:\Git\arrow> git checkout refs/tags/apache-arrow-9.0.0 --
Expand Down Expand Up @@ -118,12 +120,12 @@ C:\Git\arrow\cpp\build> MKLINK %QHOME%\w64\parquet.dll %BUILD_HOME%\bin\parquet.

It is recommended that a user install this interface through a release. This is completed in a number of steps:

1. Ensure you have downloaded/installed the Arrow C++ API following the instructions [here](https://github.com/KxSystems/arrowkdb#third-party-library-installation).
2. Download a release from [here](https://github.com/KxSystems/arrowkdb/releases) for your system architecture.
1. Ensure you have downloaded/installed the Arrow C++ API following the [instructions](#third-party-library-installation).
2. [Download a release](https://github.com/KxSystems/arrowkdb/releases) for your system architecture.
3. Install script `arrowkdb.q` to `$QHOME`, and binary file `lib/arrowkdb.(so|dll)` to `$QHOME/[mlw](64)`, by executing the following from the Release directory:

```bash
## Linux/MacOS
## Linux/macOS
chmod +x install.sh && ./install.sh

## Windows
Expand All @@ -139,7 +141,7 @@ In order to successfully build and install this interface from source, the follo
1. `ARROW_INSTALL` = Location of the Arrow C++ API release (only required if Arrow is not installed globally on the system, e.g. on Windows where Arrow was built from source)
2. `QHOME` = Q installation directory (directory containing `q.k`)

From a shell prompt (on Linux/MacOS) or Visual Studio command prompt (on Windows), clone the `arrowkdb` source from github:
From a shell prompt (on Linux/macOS) or Visual Studio command prompt (on Windows), clone the `arrowkdb` source from github:

```bash
git clone https://github.com/KxSystems/arrowkdb.git
Expand Down Expand Up @@ -175,14 +177,14 @@ cmake --build . --config Release --target install

## Documentation

Documentation outlining the functionality available for this interface can be found [here](https://code.kx.com/q/interfaces/arrow/).
Documentation outlining the functionality available for this interface can be found in the [`docs`](docs/index.md) folder.



## Status

The arrowkdb interface is provided here under an Apache 2.0 license.

If you find issues with the interface or have feature requests, please consider raising an issue [here](https://github.com/KxSystems/arrowkdb/issues).
If you find issues with the interface or have feature requests, please consider [raising an issue](https://github.com/KxSystems/arrowkdb/issues).

If you wish to contribute to this project, please follow the contributing guide [here](https://github.com/KxSystems/arrowkdb/blob/main/CONTRIBUTING.md).
If you wish to contribute to this project, please follow the [contribution guide](CONTRIBUTING.md).
Binary file added apache_arrow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 0 additions & 10 deletions docs/arrow-types.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,8 @@
---
title: Type mapping between Arrow and kdb+
description: The data layout of an Arrow table is defined by its schema.
author: Neal McDonnell
date: February 2021
---
# Type mapping between Arrow and kdb+


The data layout of an Arrow table is defined by its schema. The schema is composed from a list of fields, one for each column in the table. The field describes the name of the column and its datatype. This page examines each of these and details how they are mapped in kdb+.

:fontawesome-brands-github:
[KxSystems/arrowkdb](https://github.com/KxSystems/arrowkdb)


## Arrow datatypes

Currently Arrow supports over 35 datatypes including concrete, parameterized and nested datatypes.
Expand Down
15 changes: 3 additions & 12 deletions docs/examples.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,10 @@
---
title: Example usage of interface | Arrow/Parquet interface
description: Examples of how to read and write Parquet files, Arrow files and Arrow streams from a kdb+ session
author: Neal McDonnell
date: February 2021
---
# Example usage of interface

_Examples of how to read and write Parquet files, Arrow files and Arrow streams from a kdb+ session_


The repository has examples with more functionality.

:fontawesome-brands-github:
[KxSystems/arrowkdb/examples](https://github.com/KxSystems/arrowkdb/tree/master/examples)


## Inferred schemas

Expand Down Expand Up @@ -542,6 +533,6 @@ multi_comments:

It is left as an exercise to write the schema and array data to Parquet or Arrow files.

??? tip "Remember to use Parquet v2.0"

Otherwise the `timestamp(ns)` datatype will be converted to `timestamp(us)` resulting in a loss of precision.
> :warning: **Remember to use Parquet v2.0**
>
> Otherwise the `timestamp(ns)` datatype will be converted to `timestamp(us)` resulting in a loss of precision.
26 changes: 6 additions & 20 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,6 @@
---
title: Using Apache Arrow/Parquet data with kdb+
description: Apache Arrow is a software-development platform for building high-performance applications that process and transport large datasets
author: Neal McDonnell
date: February 2021
---
![Arrow](../img/apache_arrow.png)
![Arrow](../apache_arrow.png)
# Using Apache Arrow/Parquet data with kdb+

:fontawesome-brands-github:
[KxSystems/arrowkdb](https://github.com/KxSystems/arrowkdb)



[Apache Arrow](https://arrow.apache.org/) is a software-development platform for building high-performance applications that process and transport large datasets. It is designed to both improve the performance of analytical algorithms and the efficiency of moving data from one system (or programming language to another).

Expand Down Expand Up @@ -57,22 +47,18 @@ Users can read and write Arrow tables created from kdb+ data using:

Separate APIs are provided where the Arrow table is either created from a kdb+ table using an inferred schema or from an Arrow schema and the table’s list of array data.

:fontawesome-regular-hand-point-right:
:point_right:
[API reference](reference.md)
<br>
:fontawesome-regular-hand-point-right:
:point_right:
[Example implementations](examples.md)
<br>
:fontawesome-brands-github:
[Install guide](https://github.com/KxSystems/arrowkdb#installation)
[Install guide](../README.md#installation)


## Project

The `arrowkdb` interface is published under an Apache 2.0 license.

:fontawesome-brands-github:
[Raise an issue](https://github.com/KxSystems/arrowkdb/issues)
<br>
:fontawesome-brands-github:
[Contribute](https://github.com/KxSystems/arrowkdb/blob/master/CONTRIBUTING.md)
- [Raise an issue](https://github.com/KxSystems/arrowkdb/issues)
- [Contribute](../CONTRIBUTING.md)
Loading

0 comments on commit a40803e

Please sign in to comment.