Skip to content

Commit

Permalink
GitBook: [master] 186 pages and 77 assets modified
Browse files Browse the repository at this point in the history
  • Loading branch information
avaidyanatha authored and gitbook-bot committed Oct 8, 2021
1 parent c8c9905 commit ae32ecb
Show file tree
Hide file tree
Showing 228 changed files with 3,142 additions and 2,837 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Introduction

[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/airbytehq/airbyte/Airbyte%20CI)](https://github.com/airbytehq/airbyte/actions/workflows/gradle.yml) [![License](https://img.shields.io/static/v1?label=license&message=MIT&color=brightgreen)](./LICENSE) [![License](https://img.shields.io/static/v1?label=license&message=ELv2&color=brightgreen)](./LICENSE)
[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/airbytehq/airbyte/Airbyte%20CI)](https://github.com/airbytehq/airbyte/actions/workflows/gradle.yml) [![License](https://img.shields.io/static/v1?label=license&message=MIT&color=brightgreen)](https://github.com/airbytehq/airbyte/tree/a9b1c6c0420550ad5069aca66c295223e0d05e27/LICENSE/README.md) [![License](https://img.shields.io/static/v1?label=license&message=ELv2&color=brightgreen)](https://github.com/airbytehq/airbyte/tree/a9b1c6c0420550ad5069aca66c295223e0d05e27/LICENSE/README.md)

![](docs/.gitbook/assets/airbyte_new_logo.svg)

Expand All @@ -20,7 +20,7 @@ Airbyte is on a mission to make data integration pipelines a commodity.
* **No more security compliance process** to go through as Airbyte is self-hosted.
* **No more pricing indexed on volume**, as cloud-based solutions offer.

Here's a list of our [connectors with their health status](docs/integrations).
Here's a list of our [connectors with their health status](docs/integrations/).

## Quick start

Expand Down Expand Up @@ -52,7 +52,7 @@ If you want to schedule a 20-min call with our team to help you get set up, plea

We love contributions to Airbyte, big or small.

See our [Contributing guide](docs/contributing-to-airbyte/) on how to get started. Not sure where to start? We’ve listed some [good first issues](https://github.com/airbytehq/airbyte/labels/good%20first%20issue) to start with. If you have any questions, please open a draft PR or visit our [slack channel](slack.airbyte.io) where the core team can help answer your questions.
See our [Contributing guide](docs/contributing-to-airbyte/) on how to get started. Not sure where to start? We’ve listed some [good first issues](https://github.com/airbytehq/airbyte/labels/good%20first%20issue) to start with. If you have any questions, please open a draft PR or visit our [slack channel](https://github.com/airbytehq/airbyte/tree/a9b1c6c0420550ad5069aca66c295223e0d05e27/slack.airbyte.io) where the core team can help answer your questions.

**Note that you are able to create connectors using the language you want, as Airbyte connections run as Docker containers.**

Expand All @@ -73,5 +73,5 @@ Check out our [roadmap](docs/project-overview/roadmap.md) to get informed on wha

## License

See the [LICENSE](docs/project-overview/licenses/README.md) file for licensing information.
See the [LICENSE](docs/project-overview/licenses/) file for licensing information.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added docs/.gitbook/assets/launch (3) (3) (4).png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added docs/.gitbook/assets/schema (3) (3) (4).png
File renamed without changes
11 changes: 6 additions & 5 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
* [Set up a Connection](quickstart/set-up-a-connection.md)
* [Deploying Airbyte](deploying-airbyte/README.md)
* [Local Deployment](deploying-airbyte/local-deployment.md)
* [On Airbyte Cloud](deploying-airbyte/on-cloud.md)
* [On Airbyte Cloud](deploying-airbyte/on-cloud.md)
* [On AWS \(EC2\)](deploying-airbyte/on-aws-ec2.md)
* [On AWS ECS \(Coming Soon\)](deploying-airbyte/on-aws-ecs.md)
* [On Azure\(VM\)](deploying-airbyte/on-azure-vm-cloud-shell.md)
Expand Down Expand Up @@ -139,14 +139,14 @@
* [Databricks](integrations/destinations/databricks.md)
* [DynamoDB](integrations/destinations/dynamodb.md)
* [Chargify](integrations/destinations/keen.md)
* [Google Cloud Storage (GCS)](integrations/destinations/gcs.md)
* [Google Cloud Storage \(GCS\)](integrations/destinations/gcs.md)
* [Google PubSub](integrations/destinations/pubsub.md)
* [Kafka](integrations/destinations/kafka.md)
* [Keen](integrations/destinations/keen.md)
* [Keen](integrations/destinations/keen-1.md)
* [Local CSV](integrations/destinations/local-csv.md)
* [Local JSON](integrations/destinations/local-json.md)
* [MeiliSearch](integrations/destinations/meilisearch.md)
* [MongoDB](integrations/destinations/mongodb.md)
* [MongoDB](integrations/destinations/mongodb.md)
* [MSSQL](integrations/destinations/mssql.md)
* [MySQL](integrations/destinations/mysql.md)
* [Oracle DB](integrations/destinations/oracle.md)
Expand Down Expand Up @@ -179,7 +179,7 @@
* [HTTP-API-based Connectors](connector-development/cdk-python/http-streams.md)
* [Python Concepts](connector-development/cdk-python/python-concepts.md)
* [Stream Slices](connector-development/cdk-python/stream-slices.md)
* [Connector Development Kit \(Javascript\)](connector-development/cdk-faros-js/README.md)
* [Connector Development Kit \(Javascript\)](connector-development/cdk-faros-js.md)
* [Airbyte 101 for Connector Development](connector-development/airbyte101.md)
* [Testing Connectors](connector-development/testing-connectors/README.md)
* [Source Acceptance Tests Reference](connector-development/testing-connectors/source-acceptance-tests-reference.md)
Expand Down Expand Up @@ -227,3 +227,4 @@
* [On Setting up a New Connection](troubleshooting/new-connection.md)
* [On Running a Sync](troubleshooting/running-sync.md)
* [On Upgrading](troubleshooting/on-upgrading.md)

28 changes: 17 additions & 11 deletions docs/connector-development/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ To build a new connector in Java or Python, we provide templates so you don't ne

**Note: you are not required to maintain the connectors you create.** The goal is that the Airbyte core team and the community help maintain the connector.

## Python Connector-Development Kit (CDK)
## Python Connector-Development Kit \(CDK\)

You can build a connector very quickly in Python with the [Airbyte CDK](cdk-python/README.md), which generates 75% of the code required for you.
You can build a connector very quickly in Python with the [Airbyte CDK](cdk-python/), which generates 75% of the code required for you.

## TS/JS Connector-Development Kit (Faros AI Airbyte CDK)
## TS/JS Connector-Development Kit \(Faros AI Airbyte CDK\)

You can build a connector in TypeScript/JavaScript with the [Faros AI CDK](./cdk-faros-js/README.md), which generates and boostraps most of the code required for HTTP Airbyte sources.
You can build a connector in TypeScript/JavaScript with the [Faros AI CDK](https://github.com/airbytehq/airbyte/tree/01b905a38385ca514c2d9c07cc44a8f9a48ce762/docs/connector-development/cdk-faros-js/README.md), which generates and boostraps most of the code required for HTTP Airbyte sources.

## The Airbyte specification

Expand All @@ -25,7 +25,7 @@ Before building a new connector, review [Airbyte's data protocol specification](
To add a new connector you need to:

1. Implement & Package your connector in an Airbyte Protocol compliant Docker image
2. Add integration tests for your connector. At a minimum, all connectors must pass [Airbyte's standard test suite](testing-connectors/README.md), but you can also add your own tests.
2. Add integration tests for your connector. At a minimum, all connectors must pass [Airbyte's standard test suite](testing-connectors/), but you can also add your own tests.
3. Document how to build & test your connector
4. Publish the Docker image containing the connector

Expand All @@ -36,11 +36,13 @@ Each requirement has a subsection below.
If you are building a connector in any of the following languages/frameworks, then you're in luck! We provide autogenerated templates to get you started quickly:

#### Sources

* **Python Source Connector**
* [**Singer**](https://singer.io)**-based Python Source Connector**. [Singer.io](https://singer.io/) is an open source framework with a large community and many available connectors \(known as taps & targets\). To build an Airbyte connector from a Singer tap, wrap the tap in a thin Python package to make it Airbyte Protocol-compatible. See the [Github Connector](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-github-singer) for an example of an Airbyte Connector implemented on top of a Singer tap.
* **Generic Connector**: This template provides a basic starting point for any language.

#### Destinations

* **Java Destination Connector**
* **Python Destination Connector**

Expand All @@ -58,7 +60,7 @@ and choose the relevant template by using the arrow keys. This will generate a n
Search the generated directory for "TODO"s and follow them to implement your connector. For more detailed walkthroughs and instructions, follow the relevant tutorial:

* [Speedrun: Building a HTTP source with the CDK](tutorials/cdk-speedrun.md)
* [Building a HTTP source with the CDK](tutorials/cdk-tutorial-python-http)
* [Building a HTTP source with the CDK](tutorials/cdk-tutorial-python-http/)
* [Building a Python source](tutorials/building-a-python-source.md)
* [Building a Python destination](tutorials/building-a-python-destination.md)
* [Building a Java destination](tutorials/building-a-java-destination.md)
Expand All @@ -67,9 +69,9 @@ As you implement your connector, make sure to review the [Best Practices for Con

### 2. Integration tests

At a minimum, your connector must implement the acceptance tests described in [Testing Connectors](testing-connectors/README.md)
At a minimum, your connector must implement the acceptance tests described in [Testing Connectors](testing-connectors/)

**Note: Acceptance tests are not yet available for Python destination connectors. Coming [soon](https://github.com/airbytehq/airbyte/issues/4698)!**
**Note: Acceptance tests are not yet available for Python destination connectors. Coming** [**soon**](https://github.com/airbytehq/airbyte/issues/4698)**!**

### 3. Document building & testing your connector

Expand All @@ -88,10 +90,12 @@ When you submit a PR to Airbyte with your connector, the reviewer will use the c
2. `:airbyte-integrations:connectors:source-<name>:integrationTest` should run integration tests including Airbyte's Standard test suite.

### 4. Publish the connector
Typically this will be handled as part of code review by an Airbyter. There is a section below on what steps are needed for publishing a connector and will mostly be used by Airbyte employees publishing the connector.

Typically this will be handled as part of code review by an Airbyter. There is a section below on what steps are needed for publishing a connector and will mostly be used by Airbyte employees publishing the connector.

## Updating an existing connector
The steps for updating an existing connector are the same as for building a new connector minus the need to use the autogenerator to create a new connector. Therefore the steps are:

The steps for updating an existing connector are the same as for building a new connector minus the need to use the autogenerator to create a new connector. Therefore the steps are:

1. Iterate on the connector to make the needed changes
2. Run tests
Expand All @@ -100,7 +104,7 @@ The steps for updating an existing connector are the same as for building a new

## Publishing a connector

Once you've finished iterating on the changes to a connector as specified in its `README.md`, follow these instructions to ship the new version of the connector with Airbyte out of the box.
Once you've finished iterating on the changes to a connector as specified in its `README.md`, follow these instructions to ship the new version of the connector with Airbyte out of the box.

1. Bump the version in the `Dockerfile` of the connector \(`LABEL io.airbyte.version=X.X.X`\).
2. Update the connector definition in the Airbyte connector index to use the new version:
Expand All @@ -125,10 +129,12 @@ Once you've finished iterating on the changes to a connector as specified in its
6. The new version of the connector is now available for everyone who uses it. Thank you!

## Using credentials in CI

In order to run integration tests in CI, you'll often need to inject credentials into CI. There are a few steps for doing this:

1. **Place the credentials into Lastpass**: Airbyte uses a shared Lastpass account as the source of truth for all secrets. Place the credentials **exactly as they should be used by the connector** into a secure note i.e: it should basically be a copy paste of the `config.json` passed into a connector via the `--config` flag. We use the following naming pattern: `<source OR destination> <name> creds` e.g: `source google adwords creds` or `destination snowflake creds`.
2. **Add the credentials to Github Secrets**: To inject credentials into a CI workflow, the first step is to add it to Github Secrets, specifically within the ["more-secrets" environment](https://github.com/airbytehq/airbyte/settings/environments/276695501/edit). Admin access to the Airbyte repo is required to do this. All Airbyte engineers have admin access and should be able to do this themselves. External contributors or contractors will need to request this from their team lead or project manager who should have admin access. Follow the same naming pattern as all the other secrets e.g: if you are placing credentials for source google adwords, name the secret `SOURCE_GOOGLE_ADWORDS_CREDS`. After doing this step, the secret will be available in the relevant Github workflows using the workflow secrets syntax.
3. **Inject the credentials into test and publish CI workflows**: edit the files `.github/workflows/publish-command.yml` and `.github/workflows/test-command.yml` to inject the secret into the CI run. This will make these secrets available to the `/test` and `/publish` commands.
4. **During CI, write the secret from env variables to the connector directory**: edit `tools/bin/ci_credentials.sh` to write the secret into the `secrets/` directory of the relevant connector.
5. That should be it.

4 changes: 2 additions & 2 deletions docs/connector-development/airbyte101.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@

## The Airbyte Catalog

The Airbyte catalog defines the relationship between your incoming data's schema and the schema of your output stream. This
is an incredibly important concept to understand as a connector dev, so check out the AirbyteCatalog [here](../understanding-airbyte/beginners-guide-to-catalog.md).
The Airbyte catalog defines the relationship between your incoming data's schema and the schema of your output stream. This is an incredibly important concept to understand as a connector dev, so check out the AirbyteCatalog [here](../understanding-airbyte/beginners-guide-to-catalog.md).

1 change: 1 addition & 0 deletions docs/connector-development/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,4 @@ When reviewing connectors, we'll use the following "checklist" to verify whether
### Rate Limiting

Most APIs enforce rate limits. Your connector should gracefully handle those \(i.e: without failing the connector process\). The most common way to handle rate limits is to implement backoff.

Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# Connector Development Kit (TypeScript/JavaScript)
# Connector Development Kit \(Javascript\)

The [Faros AI TypeScript/JavaScript CDK](https://github.com/faros-ai/airbyte-connectors/tree/main/faros-airbyte-cdk) allows you to build Airbyte connectors quickly similarly to how our [Python CDK](../cdk-python) does. This CDK currently offers support for creating Airbyte source connectors for:
The [Faros AI TypeScript/JavaScript CDK](https://github.com/faros-ai/airbyte-connectors/tree/main/faros-airbyte-cdk) allows you to build Airbyte connectors quickly similarly to how our [Python CDK](cdk-python/) does. This CDK currently offers support for creating Airbyte source connectors for:

- HTTP APIs
* HTTP APIs

## Resources

[This document](https://github.com/faros-ai/airbyte-connectors/blob/main/sources/README.md) is the main guide for developing an Airbyte source with the Faros CDK.

An example of a source built with the Faros AI CDK can be found [here](https://github.com/faros-ai/airbyte-connectors/tree/main/sources/example-source). It's recommended that you follow along with the example source while building for the first time.
An example of a source built with the Faros AI CDK can be found [here](https://github.com/faros-ai/airbyte-connectors/tree/main/sources/example-source). It's recommended that you follow along with the example source while building for the first time.

12 changes: 6 additions & 6 deletions docs/connector-development/cdk-python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The CDK provides an improved developer experience by providing basic implementat

This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.io/architecture/airbyte-specification) before proceeding.

If you have any issues with troubleshooting or want to learn more about the CDK from the Airbyte team, head to the #connector-development channel in [our Slack](https://airbytehq.slack.com/ssb/redirect) to inquire further!
If you have any issues with troubleshooting or want to learn more about the CDK from the Airbyte team, head to the \#connector-development channel in [our Slack](https://airbytehq.slack.com/ssb/redirect) to inquire further!

## Getting Started

Expand All @@ -29,23 +29,23 @@ Additionally, you can follow [this tutorial](https://docs.airbyte.io/connector-d

#### Basic Concepts

If you want to learn more about the classes required to implement an Airbyte Source, head to our [basic concepts doc](./basic-concepts.md).
If you want to learn more about the classes required to implement an Airbyte Source, head to our [basic concepts doc](basic-concepts.md).

#### Full Refresh Streams

If you have questions or are running into issues creating your first full refresh stream, head over to our [full refresh stream doc](./full-refresh-stream.md). If you have questions about implementing a `path` or `parse_response` function, this doc is for you.
If you have questions or are running into issues creating your first full refresh stream, head over to our [full refresh stream doc](full-refresh-stream.md). If you have questions about implementing a `path` or `parse_response` function, this doc is for you.

#### Incremental Streams

Having trouble figuring out how to write a `stream_slices` function or aren't sure what a `cursor_field` is? Head to our [incremental stream doc](./incremental-stream.md).
Having trouble figuring out how to write a `stream_slices` function or aren't sure what a `cursor_field` is? Head to our [incremental stream doc](incremental-stream.md).

#### Practical Tips

Airbyte recommends using the CDK template generator to develop with the CDK. The template generates created all the required scaffolding, with convenient TODOs, allowing developers to truly focus on implementing the API.

For tips on useful Python knowledge, see the [Python Concepts](./python-concepts.md) page.
For tips on useful Python knowledge, see the [Python Concepts](python-concepts.md) page.

You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](../tutorials/cdk-tutorial-python-http)
You can find a complete tutorial for implementing an HTTP source connector in [this tutorial](../tutorials/cdk-tutorial-python-http/)

### Example Connectors

Expand Down
Loading

0 comments on commit ae32ecb

Please sign in to comment.