Skip to content

Commit

Permalink
Merge branch 'opensearch-project:main' into kinesis-workflow-action
Browse files Browse the repository at this point in the history
  • Loading branch information
sb2k16 authored Dec 17, 2024
2 parents 46ae116 + 956a89a commit 0abb616
Show file tree
Hide file tree
Showing 884 changed files with 63,963 additions and 54,138 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
* @chenqi0805 @engechas @graytaylor0 @dinujoh @kkondaka @KarstenSchnitter @dlvenable @oeyh
* @sb2k16 @chenqi0805 @engechas @graytaylor0 @dinujoh @kkondaka @KarstenSchnitter @dlvenable @oeyh
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,6 @@ data-prepper-main/src/test/resources/logstash-conf/logstash-filter.yaml

# Python virtual environments
.venv

# output folder created when we run test cases
**/out/
1 change: 1 addition & 0 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje

| Maintainer | GitHub ID | Affiliation |
| -------------------- | --------------------------------------------------------- | ----------- |
| Souvik Bose | [sb2k16](https://github.com/sb2k16) | Amazon |
| Qi Chen | [chenqi0805](https://github.com/chenqi0805) | Amazon |
| Chase Engelbrecht | [engechas](https://github.com/engechas) | Amazon |
| Taylor Gray | [graytaylor0](https://github.com/graytaylor0) | Amazon |
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
<img src="https://raw.githubusercontent.com/opensearch-project/data-prepper/main/docs/images/DataPrepper_auto.svg" height="64px" alt="Data Prepper">
<img src="https://raw.githubusercontent.com/opensearch-project/data-prepper/main/docs/images/DataPrepper_auto.svg" height="64px" alt="OpenSearch Data Prepper">

[![codecov](https://codecov.io/gh/opensearch-project/data-prepper/branch/main/graph/badge.svg?token=IS7GOIY622)](https://codecov.io/gh/opensearch-project/data-prepper)
# Data Prepper
# OpenSearch Data Prepper

We envision Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently Data Prepper supports processing of distributed trace data and log ingestion with plans to support metric data in the future.
We envision OpenSearch Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently OpenSearch Data Prepper supports processing of distributed trace data and log ingestion with plans to support metric data in the future.

Please read the [Overview](docs/overview.md) to understand what Data Prepper is and how it works.
Please read the [Overview](docs/overview.md) to understand what OpenSearch Data Prepper is and how it works.

## Getting Started

Our [Getting Started](docs/getting_started.md) guide is the best starting point for anybody who wants to run Data Prepper.
Our [Getting Started](docs/getting_started.md) guide is the best starting point for anybody who wants to run OpenSearch Data Prepper.

Please read the [Trace Analytics](docs/trace_analytics.md) guide or [Log Analytics](docs/log_analytics.md) to get started with using Data Prepper for trace or log analytics use cases.
Please read the [Trace Analytics](docs/trace_analytics.md) guide or [Log Analytics](docs/log_analytics.md) to get started with using OpenSearch Data Prepper for trace or log analytics use cases.

## Project Resources

* [Downloads](https://opensearch.org/downloads.html)
* [Documentation](https://opensearch.org/docs/latest/clients/data-prepper/index/)
* [Configuration Reference](https://opensearch.org/docs/latest/clients/data-prepper/data-prepper-reference/)
* Need help? Try the [Data Prepper category](https://discuss.opendistrocommunity.dev/c/data-prepper/61) in the OpenSearch forums
* Need help? Try the [OpenSearch Data Prepper category](https://discuss.opendistrocommunity.dev/c/data-prepper/61) in the OpenSearch forums
* [Project Principles](https://opensearch.org/#principles)

## Contribute
Expand Down
22 changes: 11 additions & 11 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Releasing

This document outlines the process for releasing Data Prepper.
It is a guide for maintainers of the Data Prepper project to release a new version.
This document outlines the process for releasing OpenSearch Data Prepper.
It is a guide for maintainers of the OpenSearch Data Prepper project to release a new version.

## Overview

Expand All @@ -16,13 +16,13 @@ This document has three broad categories of steps to follow:

### Branch setup

Data Prepper uses a release branch for releasing.
OpenSearch Data Prepper uses a release branch for releasing.
The [Developer Guide](docs/developer_guide.md#backporting) discusses this in detail.

The repository has a release branch for a major/minor version.
Patch versions will continue on the same branch.
For example, Data Prepper `2.6.0` was released from the `2.6` branch.
Additionally, Data Prepper `2.6.1` and `2.6.2` were also released from the `2.6` branch.
For example, OpenSearch Data Prepper `2.6.0` was released from the `2.6` branch.
Additionally, OpenSearch Data Prepper `2.6.1` and `2.6.2` were also released from the `2.6` branch.

If you are creating a new major/minor release, then you will need to create the release branch.
Use GitHub to create a new branch.
Expand All @@ -42,7 +42,7 @@ Steps:

### Update versions

The Data Prepper version is defined in the [`gradle.properties`](https://github.com/opensearch-project/data-prepper/blob/main/gradle.properties) file.
The OpenSearch Data Prepper version is defined in the [`gradle.properties`](https://github.com/opensearch-project/data-prepper/blob/main/gradle.properties) file.
We must update this whenever we create a new release.
We will need two PRs to update it.

Expand Down Expand Up @@ -76,7 +76,7 @@ Note: This step can be automated through [#4877](https://github.com/opensearch-p
### Update the THIRD-PARTY file

We should update the `THIRD-PARTY` file for every release.
Data Prepper has a GitHub action that will generate this and create a PR with the updated file.
OpenSearch Data Prepper has a GitHub action that will generate this and create a PR with the updated file.

Steps:
* Go the [Third Party Generate action](https://github.com/opensearch-project/data-prepper/actions/workflows/third-party-generate.yml)
Expand Down Expand Up @@ -118,10 +118,10 @@ Also tag this with the `backport {major}.{minor}` to create a PR that you can me

## <a name="performing-a-release">Performing a release</a>

This section outlines how to perform a Data Prepper release using GitHub Actions and the OpenSearch build infrastructure.
The audience for this section are Data Prepper maintainers.
This section outlines how to perform a OpenSearch Data Prepper release using GitHub Actions and the OpenSearch build infrastructure.
The audience for this section are OpenSearch Data Prepper maintainers.

### Start the release Data Prepper action
### Start the release OpenSearch Data Prepper action

To run the release, go to the [Release Artifacts](https://github.com/opensearch-project/data-prepper/actions/workflows/release.yml)
GitHub Action.
Expand Down Expand Up @@ -159,7 +159,7 @@ This includes building the artifacts, testing, drafting a GitHub release, and pr

The release build will create a new GitHub issue requesting to release the project.
This needs two maintainers to approve.
To approve, load [Data Prepper issues](https://github.com/opensearch-project/data-prepper/issues).
To approve, load [OpenSearch Data Prepper issues](https://github.com/opensearch-project/data-prepper/issues).
Look for and open a new issue starting with _Manual approval required for workflow_.
Verify that the metadata looks correct and that we want to release.
Add a new comment on the issue with the word _approve_ or _approved_ in it.
Expand Down
Loading

0 comments on commit 0abb616

Please sign in to comment.