-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor: fix minor bugs, remove dagster-delta-polars, improve docs
- Loading branch information
1 parent
52102d9
commit a90dfbb
Showing
58 changed files
with
3,542 additions
and
1,067 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,2 @@ | ||
libraries/dagster-delta/ @ion-elgreco | ||
libraries/dagster-delta-polars/ @ion-elgreco @sverbruggen | ||
.github/ @ion-elgreco |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
"""Ensures that the files in `dist/` are prefixed with ${{ github.ref_name }} | ||
Tag must adhere to naming convention of distributed files. For example, the tag | ||
`dagster_delta-0.1.2` must match the prefix of the files in the `dist/` folder: | ||
-rw-r--r--@ 2.0K Oct 23 14:06 dagster_delta-0.1.2-py3-none-any.whl | ||
-rw-r--r--@ 1.6K Oct 23 14:06 dagster_delta-0.1.2.tar.gz | ||
USAGE | ||
$ python .github/validate-release-version.py libraries/dagster-delta/dist dagster_delta-0.1.3 | ||
""" | ||
|
||
import sys | ||
import os | ||
|
||
|
||
if len(sys.argv) != 3: | ||
print("Requires positional arguments: <path to dist> <github.ref_name>") | ||
sys.exit(1) | ||
|
||
dist_path = sys.argv[1] | ||
github_ref_name = sys.argv[2] | ||
|
||
if not os.path.exists(dist_path): | ||
print("Release directory `dist/` must exist") | ||
sys.exit(1) | ||
|
||
for filename in os.listdir(dist_path): | ||
if filename.startswith("."): | ||
continue | ||
if not filename.startswith(github_ref_name): | ||
print(f"{filename} does not start with prefix {github_ref_name}") | ||
sys.exit(1) | ||
|
||
|
||
print(f"Success: all files in `dist/` are prefixed with {github_ref_name}") |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
name: quality-check-dagster-delta | ||
on: | ||
pull_request: | ||
types: [opened, synchronize, reopened] | ||
paths: | ||
- 'libraries/dagster-delta/**' | ||
|
||
jobs: | ||
check: | ||
uses: ./.github/workflows/template-quality-check.yml | ||
with: | ||
working_directory: ./libraries/dagster-delta |
12 changes: 12 additions & 0 deletions
12
.github/workflows/quality-dagster-unity-catalog-polars.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
name: quality-check-dagster-unity-catalog-polars | ||
on: | ||
pull_request: | ||
types: [opened, synchronize, reopened] | ||
paths: | ||
- 'libraries/dagster-unity-catalog-polars/**' | ||
|
||
jobs: | ||
check: | ||
uses: ./.github/workflows/template-quality-check.yml | ||
with: | ||
working_directory: ./libraries/dagster-unity-catalog-polars |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: build-and-release-dagster-delta-polars | ||
|
||
on: | ||
push: | ||
tags: | ||
- 'dagster_delta_polars-*.*.*' | ||
|
||
jobs: | ||
build-and-release-dagster-delta-polars: | ||
uses: ./.github/workflows/template-release.yml | ||
with: | ||
library_name: dagster-delta-polars | ||
working_directory: ./libraries/dagster-delta-polars | ||
secrets: | ||
pypi_token: ${{ secrets.PYPI_API_TOKEN_DDP }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: build-and-release-dagster-delta | ||
|
||
on: | ||
push: | ||
tags: | ||
- 'dagster_delta-*.*.*' | ||
|
||
jobs: | ||
build-and-release-dagster-delta: | ||
uses: ./.github/workflows/template-release.yml | ||
with: | ||
library_name: dagster-delta | ||
working_directory: ./libraries/dagster-delta | ||
secrets: | ||
pypi_token: ${{ secrets.PYPI_API_TOKEN_DD }} |
15 changes: 15 additions & 0 deletions
15
.github/workflows/release-dagster-unity-catalog-polars.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: build-and-release-dagster-unity-catalog-polars | ||
|
||
on: | ||
push: | ||
tags: | ||
- 'dagster_unity_catalog_polars-*.*.*' | ||
|
||
jobs: | ||
build-and-release-dagster-unity-catalog-polars: | ||
uses: ./.github/workflows/template-release.yml | ||
with: | ||
library_name: dagster-unity-catalog-polars | ||
working_directory: ./libraries/dagster-unity-catalog-polars | ||
secrets: | ||
pypi_token: ${{ secrets.PYPI_API_TOKEN_DDUC }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
name: quality-check | ||
|
||
on: | ||
workflow_call: | ||
inputs: | ||
working_directory: | ||
required: true | ||
type: string | ||
|
||
jobs: | ||
check: | ||
runs-on: ubuntu-latest | ||
steps: | ||
|
||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Install uv | ||
uses: astral-sh/setup-uv@v3 | ||
|
||
- name: Install python | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv python install 3.9 | ||
|
||
- name: Sync dependencies | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv sync | ||
|
||
- name: Ruff (lint) | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv run ruff check | ||
|
||
- name: Ruff (formatting) | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv run ruff format --check . | ||
|
||
- name: Pyright | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv run pyright | ||
|
||
- name: Pytest | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv run pytest |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
# References | ||
# | ||
# https://docs.astral.sh/uv/guides/integration/github/ | ||
# https://docs.astral.sh/uv/guides/publish/#preparing-your-project-for-packaging | ||
# https://docs.pypi.org/trusted-publishers/adding-a-publisher/ | ||
# | ||
|
||
name: build-and-release | ||
|
||
on: | ||
workflow_call: | ||
inputs: | ||
library_name: | ||
required: true | ||
type: string | ||
working_directory: | ||
required: true | ||
type: string | ||
secrets: | ||
pypi_token: | ||
required: true | ||
jobs: | ||
build: | ||
name: python | ||
runs-on: ubuntu-latest | ||
environment: production | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Install uv | ||
uses: astral-sh/setup-uv@v3 | ||
|
||
- name: Install Python | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv python install | ||
|
||
- name: Build | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv build | ||
|
||
- name: Validate release version | ||
run: python .github/validate-release-version.py ${{ inputs.working_directory }}/dist ${{ github.ref_name }} | ||
|
||
- name: Publish | ||
working-directory: ${{ inputs.working_directory }} | ||
run: uv publish | ||
env: | ||
UV_PUBLISH_TOKEN: ${{ secrets.pypi_token }} |
Oops, something went wrong.