Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch: Rioxarray Raster #13

Draft
wants to merge 4 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docker/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
.travis.yaml
.swagger-codegen-ignore
README.md
tox.ini
git_push.sh
test-requirements.txt
setup.py

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
venv/
.python-version

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

#Ipython Notebook
.ipynb_checkpoints
50 changes: 50 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
FROM python:3.9-slim

RUN apt-get update && apt-get install -y \
python3 \
python3-dev \
gcc \
libssl-dev \
libjpeg-dev \
libjpeg62-turbo-dev \
zlib1g-dev \
libxslt1-dev \
libnetcdf-dev \
libhdf5-dev \
hdf5-helpers \
&& pip3 install --upgrade pip \
&& apt-get clean

# Create a new user
RUN adduser --quiet --disabled-password --shell /bin/sh --home /home/dockeruser --gecos "" --uid 1000 dockeruser
USER dockeruser
ENV HOME /home/dockeruser
ENV PYTHONPATH "${PYTHONPATH}:/home/dockeruser/.local/bin"
ENV PATH="/home/dockeruser/.local/bin:${PATH}"

# Add artifactory as a trusted pip index
RUN mkdir $HOME/.pip

# The 'SOURCE' argument is what will be used in 'pip install'.
ARG SOURCE

# Set this argument if running the pip install on a local directory, so
# the local dist files are copied into the container.
ARG DIST_PATH

USER root
RUN mkdir -p /worker && chown dockeruser /worker
USER dockeruser
WORKDIR /worker

COPY --chown=dockeruser $DIST_PATH $DIST_PATH
USER dockeruser
RUN pip3 install --use-feature=no-binary-enable-wheel-cache --force $SOURCE --user \
&& rm -rf $DIST_PATH

# Patch for rioxarray_raster
COPY ../net2cog/rioxarray_raster_patch.diff /home/dockeruser/.local/lib/python3.9/site-packages/rioxarray/raster_writer.py

# Run the subsetter
#ENTRYPOINT ["netcdf_harmony"]
CMD ["bash"]
73 changes: 73 additions & 0 deletions docker/build-docker.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env bash

# This script is intended to be run by the CI/CD pipeline to build a specific version of the L2SS Service application.

set -Eeo pipefail

LOCAL_BUILD=false

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
-n|--service-name)
service_name="$2"
shift # past argument
shift # past value
;;
-v|--service-version)
service_version="$2"
shift # past argument
shift # past value
;;
--local)
LOCAL_BUILD=true
shift # past argument
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

USAGE="USAGE: build-docker.sh -n|--service-name service_name -v|--service-version service_version [--local]"

# shellcheck disable=SC2154
if [[ -z "${service_name}" ]]; then
echo "service_name required. Name of the service as found in pyproject.toml (e.g. podaac-staging)" >&2
echo "$USAGE" >&2
exit 1
fi

# shellcheck disable=SC2154
if [[ -z "${service_version}" ]]; then
echo "service_version required. Version of software to install (e.g. 0.1.0-a1+12353)." >&2
echo "$USAGE" >&2
exit 1
fi

set -u

SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
PROJECT_DIR="$(dirname "${SCRIPTPATH}")"
DIST_PATH="dist/"

repositoryName=podaac/podaac-cloud/${service_name}

# Docker tags can't include '+' https://github.com/docker/distribution/issues/1201
dockerTagVersion=$(echo "${service_version}" | tr "+" _)

# Build the image
if [ "$LOCAL_BUILD" = true ] ; then
wheel_filename="$(echo "${service_name}" | tr "-" _)-${service_version}-py3-none-any.whl"
docker build -t "${repositoryName}":"${dockerTagVersion}" --build-arg DIST_PATH="${DIST_PATH}" --build-arg SOURCE="${DIST_PATH}${wheel_filename}" -f "$SCRIPTPATH"/Dockerfile "$PROJECT_DIR" 1>&2
else
docker build -t "${repositoryName}":"${dockerTagVersion}" --build-arg SOURCE="${service_name}[podaac,harmony]==${service_version}" -f "$SCRIPTPATH"/Dockerfile "$SCRIPTPATH" 1>&2
fi


echo "${repositoryName}":"${dockerTagVersion}"
12 changes: 12 additions & 0 deletions net2cog/rioxarray_raster_patch.diff
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
--- /usr/local/lib/python3.8/site-packages/rioxarray/raster_writer.py 2023-10-19 11:07:08
+++ raster_writer.py 2023-10-19 11:06:12
@@ -123,7 +123,8 @@
if the value of the nodata value changed.
"""
# Complex-valued rasters can have real-valued nodata
- if str(new_dtype).startswith("c"):
+ new_dtype_str = str(new_dtype)
+ if new_dtype_str.startswith("c") or "date" in new_dtype_str or "time" in new_dtype_str:
nodata = original_nodata
else:
original_nodata = float(original_nodata)