Skip to content

Commit

Permalink
Merge pull request #4 from nsidc/setup-quarto
Browse files Browse the repository at this point in the history
Setup quarto
  • Loading branch information
rmarow authored Sep 21, 2023
2 parents 69e0e7f + 4dbcb0d commit 0064110
Show file tree
Hide file tree
Showing 13 changed files with 354 additions and 0 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/.quarto/
_site/
23 changes: 23 additions & 0 deletions _quarto.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
project:
type: website

website:
title: "NSIDC Technical Blog"
navbar:
right:
- text: "Home"
file: index.qmd
- about.qmd
- icon: github
href: https://github.com/nsidc
- icon: twitter
href: https://twitter.com/NSIDC
format:
html:
theme:
light: ["flatly", default]
dark: ["darkly", default]
css: styles.css



29 changes: 29 additions & 0 deletions about.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: "About"
image: https://nsidc.org/themes/custom/nsidc/logo.svg
about:
template: jolla
links:
- icon: twitter
text: Twitter
href: https://twitter.com/NSIDC
- icon: linkedin
text: LinkedIn
href: https://www.linkedin.com/company/national-snow-and-ice-data-center-nsidc-/mycompany/
- icon: github
text: Github
href: https://github.com/nsidc

---
Welcome to the NSIDC Technology Blog.
The development team at [The National Snow and Ice Data Center (NSIDC)](http://www.nsidc.org) builds software in support of the NSIDC mission:

>Advancing Knowlede of Earth's Frozen Regions
### Technology at NSIDC

Development at NSIDC is a broad mix of technology stacks and applications. Currently we write new code in Python, JavaScript, and Ruby but our codebase includes IDL, MATLAB, C, C++, Java, and much more. We deploy our code using tools like Jenkins, Vagrant, Puppet, and VSphere to an internal cloud. All of this and more will be covered in future posts!

### What do we build?

We work on a wide range of projects from small data analysis, large data processing pipelines, to modern interactive JavaScript web applications.
14 changes: 14 additions & 0 deletions index.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: "NSIDC Technical Blog"
listing:
contents: posts
sort: "date desc"
type: default
categories: true
sort-ui: false
filter-ui: false
page-layout: full
title-block-banner: true
---


Binary file added nsidc_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions posts/_metadata.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# options specified here will apply to all posts in this folder

# freeze computational output
# (see https://quarto.org/docs/projects/code-execution.html#freeze)
freeze: true

# Enable banner style title blocks
title-block-banner: true
254 changes: 254 additions & 0 deletions posts/conda-ptyhon-at-nsidc/index.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
---
title: "Conda Python at NSIDC: Changing How We Do Python"
author: "Kyle W Purdon"
date: "2015-04-29"
categories: [python, deployment, anaconda]
image: "python-conda.png"
---


[Anaconda (Conda Python)](http://continuum.io/downloads) from [Continuum Analytics](http://continuum.io/) is changing the way we develop and deploy python libraries and applications at NSIDC. This post will give a brief overview of how conda has changed our Python workflow.

<!-- more -->

What is Conda/Anaconda?
---

### Conda

Conda is an alternative to the traditional Python development, packaging, and deployment environment. It replaces [pip](https://pypi.python.org/pypi/pip) with [conda](http://conda.pydata.org/docs/), [virtualenv](https://virtualenv.pypa.io/en/latest/) with [conda env](http://conda.pydata.org/docs/faq.html#env), [PyPI](https://pypi.python.org/pypi) with [binstar](https://binstar.org), and [sdist/wheel packages](https://packaging.python.org/en/latest/) with [conda packages](http://conda.pydata.org/docs/build_tutorials/pkgs.html). That's right, everything! However, you can still use all of the standard python tools with conda. For example you can pip install a package into a conda environmnet and a conda package is just a binary package wrapped around a standard python sdist package.

### Anaconda

Anaconda is a set of (as of this post) 195 of the most popular packages for science, math, engineering, and data analysis. Basically, it's a curated set of packages you will need to get started with python. You do not have to use Anaconda to use conda. Miniconda is a lightweight version that provides all of the base tools without all of the packages that anaconda provides. We'll talk more about this later!

---

Development Environment Basics
---

The first step in our transition to using conda was getting a basic development environment set up. With conda this is extraordinarily easy. Most of us develop on OSX, but this process is nearly the same for Linux and even Windows machines as well. For the most up to date installation instructions [see this page](http://continuum.io/downloads).

### Installing Miniconda

Miniconda is the best place to start if you're an experienced developer. It gives you the most lightweight, clean base to build off of. To install miniconda simply:

{% highlight bash %}
$ wget http://repo.continuum.io/miniconda/Miniconda-latest-MacOSX-x86_64.sh
$ bash Miniconda-latest-MacOSX-x86_64.sh
{% endhighlight %}

Follow the installation process and make sure to add the installation to your PATH.

{% highlight bash %}
$ export PATH="/Users/[username]/miniconda/bin:$PATH" # add this to .bashrc
{% endhighlight %}

In addition I recommend installing argcomplete for conda command tab completion. This process is very simple:

{% highlight bash %}
$ conda install argcomplete
$ eval "$(register-python-argcomplete conda)" # add this to .bashrc
{% endhighlight %}

### Conda Environments

The first thing you'll want to figure out is how to use conda environments (the virtualenv equivelant). Luckily this is also very simple! Let's create a new environment and install some packages:

{% highlight bash %}
$ conda create -n test numpy
{% endhighlight %}

This command just created a new conda environment called `test` and installed the numpy package. To activate this environment simply run:

{% highlight bash %}
$ . activate test # source activate test
{% endhighlight %}

Run the following command to see a list of installed packages:

{% highlight bash %}
$ conda list
{% endhighlight %}

You'll see that `python 2.7.9` is listed as a dependency. Python itself is actually just a package to conda. This means in any environment at any time you can change your python version and it will update dependencies for you. What this means is that conda env's can also act as a replacement for [pyenv](https://github.com/yyuu/pyenv).

There is much more information on conda environments [here](http://conda.pydata.org/docs/faq.html#env)

#### TIP: `conda install anaconda` will install all of the packages in Anaconda!

---

Packaging Basics
---

The simplest way to explain how conda packaging for Python works is a simple example. Let's assume we have to following project structure:

{% highlight bash %}
├── hello
│   ├── __init__.py
│   └── hello.py
├── meta.yaml
└── setup.py
{% endhighlight %}

Other than the file `meta.yaml` this is just a standard python package. Here is the file `hello.py`:

{% highlight python %}
import click


@click.command()
@click.argument('name')
def hello(name):
print('Hello, {}'.format(name))

if __name__ == '__main__':
hello()
{% endhighlight %}

I have included [click](http://click.pocoo.org/4/) here so that we can see how dependencies are handled in the packaging process. The file `setup.py` follows:

{% highlight python %}
from setuptools import setup, find_packages

setup(
name='',
version='',
description='',
url='',
author='',
author_email='',
license='',
packages=find_packages()
)
{% endhighlight %}

You'll notice here that none of the information is filled out. This is a piece we at NSIDC have not really figured out. All of the metadata (name, version, ...) is actually stored in the conda file `meta.yaml` but the `setup.py` is still used to build the python package (which gets wrapped up in the conda package). If you're reading this and have some clarification on this issue please let us know! Finally, the `meta.yaml`:

{% highlight yaml %}
package:
name: hello
version: "0.0.1"

source:
path: .

build:
script: flake8 */*.py && python setup.py install
entry_points:
- hello = hello.hello:hello

requirements:
build:
- python 2.7*
- flake8
run:
- python 2.7*
- click

test:
commands:
- hello --help
{% endhighlight %}

With all of these files in place we can build a conda package. Note you will need to `conda install conda-build` first. To build the package simply run the following command from the directory with `meta.yaml`:

{% highlight bash %}
$ conda build .
{% endhighlight %}

This command will read the `meta.yaml` and execute the build using all of the information we have given it. Note the `script` entry tells build to run flake8 and then setup.py install. This will check the syntax and then build/install the python package using `setup.py`. One great thing about conda is that it does its builds in isolated environments by default! Once the build is complete you will get some output that tells you where the package was built and what to do next. In my case the package built was:

`/Users/kpurdon/miniconda/conda-bld/osx-64/hello-0.0.1-py27_0.tar.bz2`

You should note that this package is an osx-64 specific binary. In the next section (Deployment Basics) I will demonstrate how to build packages for all architectures. For now let's just install the new package:

{% highlight bash %}
$ conda install /Users/kpurdon/miniconda/conda-bld/osx-64/hello-0.0.1-py27_0.tar.bz2
{% endhighlight %}

The package should install, including its dependencies (Python 2.7* and Click). We can now execute the entry-point command `hello` we defined in `meta.yaml` and see the expected output:

{% highlight bash %}
$ hello Kyle
Hello, Kyle
{% endhighlight %}

### Dependencies Not on Binstar

At some point you will run into a situation where a python dependency you need is not yet converted to a conda package and released on binstar. For development you can just `pip install [pkg]` into your conda env, but for packaging you will want the package to be in conda and on binstar. Luckily this is very easy!

{% highlight bash %}
$ conda skeleton pypi [somepackage]
$ conda build [somepackage]
# now follow the binstar upload instructions in the next section!
{% endhighlight %}

With just a few simple commands you can take a python package that is on PyPI, create a conda package, build it, and be ready to upload it to your own channel on binstar!

---

Deployment Basics
---

Once we have a conda package we'll need to deploy it to some package repository. Normally you would upload a Python package to PyPI. The conda equivalent to PyPI is [Binstar](https://binstar.org). You can also create a local repository using [custom channels](http://conda.pydata.org/docs/custom-channels.html).

### Creating Multi-Arch Packages

In the packaging example the following file was created:

`/Users/kpurdon/miniconda/conda-bld/osx-64/hello-0.0.1-py27_0.tar.bz2`

This is a package specific to the osx-64 architecture. To create packages for all conda supported architectures from this package is very simple:

{% highlight bash %}
$ cd /Users/kpurdon/miniconda/conda-bld/
$ conda convert osx-64/hello-0.0.1-py27_0.tar.bz2 -p all
{% endhighlight %}

After running this command all the packages we'll need to upload should be created. You can check this with the following command:

{% highlight bash %}
$ find . -name hello*
./linux-32/hello-0.0.1-py27_0.tar.bz2
./linux-64/hello-0.0.1-py27_0.tar.bz2
./osx-64/hello-0.0.1-py27_0.tar.bz2
./win-32/hello-0.0.1-py27_0.tar.bz2
./win-64/hello-0.0.1-py27_0.tar.bz2
{% endhighlight %}

### Uploading To Binstar

Before we can upload to Binstar we'll need to complete two simple tasks:

1. Register at [Binstar.org](https://binstar.org)
2. `conda install binstar`

Now all we need to do to upload all of these packages to Binstar is run the following command:

{% highlight bash %}
$ binstar upload */hello-0.0.1-py27_0.tar.bz2
{% endhighlight %}

You should now see the package on binstar, here is an example:

![Binstar hello](/assets/binstar_hello.png)

---

Wrap-Up
---

### Scientific Python, Finally Just Works.

In addition to the very clean development and packaging worflow offered by conda the greatest benefit is how it handles non-python native packages (Numpy, SciPy, ...). No longer are there ANY issues requiring packages as dependencies or needing to compile packages each time. A simple `conda install scipy` just works, every time. For those of you that work extensively with these scientific python packages this is a breakthrough.

### Conclusion

Conda has so far solved nearly all of our issues with packaging and deploying Python packages. It has replaced our development environments, and continues to surprise us with new features frequently. If you do any work with scientific python you should be installing conda right now.

There is SO MUCH MORE to be covered in the conda ecosystem. Here are a couple topics we hope to cover in future posts:

* Automating miniconda installation on virtual machines with Puppet.
* An automation workflow for snapshot (development) packages.:w
Binary file added posts/conda-ptyhon-at-nsidc/python-conda.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added posts/post-with-code/image.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 10 additions & 0 deletions posts/post-with-code/index.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
title: "Post With Code"
author: "Harlow Malloc"
date: "2023-09-21"
categories: [news, code, analysis]
image: "image.jpg"
draft: true
---

This is a post with executable code.
13 changes: 13 additions & 0 deletions posts/welcome/index.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
title: "Welcome To My Blog"
author: "Tristan O'Malley"
date: "2023-09-18"
categories: [news]
draft: true
---

This is the first post in a Quarto blog. Welcome!

![](thumbnail.jpg)

Since this post doesn't specify an explicit `image`, the first image in the post will be used in the listing page of posts.
Binary file added posts/welcome/thumbnail.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions styles.css
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/* css styles */

0 comments on commit 0064110

Please sign in to comment.