Update release process for Windows hosts

Makefile

@@ -1,5 +1,5 @@
 # TO EDIT
-TAG ?= v1.1.0
+TAG ?= v1.1.4
 
 GH_USERNAME ?= TomAugspurger
 PANDAS_VERSION=$(TAG:v%=%)

README.md

@@ -1,10 +1,10 @@
 Release automation for pandas.
 
-**Windows users should follow the steps in `windows.md`**
+**Windows users should follow [these](./windows.md) steps or [these](./windows-wsl.md) steps for WSL 2 (experimental).**
 
 ## Steps to a release
 
-- [  ] Manually update 
+- [  ] Manually update
   - [  ] `TAG` in `Makefile`
   - [  ] `GH_USERNAME` in `Makefile`
 

docker-files/windows/build/Dockerfile

@@ -0,0 +1,7 @@
+FROM pandas-build:latest
+
+ARG TAG
+
+ENV TAG=$TAG
+
+WORKDIR /pandas-release

docker-files/windows/docs/Dockerfile

@@ -0,0 +1,24 @@
+FROM continuumio/miniconda3:latest
+
+RUN apt-get update && apt-get install -y texlive-full xclip \
+ && apt-get clean
+
+# TODO: don't need to clone complete repository, just need environment.yml
+RUN git clone https://github.com/pandas-dev/pandas.git
+
+# following maybe necessary to prevent segfaults
+RUN conda update -n base -c defaults conda
+
+# this is also done in scripts\build-docs.sh (should be done earlier but won't use cached build and texlive is big download)
+RUN apt-get update && apt-get install -y build-essential && apt-get clean
+
+# NOTE: This builds environment from master will update for correct branch in container
+RUN conda env create --file=/pandas/environment.yml --name=pandas
+
+RUN rm -r pandas
+
+RUN mkdir -p /pandas-release/pandas
+
+RUN ln -s /pandas-release/pandas /pandas
+
+WORKDIR /pandas-release

windows-wsl.md

@@ -0,0 +1,104 @@
+# README for MS Windows Users using WSL 2.
+
+## Preparing for Your First Release.
+
+see https://docs.docker.com/docker-for-windows/wsl/#prerequisites for prerequisites
+
+Ensure the following repositories are forked to your GitHub account
+  - https://github.com/pandas-dev/pandas-release
+  - https://github.com/conda-forge/pandas-feedstock
+  - https://github.com/MacPython/pandas-wheels
+  - https://github.com/pandas-dev/pandas
+
+
+### Setting up WSL 2.
+
+Install Docker (This will activate WSL2)
+
+Install Ubuntu 20.04 LTS from the Windows Store
+
+Launch Ubuntu 20.04 LTS from the start menu to set-up for first use.
+
+Launch Docker desktop
+
+go to Settings > Resources > WSL Integration.
+
+You might see
+
+```
+You don't have any WSL 2 distro. Please convert a WSL 1 distro to WSL 2, or install a new distro and it will appear here.
+
+More info
+```
+
+click on More Info and follow steps.
+
+```
+PS C:\Users\simon> wsl -l -v
+  NAME                   STATE           VERSION
+* docker-desktop         Running         2
+  Ubuntu-20.04           Running         1
+  docker-desktop-data    Running         2
+PS C:\Users\simon> wsl --set-version Ubuntu-20.04 2
+Conversion in progress, this may take a few minutes...
+For information on key differences with WSL 2 please visit https://aka.ms/wsl2
+Conversion complete.
+PS C:\Users\simon> wsl --set-default-version 2
+For information on key differences with WSL 2 please visit https://aka.ms/wsl2
+PS C:\Users\simon> wsl --set-default Ubuntu-20.04
+PS C:\Users\simon> wsl -l -v
+  NAME                   STATE           VERSION
+* Ubuntu-20.04           Running         2
+  docker-desktop         Running         2
+  docker-desktop-data    Running         2
+PS C:\Users\simon>
+```
+
+### Preparing the release environment.
+
+Start Ubuntu 20.04 LTS
+
+configure git manually
+
+```
+git config --global user.email "<your-email>"
+git config --global user.name "<your-name>"
+```
+
+or copy the configuration from windows.
+```
+cp /mnt/c/Users/<your-windows-username>/.gitconfig .
+```
+
+also copy your ssh config from windows and set permissions
+```
+cp -r /mnt/c/Users/<your-windows-username>/.ssh/ .
+chmod 400 .ssh/id_rsa
+```
+
+clone your fork of the pandas-release repo and set upstream
+```
+git clone [email protected]:<your-github-username>/pandas-release.git
+cd pandas-release
+git remote add upstream https://github.com/pandas-dev/pandas-release.git
+```
+
+download conda from https://docs.conda.io/en/latest/miniconda.html#linux-installers and
+install.
+```
+cd
+ln -s /mnt/c/Users/<your-windows-username>/Downloads/ ~/downloads
+sha256sum downloads/Miniconda3-latest-Linux-x86_64.sh
+bash downloads/Miniconda3-latest-Linux-x86_64.sh
+```
+
+close terminal and reopen to activate conda and install pandas-release conda environment
+```
+conda list
+conda update conda
+cd pandas-release
+conda env create -f environment.yml
+conda activate pandas-release
+```
+
+The linux environment is now configured on WSL. Now follow the steps in [README.md](./README.md)

windows.md

@@ -7,10 +7,10 @@ Install Docker
 Ensure the following repositories are forked to your GitHub account
   - https://github.com/conda-forge/pandas-feedstock
   - https://github.com/MacPython/pandas-wheels
-  - https://github.com/pandas-dev/pandas   
+  - https://github.com/pandas-dev/pandas
 
 Open an Anaconda Prompt
-<!-- 
+<!--
 TODO: resolve git bash Docker volume issues so that make can be used on host
  -->
 
@@ -34,17 +34,27 @@ docker volume rm pandas-release
 ```
 
 **change TAG to the release version**
+
 ```
-docker run -it --env TAG=v1.0.5 --name=pandas-release -v pandas-release:/pandas-release pandas-release /bin/bash
+docker run -it --env TAG=v1.1.4 --name=pandas-release -v pandas-release:/pandas-release pandas-release /bin/bash
 ```
 
 The Docker release container should be now be running.
 
-Make sure the repos are up-to-date
-<!-- 
-TODO: also make sure conda environment is up-to-date and pandas-release repo is up-to-date if
-re-using an older Docker image
- -->
+if using an older Docker image make sure environments and pandas-release repo are up-to-date.
+
+<!-- TODO: maybe add to Makefile -->
+```
+apt-get update && apt-get clean
+git -C /pandas-release pull --ff-only
+
+conda update -n base conda -y
+
+conda env update -n base --file=/pandas-release/environment.yml
+```
+
+Make sure the repos are up-to-date.
+
 ```
 make update-repos
 ```
@@ -56,6 +66,7 @@ make tag
 ```
 
 Stop the container.
+
 ```
 exit
 ```
@@ -64,88 +75,63 @@ exit
 
 
 Create the Docker image for the sdist build, pip test and conda test containers
-<!-- 
-TODO: maybe update the image with apt-get for cached build
- -->
+
+**change TAG to the release version**
+
+<!-- TODO: setting TAG for the second time here -->
+
 ```
-docker build -t pandas-build .
+docker build -t pandas-build --no-cache .
+
+docker build -t pandas-test --build-arg TAG=v1.1.4 -f docker-files/windows/build/Dockerfile .
 ```
 
 ## Build the sdist
-<!-- 
-TODO: some of the next steps are repetative. set WORKDIR and symlink to /pandas in pandas-build Docker image instead
-TODO: add container name (as in Makefile) and do not destroy container on exit
+<!--
+TODO: some of the next steps are repetative. symlink to /pandas in pandas-build Docker image instead
  -->
 ```
-docker run -it --rm -v pandas-release:/pandas-release pandas-build /bin/bash
-
-ln -s pandas-release/pandas pandas
-
-cd pandas-release/
-
-./scripts/build_sdist.sh
-
-exit
+docker run --name=pandas-sdist-build -v pandas-release:/pandas-release pandas-test /bin/bash -c "ln -s /pandas-release/pandas /pandas;./scripts/build_sdist.sh"
 ```
 
 ## Pip Tests
-<!-- 
+<!--
 TODO: avoid need to pass explicit filename below
-TODO: add container name (as in Makefile) and do not destroy container on exit
  -->
 
 **change filename to the release version**
 
 ```
-docker run -it --rm -v pandas-release:/pandas-release pandas-build /bin/bash
-
-ln -s pandas-release/pandas pandas
-
-cd pandas-release/
-
-./scripts/pip_test.sh /pandas/dist/pandas-1.0.5.tar.gz
-
-exit
-
+docker run --name=pandas-pip-test -v pandas-release:/pandas-release pandas-test /bin/bash -c "ln -s /pandas-release/pandas /pandas;./scripts/pip_test.sh /pandas/dist/pandas-1.1.4.tar.gz"
 ```
 
 ## Conda Tests
-<!-- 
-TODO: add container name (as in Makefile) and do not destroy container on exit
+<!--
 TODO: avoid need to re-type version below
  -->
  **change PANDAS_VERSION to the release version**
 
 ```
-docker run -it --rm --env PANDAS_VERSION=1.0.5 -v pandas-release:/pandas-release pandas-build /bin/bash
-
-ln -s pandas-release/pandas pandas
-
-cd pandas-release/
-
-conda build --numpy=1.17.3 --python=3.8 ./recipe --output-folder=/pandas/dist
-
-exit
-
+docker run --name=pandas-conda-test --env PANDAS_VERSION=1.1.4 -v pandas-release:/pandas-release pandas-test /bin/bash -c "ln -s /pandas-release/pandas /pandas;conda build --numpy=1.17.3 --python=3.8 ./recipe --output-folder=/pandas/dist"
 ```
 
 ## Copy the sdist File from the Docker Volume to the Local Host.
-<!-- 
+<!--
 TODO: avoid need to enter specific filename below (maybe just copy contents of dist directory instead)
  -->
 **change filename to the release version**
 
 ```
-docker run -t --rm -v %cd%:/local -v pandas-release:/pandas-release pandas-release /bin/bash -c "cp /pandas-release/pandas/dist/pandas-1.0.5.tar.gz /local/"
+docker run -t --rm -v %cd%:/local -v pandas-release:/pandas-release pandas-release /bin/bash -c "cp /pandas-release/pandas/dist/pandas-1.1.4.tar.gz /local/"
 ```
 
-## Push the Tag. 
+## Push the Tag.
 
 **No going back now.**
 
 Restart the release container.
-<!-- 
-TODO: does this need to be in interactive mode 
+<!--
+TODO: does this need to be in interactive mode
  -->
 ```
 docker start pandas-release -i
@@ -176,6 +162,8 @@ Restart the release container.
 ```
 docker start pandas-release -i
 
+apt-get install vim
+
 make conda-forge
 
 make wheels
@@ -190,22 +178,14 @@ Note that `make wheels` actually pushes a job to MacPython to produce wheels whi
 
 ## Build the Docs.
 You can cheat and re-tag / rebuild these if needed.
-<!-- 
-TODO build an intermediate doc image (and why pandas conda env not in Docker image?)
- -->
 ```
-docker run -it --name=pandas-docs -v pandas-release:/pandas-release pandas-docs /bin/bash
-
-rm -r pandas
+docker build -t pandas-docs -f docker-files\windows\docs\Dockerfile .
 
-ln -s pandas-release/pandas pandas
-
-cd pandas-release/
+docker run -it --name=pandas-docs -v pandas-release:/pandas-release pandas-docs /bin/bash
 
-# following maybe necessary to prevent segfaults
-conda update -n base -c defaults conda
+conda activate pandas
 
-conda env create --file=/pandas/environment.yml --name=pandas
+conda env update -n pandas --file=/pandas/environment.yml
 
 ./scripts/build-docs.sh
 
@@ -215,7 +195,7 @@ exit
 Copy the built doc files to host and manually inspect html and pdf docs.
 
 **first remove the local pandas-docs directory (just manually use file manager for now)**
-<!-- 
+<!--
 TODO: maybe add web server to container
 TODO: add steps to clean the pandas-docs directory from the docker container before copy
  -->
@@ -224,7 +204,7 @@ docker run -t --rm -v %cd%:/local -v pandas-release:/pandas-release pandas-relea
 ```
 
 ## Upload the Docs
-<!-- 
+<!--
 TODO: add steps to update website and reorder so that docs are uploaded b4 github release
 TODO: add the ssh keys to the Docker image or on container creation
  -->