Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building in docker #40

Merged
merged 18 commits into from
Mar 5, 2019
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# tagged version, not latest!
FROM rocker/geospatial:3.5.2

# install node and npm (see https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions)
RUN sudo apt-get install -y curl &&\
sudo apt-get install -y gnupg &&\
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - &&\
sudo apt-get update &&\
sudo apt-get install -y nodejs &&\
sudo apt-get install -y build-essential

RUN sudo npm install -g\
webpack\
webpack-cli\
d3-geo-projection

#bring in DOI root cert. Remove this statement for non-USGS persons
RUN /usr/bin/wget -O /usr/lib/ssl/certs/DOIRootCA.crt http://sslhelp.doi.net/docs/DOIRootCA2.cer && \
ln -sf /usr/lib/ssl/certs/DOIRootCA.crt /usr/lib/ssl/certs/`openssl x509 -hash -noout -in /usr/lib/ssl/certs/DOIRootCA.crt`.0 && \
echo "\\n\\nca-certificate = /usr/lib/ssl/certs/DOIRootCA.crt" >> /etc/wgetrc;

RUN Rscript -e 'installed.packages()'
#Note that version rocker images are already set up to use the MRAN mirror corresponding to the
#date of the R version, so package dates are already set (unless forcing another repo)
RUN Rscript -e 'devtools::install_github("richfitz/remake")' && \
Rscript -e 'install.packages("grithub", repos = c(getOption("repos"), "https://owi.usgs.gov/R"))' && \
Rscript -e 'devtools::install_github("USGS-VIZLAB/[email protected]")'
#note that most common packages will already be installed as part of the geospatial image
RUN install2.r --error \
sbtools \
geojsonio \
dataRetrieval\
bit64

RUN mkdir /home/rstudio/gage-conditions
RUN chown rstudio /home/rstudio/gage-conditions
WORKDIR /home/rstudio/gage-conditions
35 changes: 35 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
pipeline {
agent none
stages {
stage('Checkout repo') {
agent any
steps {
sh 'wget -O DOIRootCA2.cer http://sslhelp.doi.net/docs/DOIRootCA2.cer'
git "https://github.com/wdwatkins/gage-conditions"
}
}
stage('build viz') {
agent {
dockerfile {
args '-v ${WORKSPACE}:/home/rstudio/gage-conditions'
}
}
steps {
sh 'mkdir -p target'
sh '''
Rscript -e 'vizlab::vizmake("dv_sites_sp")'
Rscript -e "remake::make(remake_file = 'map_preprocess.yml')"
Rscript -e "vizlab::vizmake()"
'''
}
}
stage('push to S3') {
agent any
steps {
sh 'aws s3 sync ./target/ s3://dev-owi.usgs.gov/vizlab/water-use-15/ --exclude "*.svg" --exclude "*.json"; \
aws s3 sync ./target/ s3://dev-owi.usgs.gov/vizlab/water-use-15/ --exclude "*" --include "*.svg" --content-type "image/svg+xml"; \
aws s3 sync ./target/ s3://dev-owi.usgs.gov/vizlab/water-use-15/ --exclude "*" --include "*.json" --content-type "application/json"'
}
}
}
}
18 changes: 18 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
version: "3"
services:
docker-dev-mode:
image: gage-conditions
build:
context: .
dockerfile: Dockerfile
ports:
- "8787:8787"
volumes:
- gage-conditions-data:/home/rstudio/gage-conditions
environment:
- ROOT=TRUE
- PASSWORD=mypass

volumes:
gage-conditions-data:

2 changes: 1 addition & 1 deletion scripts/fetch/dv_data.R
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fetch.dv_data <- function(viz){
sites <- deps[["sites"]]

startDate <- paste0(year, "-01-01")
endDate <- paste0(year, "-12-31")
endDate <- paste0(year, "-01-02")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Safe to revert this change now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, although really I'd say these dates should all be moved into the viz.yaml so they can be set in a single place

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair. Up to you how you do it, but note that we're not currently sure that anybody will be developing this interactive viz any more this year, so code quality/maintainability matters somewhat less than usual.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, probably wouldn't bother doing it now


dv_sites_data <- lapply(sites, FUN = function(x){
d <- renameNWISColumns(
Expand Down
2 changes: 1 addition & 1 deletion scripts/fetch/dv_sites.R
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fetch.dv_sites <- function(viz){
year <- deps[["year"]][["year"]]

startDate <- paste0(year, "-01-01")
endDate <- paste0(year, "-12-31")
endDate <- paste0(year, "-01-02")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert this change?

hucs <- zeroPad(1:21, 2)

sites <- c()
Expand Down
8 changes: 7 additions & 1 deletion viz.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ info:
version: 0.9-7
dataRetrieval:
repo: CRAN
version: 2.7.4
version: 2.7.3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the older version work better?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was just to avoid having to set package installs to GRAN, since CRAN only has 2.7.3

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, OK

dplyr:
repo: CRAN
version: 0.7.4
Expand All @@ -28,6 +28,12 @@ info:
sp:
repo: CRAN
version: 1.2.7
sbtools:
repo: CRAN
version: 1.1.6
geojsonio:
repo: CRAN
version: 0.6.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be nice to think about removing all of these declared package dependencies as we switch to Docker (because now the Dockerfile is where the decisions actually get made about which package version gets used). Is now too soon to do so, and if not, is it as simple as deleting all the lines in this subsection, or does vizlab give errors if you do that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just would need to remove the checkVizPackages() call invizmake(), since that requires there to be a packages yaml block. Otherwise, I think we could probably just drop the packages section, unless we don't feel confident relying on the package versions that come through the rocker images.

There is probably a fair bit of code associated with checkVizPackages that could either be removed from vizlab, unless we wanted to preserve a way of checking package versions against some reference.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks for thinking it through. I'll create a GitHub issue in vizlab for this, and we can resolve it (including cleaning up the extra code) after we've had a chance to kick the tires on the Docker option for a while.

What happens if you make a packages YAML block but leave it empty?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this issue: USGS-VIZLAB/vizlab#395

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That just returns a null from checkVizPackages, I think that would work

contributors:
-
name: Lindsay Carr
Expand Down