Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Isolate data102 hub to have its own image #5054

Merged
merged 7 commits into from
Sep 21, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,12 @@ jobs:
hubploy deploy --timeout 30m data101 hub ${CIRCLE_BRANCH}
no_output_timeout: 30m

- run:
shaneknapp marked this conversation as resolved.
Show resolved Hide resolved
name: Deploy data102
command: |
hubploy deploy --timeout 30m data102 hub ${CIRCLE_BRANCH}
no_output_timeout: 30m

- run:
name: Deploy shiny
command: |
Expand Down Expand Up @@ -487,6 +493,14 @@ workflows:
ignore:
- staging
- prod
- hubploy/build-image:
deployment: data102
name: data102 image build
filters:
branches:
ignore:
- staging
- prod
#- hubploy/build-image:
# deployment: stat159
# name: stat159 image build
Expand Down Expand Up @@ -664,15 +678,6 @@ workflows:
branches:
only:
- staging
- hubploy/build-image:
balajialg marked this conversation as resolved.
Show resolved Hide resolved
deployment: data100
name: data100 image build
push: true
# Filters can only be per-job? wtf
filters:
branches:
only:
- staging
- hubploy/build-image:
deployment: data101
name: data101 image build
Expand Down Expand Up @@ -791,7 +796,8 @@ workflows:
- data8 image build
- data100 image build
- data101 image build
- astro image build
- data102 image build
- astro image build
balajialg marked this conversation as resolved.
Show resolved Hide resolved
#- stat159 image build
- stat20 image build
- ischool image build
Expand Down
6 changes: 4 additions & 2 deletions deployments/data102/hubploy.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
images:
images:
- name: us-central1-docker.pkg.dev/ucb-datahub-2018/user-images/primary-user-image
path: ../datahub/images/default
- name: us-central1-docker.pkg.dev/ucb-datahub-2018/user-images/data102-user-image
path: image/
repo2docker:
base_image: docker.io/library/buildpack-deps:jammy

registry:
provider: gcloud
Expand Down
94 changes: 94 additions & 0 deletions deployments/data102/image/apt.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Some linux packages for basic terminal work, particularly
# oriented at users new to Unix/cmd line environments.

# Basic unix tools
man
man-db
manpages-posix
manpages-dev
manpages-posix-dev

# Download tools
curl
wget

# Core text editors on a *nix box: vim and emacs
balajialg marked this conversation as resolved.
Show resolved Hide resolved
vim
emacs-nox
emacs-goodies-el
python-mode
balajialg marked this conversation as resolved.
Show resolved Hide resolved

# A couple of CLI editors that are easier than vim
# micro # currently not working on 18.04
nano
jed
jed-extra

# powerful terminal-based file manager, better than the one in JLab
mc

# for easily managing multiple repositories with one command (perl-doc
# is needed for its help pages to work)
mr
perl-doc

# Regular build tools for compiling common stuff
build-essential
gfortran

# Dependencies for nbconvert
texlive-xetex
texlive-fonts-recommended
texlive-plain-generic
# https://github.com/berkeley-dsep-infra/datahub/issues/3719
texlive-lang-chinese
lmodern

# Other useful document-related tools
pandoc
latexdiff

# Some useful git utilities use basic Ruby
ruby

# Other niceties for command-line work and life
ack # powerful grep-like tool
pydf # colorized disk usage
tmux
screen
htop
nnn # cmd line file manager
zsh
rsync
tig # console UI for git
multitail

# For later, these are not available in 18.04
#browsh # text-based web browser, occasionally handy
#dasel # json/yml/csv/etc data wrangling at the terminal
#fzf # fuzzy file finder

## This section adds tools for desktop environment usage
balajialg marked this conversation as resolved.
Show resolved Hide resolved
dbus-x11
xorg
xubuntu-icon-theme
xfce4
xfce4-goodies
xclip
xsel
firefox
chromium-browser

# GUI text editors
emacs
vim-gtk3
gedit

# Git clients and tools
git-gui
gitg
qgit
meld

# For jupyter-tree-download. Ref: https://github.com/berkeley-dsep-infra/datahub/issues/3979
zip
44 changes: 44 additions & 0 deletions deployments/data102/image/environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: data102-FA23

channels:
- conda-forge

dependencies:
- python==3.11.0
- git==2.39.1
- jupyter-resource-usage==1.0.0
- jupyterlab==4.0.4
- jupyterlab-favorites==3.0.0
- jupyterlab_server==2.23.0
- jupyterlab_widgets==3.0.8
- jupyter_server==2.7.0
- nbgitpuller==1.1.1
- notebook==7.0.2
- ipywidgets==8.0.7
- jinja2==3.1.2
- jupyterlab==4.0.4
balajialg marked this conversation as resolved.
Show resolved Hide resolved
- markupsafe==2.1.3
- matplotlib==3.7.1
- numpy==1.24.2
- nbformat==5.9.2
- nbresuse==0.4.0
- otter-grader==4.2.0
- pandas==2.0.2
- pandoc==3.1.3
# data102, Fall 2023, https://github.com/berkeley-dsep-infra/datahub/issues/5037
- pymc==5.8.1
- scikit-learn==1.2.2
- scipy==1.10.1
- seaborn==0.12.2
- statsmodels==0.14.0
- pip
- pip:
# - -r infra-requirements.txt
- ipywidgets==8.0.7
# disable until fixed (probably this: https://github.com/jupyterlab/jupyter-collaboration/issues/162)
# - jupyter_collaboration==1.0.1
- jupyterhub==4.0.2
- nbconvert==7.6.0
- gh-scoped-creds==4.1
- nb2pdf==0.6.2
- nbpdfexport==0.2.1
32 changes: 32 additions & 0 deletions deployments/data102/image/infra-requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# WARNING: Original source at scripts/infra-packages/requirements.txt
# PLEASE DO NOT EDIT ELSEWHERE
# After editing scripts/infra-packages/requirements.txt, please run
# scripts/infra-packages/sync.bash.

# This file pins versions of notebook related python packages we want
# across all hubs. This makes sure we don't need to upgrade them
# everwhere one by one.

# FIXME: Freeze this to get exact versions of all dependencies
notebook==6.4.12
jupyterlab==3.4.5
retrolab==0.3.21
nbgitpuller==1.1.0
jupyter-resource-usage==0.6.1
# Matches version in images/hub/Dockerfile
jupyterhub==3.1.0
appmode==0.8.0
ipywidgets==7.7.2
otter-grader==4.2.0
jupyter-tree-download==1.0.1
git-credential-helpers==0.2
# Enough people like this, let's load it in.
jupyter-contrib-nbextensions==0.5.1
jupyter_nbextensions_configurator==0.4.1
# Measure popularity of different packages in our hubs
# https://discourse.jupyter.org/t/request-for-implementation-instrument-libraries-actively-used-by-users-on-a-jupyterhub/7994?u=yuvipanda
popularity-contest==0.4.1
# RISE is useful for presentations - see https://github.com/berkeley-dsep-infra/datahub/issues/2527
RISE==5.7.1
# syncthing for dropbox-like functionality
jupyter-syncthing-proxy==1.0.3