Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

env for worker procs #76

Merged
merged 3 commits into from
Sep 14, 2024
Merged

env for worker procs #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
64f8cec
env for worker procs
sosiristseng Sep 14, 2024
c870c2d
3 jobs
sosiristseng Sep 14, 2024
a9a03f5
no cache_num
sosiristseng Sep 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,9 @@ concurrency:
cancel-in-progress: true

env:
NBCONVERT_JOBS: '4'
LITERATE_PROC: '4'
NBCONVERT_JOBS: '3'
LITERATE_PROC: '3'
ALLOWERRORS: 'false'
CACHE_NUM: '1'
JULIA_CONDAPKG_BACKEND: 'Null'
JULIA_CI: 'true'
NBCACHE: '.cache'
Expand Down Expand Up @@ -46,7 +45,7 @@ jobs:
id: hash
run: |
echo "value=${{ hashFiles('Project.toml', 'Manifest.toml', 'src/**') }}" >> "$GITHUB_OUTPUT"
echo "ver=${{ runner.os }}-julia-${{ env.CACHE_NUM }}-${{ steps.read_toml.outputs.value }}" >> "$GITHUB_OUTPUT"
echo "ver=${{ runner.os }}-julia-${{ steps.read_toml.outputs.value }}" >> "$GITHUB_OUTPUT"
- name: Cache executed notebooks
uses: actions/cache@v4
id: cache-nb
Expand All @@ -61,7 +60,7 @@ jobs:
version: ${{ steps.read_toml.outputs.value }}
- name: Restore Julia packages
uses: actions/cache/restore@v4
if: ${{ runner.environment == 'github-hosted' }}
if: ${{ runner.environment == 'github-hosted'}}
id: cache-julia
with:
path: ~/.julia
Expand All @@ -72,7 +71,7 @@ jobs:
if: ${{ runner.environment == 'self-hosted' || steps.cache-julia.outputs.cache-hit != 'true' }}
shell: julia --color=yes {0}
run: |
using Pkg, Dates
using Pkg
Pkg.add(["IJulia", "Literate", "PrettyTables", "JSON"])
Pkg.activate(".")
Pkg.instantiate()
Expand All @@ -85,7 +84,7 @@ jobs:
key: ${{ steps.cache-julia.outputs.cache-primary-key }}
- name: Run notebooks
if: ${{ steps.cache-nb.outputs.cache-hit != 'true' }}
run: julia --color=yes -p ${{ env.LITERATE_PROC }} --heap-size-hint=3G run.jl
run: julia --project=@. --color=yes -p ${{ env.LITERATE_PROC }} run.jl
- name: Copy back built notebooks
run: cp --verbose -rf ${{ env.NBCACHE }}/docs/* docs/
- name: Build website
Expand Down
5 changes: 1 addition & 4 deletions run.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ using IJulia
@everywhere begin
ENV["GKSwstype"] = "100"
using Literate, Pkg, JSON
Pkg.activate(Base.current_project())
end

# Strip SVG output from a Jupyter notebook
Expand Down Expand Up @@ -100,9 +99,7 @@ function main(;
ts_lit = pmap(litnbs; on_error=ex -> NaN) do nb
@elapsed run_literate(nb, cachedir; rmsvg)
end

# Remove worker processes to release some memory
rmprocs(workers())
rmprocs(workers()) # Remove worker processes to release some memory

# Debug notebooks one by one if there are errors
for (nb, t) in zip(litnbs, ts_lit)
Expand Down