Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mask runner check for existing files to skip processing #676

Closed
wants to merge 135 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
6934d27
added Dockerfile and GH action
sfarrens Nov 16, 2022
82bc72a
added make dependency
sfarrens Nov 16, 2022
29c609e
docker testing
martinkilbinger Nov 1, 2023
da18970
Merge remote-tracking branch 'upstream/develop' into docker_image
martinkilbinger Nov 3, 2023
5770367
Dockerfile tests
martinkilbinger Nov 8, 2023
3166e3c
Added cent-os to CI tests
martinkilbinger Nov 8, 2023
373722e
Added pyproject
martinkilbinger Nov 9, 2023
b025b6a
canfar pyproject update
martinkilbinger Nov 10, 2023
ba135f2
cfis vos configs: missing keywords, DR5 added
martinkilbinger Nov 10, 2023
07cafb4
Testing dockerfile with only conda env installed
martinkilbinger Nov 15, 2023
92a999c
Fixing numpy and astropy versions
martinkilbinger Nov 15, 2023
2b6600d
installation on canfar working: removed most versions from yml file; …
martinkilbinger Nov 17, 2023
b6e5c97
Dockerfile for image 0:7, 0:8
martinkilbinger Nov 17, 2023
bc66565
Installation of sextractor and psfex with conda
martinkilbinger Nov 19, 2023
0b3aac6
script to call curl with canfar contained updated, adding NCORE as ar…
martinkilbinger Nov 19, 2023
69317cb
added init scripts for canfar; job_sp added n_smp for further jobs
martinkilbinger Nov 19, 2023
d35bd69
script to call curl for canfar container: added resources
martinkilbinger Nov 19, 2023
d93a60e
testing input numbers
martinkilbinger Nov 19, 2023
fb83a27
removed debug msg
martinkilbinger Nov 20, 2023
e33972b
numpy error fixed with version upgrade
martinkilbinger Nov 20, 2023
edbec76
Merge branch 'science_portal_run' of github.com:martinkilbinger/shape…
martinkilbinger Nov 20, 2023
9aa16be
Getting ready for exclusive-one-tile processing
martinkilbinger Nov 22, 2023
e5d0335
Added command line option to specify exclusive ID for processing
martinkilbinger Nov 22, 2023
49d8880
Merge remote-tracking branch 'origin/exclusive' into science_portal_run
martinkilbinger Nov 22, 2023
d02347e
curl script exclusive ID
martinkilbinger Nov 24, 2023
12e6b63
Dockerfile conda -> source activate
martinkilbinger Nov 24, 2023
b05f2a5
comment added
martinkilbinger Nov 24, 2023
45942d5
Updated Dockerfile
martinkilbinger Nov 24, 2023
7d1891e
running with exclusive ID
martinkilbinger Nov 24, 2023
02affa8
exp runs
martinkilbinger Nov 25, 2023
d8ad328
local curl script NCORES -> 1
martinkilbinger Nov 25, 2023
60c3cd1
Renamed science-portal scripts (local/remote)
martinkilbinger Nov 28, 2023
0737c0f
trying to run Pi
martinkilbinger Dec 1, 2023
7dba136
Merge remote-tracking branch 'origin/science_portal_run' into exclusive
martinkilbinger Dec 1, 2023
c65d75a
numpy -> 1.22 to avoid asscalar bug
martinkilbinger Dec 1, 2023
1e7b633
aux script to create links for exposure output runs for tile
martinkilbinger Dec 3, 2023
b006368
Merge remote-tracking branch 'origin/science_portal_run' into exclusive
martinkilbinger Dec 3, 2023
85f0c99
Added aux script to update runs log file
martinkilbinger Dec 3, 2023
d0c3d9f
update runs log file script: deal with multiple runs of same module
martinkilbinger Dec 3, 2023
323d44a
run_log: added function get_all_dirs
martinkilbinger Dec 4, 2023
dedbc3f
Fixed new function get_all_dirs
martinkilbinger Dec 4, 2023
e67f549
Fixed (as for MCCD) FITS key bug
martinkilbinger Dec 4, 2023
55677e7
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 4, 2023
b6cb27d
vignet makers: can use last and all in additional input inage directo…
martinkilbinger Dec 4, 2023
096af60
Dockerfile + jupyter, activate
martinkilbinger Dec 8, 2023
3c37250
file handler raises error if no process
martinkilbinger Dec 8, 2023
667eb7b
combine mask outputs
martinkilbinger Dec 8, 2023
bc20ab2
canfar curl command: added kind (tile, exp) as option
martinkilbinger Dec 13, 2023
1b4b3bf
curl remote job script init_run_exclusive_canfar: command line option…
martinkilbinger Dec 13, 2023
bd82f95
curl local command: added -k kind
martinkilbinger Dec 14, 2023
33ea4a1
summary missing ID 32 fixed (?)
martinkilbinger Dec 15, 2023
50df9a7
canfar scripts command line options
martinkilbinger Dec 15, 2023
ba17ebc
curl canfar local script added job, kind
martinkilbinger Dec 15, 2023
394e464
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 15, 2023
94428b4
Merge pull request #4 from martinkilbinger/science_portal_run
martinkilbinger Dec 15, 2023
9ac9dc1
SP ngmxix (job 128) running on canfar
martinkilbinger Dec 17, 2023
1d25a8b
curl canfar local script updated
martinkilbinger Dec 17, 2023
3d2b127
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 17, 2023
ca6b602
curl scripts updated
martinkilbinger Dec 19, 2023
cf69907
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 19, 2023
c643d0c
run summary more OO
martinkilbinger Dec 19, 2023
523b10f
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 19, 2023
53d9d35
update_runs_log_file script: fixed bug when run dir is empty
martinkilbinger Dec 21, 2023
ce1179e
improved canfar job scripts; fixed some make cat bugs
martinkilbinger Dec 21, 2023
a005119
added curl to Dockerimage
martinkilbinger Dec 21, 2023
64f04cf
Merge remote-tracking branch 'origin/exclusive' into exclusive
martinkilbinger Dec 21, 2023
22ee0f3
Remove temp hack from mask
martinkilbinger Dec 21, 2023
c62861d
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 21, 2023
b682e7c
removed unused code from mask
martinkilbinger Dec 21, 2023
9b9dbf1
Merge pull request #3 from martinkilbinger/exclusive
martinkilbinger Dec 21, 2023
dd57fd8
Merge remote-tracking branch 'origin/develop' into develop
martinkilbinger Dec 21, 2023
7fc3645
curl scripts updated
martinkilbinger Dec 23, 2023
15a4b7b
combine psf validation files: preles now with prepare_tiles_for_final…
martinkilbinger Dec 25, 2023
0d56e23
Update post_processing.md
martinkilbinger Dec 25, 2023
fde5982
prepare tiles script loop tests
martinkilbinger Dec 25, 2023
9c5cad4
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 25, 2023
ac0a339
curl canfar local script minor change
martinkilbinger Dec 25, 2023
4048bf1
added vos doc md file
martinkilbinger Dec 25, 2023
0a3f80e
Update post_processing.md
martinkilbinger Dec 25, 2023
d81bdf2
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
672001d
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
cc96a0c
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
87bb2b7
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
76c08a6
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
faa0836
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
85f6d91
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
4f81635
combine runs script renamed
martinkilbinger Dec 25, 2023
5574393
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 25, 2023
d4cb47c
Update post_processing.md
martinkilbinger Dec 25, 2023
2ffba3b
P3 proceesing to final cat
martinkilbinger Dec 26, 2023
67c966c
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 26, 2023
efb2799
Merge branch 'develop' into p3
martinkilbinger Dec 26, 2023
1f36bf5
Merge pull request #5 from martinkilbinger/p3
martinkilbinger Dec 26, 2023
9f81502
config files updated
martinkilbinger Jan 6, 2024
6eb11cf
PSFEx interp runner: allowing all: for ME_DOT_PSF_PDIRS
martinkilbinger Jan 15, 2024
485da58
init run exc script: added -d option; updates
martinkilbinger Jan 15, 2024
ef3a2c9
job sp canfar script: remove old vos upload code
martinkilbinger Jan 15, 2024
a20cb4f
summary create library and param files
martinkilbinger Jan 15, 2024
29b463b
minor changes
martinkilbinger Jan 15, 2024
cb67e8f
curl local script: now working with -e ID and -f file_IDs, in dry and…
martinkilbinger Jan 15, 2024
f0acd65
summary: fixed main path, verbose
martinkilbinger Jan 16, 2024
1bc602b
Started canfar howto
martinkilbinger Jan 16, 2024
d6ff437
Added summary run notebook
martinkilbinger Jan 16, 2024
1a81ae3
psfex_interp: continue instead of error if one of the .psf files not …
martinkilbinger Jan 17, 2024
93d6c6e
fixed symlink config files
martinkilbinger Jan 17, 2024
9c0b9a5
Updated summary run
martinkilbinger Jan 17, 2024
b19ad03
revert to main psfex in link exp for tiles script
martinkilbinger Jan 17, 2024
af38556
curl job script: remoging session logs
martinkilbinger Jan 17, 2024
8c9921f
comments
martinkilbinger Jan 17, 2024
18d104c
Merge pull request #6 from martinkilbinger/P7
martinkilbinger Jan 17, 2024
4276fea
Update canfar.md
martinkilbinger Jan 19, 2024
3f15533
Update canfar.md
martinkilbinger Jan 19, 2024
cfefcfb
Update canfar.md
martinkilbinger Jan 19, 2024
24480e2
Update canfar.md
martinkilbinger Jan 19, 2024
48e6114
Merge pull request #8 from martinkilbinger/martinkilbinger-patch-1
martinkilbinger Jan 19, 2024
cd7c61b
Merge pull request #7 from martinkilbinger/martinkilbinger-canfar-doc
martinkilbinger Jan 19, 2024
9a964c5
updated canfar doc
martinkilbinger Jan 26, 2024
e0100ca
script with akaha lib to count headlerss jobs
martinkilbinger Jan 26, 2024
984ec90
cleaned up curl submit script
martinkilbinger Jan 26, 2024
842e382
minor modifs to summary
martinkilbinger Jan 26, 2024
d9846e9
Removed VM_HOME; jon sp canfar cleaned up
martinkilbinger Jan 26, 2024
c5deb34
Dockerfile cleand up
martinkilbinger Jan 30, 2024
d507780
summary params minor bug fixed
martinkilbinger Feb 1, 2024
b4e0aa0
major bugx fixed: N_SMP was n_SMP, not propagated
martinkilbinger Feb 1, 2024
f8a07d7
major bugx fixed: N_SMP was n_SMP, not propagated
martinkilbinger Feb 1, 2024
30116b8
summary nb
martinkilbinger Feb 1, 2024
9e44fa6
sumamry updated
martinkilbinger Feb 11, 2024
3ab923f
terminal title from within curl script
martinkilbinger Feb 11, 2024
ee5b3ad
Update tiles_P7.txt
martinkilbinger Feb 11, 2024
79cb00a
Merge pull request #9 from martinkilbinger/P7
martinkilbinger Feb 11, 2024
92e4308
merged Dockerfile from docker_image
martinkilbinger Feb 11, 2024
1d928d4
monitoring and job handling scripts; small modifs, output; canfar pip…
martinkilbinger Feb 23, 2024
403e8c5
Merge pull request #10 from martinkilbinger/curl
martinkilbinger Feb 23, 2024
a6d8b52
combine run script: flag_tile and flag_exp as diff options
martinkilbinger Mar 5, 2024
1a2d47d
mask runer excluding processed files working
martinkilbinger Mar 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update post_processing.md
cleaned up psf post proc
  • Loading branch information
martinkilbinger authored Dec 25, 2023
commit d4cb47ce14ed9f8fcdd38af42cf4f68488f22560
77 changes: 13 additions & 64 deletions docs/source/post_processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,59 +10,7 @@ catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plo
If main ShapePipe processing happened at the old canfar VM system (e.g. CFIS v0 and v1), go
[here](vos_retrieve.md) for details how to retrieve the ShapePipe output files.

Some of the following steps pertain specifically to runs carried out on [canfar](https://www.canfar.net/en),
but most are general.

1. Retrieve `ShapePipe` result files

For a local run on the same machine as for post-processing, nothing needs to be done.
In some cases, the run was carried out on a remote machine or cluster, and the resulting `ShapePipe`
output files need to be retrieved.

In the specific case of canfar_avail_results.py, this is done as follows.

A. Check availability of results

A `canfar` job can submit a large number of tiles, whose processing time can vary a lot.
We assume that the submitted tile ID list is available locally via the ascii file `tile_numbers.txt`.
To check which tiles have finished running, and whose results have been uploaded, use
```bash
canfar_avail_results -i tile_numbers.txt -v -p PSF --input_path INPUT_PATH
```
where PSF is one in [`psfex`|`mccd`], and INPUT_PATH the input path on vos, default `vos:cfis/cosmostat/kilbinger/results`.
See `-h` for all options.

B. Download results

All results files will be downloaded with
```bash
canfar_download_results -i tile_numbers.txt -v -p PSF --input_vos INPUT_VOS
```
Use the same options as for same as for `canfar_avail_results`.

This command can be run in the same directory at subsequent times, to complete an ongoing run: Only newer files will be downloaded
from the `vos` directory. This also assures that partially downloaded or corrupt files will be replaced.

Checking the `vos` directorty can be slow for large patches.
To only download files that are not yet present locally (in `.`), first write the missing ones to an ascii file, using again the
script `canfar_avail_results`, but this time with `.` as input path:
```bash
canfar_avail_results -i tile_numbers.txt --input_path . -p PSF -v -o missing.txt
'''
Then, download only the missing files with
```bash
canfar_download_results -i missing.txt --input_vos cosmostat/kilbinger/results_mccd_oc2 -p mccd -v
```

C. Un-tar results
```bash
untar_results -p PSF
```
On success, `ShapePipe` output `fits` and `log` files will be now in various subdirs of the `output` directory.

At this step all required `ShapePipe` resulting output files are available in the current working directory.

2. Optional: Split output in sub-samples
1. Optional: Split output into sub-samples

An optional intermediate step is to create directories for sub-samples, for example one directory
for each patch on the sky. This will create symbolic links to the results `.tgz` files downloaded in
Expand All @@ -73,33 +21,34 @@ At this step all required `ShapePipe` resulting output files are available in th
```
The following steps will then be done in the directory `tiles_W3`.

3. Run PSF diagnostics, create merged catalogue
2. Run PSF diagnostics, create merged catalogue

Type
```bash
post_proc_sp -p PSF
```
to automatically perform a number of post-processing steps. Chose the PSF model with the option
to automatically perform a number of post-processing steps. Choose the PSF model with the option
`-p psfex|mccd`. In detail, these are (and can also be done individually
by hand):

A. Analyse psf validation files
1. Analyse psf validation files

```bash
prepare_star_cat -p PSF
combine_runs -t psf -p PSF
```
with options as for `post_proc_sp`.
This script identifies all psf validation files (from all processed tiles downloaded to `pwd`), creates symbolic links,
merges the catalogues, and creates plots of PSF ellipticity, size, and residuals over the focal plane.
This script creates a new combined psf run in the ShapePipe `output` directory, by identifying all psf validation files
and creating symbolic links. The run log file is updated.

B. Create plots of the PSF and their residuals in the focal plane, as a diagnostic of the overall PSF model.
As a scale-dependend test, which propagates directly to the shear correlation function, the rho statistics are computed,
see {cite:p}`rowe:10` and {cite:p}`jarvis:16`,
3. Merge individual psf validation files into one catalogue. Create plots of the PSF and their residuals in the focal plane,
as a diagnostic of the overall PSF model.
As a scale-dependend test, which propagates directly to the shear correlation function, the rho statistics are computed,
see {cite:p}`rowe:10` and {cite:p}`jarvis:16`,
```bash
shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini
```

C. Prepare output directory
4. Prepare output directory

Create links to all 'final_cat' result files with
```bash
Expand All @@ -108,7 +57,7 @@ At this step all required `ShapePipe` resulting output files are available in th
The corresponding output directory that is created is `output/run_sp_combined/make_catalog_runner/output`.
On success, it contains links to all `final_cat` output catalogues

D. Merge final output files
5. Merge final output files

Create a single main shape catalog:
```bash
Expand Down