Skip to content

Latest commit

 

History

History
111 lines (82 loc) · 2.02 KB

cleanup.md

File metadata and controls

111 lines (82 loc) · 2.02 KB

🚧 Under construction !!! 🚧

[[TOC]]

Quotas on servers

Your home, scratch and the project folders in /work/ have a quota for the number of files and occupied space

# occupied space and quotas
df -h
# number of files and quotas
df -h -i

To free up space:

  • remove temporary data
  • compress and archive not used data
    • compression: to save space
    • archiving: to reduce the number of files
  • move data
    • final results: move to XXXXX (see here)
    • temporary/preliminary data: scratch (see here)

Cleanup

Conda

conda environments can be very large and contain a lot of files.

# list installed environments
conda info --envs
# remove not used environments
rm -r path/to/conda/env/
# remove unused packages and caches
conda remove --all

Snakemake

TODO

  • .snakemake
    • logs
    • conda env.s
    • meta files
    • ...

Compressing and archiving data

TODO: info and references

  • zip
  • tar
  • gzip, pigz
  • bzip2, pbzip2
# tar: archive folder
tar -cvf path/to/archive.tar path/to/folder/
# tar: archive files
tar -cvf path/to/archive.tar /path/to/file1 /path/to/file2 ...

# tar: list contents
tar -tvf path/to/archive.tar

# extract archive
tar -xvf path/to/archive.tar
# zip: compress and archive folders
zip -r path/to/archive.zip path/to/folder/
# zip: compress and archive files
zip path/to/archive.zip /path/to/file1 /path/to/file2 ...

# zip: update an archive
zip -u path/to/archive.zip /path/to/file1 /path/to/file2 ...

# zip: list contents
unzip -l path/to/archive.zip

# zip: extract and decompress
unzip path/to/archive.zip
# gzip: compress files
TODO

# tar + pigz: archive and compress a folder
tar -cvf - path/to/folder/ | pigz --best -p {cpus} > path/to/archive.tar.gz

# tar + gzip: list contents
tar -ztvf path/to/archive.tar.gz

# tar + gzip: extract files
tar -zxvf path/to/archive.tar.gz
# bzip2
TODO

# pbzip2
TODO