Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow downloading the data without ssl certificate verification (#43) #44

Merged
merged 2 commits into from
Jan 17, 2024
Merged

feat: Allow downloading the data without ssl certificate verification (#43) #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
510a620
feat: Implement mode without cert check
gromdimon Jan 17, 2024
16e8e4c
readme
gromdimon Jan 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,26 @@ cp docker-compose.override.yml-dev docker-compose.override.yml

### Download Dev Data

Now you need to obtain the data to serve by the mehari, viguno, and annonars container.
For this, we have prepared strongly reduced data sets (overall less than 2GB rather than hundreds of GB of data).
To serve data via the mehari, viguno, and annonars containers, you need to obtain the required datasets. We have prepared significantly reduced datasets (totaling less than 2GB as opposed to hundreds of GB) for development purposes.

We provide a script that will setup the necessary directories, download the data, and create symlinks.
We provide a script that sets up the necessary directories, downloads the data, and creates symlinks.

By default, the script verifies SSL certificates when downloading data. If you encounter SSL verification issues or operate in an environment where SSL verification is not required, you can disable SSL verification by setting the `NO_VERIFY_SSL` variable to `1` when running the script.

To download the data with SSL verification (default behavior):

```bash session
bash download-data.sh
```

To download the data without SSL verification:

```bash session
NO_VERIFY_SSL=1 bash download-data.sh
```

Note: Disabling SSL verification can make the connection less secure. Use this option only if you understand the risks and it is necessary for your environment.

### Setup Configuration

The next step step is to create the configuration files in `.dev/config`.
Expand Down
10 changes: 10 additions & 0 deletions download-data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ export QUIET=${QUIET-0}
export VERBOSE=${VERBOSE-0}
# Dry-run mode.
export DRY_RUN=${DRY_RUN-0}
# No-verify-ssl mode: do not verify SSL certificates.
export NO_VERIFY_SSL=${NO_VERIFY_SSL-0}
# Download options: reduced-dev (default), reduced-exomes, full.
export DOWNLOAD=${DOWNLOAD-reduced-dev}
# Directory for static data.
Expand All @@ -25,6 +27,13 @@ export DATA_DIR=${DATA_DIR-$DIR_PREFIX/volumes/$STATIC_DIR/data}
# S3 endpoing URL.
export S3_ENDPOINT_URL=https://ceph-s3-public.cubi.bihealth.org

# Set S5CMD_NO_VERIFY_SSL_ARG based on NO_VERIFY_SSL
if [ "$NO_VERIFY_SSL" -eq 1 ]; then
S5CMD_NO_VERIFY_SSL_ARG="--no-verify-ssl"
else
S5CMD_NO_VERIFY_SSL_ARG=""
fi

# -- Versions -----------------------------------------------------------------

# annonars
Expand Down Expand Up @@ -251,6 +260,7 @@ while read -r line; do
run s5cmd \
--endpoint-url=$S3_ENDPOINT_URL \
--no-sign-request \
$S5CMD_NO_VERIFY_SSL_ARG \
sync \
"s3://varfish-public/$(prefix_for $line)/$line/*" \
$DATA_DIR/download/$line \
Expand Down
Loading