Skip to content

Latest commit

 

History

History
267 lines (218 loc) · 10.6 KB

README.adoc

File metadata and controls

267 lines (218 loc) · 10.6 KB
Raw

Quarkus.io Search

A Quarkus-powered application that allows full-text search in websites of the Quarkus organization such as https://quarkus.io.

Architecture

The application is deployed on an OpenShift cluster, next to an Elasticsearch instance.

It fetches the sources from quarkus.io and localized variants (pt.quarkus.io, …​) to index them, and exposes the search feature through a REST API.

The webpage at https://quarkus.io/guides calls that API to perform search.

Architecture diagram

Development

Using the built-in quarkus.io sample

Note

By default, this will use a small sample of quarkus.io included in this repository, which has the advantage of being faster to index on startup, but is incomplete.

To index the actual data, see Using the actual quarkus.io data.

To launch the application in dev mode:

quarkus dev
# OR
./mvnw quarkus:dev

Then head over to http://0.0.0.0:9000/q/swagger-ui/ to try the service.

Using the actual quarkus.io data

To use the actual quarkus.io data without downloading it on each startup, you should clone the quarkus.io repository manually and add a .env file to your working directory:

_DEV_QUARKUSIO_GIT_URI=file:/home/myself/path/to/quarkusio.github.io
_DEV_QUARKUSIO_LOCALIZED_JA_GIT_URI=file:/home/myself/path/to/ja.quarkus.io
_DEV_QUARKUSIO_LOCALIZED_ES_GIT_URI=file:/home/myself/path/to/es.quarkus.io
_DEV_QUARKUSIO_LOCALIZED_PT_GIT_URI=file:/home/myself/path/to/pt.quarkus.io
_DEV_QUARKUSIO_LOCALIZED_CN_GIT_URI=file:/home/myself/path/to/cn.quarkus.io
# Avoid reindexing on startup if indexes already contain data.
# You can remove this if you're fine with waiting a few minutes on each live reload.
_DEV_INDEXING_ON_STARTUP_WHEN=indexes-empty

Then the application will use data from that clone when indexing on startup.

If you also intend to run quarkus.io locally, and want links of search hits to redirect to your local instance of quarkus.io, you should also add this property:

_DEV_QUARKUSIO_WEB_URI=http://localhost:4000

By default, in dev mode, any parsing/indexing warnings and errors are going to be simply logged. But in case you want to get index errors reported to a GitHub issue, next properties should be added:

_DEV_INDEXING_REPORTING_TYPE=github-issue
# see about tokens https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
INDEXING_REPORTING_GITHUB_TOKEN={your-generated-github-token}
INDEXING_REPORTING_GITHUB_ISSUE_REPOSITORY={your-github-user}/search.quarkus.io
INDEXING_REPORTING_GITHUB_ISSUE_ID={github-issue-id-in-your-repository}
INDEXING_REPORTING_GITHUB_WARNING_REPEAT_DELAY=10m

Testing

Testing (and, by default dev mode) will use a small sample of quarkus.io included in this repository at src/test/resources/quarkusio-sample.zip (alongside with the localized site samples such as quarkusio-sample-cn.zip, quarkusio-sample-es.zip, quarkusio-sample-ja.zip, quarkusio-sample-pt.zip).

This sample must itself be a git repository, so generating it can get technical.

Fortunately, there is a tool included in this project to generate the sample. To update the sample:

  • Add the references of the guides you want included in the sample to GuidesRef.

  • Run QuarkusIOSample.main() using your IDE, with the project root as the current working directory.

    If you set up your .env file as descripted in Using the actual quarkus.io data, you don’t need to pass any argument.

    Otherwise, pass as the path to your clone of quarkus.io, followed by pairs of language code and path to corresponding localized site, e.g.path/to/quarkus.io ja path/to/ja.quarkus.io es path/to/es.quarkus.io […​]. Add pairs for all cn, es, ja, pt language codes to update all samples at the same time.

Production

OpenShift configuration

The production environment uses OpenShift. You can generate the OpenShift configuration by running this:

quarkus build -DskipTests
# OR
./mvnw package -DskipTests

The generated file will be in target/kubernetes/openshift.yml.

Running it with podman

If you want to start a (production) container locally with podman, you can build one with the following command:

quarkus build -DskipTests -Dquarkus.container-image.build=true
# OR
./mvnw install -DskipTests -Dquarkus.container-image.build=true

Then start it this way:

podman pod create -p 8080:8080 -p 9000:9000 -p 9200:9200 --name search.quarkus.io
# Start multiple Elasticsearch containers
podman container run -d --name search-backend-0 --pod search.quarkus.io \
    --cpus=2 --memory=2g \
    -e "node.name=search-backend-0" \
    -e "discovery.seed_hosts=localhost" \
    -e "cluster.initial_cluster_manager_nodes=search-backend-0,search-backend-1,search-backend-2" \
    -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
    -e "xpack.security.enabled=false" \
    -e "cluster.routing.allocation.disk.threshold_enabled=false" \
    elasticsearch-custom:latest
podman container run -d --name search-backend-1 --pod search.quarkus.io \
    --cpus=2 --memory=2g \
    -e "node.name=search-backend-1" \
    -e "discovery.seed_hosts=localhost" \
    -e "cluster.initial_cluster_manager_nodes=search-backend-0,search-backend-1,search-backend-2" \
    -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
    -e "xpack.security.enabled=false" \
    -e "cluster.routing.allocation.disk.threshold_enabled=false" \
    elasticsearch-custom:latest
podman container run -d --name search-backend-2 --pod search.quarkus.io \
    --cpus=2 --memory=2g \
    -e "node.name=search-backend-2" \
    -e "discovery.seed_hosts=localhost" \
    -e "cluster.initial_cluster_manager_nodes=search-backend-0,search-backend-1,search-backend-2" \
    -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
    -e "xpack.security.enabled=false" \
    -e "cluster.routing.allocation.disk.threshold_enabled=false" \
    elasticsearch-custom:latest
# Then the app; this will fetch the actual data on startup (might take a while):
podman container run -it --rm --name search.quarkus.io --pod search.quarkus.io search-quarkus-io:999-SNAPSHOT
# OR, if you already have locals clones of *.quarkus.io:
# (you might need to run quarkus dev with those repos first to get them all in sync)
REPOS_DIR=$HOME/path/to/dir/containing/repos
podman container run -it --rm --name search.quarkus.io --pod search.quarkus.io \
    --cpus=1 --memory=1g \
    -v $REPOS_DIR/quarkusio.github.io:/mnt/quarkus.io:ro,z \
    -v $REPOS_DIR/cn.quarkus.io:/mnt/cn.quarkus.io:ro,z \
    -v $REPOS_DIR/es.quarkus.io:/mnt/es.quarkus.io:ro,z \
    -v $REPOS_DIR/ja.quarkus.io:/mnt/ja.quarkus.io:ro,z \
    -v $REPOS_DIR/pt.quarkus.io:/mnt/pt.quarkus.io:ro,z \
    -e INDEXING_REPORTING_TYPE=log \
    -e GITHUB_OAUTH=ignored \
    -e GITHUB_STATUS_ISSUE_ID=1 \
    -e QUARKUSIO_GIT_URI=file:/mnt/quarkus.io \
    -e QUARKUSIO_LOCALIZED_CN_GIT_URI=file:/mnt/cn.quarkus.io \
    -e QUARKUSIO_LOCALIZED_ES_GIT_URI=file:/mnt/es.quarkus.io \
    -e QUARKUSIO_LOCALIZED_JA_GIT_URI=file:/mnt/ja.quarkus.io \
    -e QUARKUSIO_LOCALIZED_PT_GIT_URI=file:/mnt/pt.quarkus.io \
    search-quarkus-io:999-SNAPSHOT

Deployment

Current process

Maintainers can review the application and update configuration/secrets on the OpenShift console.

There are two namespaces containing two separate deployments at the moment:

Deployment will happen automatically when pushing to the relevant branch.

Be careful about which configuration you change in the UI, as deployment may overwrite part of the topology.

Setting it up

Most of the process is automated, but if you need to deploy to a new cluster, you will need to set up a few things manually:

  1. Service account for GitHub Actions deployment. The account credentials (username/token) need to be registered as GitHub Actions secrets, as well as the cluster URI. See .github/workflows/deploy.yml.

  2. Namespace The OpenShift namespace needs to be registered as a GitHub Actions environment variable. See .github/workflows/deploy.yml.

  3. Config maps and secrets.

    search-quarkus-io-config

    Environment variables for the application.
    Put in there whatever configuration you need for your specific cluster.
    In particular:

    • GITHUB_STATUS_ISSUE_ID: The number of an issue on quarkusio/search.quarkus.io where indexing status should be reported. See indexing.reporting.github configuration properties for more details.

    search-quarkus-io-secret

    Secret environment variables for the application.
    Put in there whatever secret configuration you need for your specific cluster.
    In particular:

    • GITHUB_OAUTH: a GitHub token that allows commenting/reopening/closing a GitHub issue on quarkusio/search.quarkus.io. See indexing.reporting.github configuration properties for more details.

    search-backend-config

    Environment variables for the Elasticsearch instances.
    Put in there whatever configuration you need for your specific cluster.

    search-backend-secret

    Secret environment variables for the Elasticsearch instances.
    Put in there whatever secret configuration you need for your specific cluster.

License

This project is licensed under the Apache License Version 2.0.

The web assets in src/main/resources/web are licensed under the Creative Commons Attribution 3.0 International License.