From 4d37d9f4b5d30d4408f9d0a90c279425513ad338 Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Thu, 14 Dec 2023 13:37:52 -0800 Subject: [PATCH 1/4] Fix unpredictable markdown link test behavior, clarify test Testing the container-based Bodhi development environment I'm working on exposed a subtle behavior issue in this test/code. The tests have a global fixture - `mock_settings_env_vars` - which sets an environment variable, and a setup method - `BaseTestCaseMixin._setup_method` - which updates `bodhi.server.config.config` (which is a mutable singleton) from the file that env var points to (`tests/testing.ini`). Before the first time the setup method is run, when running the tests in a development environment, `bodhi.server.config.config` will contain the values from `/home/vagrant/development.ini`, the configuration intended for the development deployment, and so any code that's run at this point - including code at the global level in modules that get imported before any test case is run, I believe - uses those values, not the ones from testing.ini. You can test this by ensuring the value is different in the two config files, then adding a line `raise ValueError(config['base_address'])` first at the top of `ffmarkdown.py` - where `UPDATE_RE` was defined, before this commit - and running the test, then moving the line into the `extendMarkdown` method - where this commit moves it to - and running the test. When we check the value at the top of the file, it will be the one from `/home/vagrant/development.ini`; when we check the value from inside `extendMarkdown`, it is the value from `testing.ini`. I think this is because code at the global level gets executed immediately on the first import of the module, which happens before any test case has started to run and so before the setup method has changed the config settings; but `extendMarkdown` will only get called *after* the test case has started running and the setup method has been called. To confuse matters, the test case isn't super explicit about what it's really doing, because it just hardcodes the base_address value from testing.ini into the URL. Changing the test case to show that it's using `config['base_address']` makes it clearer, and moving the update regex definition inside `extendMarkdown` ensures the test and the app code agree on what that value actually is. The regex is not used anywhere else. Alternatively we could leave the regex definition at the global level and have the test code read base_address from `base.original_config`, which is a copy of the 'original' config that `base.py` takes before doing anything to it (so probably somebody ran into this before). But I think I like this way. I noticed this issue because I had to change the base_address value in `/home/vagrant/development.ini` in my container-based environment. In the VM-based environment the base_address value in both development.ini and testing.ini is the same, so the issue is hidden. Signed-off-by: Adam Williamson --- bodhi-server/bodhi/server/ffmarkdown.py | 13 +++++++++---- bodhi-server/tests/views/test_generic.py | 4 +++- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/bodhi-server/bodhi/server/ffmarkdown.py b/bodhi-server/bodhi/server/ffmarkdown.py index 08597245b3..3839166959 100644 --- a/bodhi-server/bodhi/server/ffmarkdown.py +++ b/bodhi-server/bodhi/server/ffmarkdown.py @@ -41,9 +41,6 @@ BUGZILLA_RE = r'([a-zA-Z]+)(#[0-9]{5,})' -UPDATE_RE = (r'(?:(? str: @@ -204,7 +201,15 @@ def extendMarkdown(self, md: markdown.Markdown) -> None: Args: md: An instance of the Markdown class. """ + # it is intentional that update_re is defined here, not in + # global scope at the top of this file. When testing, defining + # it early results in config['base_address'] being read before + # the test setup method has changed the config to use the + # values from testing.ini and the value may not be as expected + update_re = (r'(?:(?

See ' '' From 2b3838222be6cbed68b9c920f211ccc19ea76767 Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Fri, 22 Dec 2023 14:01:11 -0800 Subject: [PATCH 2/4] When using dummy ACLs, assume update owner can edit it This makes things behave more intuitively in an interactive development environment which uses dummy ACLs (it's weird if you're logged in as bob and you can't edit an update owned by bob). Signed-off-by: Adam Williamson --- bodhi-server/bodhi/server/validators.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/bodhi-server/bodhi/server/validators.py b/bodhi-server/bodhi/server/validators.py index 2548a6ed8a..9593d290a0 100644 --- a/bodhi-server/bodhi/server/validators.py +++ b/bodhi-server/bodhi/server/validators.py @@ -609,6 +609,14 @@ def validate_acls(request, **kwargs): committers = ['ralph', 'bowlofeggs', 'guest'] if config['acl_dummy_committer']: committers.append(config['acl_dummy_committer']) + # let's also assume the update's owner can edit it + update = request.validated.get('update') + if not update: + alias = request.validated.get('edited') + if alias: + update = Update.get(alias) + if update: + committers.append(update.user.name) if user.name in committers: has_access = True people = committers From 16a5bfcb3075944fdd15b5cdbc0add7d419ea636 Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Fri, 22 Dec 2023 14:02:31 -0800 Subject: [PATCH 3/4] docs/Makefile: use python3 not python I think we can stop caring about python2 now, and this means we don't have to install python-unversioned-command any time we want to use this. Signed-off-by: Adam Williamson --- docs/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Makefile b/docs/Makefile index b9269ad048..9092269efa 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -3,7 +3,7 @@ # You can set these variables from the command line. SPHINXOPTS = -PYTHON = python +PYTHON = python3 SPHINXBUILD = $(PYTHON) -m sphinx PAPER = BUILDDIR = _build From baf45b49a686a253ceb825b48dab8d0dbafb59ad Mon Sep 17 00:00:00 2001 From: Adam Williamson Date: Wed, 13 Dec 2023 12:01:51 -0800 Subject: [PATCH 4/4] Add BCD: a standalone, container-based dev environment The current development environment requires four separate VMs (three from tinystage and one for Bodhi itself), plus some containers running inside the Bodhi VM. It's pretty heavy! This provides an alternative development environment that is standalone and entirely based on containers: a postgres container, a waiverdb container, a greenwave container, a rabbitmq container, an ipsilon container, and a bodhi container that functions similarly to the bodhi VM from the existing environment. There is no FreeIPA or FAS backing the ipsilon instance, we just use ipsilon's testauth mode, with some configuration to allow for testing different group memberships. The containers are Podman containers orchestrated by an ansible playbook, around which the `bcd` command is a light wrapper. They share a network namespace via a pod, and everything is accessed from the host as `localhost`, without SSL or Apache (these could be added if we really want, but it seemed unnecessary). Signed-off-by: Adam Williamson --- bcd | 125 +++++++++++++++ .../containers/bodhi/Containerfile | 95 ++++++++++++ devel/ansible-podman/containers/bodhi/bashrc | 59 +++++++ .../containers/bodhi/bodhi-wait.py | 29 ++++ .../containers/bodhi/bodhi.service | 20 +++ .../containers/bodhi/celery.service | 12 ++ .../containers/bodhi/config.toml | 51 ++++++ .../containers/bodhi/development.ini.example | 1 + devel/ansible-podman/containers/bodhi/motd | 24 +++ .../containers/bodhi/printer.toml | 44 ++++++ .../containers/bodhi/remote.toml | 57 +++++++ devel/ansible-podman/containers/bodhi/vimrc | 3 + devel/ansible-podman/playbook.yml | 5 + .../roles/podman/files/greenwave-settings.py | 1 + .../roles/podman/files/init-db.sh | 9 ++ .../roles/podman/files/run_waiverdb.sh | 3 + .../roles/podman/files/waiverdb-settings.py | 1 + .../podman/tasks/clear_ipsilon_sessions.yml | 4 + .../roles/podman/tasks/main.yml | 14 ++ .../roles/podman/tasks/prep.yml | 65 ++++++++ .../roles/podman/tasks/remove.yml | 16 ++ .../roles/podman/tasks/start.yml | 96 ++++++++++++ .../roles/podman/tasks/stop.yml | 16 ++ devel/ansible/playbook.yml | 2 +- devel/ci/integration/ipsilon/Dockerfile | 31 ++-- .../{setup-bodhi.py => setup-bodhi.py.tmpl} | 11 +- devel/docker/settings/restore_waiverdb.sh | 0 docs/developer/bcd.rst | 98 ++++++++++++ docs/developer/index.rst | 15 +- docs/developer/vagrant.rst | 2 + docs/developer/virtualenv.rst | 146 ------------------ docs/index.rst | 2 +- 32 files changed, 888 insertions(+), 169 deletions(-) create mode 100755 bcd create mode 100644 devel/ansible-podman/containers/bodhi/Containerfile create mode 100644 devel/ansible-podman/containers/bodhi/bashrc create mode 100755 devel/ansible-podman/containers/bodhi/bodhi-wait.py create mode 100644 devel/ansible-podman/containers/bodhi/bodhi.service create mode 100644 devel/ansible-podman/containers/bodhi/celery.service create mode 100644 devel/ansible-podman/containers/bodhi/config.toml create mode 120000 devel/ansible-podman/containers/bodhi/development.ini.example create mode 100644 devel/ansible-podman/containers/bodhi/motd create mode 100644 devel/ansible-podman/containers/bodhi/printer.toml create mode 100644 devel/ansible-podman/containers/bodhi/remote.toml create mode 100644 devel/ansible-podman/containers/bodhi/vimrc create mode 100644 devel/ansible-podman/playbook.yml create mode 120000 devel/ansible-podman/roles/podman/files/greenwave-settings.py create mode 100755 devel/ansible-podman/roles/podman/files/init-db.sh create mode 100755 devel/ansible-podman/roles/podman/files/run_waiverdb.sh create mode 120000 devel/ansible-podman/roles/podman/files/waiverdb-settings.py create mode 100644 devel/ansible-podman/roles/podman/tasks/clear_ipsilon_sessions.yml create mode 100644 devel/ansible-podman/roles/podman/tasks/main.yml create mode 100644 devel/ansible-podman/roles/podman/tasks/prep.yml create mode 100644 devel/ansible-podman/roles/podman/tasks/remove.yml create mode 100644 devel/ansible-podman/roles/podman/tasks/start.yml create mode 100644 devel/ansible-podman/roles/podman/tasks/stop.yml rename devel/ci/integration/ipsilon/{setup-bodhi.py => setup-bodhi.py.tmpl} (90%) mode change 100644 => 100755 devel/docker/settings/restore_waiverdb.sh create mode 100644 docs/developer/bcd.rst delete mode 100644 docs/developer/virtualenv.rst diff --git a/bcd b/bcd new file mode 100755 index 0000000000..f7992abdbb --- /dev/null +++ b/bcd @@ -0,0 +1,125 @@ +#!/usr/bin/python + +import argparse +import os +import subprocess +import sys + +CONTAINERS = ( + "database", + "waiverdb", + "greenwave", + "rabbitmq", + "ipsilon", + "bodhi", +) + + +def ansible(args): + """Run an ansible playbook command based on the parser name.""" + # this is the subcommand that was run - 'run', 'stop' etc. + here = os.path.abspath(os.path.dirname(__file__)) + ret = subprocess.run( + ( + "ansible-playbook", + f"{here}/devel/ansible-podman/playbook.yml", + f"-e bodhi_dev_{args.subcommand}=true" + ) + ) + sys.exit(ret.returncode) + + +def logs(args): + fullc = f"bodhi-dev-{args.container}" + ret = subprocess.run(("podman", "logs", fullc)) + sys.exit(ret.returncode) + + +def shell(args): + fullc = f"bodhi-dev-{args.container}" + ret = subprocess.run(("podman", "exec", "-it", fullc, "/bin/bash")) + sys.exit(ret.returncode) + + +def parse_args(): + """Parse arguments with argparse.""" + parser = argparse.ArgumentParser( + description=( + "Bodhi Container Development environment. Controls a complete Bodhi development " + "environment in Podman containers orchestrated by Ansible." + ) + ) + subparsers = parser.add_subparsers(dest="subcommand") + subparsers.required = True + parser_run = subparsers.add_parser( + "run", + description="Prepare and run the environment", + aliases=["up"] + ) + parser_run.set_defaults(func=ansible) + parser_stop = subparsers.add_parser( + "stop", + description="Stop the environment (does not remove containers)", + aliases=["halt"] + ) + parser_stop.set_defaults(func=ansible) + parser_remove = subparsers.add_parser( + "remove", + description="Stop and remove all containers", + aliases=["destroy"] + ) + parser_remove.set_defaults(func=ansible) + parser_cis = subparsers.add_parser( + "cis", + description="Clear Ipsilon sessions (to allow you to log in as a different user)" + ) + parser_cis.set_defaults(func=ansible) + parser_shell = subparsers.add_parser( + "shell", + description="Open a shell in a container. Container must be running" + ) + parser_shell.add_argument( + "container", + help="The container to open a shell in (default: bodhi)", + default="bodhi", + nargs='?', + choices=CONTAINERS + ) + parser_shell.set_defaults(func=shell) + parser_logs = subparsers.add_parser( + "logs", + description="Show logs of the specified container (does not work on bodhi, use journalctl)" + ) + parser_logs.add_argument( + "container", + help="The container to show logs for", + choices=[cont for cont in CONTAINERS if cont != "bodhi"], + ) + parser_logs.set_defaults(func=logs) + parser_prep = subparsers.add_parser( + "prep", + description="Run preparation steps only" + ) + parser_prep.set_defaults(func=ansible) + parser_start = subparsers.add_parser( + "start", + description="Start containers only (do not run prep, will fail if prep has not already run)" + ) + parser_start.set_defaults(func=ansible) + return parser.parse_args() + + +def main(): + """Main loop.""" + try: + args = parse_args() + args.func(args) + except KeyboardInterrupt: + sys.stderr.write("Interrupted, exiting...\n") + sys.exit(1) + + +if __name__ == "__main__": + main() + +# vim: set textwidth=100 ts=8 et sw=4: diff --git a/devel/ansible-podman/containers/bodhi/Containerfile b/devel/ansible-podman/containers/bodhi/Containerfile new file mode 100644 index 0000000000..4ea3eed24a --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/Containerfile @@ -0,0 +1,95 @@ +FROM quay.io/fedora/fedora:latest +LABEL \ + summary="Bodhi development environment" \ + description="Distribution software update management system" \ + maintainer="Red Hat, Inc." \ + license="GPLv2+" \ + url="https://github.com/fedora-infra/bodhi" \ + vcs-type="git" \ + io.k8s.display-name="Bodhi DE" +MAINTAINER adamwill +RUN set -exo pipefail \ + # to get bodhi's runtime deps installed, install the official + # packages then remove them (in a single step so we don't get a + # layer with them included) + && dnf install -y --setopt install_weak_deps=false --nodocs bodhi-server bodhi-client python3-bodhi-messages \ + && dnf remove -y --noautoremove bodhi-server bodhi-client python3-bodhi-messages \ + # install test and dev env deps (and any new deps not yet in the + # package) + && dnf install -y --setopt install_weak_deps=false --nodocs \ + # missing runtime deps + # there are errors in the logs if this isn't installed + procps-ng \ + # dev env creation and control deps + poetry \ + postgresql \ + python3-pip \ + sed \ + systemd \ + # dev env QOL and debug deps + bash-completion \ + htop \ + httpie \ + nano \ + nmap-ncat \ + pcp-system-tools \ + python3-ipdb \ + python3-pydocstyle \ + screen \ + tmux \ + tree \ + vim-enhanced \ + # doc build deps + graphviz \ + make \ + python3-sqlalchemy_schemadisplay \ + python3-sphinx \ + # test deps + createrepo_c \ + pre-commit \ + python3-createrepo_c \ + python3-diff-cover \ + python3-pytest \ + python3-pytest-cov \ + python3-pytest-mock \ + python3-responses \ + python3-webtest \ + # pre-commit uses flake8, mypy, pydocstyle and ruff-pre-commit, + # but it always pulls them from repos, never uses packages + && dnf clean all \ + && rm -rf /var/cache/* /var/log/dnf* + +# note we use the devel/ directory as the build context so we can access +# development.ini.example here +COPY ./ansible-podman/containers/bodhi/celery.service /etc/systemd/system/celery.service +COPY ./ansible-podman/containers/bodhi/bodhi.service /etc/systemd/system/bodhi.service +COPY ./ansible-podman/containers/bodhi/motd /etc/motd +COPY ./ansible-podman/containers/bodhi/bashrc /root/.bashrc +COPY ./ansible-podman/containers/bodhi/vimrc /root/.vimrc +COPY ./ansible-podman/containers/bodhi/config.toml /etc/fedora-messaging/config.toml +COPY ./ansible-podman/containers/bodhi/printer.toml /etc/fedora-messaging/printer.toml +COPY ./ansible-podman/containers/bodhi/remote.toml /etc/fedora-messaging/remote.toml +COPY ./ansible-podman/containers/bodhi/bodhi-wait.py /usr/local/bin/bodhi-wait.py +COPY ./development.ini.example /etc/bodhi/production.ini + +RUN set -ex \ + && sed -i -e "s,celery_config.*,celery_config = /bodhi/bodhi-server/celeryconfig.py,g" /etc/bodhi/production.ini \ + && sed -i -e "s,pungi.basepath.*,pungi.basepath = /bodhi/devel/ci/integration/bodhi/,g" /etc/bodhi/production.ini \ + && sed -i -e "s,base_address.*,base_address = http://localhost.localdomain:6543/,g" /etc/bodhi/production.ini \ + && sed -i -e "s,cors_origins_rw.*,cors_origins_rw = *,g" /etc/bodhi/production.ini \ + && sed -i -e "s,cors_connect_src.*,cors_connect_src = *,g" /etc/bodhi/production.ini \ + && sed -i -e "s,openid.provider.*,openid.provider = http://localhost.localdomain:6546/openid/,g" /etc/bodhi/production.ini \ + && sed -i -e "s,openid.url.*,openid.url = http://localhost.localdomain:6546/,g" /etc/bodhi/production.ini \ + && sed -i -e "s,openid_template.*,openid_template = {username}.localdomain,g" /etc/bodhi/production.ini \ + && sed -i -e "s,oidc.fedora.client_id.*,oidc.fedora.client_id = integration-tests,g" /etc/bodhi/production.ini \ + && sed -i -e "s,oidc.fedora.client_secret.*,oidc.fedora.client_secret = integration-tests,g" /etc/bodhi/production.ini \ + && sed -i -e "s,oidc.fedora.server_metadata_url.*,oidc.fedora.server_metadata_url = http://localhost.localdomain:6546/openidc/.well-known/openid-configuration,g" /etc/bodhi/production.ini + +RUN pip install pyramid_debugtoolbar +RUN ln -s /usr/bin/true /usr/bin/pungi-koji +RUN mkdir -p /srv/{composes/final,composes/stage} +RUN systemctl enable fm-consumer@config.service fm-consumer@printer.service celery.service bodhi.service +RUN systemctl disable pmcd.service pmie.service pmlogger.service pmlogger_farm.service +RUN poetry config virtualenvs.create false +EXPOSE 6543 +CMD [ "/usr/sbin/init" ] diff --git a/devel/ansible-podman/containers/bodhi/bashrc b/devel/ansible-podman/containers/bodhi/bashrc new file mode 100644 index 0000000000..16e3d5d6a9 --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/bashrc @@ -0,0 +1,59 @@ +# .bashrc + +# Source global definitions +if [ -f /etc/bashrc ]; then + . /etc/bashrc +fi + +# Uncomment the following line if you don't like systemctl's auto-paging feature: +# export SYSTEMD_PAGER= + +shopt -s expand_aliases +alias bdocs="make -C /bodhi/docs clean && make -C /bodhi/docs html && make -C /bodhi/docs man" +alias blog="journalctl -u bodhi -u fm-consumer@config" +alias brestart="systemctl restart bodhi.service celery.service fm-consumer@config.service fm-consumer@printer.service && echo 'The application is running on http://localhost.localdomain:6543'" +alias bstart="systemctl start bodhi.service celery.service fm-consumer@config.service fm-consumer@printer.service && echo 'The application is running on http://localhost.localdomain:6543'" +alias bstop="systemctl stop bodhi.service celery.service fm-consumer@config.service fm-consumer@printer.service" +alias blint="pre-commit run -a" +alias bmessages="journalctl -u fm-consumer@printer -f" + + +function bresetdb { + bstop; + psql -U postgres -h localhost -c "DROP DATABASE bodhi2"; + createdb -U postgres -h localhost bodhi2; + if [ ! -f "/tmp/bodhi2.dump.xz" ] ; then + curl -o /tmp/bodhi2.dump.xz https://infrastructure.fedoraproject.org/infra/db-dumps/bodhi2.dump.xz + fi + xzcat /tmp/bodhi2.dump.xz | psql -U postgres -h localhost bodhi2; + pushd /bodhi/bodhi-server; + # we call 'python3' explicitly to dodge some options in the + # shebang which break finding our bodhi modules + python3 /usr/bin/alembic upgrade head; + popd; + bstart; +} + + +function btest { + find /bodhi -name "*.pyc" -delete; + pushd /bodhi + blint || return $? + bdocs || return $? + for module in bodhi-messages bodhi-client bodhi-server; do + pushd $module + python3 -m pytest $@ tests || return $? + popd + done + diff-cover bodhi-*/coverage.xml --compare-branch=develop --fail-under=100 + popd +} + + +export BODHI_URL="http://localhost.localdomain:6543/" +export BODHI_OPENID_PROVIDER="http://localhost.localdomain:6546/openidc" +export PYTHONWARNINGS="once" +export BODHI_CI_ARCHIVE_PATH="/bodhi-ci-test_results/" + +cat /etc/motd +cd /bodhi diff --git a/devel/ansible-podman/containers/bodhi/bodhi-wait.py b/devel/ansible-podman/containers/bodhi/bodhi-wait.py new file mode 100755 index 0000000000..8825094deb --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/bodhi-wait.py @@ -0,0 +1,29 @@ +#!/usr/bin/python3 + +import time + +from bodhi.server.config import config +from sqlalchemy import engine_from_config +from sqlalchemy.exc import OperationalError + +config.load_config() +engine = engine_from_config(config) + + +# stolen from waiverdb, GPLv2+, thanks Dan Callaghan +def wait_for_db(): + poll_interval = 10 # seconds + while True: + try: + engine.connect() + except OperationalError as e: + print('Failed to connect to database: {}'.format(e)) + print(f'Sleeping for {poll_interval} seconds...') + time.sleep(poll_interval) + print('Retrying...') + else: + break + + +if __name__ == '__main__': + wait_for_db() diff --git a/devel/ansible-podman/containers/bodhi/bodhi.service b/devel/ansible-podman/containers/bodhi/bodhi.service new file mode 100644 index 0000000000..5f1979875f --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/bodhi.service @@ -0,0 +1,20 @@ +[Unit] +Description=bodhi +After=network-online.target +Wants=network-online.target + +[Service] +Environment=PYTHONWARNINGS=once +WorkingDirectory=/bodhi/bodhi-server +ExecStartPre=/usr/bin/poetry -C /bodhi/bodhi-messages install --only-root +ExecStartPre=/usr/bin/poetry -C /bodhi/bodhi-client install --only-root +ExecStartPre=/usr/bin/poetry -C /bodhi/bodhi-server install --only-root +ExecStartPre=/usr/local/bin/bodhi-wait.py +# we don't run alembic and pserve directly from /usr/bin as some +# options in their shebangs break finding our bodhi modules +ExecStartPre=python3 -m alembic -c alembic.ini upgrade head +ExecStart=python3 -m pyramid.scripts.pserve /etc/bodhi/production.ini --reload +TimeoutStartSec=600 + +[Install] +WantedBy=multi-user.target diff --git a/devel/ansible-podman/containers/bodhi/celery.service b/devel/ansible-podman/containers/bodhi/celery.service new file mode 100644 index 0000000000..1752890ed0 --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/celery.service @@ -0,0 +1,12 @@ +[Unit] +Description=celery +After=bodhi.service +Wants=bodhi.service + +[Service] +Environment=BODHI_CONFIG=/etc/bodhi/development.ini +WorkingDirectory=/bodhi/bodhi-server +ExecStart=/usr/bin/poetry run celery -A bodhi.server.tasks.app worker -l info -Q celery,has_koji_mount -B + +[Install] +WantedBy=multi-user.target diff --git a/devel/ansible-podman/containers/bodhi/config.toml b/devel/ansible-podman/containers/bodhi/config.toml new file mode 100644 index 0000000000..64a43871a1 --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/config.toml @@ -0,0 +1,51 @@ +amqp_url = "amqp://" +publish_exchange = "amq.topic" +callback = "bodhi.server.consumers:Consumer" + +[[bindings]] +queue = "bodhi_local_queue" +exchange = "amq.topic" +routing_keys = [ + "org.fedoraproject.*.bodhi.update.edit", + "org.fedoraproject.*.bodhi.update.request.testing", + "org.fedoraproject.*.waiverdb.waiver.new" +] + +[tls] +ca_cert = "/etc/pki/tls/certs/ca-bundle.crt" +keyfile = "/my/client/key.pem" +certfile = "/my/client/cert.pem" + +[client_properties] +app = "Bodhi dev local" + +[queues.bodhi_local_queue] +durable = true +auto_delete = false +exclusive = false +arguments = {} + +[qos] +prefetch_size = 0 +prefetch_count = 25 + +[log_config] +version = 1 +disable_existing_loggers = true + +[log_config.formatters.simple] +format = "[%(name)s %(levelname)s] %(message)s" + +[log_config.handlers.console] +class = "logging.StreamHandler" +formatter = "simple" +stream = "ext://sys.stdout" + +[log_config.loggers.fedora_messaging] +level = "INFO" +propagate = false +handlers = ["console"] + +[log_config.root] +level = "WARNING" +handlers = ["console"] diff --git a/devel/ansible-podman/containers/bodhi/development.ini.example b/devel/ansible-podman/containers/bodhi/development.ini.example new file mode 120000 index 0000000000..c8ffca38ae --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/development.ini.example @@ -0,0 +1 @@ +../../../development.ini.example \ No newline at end of file diff --git a/devel/ansible-podman/containers/bodhi/motd b/devel/ansible-podman/containers/bodhi/motd new file mode 100644 index 0000000000..73cb03d6e7 --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/motd @@ -0,0 +1,24 @@ + +Welcome to the Bodhi development environment! Here are some helpful commands: + +bdocs: Build Bodhi's documentation. +blint: Run a series of linter checks. +btest: Run Bodhi's test suite (includes blint and bdocs). +bmessages: Display the log of Bodhi's messages on the bus. +blog: View Bodhi's log. (Support all the systemctl options, such as -lf). +bresetdb: Drop and reimport the database. +brestart: Restart the Bodhi service. +bodhi-shell: Get a handy python shell initialized with Bodhi models. +bstart: Start the Bodhi service. +bstop: Stop the Bodhi service. + +The BODHI_URL and BODHI_OPENID_PROVIDER variables are set so the +the bodhi client will use the local development server and all +requests should work correctly. + +You can run `systemctl start fm-consumer@remote.service` to run a +callback consumer that listens to real messages on the production +bus from Koji and ResultsDB, though this dev env may not be able +to correctly process all messages. + +Happy hacking! diff --git a/devel/ansible-podman/containers/bodhi/printer.toml b/devel/ansible-podman/containers/bodhi/printer.toml new file mode 100644 index 0000000000..a26c55275f --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/printer.toml @@ -0,0 +1,44 @@ +amqp_url = "amqp://" +callback = "fedora_messaging.example:printer" + +[[bindings]] +queue = "bodhi_printer_queue" +exchange = "amq.topic" +routing_keys = [ + # this means 'take any bodhi message', so we print them all + "org.fedoraproject.*.bodhi.#", +] + +[client_properties] +app = "Bodhi dev printer" + +[queues.bodhi_printer_queue] +durable = true +auto_delete = false +exclusive = false +arguments = {} + +[qos] +prefetch_size = 0 +prefetch_count = 25 + +[log_config] +version = 1 +disable_existing_loggers = true + +[log_config.formatters.simple] +format = "[%(name)s %(levelname)s] %(message)s" + +[log_config.handlers.console] +class = "logging.StreamHandler" +formatter = "simple" +stream = "ext://sys.stdout" + +[log_config.loggers.fedora_messaging] +level = "INFO" +propagate = false +handlers = ["console"] + +[log_config.root] +level = "WARNING" +handlers = ["console"] diff --git a/devel/ansible-podman/containers/bodhi/remote.toml b/devel/ansible-podman/containers/bodhi/remote.toml new file mode 100644 index 0000000000..1f7798c216 --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/remote.toml @@ -0,0 +1,57 @@ +amqp_url = "amqps://fedora:@rabbitmq.fedoraproject.org/%2Fpublic_pubsub" +callback = "bodhi.server.consumers:Consumer" + +[exchanges."amq.topic"] +type = "topic" +durable = true +auto_delete = false +arguments = {} + +# Note the double brackets below. +# To add another binding, add another [[bindings]] section. +[[bindings]] +queue = "bodhi_remote_queue" +exchange = "amq.topic" +routing_keys = [ + "org.fedoraproject.*.buildsys.tag", + "org.fedoraproject.*.resultsdb.result.new" +] + +[tls] +ca_cert = "/etc/fedora-messaging/cacert.pem" +keyfile = "/etc/fedora-messaging/fedora-key.pem" +certfile = "/etc/fedora-messaging/fedora-cert.pem" + +[client_properties] +app = "Bodhi dev remote" + +[queues.bodhi_remote_queue] +durable = true +auto_delete = false +exclusive = false +arguments = {} + +[qos] +prefetch_size = 0 +prefetch_count = 25 + +[log_config] +version = 1 +disable_existing_loggers = true + +[log_config.formatters.simple] +format = "[%(name)s %(levelname)s] %(message)s" + +[log_config.handlers.console] +class = "logging.StreamHandler" +formatter = "simple" +stream = "ext://sys.stdout" + +[log_config.loggers.fedora_messaging] +level = "INFO" +propagate = false +handlers = ["console"] + +[log_config.root] +level = "WARNING" +handlers = ["console"] diff --git a/devel/ansible-podman/containers/bodhi/vimrc b/devel/ansible-podman/containers/bodhi/vimrc new file mode 100644 index 0000000000..ab0fc4607c --- /dev/null +++ b/devel/ansible-podman/containers/bodhi/vimrc @@ -0,0 +1,3 @@ +set expandtab +set tabstop=4 +set shiftwidth=4 diff --git a/devel/ansible-podman/playbook.yml b/devel/ansible-podman/playbook.yml new file mode 100644 index 0000000000..349e1b3c9d --- /dev/null +++ b/devel/ansible-podman/playbook.yml @@ -0,0 +1,5 @@ +--- +- hosts: localhost + become: false + roles: + - role: podman diff --git a/devel/ansible-podman/roles/podman/files/greenwave-settings.py b/devel/ansible-podman/roles/podman/files/greenwave-settings.py new file mode 120000 index 0000000000..b1330bde24 --- /dev/null +++ b/devel/ansible-podman/roles/podman/files/greenwave-settings.py @@ -0,0 +1 @@ +../../../../docker/settings/greenwave-settings.py \ No newline at end of file diff --git a/devel/ansible-podman/roles/podman/files/init-db.sh b/devel/ansible-podman/roles/podman/files/init-db.sh new file mode 100755 index 0000000000..ed83df47df --- /dev/null +++ b/devel/ansible-podman/roles/podman/files/init-db.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -e + +psql -U postgres -c "CREATE USER waiverdb CREATEDB;" +psql -U postgres -c "CREATE USER bodhi2 CREATEDB;" +psql -U postgres -c "CREATE DATABASE bodhi2;" +xzcat /docker-entrypoint-initdb.d/waiverdb.dump.xz | psql -U postgres +xzcat /docker-entrypoint-initdb.d/bodhi2.dump.xz | psql bodhi2 -U postgres +touch /tmp/.init-done diff --git a/devel/ansible-podman/roles/podman/files/run_waiverdb.sh b/devel/ansible-podman/roles/podman/files/run_waiverdb.sh new file mode 100755 index 0000000000..5b6dd30b86 --- /dev/null +++ b/devel/ansible-podman/roles/podman/files/run_waiverdb.sh @@ -0,0 +1,3 @@ +#!/bin/sh +. /venv/bin/activate +waiverdb wait-for-db && waiverdb db upgrade && gunicorn --bind 0.0.0.0:6544 --access-logfile=- --enable-stdio-inheritance waiverdb.wsgi:app diff --git a/devel/ansible-podman/roles/podman/files/waiverdb-settings.py b/devel/ansible-podman/roles/podman/files/waiverdb-settings.py new file mode 120000 index 0000000000..fe71516d20 --- /dev/null +++ b/devel/ansible-podman/roles/podman/files/waiverdb-settings.py @@ -0,0 +1 @@ +../../../../docker/settings/waiverdb-settings.py \ No newline at end of file diff --git a/devel/ansible-podman/roles/podman/tasks/clear_ipsilon_sessions.yml b/devel/ansible-podman/roles/podman/tasks/clear_ipsilon_sessions.yml new file mode 100644 index 0000000000..92748327a3 --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/clear_ipsilon_sessions.yml @@ -0,0 +1,4 @@ +- name: Clear all sessions in Ipsilon container + containers.podman.podman_container_exec: + name: bodhi-dev-ipsilon + command: "sh -c 'rm -f /var/lib/ipsilon/root/sessions/*'" diff --git a/devel/ansible-podman/roles/podman/tasks/main.yml b/devel/ansible-podman/roles/podman/tasks/main.yml new file mode 100644 index 0000000000..4681bdd7dd --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/main.yml @@ -0,0 +1,14 @@ +- include_tasks: prep.yml + when: "bodhi_dev_prep is defined or bodhi_dev_run is defined" + +- include_tasks: start.yml + when: "bodhi_dev_start is defined or bodhi_dev_run is defined" + +- include_tasks: stop.yml + when: "bodhi_dev_stop is defined" + +- include_tasks: remove.yml + when: "bodhi_dev_remove is defined" + +- include_tasks: clear_ipsilon_sessions.yml + when: "bodhi_dev_cis is defined" diff --git a/devel/ansible-podman/roles/podman/tasks/prep.yml b/devel/ansible-podman/roles/podman/tasks/prep.yml new file mode 100644 index 0000000000..565f1454d5 --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/prep.yml @@ -0,0 +1,65 @@ +- name: Create temp dir + ansible.builtin.file: + path: /tmp/bodhi-dev + state: directory + mode: '0755' + +- name: Get Bodhi database dump header + ansible.builtin.uri: + url: "https://infrastructure.fedoraproject.org/infra/db-dumps/bodhi2.dump.xz" + method: HEAD + register: "urlstat" + changed_when: false + +- name: Check stat of local Bodhi database dump file + ansible.builtin.stat: + path: "/tmp/bodhi-dev/bodhi2.dump.xz" + register: "filestat" + changed_when: false + +- name: Retrieve Bodhi database dump + ansible.builtin.get_url: + url: "https://infrastructure.fedoraproject.org/infra/db-dumps/bodhi2.dump.xz" + dest: "/tmp/bodhi-dev/bodhi2.dump.xz" + force: yes + # thanks, github @Constantin07 + # https://github.com/ansible/ansible/issues/30003#issuecomment-328855553 + when: "not filestat.stat.exists or urlstat.content_length|int != filestat.stat.size|int" + +- name: Retrieve waiverdb dump and greenwave policy + ansible.builtin.get_url: + url: "{{ item }}" + dest: "/tmp/bodhi-dev/{{ item | basename }}" + force: yes + loop: + - https://infrastructure.fedoraproject.org/infra/db-dumps/waiverdb.dump.xz + - https://pagure.io/fedora-infra/ansible/raw/main/f/roles/openshift-apps/greenwave/templates/fedora.yaml + +- name: Copy various required files into place + ansible.builtin.copy: + src: "{{ item }}" + dest: "/tmp/bodhi-dev/{{ item }}" + with_items: + - waiverdb-settings.py + - greenwave-settings.py + +- name: Adjust waiverdb hostname in greenwave settings for this environment + ansible.builtin.replace: + path: /tmp/bodhi-dev/greenwave-settings.py + regexp: 'waiverdb:6544' + replace: 'bodhi-dev-waiverdb:6544' + +- name: Adjust database hostname in waiverdb settings for this environment + ansible.builtin.replace: + path: /tmp/bodhi-dev/waiverdb-settings.py + regexp: 'wdb:5432' + replace: 'bodhi-dev-database:5432' + +- name: Copy various required executable files into place + ansible.builtin.copy: + src: "{{ item }}" + dest: "/tmp/bodhi-dev/{{ item }}" + mode: 0755 + with_items: + - init-db.sh + - run_waiverdb.sh diff --git a/devel/ansible-podman/roles/podman/tasks/remove.yml b/devel/ansible-podman/roles/podman/tasks/remove.yml new file mode 100644 index 0000000000..1abe276e1d --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/remove.yml @@ -0,0 +1,16 @@ +- name: "Remove all containers" + containers.podman.podman_container: + name: "{{ item }}" + state: absent + with_items: + - bodhi-dev-bodhi + - bodhi-dev-ipsilon + - bodhi-dev-rabbitmq + - bodhi-dev-greenwave + - bodhi-dev-waiverdb + - bodhi-dev-database + +- name: "Remove the pod" + containers.podman.podman_pod: + name: bodhi-dev + state: absent diff --git a/devel/ansible-podman/roles/podman/tasks/start.yml b/devel/ansible-podman/roles/podman/tasks/start.yml new file mode 100644 index 0000000000..6c6580dca6 --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/start.yml @@ -0,0 +1,96 @@ +- name: Create the pod + containers.podman.podman_pod: + name: bodhi-dev + state: started + publish: + - "6543:6543" + - "6544:6544" + - "6545:6545" + - "6546:6546" + - "15672:15672" + +- name: Run the database container + containers.podman.podman_container: + name: bodhi-dev-database + image: "docker.io/library/postgres:latest" + env: + POSTGRES_HOST_AUTH_METHOD: "trust" + state: started + expose: + - 5432 + pod: bodhi-dev + volume: + - /tmp/bodhi-dev/init-db.sh:/docker-entrypoint-initdb.d/init-db.sh:Z + - /tmp/bodhi-dev/waiverdb.dump.xz:/docker-entrypoint-initdb.d/waiverdb.dump.xz:Z + - /tmp/bodhi-dev/bodhi2.dump.xz:/docker-entrypoint-initdb.d/bodhi2.dump.xz:Z + healthcheck: "test -f /tmp/.init-done && pg_isready -h localhost -U waiverdb -d waiverdb" + +- name: Run the waiverdb container + containers.podman.podman_container: + name: bodhi-dev-waiverdb + image: "quay.io/factory2/waiverdb:latest" + state: started + pod: bodhi-dev + volume: + - /tmp/bodhi-dev/waiverdb-settings.py:/etc/waiverdb/settings.py:Z + - /tmp/bodhi-dev/run_waiverdb.sh:/usr/local/bin/run_waiverdb.sh:Z + healthcheck: "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:6544/healthcheck')\"" + entrypoint: "/usr/local/bin/run_waiverdb.sh" + +- name: Run the greenwave container + containers.podman.podman_container: + name: bodhi-dev-greenwave + image: "quay.io/factory2/greenwave:latest" + state: started + pod: bodhi-dev + volume: + - /tmp/bodhi-dev/greenwave-settings.py:/etc/greenwave/settings.py:Z + - /tmp/bodhi-dev/fedora.yaml:/etc/greenwave/policies/fedora.yaml:Z + healthcheck: "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:6545/api/v1.0/about')\"" + command: ["gunicorn", "--bind", "0.0.0.0:6545", "--access-logfile", "-", "--error-logfile", "-", "--enable-stdio-inheritance", "greenwave.wsgi:app"] + +- name: Run the RabbitMQ container + containers.podman.podman_container: + name: bodhi-dev-rabbitmq + image: "docker.io/library/rabbitmq:3-management" + state: started + pod: bodhi-dev + +- name: Build the Ipsilon container + containers.podman.podman_image: + name: bodhi-dev-ipsilon + # dockerfile expects build context to be top-level dir + path: "{{ playbook_dir }}/../.." + build: + file: "{{ playbook_dir }}/../ci/integration/ipsilon/Dockerfile" + # I've seen the redirect URI come out both ways, not sure why, + # so let's just allow both + extra_args: "--build-arg redirect=http://localhost.localdomain:6543/oidc/authorize,http://localhost:6543/oidc/authorize --build-arg hostname=localhost.localdomain:6546 --build-arg clienturi=http://localhost.localdomain:6543/ --build-arg listen=6546" + +- name: Run the Ipsilon container + containers.podman.podman_container: + name: bodhi-dev-ipsilon + image: bodhi-dev-ipsilon + state: started + pod: bodhi-dev + +- name: Build the Bodhi container + containers.podman.podman_image: + name: bodhi-dev-bodhi + # we use devel/ as the build context so we're above development.ini.example + path: "{{ playbook_dir }}/.." + build: + file: "{{ playbook_dir }}/containers/bodhi/Containerfile" + +- name: Run the Bodhi container + containers.podman.podman_container: + name: bodhi-dev-bodhi + image: bodhi-dev-bodhi + state: started + pod: bodhi-dev + cap_add: + - AUDIT_WRITE + security_opt: + - label:disable + volume: + - "{{ playbook_dir }}/../..:/bodhi" diff --git a/devel/ansible-podman/roles/podman/tasks/stop.yml b/devel/ansible-podman/roles/podman/tasks/stop.yml new file mode 100644 index 0000000000..e6026f8d0e --- /dev/null +++ b/devel/ansible-podman/roles/podman/tasks/stop.yml @@ -0,0 +1,16 @@ +- name: Stop all containers + containers.podman.podman_container: + name: "{{ item }}" + state: stopped + with_items: + - bodhi-dev-bodhi + - bodhi-dev-ipsilon + - bodhi-dev-rabbitmq + - bodhi-dev-greenwave + - bodhi-dev-waiverdb + - bodhi-dev-database + +- name: Stop the pod + containers.podman.podman_pod: + name: bodhi-dev + state: stopped diff --git a/devel/ansible/playbook.yml b/devel/ansible/playbook.yml index 15d8e33f03..61d9c6daa7 100644 --- a/devel/ansible/playbook.yml +++ b/devel/ansible/playbook.yml @@ -13,7 +13,7 @@ - name: Give reason for failure fail: msg: Provisioning bodhi requires the base tinystage setup to be running. - when: ping_response.rc != 0 + when: "ping_response.rc != 0" roles: - rabbitmq - bodhi diff --git a/devel/ci/integration/ipsilon/Dockerfile b/devel/ci/integration/ipsilon/Dockerfile index 859e3d443d..c686255f8d 100644 --- a/devel/ci/integration/ipsilon/Dockerfile +++ b/devel/ci/integration/ipsilon/Dockerfile @@ -1,19 +1,32 @@ FROM quay.io/fedora/fedora:38 -RUN curl -o /etc/yum.repos.d/infra-tags.repo https://pagure.io/fedora-infra/ansible/raw/main/f/files/common/fedora-infra-tags.repo -RUN dnf install -y ipsilon ipsilon-openid ipsilon-openidc patch git -RUN git clone https://pagure.io/fedora-infra/ipsilon-fedora.git /opt/ipsilon-fedora -RUN cd /opt/ipsilon-fedora && ./install.sh -RUN ipsilon-server-install --root-instance --secure no --testauth yes --openid yes --openidc yes --hostname id.dev.fedoraproject.org --openid-extensions "insecureAPI,Teams,CLAs,Simple Registration" +# should be specified as hostname:port to listen on a non-standard port +ARG hostname=id.dev.fedoraproject.org +# can be a comma-separated list +ARG redirect=http://bodhi.ci:8080/oidc/authorize +ARG clienturi=http://bodhi.ci:8080/oidc/ +# to listen on a non-standard port, specify this too, it's dumb but you can't do +# conditionals or string parsing in dockerfiles...only used for EXPOSE anyway +ARG listen=80 +RUN curl -o /etc/yum.repos.d/infra-tags.repo https://pagure.io/fedora-infra/ansible/raw/main/f/files/common/fedora-infra-tags.repo \ + && curl -o /etc/yum.repos.d/ipsilon-bodhi.repo https://adamwill.fedorapeople.org/ipsilon-bodhi-repo/ipsilon-bodhi.repo + +RUN dnf install -y ipsilon ipsilon-openid ipsilon-openidc patch git sed systemd procps-ng elinks +RUN git clone https://pagure.io/fedora-infra/ipsilon-fedora.git /opt/ipsilon-fedora \ + && cd /opt/ipsilon-fedora \ + && ./install.sh +RUN ipsilon-server-install --root-instance --secure no --testauth yes --testauth-groups "fedora-contributors,packager" --openid yes --openidc yes --hostname $hostname --openid-extensions "insecureAPI,Teams,CLAs,Simple Registration" --openidc-extensions "fedora-account,waiverdb" --openidc-default-attribute-mapping '[["*", "*"], ["_groups", "groups"], [["_extras", "cla"], "cla"], ["fullname", "name"], ["_username", "nickname"], ["_username", "preferred_username"], ["fasIRCNick", "ircnick"], ["fasLocale", "locale"], ["fasTimeZone", "zoneinfo"], ["fasTimeZone", "timezone"], ["fasWebsiteURL", "website"], ["fasGPGKeyId", "gpg_keyid"], ["ipaSshPubKey", "ssh_key"], ["fasIsPrivate", "privacy"], ["fullname", "human_name"]]' RUN sscg \ --ca-file /etc/pki/tls/certs/localhost-ca.crt \ --cert-file /etc/pki/tls/certs/localhost.crt \ --cert-key-file /etc/pki/tls/private/localhost.key \ - --hostname id.dev.fedoraproject.org \ + --hostname $hostname \ --subject-alt-name ipsilon.ci \ --subject-alt-name ipsilon -COPY devel/ci/integration/ipsilon/setup-bodhi.py /usr/local/bin/setup-bodhi.py +COPY devel/ci/integration/ipsilon/setup-bodhi.py.tmpl /tmp/setup-bodhi.py.tmpl +RUN sed -e "s|##REDIRECT##|$redirect|g" -e "s,##CLIENTURI##,$clienturi,g" /tmp/setup-bodhi.py.tmpl > /usr/local/bin/setup-bodhi.py && chmod ugo+x /usr/local/bin/setup-bodhi.py RUN python3 /usr/local/bin/setup-bodhi.py +RUN systemctl enable httpd.service COPY devel/ci/integration/ipsilon/start.sh /usr/local/bin/start.sh -RUN dnf install -y procps-ng elinks -EXPOSE 80 443 +EXPOSE 80 443 $listen +RUN systemctl enable httpd.service CMD /usr/local/bin/start.sh diff --git a/devel/ci/integration/ipsilon/setup-bodhi.py b/devel/ci/integration/ipsilon/setup-bodhi.py.tmpl similarity index 90% rename from devel/ci/integration/ipsilon/setup-bodhi.py rename to devel/ci/integration/ipsilon/setup-bodhi.py.tmpl index 5fa3a3f092..b2ec06bba1 100644 --- a/devel/ci/integration/ipsilon/setup-bodhi.py +++ b/devel/ci/integration/ipsilon/setup-bodhi.py.tmpl @@ -17,7 +17,9 @@ CLIENTS = [ { "client_id": "bodhi-client", + "client_name": "Bodhi CI / dev client", "client_secret": "", + "client_uri": "", "redirect_uris": ["urn:ietf:wg:oauth:2.0:oob"], "application_type": "native", "response_types": ["code"], @@ -26,11 +28,15 @@ "ipsilon_internal": {"trusted": True, "client_id": "bodhi-client", "type": "static"}, "client_secret_expires_at": 0, "token_endpoint_auth_method": "none", + "subject_type": "public", + "sector_identifier_uri": "", }, { "client_id": "integration-tests", + "client_name": "Bodhi CI / dev", + "client_uri": "##CLIENTURI##", "client_secret": "integration-tests", - "redirect_uris": ["http://bodhi.ci:8080/oidc/authorize"], + "redirect_uris": "##REDIRECT##".split(","), "application_type": "web", "response_types": ["code"], "grant_types": ["authorization_code"], @@ -38,6 +44,8 @@ "ipsilon_internal": {"trusted": True, "client_id": "integration-tests", "type": "static"}, "client_secret_expires_at": 0, "token_endpoint_auth_method": "client_secret_post", + "subject_type": "public", + "sector_identifier_uri": "", }, ] @@ -66,7 +74,6 @@ } } - conn = sqlite3.connect(CLIENT_STATIC_DB) with conn: # Register the bodhi apps (client & server) diff --git a/devel/docker/settings/restore_waiverdb.sh b/devel/docker/settings/restore_waiverdb.sh old mode 100644 new mode 100755 diff --git a/docs/developer/bcd.rst b/docs/developer/bcd.rst new file mode 100644 index 0000000000..41d78d033e --- /dev/null +++ b/docs/developer/bcd.rst @@ -0,0 +1,98 @@ +================================================= +BCD - the Bodhi Container Development environment +================================================= + +BCD is an Ansible-orchestrated, Podman-powered container-based development environment for Bodhi. +It lets you quickly and easily test your changes in a live Bodhi environment, without any external +dependencies, root privilege requirements, or virtual machines. It's also convenient for running +the unit tests. + +To get started on a Fedora or Enterprise Linux-ish system, do:: + + $ sudo dnf install ansible-core ansible-collection-containers-podman podman + $ ./bcd run + +Your newly provisioned bodhi development instance is now available at http://localhost.localdomain:6543/ . + +The AMQP message broker web interface is available at http://localhost:15672/. The default username +is ``guest`` and the password is ``guest``. + +The Waiverdb web interface is available at http://localhost:6544/ , and the Greenwave web interface +is available at http://localhost:6545/ . + +The Ipsilon identity service web interface is available at http://localhost:6546/ (though there +isn't much reason to use it directly in this environment). + +Other commands +^^^^^^^^^^^^^^ + +Other command commands are ``./bcd stop`` to stop all containers, ``./bcd remove`` to remove all +containers, ``bcd logs (container)`` to view the logs for a container, ``bcd shell (container)`` +to shell into a container (the Bodhi container by default), and ``./bcd cis`` to clear Ipsilon's +session cache. This is necessary to log in to Bodhi as a different user - first log out, then run +``./bcd cis``, then log in again. If you don't clear the session cache Ipsilon will just keep +logging you in as the same user. Run ``./bcd -h`` or ``./bcd (subcommand) -h`` for more help. + +Containers +^^^^^^^^^^ + +Behind the scenes, BCD uses an Ansible playbook that controls a pod of Podman containers. The +pod is called ``bodhi-dev``. The full names of the containers are ``bodhi-dev-database``, +``bodhi-dev-waiverdb``, ``bodhi-dev-greenwave``, ``bodhi-dev-rabbitmq``, ``bodhi-dev-ipsilon`` and +``bodhi-dev-bodhi``. BCD commands which take a container name use shortened names with 'bodhi-dev-' +omitted, for convenience. You can interact with the pod and the containers directly using normal +podman commands if you wish - to start and stop individual containers, for instance. + +The 'database' container runs postgresql and is initialized with a dump of real data for waiverdb +and Bodhi, both of which connect to it. The 'waiverdb' and 'greenwave' containers run those services +respectively, which are used by Bodhi for retrieving test results and test waivers and deciding +on the gating status of updates. Note that the 'greenwave' container is configured to connect to +the real, production ResultsDB and forward results from it, so the results shown and the gating +status calculated will change to reflect the real-world state. The 'rabbitmq' container runs an +instance of the RabbitMQ message broker which Bodhi will publish messages to and consume messages +from. The 'ipsilon' container runs the authentication service (see below for details). And, of +course, the 'bodhi' container runs Bodhi itself. It has your source tree mapped as a volume, so +changes you make to your source tree are immediately reflected in the development environment. +The server will automatically reload when any Python source file is changed. + +The Bodhi container uses systemd, so you can shell into it and stop or restart the bodhi service +or any of the ancillary services it runs, if you need to. + +Authentication +^^^^^^^^^^^^^^ + +The BCD environment uses an instance of the Ipsilon authentication service (as also used for +Fedora authentication in the real world). This instance is configured in testauth mode, which means +you can log in as any user at all with the password 'ipsilon'. If you log in as a user that does +not exist in the Bodhi database, it will be added. + +Note that the group memberships and email addresses for real users are not the same as in the real +world. This Ipsilon instance is configured by default to say that all users are members of the +groups "fedora-contributors" and "packagers" and also a group of the same name as their username, +and have the email address "username@example.com". These values will be changed in the Bodhi +database on login. + +There is a special mechanism for testing different group memberships. You can login with a username +like 'someuser:groups=somegroup1,somegroup2' to log in as user 'someuser' but with your groups +reported as 'somegroup1' and 'somegroup2' (you will *not* be reported as a member of 'packagers' +or 'fedora-contributors' in this case). You can also take advantage of the 'user is a member of +the group with the same name' mechanism, e.g. by logging in as 'provenpackager' to be reported as +a member of the 'provenpackager' group. + +As mentioned above, switching users can be a bit tricky, as Ipsilon likes to cache the session +(this is sort of what it's supposed to do, after all). To switch users, log out, run ``./bcd cis``, +then log in again. If things get really messy you may need to stop and remove the bodhi and ipsilon +containers to get back to a clean state. + +Quick tips about the BCD environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can shell into the running Bodhi container like this:: + + # Make sure your bodhi checkout is your shell's cwd + $ ./bcd shell + +Note the container must be running, or this will fail. Once you are inside the development +environment, there are a helpful set of commands in your ``.bashrc`` that will be printed to the +screen via the ``/etc/motd`` file. These work exactly as documented in :ref:`the Vagrant doc `, +except that ``bci`` is not available in this environment (as we can't nest containers like that). diff --git a/docs/developer/index.rst b/docs/developer/index.rst index f4fc642992..1761fcdca0 100644 --- a/docs/developer/index.rst +++ b/docs/developer/index.rst @@ -167,18 +167,13 @@ that runs ``sudo devel/ci/bodhi-ci`` for you. Create a Bodhi development environment ====================================== -There are two ways to bootstrap a Bodhi development environment. You can use Vagrant, or you can use -virtualenv on an existing host. `Vagrant`_ allows contributors to get quickly up and running with a -Bodhi development environment by automatically configuring a virtual machine. `Virtualenv`_ is -a more manual option for building a development environment on an existing system. If you aren't -sure which development environment you would like to use, Vagrant is recommended as it get you a -working system more quickly and with less effort. If you would like to use Vagrant, see the -:doc:`Bodhi Vagrant Guide `. If you would like to use Virtualenv, see the -:doc:`Bodhi Virtualenv Guide `. +You can use Bodhi's "BCD" container-based development environment, or use Vagrant to create a VM- +based development environment. Using a local Python virtualenv is no longer supported. See the +:doc:`BCD Guide ` for instructions on using BCD, or the :doc:`Bodhi Vagrant Guide ` +for instructions on using Vagrant. -If you use Vagrant, you can configure Visual Studio Code to run unit-tests inside with :doc:`Bodhi Vagrant - VS Code Guide `. +You can configure Visual Studio Code to run unit-tests inside with :doc:`Bodhi Vagrant - VS Code Guide `. .. _docs/user/release_notes.rst: https://github.com/fedora-infra/bodhi/blob/develop/docs/user/release_notes.rst#release-notes .. _type hints: https://docs.python.org/3/library/typing.html .. _Vagrant: https://www.vagrantup.com -.. _Virtualenv: https://virtualenv.pypa.io/en/stable/ diff --git a/docs/developer/vagrant.rst b/docs/developer/vagrant.rst index b0e976e193..953bb074b8 100644 --- a/docs/developer/vagrant.rst +++ b/docs/developer/vagrant.rst @@ -52,6 +52,7 @@ edit the ``fas_username`` variable in the Vagrantfile and re-provision the VM. B not be a copy of your real fas account, it will just have the same username with the default password ``password`` and fake complementary data. +.. _vagrant-tips: Quick tips about the Bodhi Vagrant environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -114,3 +115,4 @@ the host:: If you wish to use a custom ``Vagrantfile``, you can set the environment variable ``VAGRANT_VAGRANTFILE`` as a path to a script. + diff --git a/docs/developer/virtualenv.rst b/docs/developer/virtualenv.rst deleted file mode 100644 index e57850470e..0000000000 --- a/docs/developer/virtualenv.rst +++ /dev/null @@ -1,146 +0,0 @@ -========== -Virtualenv -========== - -Dependencies -^^^^^^^^^^^^ -``sudo dnf install libffi-devel postgresql-devel openssl-devel koji pcaro-hermit-fonts freetype-devel libjpeg-turbo-devel zeromq-devel git gcc redhat-rpm-config fedora-cert python2-dnf yum`` - -Setup virtualenvwrapper -^^^^^^^^^^^^^^^^^^^^^^^ -``sudo dnf -y install python-virtualenvwrapper python-createrepo_c createrepo_c`` - -Add the following to your `~/.bashrc`:: - - export WORKON_HOME=$HOME/.virtualenvs - source /usr/bin/virtualenvwrapper.sh - -Set PYTHONPATH -^^^^^^^^^^^^^^ - -Add the following to your `~/.bashrc` - -``export PYTHONPATH=$PYTHONPATH:$HOME/.virtualenvs`` - -Then on the terminal :: - - source ~/.bashrc - -Clone the source -^^^^^^^^^^^^^^^^ -:: - - git clone https://github.com/fedora-infra/bodhi.git - cd bodhi - -Bootstrap the virtualenv -^^^^^^^^^^^^^^^^^^^^^^^^ -:: - - ./bootstrap.py - workon bodhi-python2.7 - -Setting up -^^^^^^^^^^ -``python setup.py develop`` - -``pip install psycopg2 pyramid_debugtoolbar`` - -Create the `development.ini `_ file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Copy ``devel/development.ini.example`` to ``development.ini``: -:: - - cp devel/development.ini.example development.ini - -Run the test suite -^^^^^^^^^^^^^^^^^^ -``py.test`` - -Import the bodhi2 database -^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: - - curl -O https://infrastructure.fedoraproject.org/infra/db-dumps/bodhi2.dump.xz - sudo -u postgres createdb bodhi2 - sudo -u postgres psql -c "create role bodhi2;" - xzcat bodhi2.dump.xz | sudo -u postgres psql bodhi2 - -.. note:: If you do not have a PostgreSQL server running, please see the - instructions at the bottom of the file. - - -Adjust database configuration in `development.ini `_ file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Set the configuration key -`sqlalchemy.url `_ -to point to the postgresql database. Something like: -:: - - sqlalchemy.url = postgresql://postgres:anypasswordworkslocally@localhost/bodhi2 # gitleaks:allow - - -Upgrade the database -^^^^^^^^^^^^^^^^^^^^ -``alembic upgrade head`` - - -Run the web app -^^^^^^^^^^^^^^^ -``pserve development.ini --reload`` - - - -Setup the postgresql server -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -1. Install postgresql -~~~~~~~~~~~~~~~~~~~~~ -:: - - dnf install postgresql-server - - -2. Setup the Database -~~~~~~~~~~~~~~~~~~~~~ - -As a privileged user on a Fedora system run the following: -:: - - sudo postgresql-setup initdb - - -3. Adjust PostgreSQL Connection Settings -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As a privileged user on a Fedora system modify the pg_hba.conf file: -:: - - vi /var/lib/pgsql/data/pg_hba.conf - -Then adjust the content at the bottom of the file to match the following. - -:: - - # TYPE DATABASE USER ADDRESS METHOD - - # "local" is for Unix domain socket connections only - local all all peer - # IPv4 local connections are *trusted*, any password will work. - host all all 127.0.0.1/32 trust - # IPv6 local connections are *trusted*, any password will work. - host all all ::1/128 trust - -If you need to make other modifications to postgresql please make them now. - -4. Start PostgreSQL -~~~~~~~~~~~~~~~~~~~ - -As a privileged user on a Fedora system run the following: -:: - - sudo systemctl start postgresql.service - - diff --git a/docs/index.rst b/docs/index.rst index 34f47443ba..feb4ac7599 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -61,9 +61,9 @@ Contributor Guide developer/index developer/releases + developer/bcd developer/vagrant developer/vagrant_vscode - developer/virtualenv developer/models