From e50395381e564cfb65b2a35e8f230543bf2cb6b7 Mon Sep 17 00:00:00 2001 From: Christian Glombek Date: Thu, 10 Dec 2020 14:21:23 +0100 Subject: [PATCH] Update OKD proposal --- enhancements/okd-on-fedora-coreos.md | 134 +++++++-------------------- 1 file changed, 32 insertions(+), 102 deletions(-) diff --git a/enhancements/okd-on-fedora-coreos.md b/enhancements/okd-on-fedora-coreos.md index 899e0c63fcc..9bd831f5e6b 100644 --- a/enhancements/okd-on-fedora-coreos.md +++ b/enhancements/okd-on-fedora-coreos.md @@ -15,7 +15,7 @@ reviewers: - "@cgwalters" approvers: creation-date: 2019-10-21 -last-updated: 2020-05-18 +last-updated: 2020-12-17 status: provisional|implementable|implemented|deferred|rejected|withdrawn|replaced see-also: replaces: @@ -32,128 +32,58 @@ superseded-by: - [ ] Graduation criteria for dev preview, tech preview, GA - [ ] User-facing documentation is created in [openshift/docs] -## Open Questions [optional] - - 1. When is OCP migrating to Ignition config spec v3? - - Please see the Graduation Criteria section for more context on both open questions. - ## Summary OKD is the OpenShift community distribution of Kubernetes. -As part of the version 4 effort, the OKD working group has agreed to target Fedora CoreOS (FCoS) as the primary base operating system for OKD. +As part of the version 4 effort, the OKD working group has decided to target Fedora CoreOS (FCOS) as the primary base operating system for OKD control plane and worker nodes. -This document contains a proposal to create all OKD artifacts in OpenShift's Prow instance from the canonical OpenShift master and release branches alongside OCP artifacts. - -Eventually, container images for all components will be shared between OKD and OCP, with the exception of the machine-os-content container image which will be based on Fedora CoreOS content, and any container images that are based on RHEL images in OCP. These will be rebuilt on either a UBI, Fedora or CentOS base for OKD. - -A lot of this is already done. What's left is the un-forking of Machine Config Operator and (optionally) Installer repository and building variant-agnostic artifacts of them, as well as changing the Samples Operator to not go degraded on OKD clusters. +This document contains a proposal to add missing FCOS support to key components, +create OKD artifacts in OpenShift's Prow instance from the canonical OpenShift master and release branches alongside OCP CI artifacts, +and regularily create OKD release payloads from there. ## Motivation -Fedora CoreOS includes Ignition v2.x to use for the firstboot configuration of newly provisioned machines with Ignition configuration specification 3.x while RHEL CoreOS currently includes and uses Ignition v0.x supporting config spec 2.x. - -Ignition configuration is a declarative format representing the desired on-disk state of the machine. - -The installer uses Terraform to provision the bootstrap machine with a config encoded in a spec understood by the host system. After this initial provisioning of the bootstrap host, a Machine Config Server (MCS) is started on the bootsrap host to serve Ignition config to all nodes joining for the initial installation. Subsequently, the bootstrap host is destroyed and another instance of the MCS on the cluster continues to serve Ignition for additional nodes that want to join the cluster later. - -Only a limited subset of the Ignition config fields is supported by the MCS (i.e. config that describes the on-disk state like files and systemd units). - -In OCP this on-disk state representation used within the Installer and the MCS is currently encoded in Ignition spec v2.2 config types, while v3.0 config types are used in OKD. To account for the different spec types, OKD's Installer binary as well as OKD's Machine Config Operator (MCO) container image are currently built from forked codebases. - -Long-term maintainance of those forks is not wanted. The MCO fork represents an OKD 4 GA release blocker. The installer fork, while not blocking GA, should be dissolved after OCP migrates to Ignition config spec v3. +The OKD Working Group wants to deliver a community variant of OpenShift on top of Fedora CoreOS +that is released off of the latest OpenShift stable release branch in a rolling manner, +and provides the ability to update. An update release payload for OKD encapsulates a new release of the underlying operating system +as well as new releases of the cluster components. ### Goals -The goal is to create all OKD artifacts in OpenShift's Prow instance from the canonical OpenShift master and release branches. +The goals are to achieve rolling OKD 4 Releases, based on Fedora CoreOS and the current OpenShift codebase, +as well as facilitation of community participation, adoption, feedback and knowledge sharing. ### Non-Goals -The migration of existing OCP clusters to Ignition Config Spec v3 is not part of this proposal, however the functionality proposed here will facilitate that in the future. +This enhancement does not concern itself with the releases of optional operators for OKD via OperatorHub. ## Proposal -There are four parts to this proposal: Changes concerning Prow, MCO, Samples Operator and Installer. - -### 1: Setup Prow Jobs -The definitions of OKD build jobs for machine-os-content containers will be added to the the `openshift/release` repository to live side by side with the OCP job defintions. - -This is mostly done. - -### 2: Machine Config Operator Changes - -These changes are requirements for OKD 4 GA. - -#### Pulling Machine Config Daemon from release payload - -Fedora CoreOS images do not contain the Machine Config Daemon (MCD). -In OKD, the MCD (which contains the `pivot` command) is pulled onto the nodes by extracting it from the Machine Config Operator image. Then it can proceed to pivot into the current OS version specified in the machine config. - -The proposal is to align OCP with this, so the MCD is always pulled from image and is not needed in the RHEL CoreOS image compose. - -One benefit of this is that version drift between MCD from the boot image and from the current machine-os-content is ruled out, as the MCD will always come from the MCO image from the current payload. - -Note: MCO images are currently UBI7 based. For RHEL CoreOS it should be preferred to pull an MCD built inside a UBI8 container. - -#### Dual Ignition config spec v2/v3 support in Machine Config Operator - -The proposal is to add dual Ignition support to the MCO so that OKD does not require a forked version and can instead be built from the canonical branches in the machine-config-operator repository. - -In order to achive this, the following will be done: - -Firstly, the ability to parse and translate v3.x configs to v2.2 will be added. This way the MCO will be able to process v3.x config that is either provided by the installer during install, or by the cluster admin at a later point. The rendered machine config representing the current on-disk state internally will remain spec v2.2 for now. - -Secondly, the Machine Config Server will gain the ability to inspect the User Agent HTTP header of the incoming config request to serve a spec version supported by the requesting system. For this, the canonical rendered machine config will be translated up to spec v3 if the User Agent shows an Ignition version larger than 2.0.0. - -This is implemented by this PR: https://github.com/openshift/machine-config-operator/pull/1703 - -The last commit in that PR already shows that the spec used to store on-disk state internally (i.e. the rendered machine config) could already be switched over to v3 without any breaking changes in external APIs. This however is not required to enable using the same MCO images for OKD and OCP. - -### 3: Sample Operators Changes - -#### Mark RHEL based operator images as optional - -The samples operator degrades on OKD due to the fact that not all images are pullable without a valid pull secret. The behaviour should change so it does not degrade on OKD clusters for users that have not specified a valid pull secret. - -Alternatively, all RHEL based images this concers could be rebuilt on a CentOS or Fedora base, and a mechanism to pull them in case of no valid pull secret could be implemented in `cluster-samples-operator`. - -### 4: Installer Changes - -These changes are optional for the immediate goal of releasing OKD 4 GA. They should however be strongly considered in order to facilitate dropping the OKD fork of installer rather sooner than later. - -#### Add fcos.json - -Support for an `fcos.json` to exist alongside `rhcos.json` for OS image references. - -This could be enabled either via a build-time flag to produce seperate OKD installer artifacts, or an `fcos` or `okd` run-time flag to pass to the installer for it to provision Fedora CoreOS instead of RHEL CoreOS hosts (in that case it would default to contents of `rhcos.json`). - -#### Add option to disable pull secret to Installer - -OKD installs do not require a pull secret, so make it optional. - -This could be enabled either via a build-time flag to produce seperate OKD installer artifacts, or a run-time flag to signal to the installer to not require a pull secret. - -#### Switching Installer to Ignition spec v3 - -The MachineConfig API vendored in the Installer has to be updated to the current version that uses a RawExtension to store the ignition config. - -To facilitate the migration, all functions requiring Ignition types will be abstracted into one package. This is implemented by this PR: https://github.com/openshift/installer/pull/3248 - -With the MCO's ability to process both spec v2 and v3 configs, switching the installer over to spec v3 is technically possible. - -It is imperative that the timing of this change is aligned with RHEL CoreOS images switching to an Ignition v2.x binary that supports spec v3. - -Once this migration is complete, it will be possible for OKD to use an installer built off of the same sources. +This is a proposal to: +- Build OKD artifacts in OpenShift's Prow instance from the canonical OpenShift master branch and at least the latest release branch for continuous testing. +- Release OKD payloads built in and promoted from Prow from the latest release branch regularly, i.e. every 2 weeks. +- Support Fedora CoreOS as base OS in key OpenShift componentes (`machine-config-operator` and `installer`) so OKD releases can be made without requiring forked codebases. +- Provide easy-to-access ways for community members to participate: + - by providing a repository with bug report templates + - by encouraging and directing community contributors to send patches and bug fixes directly to the upstream OpenShift codebase + - create documentation and share knowledge about OKD ### Graduation Criteria -Specifically, success of this proposal will be measured by the following: -- Fedora RPM based `machine-os-content` containers are built continuously, with promotion gating for each minor version release stream. -- `machine-config-operator` sources do not require a fork to run in OKD. -- All container images that are part of operators shipped by `cluster-samples-operator` are continuously available as either Fedora or UBI-based container images. +- [x] [OKD repository](https://github.com/openshift/okd/) exists for community issue and bug triage, and development tracking. +- [x] The UBI-based container images for all components in the release payload are shared between OKD and OCP for testing in Prow (with a few exceptions, see below) +- [x] OKD build and release jobs are added to `openshift/release` +- [x] Fedora RPM- and FCOS ostree-based `machine-os-content` container images are built continuously from the [`openshift/okd-machine-os`](https://github.com/openshift/okd-machine-os/) repository, with promotion gating for each minor version release stream. +- [x] Mirror OKD release payloads from the internal registry to quay.io +- [x] Support and end-to-end CI testing is added for running `machine-config-operator`, and the `machine-config-daemon` on FCOS-based OKD clusters. +- [x] `machine-config-operator` sources do not require a fork to be built for OKD +- [x] Support and end-to-end CI testing is added for building OKD installer binaries +- [ ] `openshift-install` sources do not require a fork to be built for OKD: ([Pull Request](https://github.com/openshift/installer/pull/4453)) +- [x] All container images that are part of `cluster-samples` and `marketplace` operators are continuously available as either Fedora, CentOS or UBI-based container images. +- [ ] All container images that are part of `baremetal-operator` are continuously available as either Fedora, CentOS or UBI-based container images. ([okd#197](https://github.com/openshift/okd/issues/197), [ironic-image#123](https://github.com/openshift/ironic-image/pull/123), [ironic-ipa-downloader#56](https://github.com/openshift/ironic-image/pull/123)) ## Infrastructure Needed [optional] -- Prow CI compute +- Limited Prow CI compute on all platforms for end-to-end testing - Storage and/or registries for CI artifacts (RPMs, machine-os-content containers, and operator containers) \ No newline at end of file