Post-modular CLI usage and strategy

[janmedrek]

Created on {2023-12-29} by Jan Medrek (@janmedrek).

Decision log

Name	Description
Title	Separation of command-line functionalities
Due date	2024-01-26
Status	Accepted on 2024-01-29
Decision type	Choice
Affected decisions	None

Context

With the Modularization came changes to the Kyma ecosystem - it is no longer deployed as a single "monolithic" set of applications in a specifically prepared cluster. The requirements were loosened and interdependencies between components were removed. This leads to the (desired) situation where Kyma is just a set of loosely related components that coexist in almost any Kubernetes cluster.

With those changes, the requirements and needs for the proper tooling also shifted. A specialized set of commands that aim to deploy an environment fulfilling old Kyma requirements, and then install multiple related charts on top of that is no longer needed. Since that was the main responsibility of the Kyma CLI the strategy needs to be revisited.

The current set of commands in the CLI can be divided into two categories: end-user tools for interacting with the Kyma cluster and the ones that are used during the development of the modules or in the CI solutions.

The relevant (deprecated, outdated, help, and version omitted) command structure is:

kyma (end-user)
  dashboard
  function
    apply
    init
    run
    sync
  module
    add
    remove
    list

kyma (dev-ci)
  provision
    k3d
    gardener
  module
    create
    sign
    verify

This makes a clear division, let's tackle the end-user part first:

• The kyma dashboard command deploys and opens up a Busola dashboard in the cluster, the need for this command is set in question since the introduction of the new community modules approach and dashboard.
• The function section is a subset of commands strictly tied to one of the Kyma modules, Serverless in this case. The decision needed here is the strategy on the shipment channel for tooling brought by Kyma modules.
• The module section is strictly tied to interacting with Kyma CR, which can be easily replaced by K8S commands or visual CR editing via some kind of K8S dashboard.

Then, there is the dev-ci section:

• The kyma provision ... commands are used both in the teams' pipelines and during the development cycle of both Kyma Modules and the Modularization components
• module commands are used mainly by the Module Submission Pipeline and to some extent by the infrastructure/security setup of the Control Plane clusters/resources

Decision

Taking all of that into account, there are a few possible options for further development. Two decisions need to be taken - one for the user-facing CLI for modular Kyma and the other related to the toolkit used during development and pipeline execution.

For the end-user part, the options are simple:
A) Drop the end-user CLI completely and make Modules provide separate tooling for their features. This means that if the module wants to provide some functionalities via command line tools a separate binary is needed. Such a solution brings multiple tools that need to be used to interact with a single Kyma cluster containing various modules.
B) Have a dedicated CLI with functionalities related to the Kyma modules only. This makes it easier for the user to interact with the cluster, but the maintenance effort of a single cross-team repository is high and stands in opposition to what we wanted to achieve with modularisation (separation of components).

For the dev-toolkit part:
A) Divide the functionalities across different repositories and have them maintained by different teams. This brings multiple sources of truth and a de-centralised approach to such tooling.
B) Create a dedicated dev-toolkit or cc-utils repository in the Kyma organization and contain the scripts/commands for the existing logic there.

As for now, I am biased towards dropping the end-user CLI completely and just keeping the set of tools that are used by Kyma dev teams (which translates to the A) and B) respectively). As I've already written in the Context part - the requirements changed and I see no reason to maintain end-user-facing tooling (unless it is extensively used toward the Serverless component). For the second decision - I have no strong opinions, both results seem very similar to me - it is a rather cosmetic change but I lean towards having a single source of common CI-related commands (provisioning clusters, etc).

Summary

After the discussion, it is clear that business-oriented Kyma CLI is needed for the end-users. It should cover functionalities such as:

• Module add/delete/listing commands as pointed in this comment
• Proper storage setup simplified (PVC, emptyDir, ...)
• BTP integration and credentials synchronization
• Generating API Proxy from swagger definitions

When it comes to the module development functionalities, those can be distributed across repositories where they fit best. Module submission pipeline can be based directly on the available OCM tooling while the infra tools should be taken over by respective teams.

Consequences

• Kyma CLI will be split into two parts - user-facing and dev-toolkit
• Depending on the requirements/usage we consider dropping one or both of those parts
• Overall, end users and Kyma teams will use separate tools for their use cases instead of keeping everything together.

[ptesny]

Indeed with the kyma modularisation being almost complete, the current version of kyma cli with kyma deploy is almost of no use.
Ideally, with kyma deploy users could deploy OS Kyma in line with the official modularisation documentation, for instance:

kyma deploy --source 20.0.0 would deploy a bare bone OS Kyma with no modules
2.
kyma deploy --source 20.0.0 default-list would deploy kyma with the default modules: istio, api-gateway, etc
3.
kyma deploy --source 20.0.0 --module module1@namespace --module module2@namespace --module moduleN@namespace would deploy kyma with hand picked modules

Re the third-party modules like the transparent proxy, we could opt for simplifying their kyma deploy installation by additionally allowing to provide a SAP repository technical user and password in the command line in line with the following official guidelines, namely:

https://help.sap.com/docs/connectivity/sap-btp-connectivity-cf/using-transparent-proxy
https://help.sap.com/docs/RBSC/0a64be17478d4f5ba45d14ab62b0d74c/7e83dfc309834942b441fc2106c5b7f5.html
https://help.sap.com/docs/connectivity/sap-btp-connectivity-cf/transparent-proxy-lifecycle-management

To undeploy, we would use kyma undeploy.

PS. How to achieve 1 and 2 today:

1. install a bare bone OS Kyma

$ kyma deploy --component certificates@istio-system --source=2.20.0 --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml 
- No previous Kyma version found
- Using Kyma from the workspace directory: ~/.kyma/sources/2.20.0
- Initial setup
- Component 'CRDs' deployed
- Component 'certificates' deployed
- Kyma deployed successfully!

Kyma is installed in version:	2.20.0
Kyma installation took:		0 hours 0 minutes

Happy Kyma-ing! :)

2. istio

$ make istio-deploy
kubectl create ns kyma-system --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml --dry-run=client -o yaml | kubectl apply --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml -f -
namespace/kyma-system created
kubectl label namespace kyma-system istio-injection=enabled --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml
namespace/kyma-system labeled

kubectl apply -f https://github.com/kyma-project/istio/releases/latest/download/istio-manager.yaml --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml

Warning: resource customresourcedefinitions/istios.operator.kyma-project.io is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/istios.operator.kyma-project.io configured
serviceaccount/istio-controller-manager created
role.rbac.authorization.k8s.io/istio-leader-election-role created
clusterrole.rbac.authorization.k8s.io/istio-manager-role created
rolebinding.rbac.authorization.k8s.io/istio-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/istio-manager-rolebinding created
configmap/istio-ui.operator.kyma-project.io created
priorityclass.scheduling.k8s.io/istio-kyma-priority created
deployment.apps/istio-controller-manager created

kubectl apply -f https://github.com/kyma-project/istio/releases/latest/download/istio-default-cr.yaml --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml

istio.operator.kyma-project.io/default created

kubectl get -n kyma-system istios/default  --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml
NAME      STATE
default   READY

2. api-gateway

$ make api-gateway-deploy
kubectl create ns kyma-system --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml --dry-run=client -o yaml | kubectl apply --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml -f -
namespace/kyma-system configured
kubectl label namespace kyma-system istio-injection=enabled --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml
namespace/kyma-system not labeled

kubectl apply -f https://github.com/kyma-project/api-gateway/releases/latest/download/api-gateway-manager.yaml --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml

namespace/kyma-system configured
customresourcedefinition.apiextensions.k8s.io/apigateways.operator.kyma-project.io created
Warning: resource customresourcedefinitions/apirules.gateway.kyma-project.io is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/apirules.gateway.kyma-project.io configured
serviceaccount/api-gateway-controller-manager created
role.rbac.authorization.k8s.io/api-gateway-leader-election-role created
clusterrole.rbac.authorization.k8s.io/api-gateway-manager-role created
rolebinding.rbac.authorization.k8s.io/api-gateway-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/api-gateway-manager-rolebinding created
configmap/api-gateway-ui.operator.kyma-project.io created
priorityclass.scheduling.k8s.io/api-gateway-priority-class created
deployment.apps/api-gateway-controller-manager created

kubectl apply -f https://github.com/kyma-project/api-gateway/releases/latest/download/apigateway-default-cr.yaml --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml
apigateway.operator.kyma-project.io/default created
kubectl get apigateways/default  --kubeconfig ~/.kube/kubeconfig--btp-easy--btp.yaml
NAME      STATE
default   READY

it may take up to several minutes for the modules to become READY

[ptesny]

After some thought this is what I think any dedicated kyma cli should rather be an automation tools by category:

Provisioning/deployment

1. kyma deploy / undeploy with modules list on top of a gardener cluster; that would optionally entail creating of a gardener cluster

Status / cluster health

2. kyma health-check report an aggregated view of the state of the cluster; including the cpu and mem resevervations per each node

mTLS communication with both kyma custom domain and business custom domain(s)

1. kyma mtls-gateway- create a fixed subdomain for a kyma cluster domain for a mTLS gateway
2. kyma mtls-business - create a wildcard subdomain for a custom business domain with a mTLS gateway

Applications
kyma html5 - deploy a static web-app natively to a kyma cluster as either a standalone or a multi-tenant web application

[TorstenD-SAP]

For the modules command in the dev-ci section I think it makes sense to use native (existing) OCM tooling to create and verify module templates.

[janmedrek]

@ptesny some remarks from my side:

kyma deploy / undeploy with modules list on top of a gardener cluster; that would optionally entail creating of a gardener cluster

We already have Gardener cluster provisioning functionality, the idea would be to have it as part of the dev toolkit as the main use case for it is the pipelines.

kyma health-check report an aggregated view of the state of the cluster; including the cpu and mem resevervations per each node

There is already an excellent overview in the Kyma Dashboard, do we want to duplicate it and present it as a text in the CLI? I am not a fan of that, the information in such an overview would be just an aggregation of simple commands like inspecting Kyma CR or using kubectl to get cluster nodes.

kyma mtls-gateway- create a fixed subdomain for a kyma cluster domain for a mTLS gateway
kyma mtls-business - create a wildcard subdomain for a custom business domain with a mTLS gateway

Is there anything besides deploying some YAMLs required? Or is it just creating Certificates and entries in the Istio Gateways?

kyma html5 - deploy a static web-app natively to a kyma cluster as either a standalone or a multi-tenant web application

Shouldn't it be covered as some kind of an example? I am not familiar with that scenario, I suppose it is already a workload deployment step not exactly tied to the logic of Kyma, are there any generic tools for Kubernetes that do that?

Overall, my goal is to not duplicate functionalities of popular CLI tools such as kubectl. Looking at what Kyma is right now, after modularization there is nothing that would require such tooling as all the modules are separate entities and there is nothing to "deploy" as Kyma anymore.

The tools that you've listed would be nice additions to some kind of examples or tools repository, in my opinion, this is not enough to qualify for the official Kyma CLI tooling (as those already interact with specific modules/scenarios or just aggregate and execute simple kubectl commands underneath).

As I pointed out in the proposal I am not aware of any functionalities that would justify having a dedicated user-facing CLI for Kyma itself. It may be different for specific modules or business scenarios, but then - should it be a part of the CLI?

If you are convinced such a tool would bring value it would be great to know all the requirements that are set by our users. Then, we will be able to discuss what and where should be added and whether it is worth keeping the CLI for that. 🙂

[pbochynski]

It is the list of commands for module management I would see in the CLI:

• kyma modules - show module list (with clear indicator whitch module is installed, which is managed, and the module versions/channels)
• kyma deploy - install default modules (right now istio, api-gateway, btp-operator) with default configuration
• kyma deploy -m serverless telemetry - install serverless and telemetry with default configuration
• kyma deploy -m serverless:1.2.1 telemetry nats eventing -c regular - install modules in the version from regular channel, but serverless in the specific 1.2.1 version
• kyma deploy -m serverless --customConfig - install serverless without applying default configuration
• kyma manage -m serverless -c regular - add serverless to managed modules (add to Kyma CR)
• kyma unmanage -m serverless - removes serverless from managed modules list
• kyma undeploy -m serverless - undeploys serverless manager (if no managed objects like functions do not exist)
• kyma undeploy -m serverless --force - delete all managed resources for serverless (functions) and then undeploy module manager

I think we need also commands for deploying applications and exposing them through API gateway, but I need to think a little bit more about them.

[abbi-gaurav]

One of the power and at the same time complexities with Kyma / K8s is being able to use storage. However, customers end up doing it wrong since they are not experts. e.g. using emptyDir instead of PVC.

Perhaps there should be a simple way with Kyma CLI to set up a proper storage which hides all details and complexities

[abbi-gaurav]

Another challenge which many BTP customers faced is the development cycle. They want to develop their apps locally and then deploy them. But that implies a lot of mocking if they are consuming BTP services. Perhaps Kyma CLI can provide the credentials of BTP Services locally in easier fashion

[ptesny]

kyma api-proxy
Well I'm not that sure this would fit a cli use case.

The api proxy use case.
SAP offers access to hundreds of API packages on SAP Business Accelerator Hub.
Why not have kyma cli help generate an api proxy from any public swagger API definition?

[janmedrek]

Then it would be clear to have the module-related commands and K8S stuff as the commands in the user-facing CLI (which would already justify having it as a "product" available to the end-users).

I am a bit reluctant regarding the interaction with the BTP, should that be a part of Kyma CLI? Or should the BTP CLI be extended? To me it is a bit of a "grey area" and maintenance-wise it can be quite complicated, especially with changes and the need to align with their release cycle.

Perhaps Kyma CLI can provide the credentials of BTP Services locally in an easier fashion
@abbi-gaurav can you elaborate a little bit on this one? I am unsure what the behaviour would be - should the Kyma CLI fetch credentials from BTP and save them in the local (k3d) cluster?

@abbi-gaurav @ptesny @kwiatekus do you have any insights on the usage of Serverless commands? I am wondering what should be the strategy regarding module-specific commands.

@pbochynski do you have any opinions on the development part? Would you see it rather as a separate repository/CLI with a collection of scripts/commands? Or should those be somehow distributed across repositories?

[pbochynski]

@pbochynski do you have any opinions on the development part? Would you see it rather as a separate repository/CLI with a collection of scripts/commands? Or should those be somehow distributed across repositories?

In my opinion, CLI should not be distributed across many repositories. Commands can be located in separate files. I am looking for a team (contributors) that can own the CLI as one consistent piece of software that provides a good interface for capabilities exposed by different kyma modules.

[janmedrek]

@pbochynski What I meant is the module building, signing and verifying part, at first I would see it as a part of the Lifecycle Manager repository as a collection of scripts for the module development purposes. The raw OCM usage in the submission pipeline can later replace that.

[janmedrek]

I think we have the overall direction and the further plans for the CLI already set, I will update this ticket with the decision and follow up regarding further developments in this topic.

[luizcarlospedrosogomes]

update: https://kyma-project.io/#/02-get-started/01-quick-install