Move development to Kubernetes entirely

[pierreyves-lebrun]

I find Skaffold awesome but it doesn’t push the logic of development in Kubernetes as far as it should.

Skaffold currently requires developers to iterate on their application source code locally and then deploy to local or remote Kubernetes clusters.

Such a workflow is not in phase with modern application development in that developers shouldn’t have to locally install any project dependency or tool.

Development environments should be repeatable and easily deployable. Containerisation solved this issue many years ago by allowing developers to specify development environment in the form of dockerfiles.
IDEs like VSCode nowadays embrace such a philosophy:
https://code.visualstudio.com/docs/remote/containers

Other tools like Devspace or Okteto understood that by moving development entirely to Kubernetes:
https://okteto.com/docs/reference/faqs/index.html#how-is-okteto-different-from-other-tools-like-skaffold

I think Skaffold would greatly benefit from embracing the same idea.

[haf]

I want to bite: the reason I use skaffold over vscode containers or Google Cloud Code is that it lets me work with the tooling I use for deploying, and as such, I "just" (a big footnoote) set up the same infra locally with port-forwarding or NodePorts as I would have in k8s in the cloud.

I would be very interested in hearing what you find better with containerised development environments? Are you not using k8s in the same was as I am?

With containers I'm not sure I would achieve as good dev-prod parity as I can with skaffold and/or external tools.

If I were to make an improvement to Skaffold, it would be "groups" or folders to be run together; or a single skaffold file with keyed inner skaffold files (kind: SkaffoldList). That way I could tell a frontend dev to run: skaffold dev --group frontend and so on.

I would also make skaffold better at handling local images; right now I have to configure the image pull policy, or it won't start inside k8s (for the docker-desktop use-case).

Another bit I would improve with skaffold is how it documents "testing", as well as file synchronisation and when/how to use live rebuilds+sync versus "realistic" dockefiles to build (I'd like this as another flag)

[pierreyves-lebrun]

I am having a hard time figuring out what you mean so let me explain how I develop applications and you can do the same.

Originally I used to install all project dependencies and tooling locally. So in the case of a node project I would install all node modules and VSCode extensions on my laptop.

To be noted that I already was using Docker but still node modules needed to be installed locally as otherwise no IDE would be able to resolve my project dependencies.

Then in 2018 came VSCode remote containers development feature. That freed myself from having to locally install node modules and VSCode extensions as I could connect my IDE to a container holding all of those dependencies. To me that represents the grail of dev/prod parity as finally development is possible in the same environment as in production. No need for any developer to install any local dependency anymore.

The last piece of the puzzle was development in Kubernetes, which Skaffold almost figured out. I am saying almost because it brings us back to a split “develop locally, deploy remotely” workflow.

I’ll quote Okteto documentation:
“Skaffold automates the workflow for building, pushing and deploying your application. You iterate on your application source code locally and then deploy to local or remote Kubernetes clusters. Okteto's philosophy is to move development entirely to Kubernetes.”

This is the feature I am looking for. There shouldn’t be a environment for development and a another separate one for deployment. There should be a single environment where developers can just connect to containers deployed by Skaffold and develop inside them. That of course implies syncing code changes from the container down to the local repository, but that is something Skaffold isn’t capable of due to this pending issue.

It could be I am trying so solve the wrong problem, I actually would actually love to be proved wrong about what the ideal pipeline should look like. As of now, development in containers is an evidence to me so I would really like marry it with development in Kubernetes.

My reasoning being based on many assumptions, I understand other people may disagree with me. Feel free to explain how you are working as this will help me making sense of how other developers are using Skaffold.

[tstromberg]

Skaffold currently requires developers to iterate on their application source code locally and then deploy to local or remote Kubernetes clusters.

I'm not sure I entirely agree here: Skaffold can be quite useful when run remotely, and can be run from within a Kubernetes cluster.

Such a workflow is not in phase with modern application development in that developers shouldn’t have to locally install any project dependency or tool.

I totally agree that this is where the future of software development is heading, and I think Skaffold can definitely help enable it. This is likely why for instance, Google Cloud SHell includes Skaffold.

Development environments should be repeatable and easily deployable. Containerisation solved this issue many years ago by allowing developers to specify development environment in the form of dockerfiles.
IDEs like VSCode nowadays embrace such a philosophy:
https://code.visualstudio.com/docs/remote/containers

Other tools like Devspace or Okteto understood that by moving development entirely to Kubernetes:
https://okteto.com/docs/reference/faqs/index.html#how-is-okteto-different-from-other-tools-like-skaffold

I think Skaffold would greatly benefit from embracing the same idea.

At the moment, Skaffold is taking a more neutral building-block tool approach here. If you want to run it in Kubernetes, remotely, or locally, it should be made to work for you. Skaffold does however assume that the source code is accessible via a file system, which other systems may not.

Other than the 2-way sync, What do you feel think is missing?

[pierreyves-lebrun]

Thanks for sharing your thoughts.

Skaffold currently requires developers to iterate on their application source code locally and then deploy to local or remote Kubernetes clusters.

I'm not sure I entirely agree here: Skaffold can be quite useful when run remotely, and can be run from within a Kubernetes cluster.

I completely agree with what you are saying here, that is all good.

The point I am trying to make is that Skaffold assumes developers are going to iterate on the code outside of the containers it is going to deploy. The issue is that an increasing number of developers cannot afford to do that because their development environment lies in the very containers Skaffold is deploying.

At the moment, Skaffold is taking a more neutral building-block tool approach here. If you want to run it in Kubernetes, remotely, or locally, it should be made to work for you. Skaffold does however assume that the source code is accessible via a file system, which other systems may not.

Other than the 2-way sync, What do you feel think is missing?

I think that is essentially what is missing, once 2-way sync is implemented then devs can remote into containers, iterate on the code and get all changes synced back.

[simonoff]

I have a few thing to think.
Is a possibility to run specific commands.
For example:

I want to create a migration file in rails.
Right now without two-way sync we are using docker-compose. With two-way is possible to run command inside pod. But what if we be able to run command with specific container/pod environments and after sync changes back?

Like skaffold cmd -p development -c rails-container — rails g model TestModel name:string

[nkubala]

this is a dupe of #3960 (which I just left a small comment on), so let's move the conversation here.

@pierreyves-lebrun can you help me understand what this would look like from skaffold's perspective? I think I understand the problem you're trying to solve - eliminate the need to configure a local development environment in favor of using a prebuilt container image with all dependencies installed, deploy that k8s and develop in that pod.

what would skaffold's role in this be? it seems to me like that container would also have skaffold inside of it, and you could connect to that remotely in whatever way you've already been (through VSCode remote dev, or SSH, or whatever) and continue to use skaffold like you would normally. you could even build your development container based off of our skaffold docker image (gcr.io/k8s-skaffold/skaffold) so you have all of skaffold's dependencies curated for you.

i'm not quite sure what else we can do on our end, but you have much more experience with this than we do. i don't think anyone on our team is opposed to supporting this, we just don't quite understand how it would look :)

[nkubala]

also @simonoff, we have a design in the works for running commands inside of deployed pods/containers, so stay tuned for that. there's another open feature request for two-way sync, so we can follow up on the conversation around that over there.

[pierreyves-lebrun]

this is a dupe of #3960 (which I just left a small comment on), so let's move the conversation here.

@pierreyves-lebrun can you help me understand what this would look like from skaffold's perspective? I think I understand the problem you're trying to solve - eliminate the need to configure a local development environment in favor of using a prebuilt container image with all dependencies installed, deploy that k8s and develop in that pod.

Yes that is correct. I don't mind locally installing Skaffold by the way, that is acceptable to me.

what would skaffold's role in this be?

Essentially two way sync, more specifically this.

When I remote in a container and make changes, I need those changes to be synced back to my local file system.

it seems to me like that container would also have skaffold inside of it, and you could connect to that remotely in whatever way you've already been (through VSCode remote dev, or SSH, or whatever) and continue to use skaffold like you would normally. you could even build your development container based off of our skaffold docker image (gcr.io/k8s-skaffold/skaffold) so you have all of skaffold's dependencies curated for you.

I wasn't even thinking to go that deep. What I meant by "development" environment is a container where my all dependencies are pre installed. E.g. in the case of a TypeScript project, I don't want to locally install npm. I just want to remote into the container Skaffold deployed and issue commands like tsc or npm install.

Obviously I can already do that at the moment but commands causing side effects like npm install are useless since changes are not synced back to the local filesystem.

i'm not quite sure what else we can do on our end, but you have much more experience with this than we do. i don't think anyone on our team is opposed to supporting this, we just don't quite understand how it would look :)

No worries, I really appreciate your feedback. I also realize that a practice I was considering as common (dev in containers) isn't as widespread as I originally thought.

[simonoff]

@nkubala yeah I can say the same. The stop moment now it that two-way sync is not working.
About local development in Kubernetes:
1 We running a multipass virtual machines with microk8s or k3s(new projects using it as it much faster and use less memory)
2 We running whole application in kubernetes with helm. The only difference from production that its includes an additional container in pod which running a webpack-dev-server.
3 When we changing code locally - it synced into containers and we can use them immediately.
4 Unfortunately we still need to use a docker-compose for cases when we need to run bundle install or yarn add which is really annoying.
5 Also for now still the developers for entering the container needs to use a full kubectl command, which requires to get list of pods first and only after we can know exact name of it as we are usually using deployments.

If we will have a two-way sync and some command aliases it will be perfectly.

P.S. There is no needs to have a scaffold in images. Scaffold just deploy my app to kubernetes, monitor it, sync files and gives me possibility to run any command in any container. Super easy.

[mfrey777]

I believe I am facing the same challenge than @pierreyves-lebrun.

I am developing an application with a Python/Flask backend and a Node/React frontend. I use "skaffold dev" to fire up the kubernetes and can then develop my app/replicate my code changes.

In an ideal world, I would like to be able to go to a new PC (with VS Code and Skaffold installed), pull my code from the github repo and continue developing. For that to work, my "backend" folder should use the Python interpreter of the backend container and my "frontend" folder open in the front-end container (the same containers than the one deployed in the kubernetes).

A bit like if I have two VS Code windows open (each one with the frontend/backend open in a docker container), but using the same docker image than the one deployed to the kubernetes (so that I can connect to the database/redis running in other kubernetes deploymentes).

I face currently 2 issues:

• ability to use more than one remote development environment per project (so one per project folder and not one for the whole project)
• how to use the kubernetes deployed containers for remote development

[briandealwis]

I'm hesitant to use two-way syncing with in-container development as

1. you risk making irreproducible changes ("ooo I'll just twiddle that /etc/config/file"), and
2. your container environment is full of tools that shouldn't be part of your production container.

That said, skaffold dev has some special hooks with GCP Buildpacks that offers an experience that's very similar to what you're describing, but without developing inside the remote container. Basically the GCP Buildpacks collaborates with Skaffold to automatically configure sync rules for a set of filetypes that the GCP Buildpacks supports. For example, the NodeJS buildpacks configures *.js files amongst others. When using skaffold dev, the image is built such that the language build tools are installed, and whenever the changed files are uploaded, the normal build (npm install) will be run and the app restarted. (Note to self: I need to add TypeScript there too.)

You can configure your Dockerfile and skaffold.yaml to do the equivalent too (GCP Buildpacks uses watchexec as its file watcher). And Skaffold 1.14 now exposes Skaffold's run mode (e.g., build, dev, debug) as a Docker build variable called SKAFFOLD_RUN_MODE so you could key off that if you wanted.

[simonoff]

@briandealwis there is no risks.
About 1 - i think if you change something specific you know what do you do
About 2 - yes and no. for example for one project on node.js the onlly difference in NODE_MODE and in additional container in the pod with webpack. Thats it.

I don't need any buildpacks or something else. Just give me the ability for two-way sync. Other I will do myself.

[briandealwis]

Thanks all for the feedback here. I'm going to close this out as we have a request tracked for two-way sync in #2492, and having separate dockerfiles for dev/debug in #5844. Please feel free to open more focused issues for specific features.