permit sending rpc commands

[clementnuss]

it would be much appreciated if we were able (from the command line) to issue RPC to VMWare/ESXi. Our use case is something like:

vmtoolsd --cmd="info-get guestinfo.esxhost"

I went through the codebase but am not sure where to implement that. I gave it a shot here but am not sure if that's really the way to go, as I'm not sure if we want/can open a new nanotoolbox.Service or if we should try to reuse the already running vmtoolsd.

Let me know if my code is heading in the right direction, then I'll open a PR and I'll try to build and test it (so far I got some strange error when running make vmtoolsd... related to ARG {TOOLCHAIN} so I put that on hold before your reply.)

[jonkerj]

I like this idea!

If I were to implement this, I'd implement subcommands using spf13/cobra, one for the current function (the daemon), one for issueing these commands.

It would require a bit of refactoring in cmd/*:

• create a top-level main.go that basically imports github.com/siderolabs/talos-vmtoolsd/cmd and invokes cmd.Root.Execute() (I personally think it's good practise to have package main in root dir anyways)
• create a cmd/root.go that setup root cobra and adds daemon and rpc-cmd
• create a cmd/daemon.go that is a refactoring of current cmd/*
• create a cmd/rpc-cmd.go that contains your idea

I'm a bit short on time (and typically over-optimistic when it comes to planning), so don't count on it too much, but we could do this command refactoring in the coming two weeks.

[clementnuss]

cool! I've already worked with cobra, I'll try to implement that 👍🏼

one key point for me though: can we have multiple nanotoolbox.Service running ? or should we try to somehow implement an api on the daemon that the rpc-cmd subcommand would then query ?

I have a feeling it should work out-of-the box with another service, I'm just not sure how the underlying communication happens:

1. is there a socket for the VMWare comm?
2. and mostly, does it support parallel operations or will it crash if the daemon and the rpc-cmd try to issue an RPC all at the same time?

[clementnuss]

hi @jonkerj

what's the status here ? shall I take it over? if so could you answer my 2 questions above please ? 🙃

[jonkerj]

Oops, I totally forgot about this. It was on my "yeah, I should look into that soon"-list at some point. Sorry for that, and thanks for reminding me.

I dove into the code to answer your questions, and got the feeling some refactoring is overdue. It is definitely not the cleanest I've seen yet. I'm going to see if we can free up some time to refactor/clean it a bit in the next two weeks. If you can wait for this, we can lay the groundwork for implementing additional subcommands in the same go.

1. is there a socket for the VMWare comm?

Sort of, the connection to the hypervisor happen here, a channel pair is what you're probably looking for.

3. and mostly, does it support parallel operations or will it crash if the daemon and the rpc-cmd try to issue an RPC all at the same time?

If you are issuing the commands over the same channel pair that is used by the daemon, it could definately interfere. If you implement your command as a standalone subcommand, it will run as a separate process with its own channel pair, and should not interfere.

[clementnuss]

all right. I'll wait for the refactoring then it'll be easier 🙃

[jonkerj]

I've almost completed the refactoring, need to test, dot some i's and cross some t's and we're done. I expect to produce a PR in a few days.

[jonkerj]

We are fine in #27, v1.0.0 has been released and we believe it's less "spaghetti" now. Your query subcommand would probably need to use service.Request(), I think you can find inspiration in internal/integration/guestinfo.go for that.

As you mentioned you are interested in info-get guestinfo.esxhost, if your goal is (I'm just guessing here) to label k8s nodes according to their ESXi host for (anti-)affinity, I'd expect/hope cloud-provider-vsphere to do stuff like that, by the way.

[clementnuss]

congrats on that !

You guessed right, and I shared your hope regarde vsphere-cpi.. however it only labels the node upon initial discovery of a node (triggered by a node.cloudprovider.kubernetes.io/uninitialized taint on the node) by the cpi.

as your VMs move around all the time though the value isn't valid for long, and tainting nodes every x minutes didn't appeal us very much.

therefore we have configured a series of cronjobs on the esxi which instead adds the label the the VM info. it's then a matter of running the info-get guestinfo.esxhost command and writing the output to a file parsed by node-feature-discovery, which makes it possible to get accurate ESXi label all the time.

I will implement the query subcommand soon, will let you know and create a PR once I'll have tested it 🙃

[jonkerj]

In that case: you could write a small controller that continuously runs and updates labels accordingly, instead of k8s cronjob.

We have thought about something like that too, but we don't really have a use case for hypervisor-affinity. It might even cause misconfigurations when you don't back it up using vSphere DRS anti-affinity.