Looking for help testing v2.0.0 #385
Replies: 7 comments 19 replies
-
Beta Was this translation helpful? Give feedback.
-
|
I removed the example file since the config is on a different page now, see https://github.com/vitobotta/hetzner-k3s/blob/more-refactoring/docs/Creating_a_cluster.md. The README had become too long so I have split it into multiple pages :) |
Beta Was this translation helpful? Give feedback.
-
|
I added a link to the new README in the OP. :) |
Beta Was this translation helpful? Give feedback.
-
|
Oh sorry, I had misunderstood. I replaced with that text with a link to the cluster create page. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the edits, think it is clearer now. deleted all my comments. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, good morning and happy Wednesday! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thanks for the effort on this project! I saw this discussion so I thought I would gave a try running our company cluster (a real small one that just has Gitlab runners) and noticed a couple of issues (note this is not an upgrade, I created it from scratch):
Label error: Taint error: Here is my config, which I think is valid: I've another issue but I'm not 100% if it's related to this project so I didn't include it for now. Let me know if you need other information, cheers! (Edit: should I open an issue instead of posting here directly?) |
Beta Was this translation helpful? Give feedback.
-
|
I forgot to check taints and labels, thanks for the report. I am traveling at the moment so I will look into this when I can. |
Beta Was this translation helpful? Give feedback.
-
|
@jkoestinger This should be fixed in rc2. Can you double check when the binary for your OS is ready? https://github.com/vitobotta/hetzner-k3s/actions/runs/10098594405 |
Beta Was this translation helpful? Give feedback.
-
|
Sure, I tested on Linux and it works 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Cool, thanks for confirming 👍 |
Beta Was this translation helpful? Give feedback.
-
|
@vitobotta OK, I finally got some time to do some tests. I used the rc2 binary from the GitHub releases. Small thing: The instructions are missing the info that I had to manually assign the master server to the existing network (while the tool waited for instance to be up in a loop). It then completed. Interestingly, the autoscaled instances ( I then changed the config and had the pool not autoscaled. ( Also, side ask: How about a conversion command for the tool to convert old cluster_config to the new format. It was quite cumbersome to migrate to the new version. The two new required defaults (enable private network and instance naming) can be done by the conversion as well. WDYT? Here's my config: cluster_name: seiyuu
k3s_version: v1.30.2+k3s2
networking:
ssh:
public_key_path: "/root/.ssh/id_rsa.pub"
private_key_path: "/root/.ssh/id_rsa"
use_agent: false
allowed_networks:
ssh:
- 0.0.0.0/0
- ::/0
api:
- 0.0.0.0/0
- ::/0
private_network:
enabled: true
existing_network_name: seiyuu
subnet: 10.63.0.0/16
public_network:
ipv4: false
ipv6: false
cni:
enabled: true
encryption: true
cluster_cidr: 10.244.0.0/16
service_cidr: 10.43.0.0/16
include_instance_type_in_instance_name: true
schedule_workloads_on_masters: false
manifests:
cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml"
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.8.0/deploy/kubernetes/hcloud-csi.yml"
system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
image: debian-12
autoscaling_image: debian-12
masters_pool:
instance_type: cax11
instance_count: 1
location: fsn1
worker_node_pools:
- name: cax11-autoscale-1
instance_type: cax11
instance_count: 1
location: fsn1
autoscaling:
enabled: true
min_instances: 0
max_instances: 3
post_create_commands:
- timedatectl set-timezone Europe/Berlin
- ip route add default via 10.63.0.1
- ip route add 169.254.0.0/16 via 172.31.1.1
- rm -f /etc/resolv.conf
- echo 'nameserver 185.12.64.1' >> /etc/resolv.conf
- echo 'nameserver 185.12.64.2' >> /etc/resolv.conf
- echo 'edns edns0 trust-ad' >> /etc/resolv.conf
- echo 'search .' >> /etc/resolv.conf
- mkdir -p /etc/network/interfaces.d
- echo "auto enp7s0" > /etc/network/interfaces.d/enp7s0
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces.d/enp7s0
- echo " post-up ip route add default via 10.63.0.1" >> /etc/network/interfaces.d/enp7s0
- echo " post-up ip route add 169.254.169.254 via 172.31.1.1" >> /etc/network/interfaces.d/enp7s0
- apt update
- apt upgrade -y
- apt autoremove -y
- apt install -y apparmor apparmor-utils
and some log output: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for testing!
The IP address for the private network should be assigned automatically to all masters and workers even with an existing network. Have you been able tto reproduce this issue consistently? I will try to reproduce too using your config.
Not sure I am following, if you disabled the autoscaling for an existing pool no new instances should be created for that pool by the autoscaler, so what do you mean by "didn't join the network"? Which nodes are you referring to in this case?
Yeah I thought about it but it would be quite time consuming to implement and it's only required once per existing cluster. It's only a few settings after all. |
Beta Was this translation helpful? Give feedback.
-
Have you reproduced this only with existing clusters, not new ones? |
Beta Was this translation helpful? Give feedback.
-
|
@Funzinator Do you have a copy of the old config file in 1.1.5 format? I want to be sure I reproduce everything in the exact same conditions as you. |
Beta Was this translation helpful? Give feedback.
-
I deleted the cluster for each test run (except the network). With a worker pool without autoscaling, the private network IP got assigned to the worker pool nodes - but still not the master node(s). With autoscaling, no IP got assigned to the worker pool nodes. Every time, consistently.
yes, here you go: cluster_name: seiyuu
k3s_version: v1.30.2+k3s2
public_ssh_key_path: "/root/.ssh/id_rsa.pub"
private_ssh_key_path: "/root/.ssh/id_rsa"
use_ssh_agent: false
ssh_allowed_networks:
- 0.0.0.0/0
- ::/0
api_allowed_networks:
- 0.0.0.0/0
- ::/0
schedule_workloads_on_masters: false
enable_public_net_ipv4: false
enable_public_net_ipv6: false
private_network_subnet: 10.63.0.0/16
cluster_cidr: 10.244.0.0/16
service_cidr: 10.43.0.0/16
disable_flannel: false
enable_encryption: true
image: debian-12
autoscaling_image: debian-12
cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml"
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.8.0/deploy/kubernetes/hcloud-csi.yml"
system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
masters_pool:
instance_type: cax11
instance_count: 1
location: fsn1
worker_node_pools:
- name: cax11-autoscale-1
instance_type: cax11
instance_count: 1
location: fsn1
autoscaling:
enabled: true
min_instances: 0
max_instances: 3
post_create_commands:
- timedatectl set-timezone Europe/Berlin
- ip route add default via 10.63.0.1
- ip route add 169.254.0.0/16 via 172.31.1.1
- rm -f /etc/resolv.conf
- echo 'nameserver 185.12.64.1' >> /etc/resolv.conf
- echo 'nameserver 185.12.64.2' >> /etc/resolv.conf
- echo 'edns edns0 trust-ad' >> /etc/resolv.conf
- echo 'search .' >> /etc/resolv.conf
- mkdir -p /etc/network/interfaces.d
- echo "auto enp7s0" > /etc/network/interfaces.d/enp7s0
- echo "iface enp7s0 inet dhcp" >> /etc/network/interfaces.d/enp7s0
- echo " post-up ip route add default via 10.63.0.1" >> /etc/network/interfaces.d/enp7s0
- echo " post-up ip route add 169.254.169.254 via 172.31.1.1" >> /etc/network/interfaces.d/enp7s0
- apt update
- apt upgrade -y
- apt autoremove -y
- apt install -y apparmor apparmor-utils |
Beta Was this translation helpful? Give feedback.
-
|
@Funzinator do you mind opening an issue for each problem so it's easier to follow and handle each issue separately? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I've done a round of testing, and here is what I found:
I'll do more testing later this week (in particular, I plan to test upgrades from v1 to v2 on a live cluster). Also (let me know if you'd like me to open a separate issue for this): on my clusters, I use ZFS local PV, which requires a separate block device or partition to create the ZFS pool. To make that work with hetzner-k3s, I've built a custom version of hetzner-k3s where the cloud init YAML template is patched to disable Thanks! |
Beta Was this translation helpful? Give feedback.
-
Thanks, let's continue this specific discussion in the PR :)
Yeah unfortunately load balancers don't support firewall yet so there isn't much we can do there for now.
I changed it because well, it's newer and also solves some known issues that occur when using Cilium with 22.04. Do you see any problems with 24.04 or by PR do you mean only adding some info to the docs?
The command fails gracefully if not present in the system so it shouldn't cause any problems. I added this to enable support for old crypto which was causing problems with older version of the SSH2 library for Crystal. We may not need it anymore, not sure.
Fail2ban support as is now is not very complete as I forgot to take into account configs that customize the SSH port. Taking a note about that for v2.0.1.
Sure, sounds good as long as it's configurable and Thanks for the help! |
Beta Was this translation helpful? Give feedback.
-
|
I'm getting an error while running Config: |
Beta Was this translation helpful? Give feedback.
-
|
Same error with the plain config from https://github.com/vitobotta/hetzner-k3s/blob/abfacb14ef0293d645928b3a076105f763c524ef/docs/Creating_a_cluster.md |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jrudolph, I just tested from my local code which is more or less rc2 and I didn't have any problems with the sample configuration in that document. Can you try and see if you can reproduce with rc3 which is now available? The error you pasted is definitely a problem with the structure of the config file, which doesn't match the expectations. In a future release I will try to add some syntax validation which outputs more helpful messages. |
Beta Was this translation helpful? Give feedback.
-
|
@jrudolph I also tried with the rc2 binary just to double check and didn't have any problems. Please check the yaml. If you are still seeing the same problem, please open an issue about this specific problem. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
I tested v2.0.0.rc4 with FYI Letsencrypt/certificate manager, is not working for me in 1.15 as well, so maybe not related. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think it's related to hetzner-k3s regardless of the version. I replied in the other discussion so let's continue there about the issue with certificates :) Thanks for checking rc4! |
Beta Was this translation helpful? Give feedback.
-
If you are available to help test the RC1 of v2.0.0 when ready, please let me know in a comment below. Thanks
See new README split into multiple pages at https://github.com/vitobotta/hetzner-k3s/blob/more-refactoring/README.md.
The binaries can be downloaded here: https://github.com/vitobotta/hetzner-k3s/releases/tag/v2.0.0.rc1.
I need help testing as many scenarios as possible, i.e. with flannel or cilium, private network enabled or disabled, with or without encryption, with embedded etcd or external datastore, with the embedded registry mirror enabled, and of course it would be nice if you could help testing upgrading clusters created with a previous version of the tool (do not upgrade a production cluster of course). The changes are many so the more scenarios you can help me cover, the better. Thanks in advance!
Here's what will be the release notes once v2 is finalized:
New
Improvements
Fixes
Upgrading from v1.1.5
Important: Read these upgrade notes carefully and test the upgrade with a test cluster first, if possible.
Before upgrading:
/etc/k8s-resolv.confon ALL instances (both masters and workers); the file should include a single line:nameserver 8.8.8.8use_ssh_agentfrom the root of the config file toFollow the same pattern for these settings:
networking.private_network.enabledtotrueas all existing clusters were using a private network while the new default isfalseto allow creating larger clusters more easilyinclude_instance_type_in_instance_nametotrue; this is because historically the instance type was included in the names of the instances, causing confusion when changing instance type from the Hetzner console. Since clusters created prior to v2 used that old naming scheme, this new setting must be set totrueto preserve that behavior with v2.Contributing:
If you are a Visual Studio Code user and would like to contribute to the project, you can now more easily work on it by using a dev container with Code. Crystal and all the other dependencies are already included in the container. See docs for more details.
Beta Was this translation helpful? Give feedback.
All reactions