Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GCE scale testing on kops #16181

Merged
merged 1 commit into from
Feb 2, 2024
Merged

Add GCE scale testing on kops #16181

merged 1 commit into from
Feb 2, 2024

Conversation

upodroid
Copy link
Member

@upodroid upodroid commented Dec 20, 2023 •
edited
Loading

Part of kubernetes/enhancements#4224
Fixes: #16241

kops will be replacing the current kubeup 5k scale test, which will be developed here

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 20, 2023
@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Dec 20, 2023
@upodroid
Copy link
Member Author

/test foo

@k8s-ci-robot
Copy link
Contributor

@upodroid: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test presubmit-kops-aws-boskos
  • /test presubmit-kops-aws-boskos-kubetest2
  • /test presubmit-kops-aws-scale-amazonvpc
  • /test presubmit-kops-aws-scale-amazonvpc-using-cl2
  • /test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
  • /test presubmit-kops-gce-scale-kubenet-using-cl2
  • /test presubmit-kops-gce-small-scale-kubenet-using-cl2
  • /test pull-kops-aws-distro-al2023
  • /test pull-kops-aws-distro-amazonlinux2
  • /test pull-kops-aws-distro-debian10
  • /test pull-kops-aws-distro-debian11
  • /test pull-kops-aws-distro-debian12
  • /test pull-kops-aws-distro-flatcar
  • /test pull-kops-aws-distro-rhel8
  • /test pull-kops-aws-distro-rhel9
  • /test pull-kops-aws-distro-rocky8
  • /test pull-kops-aws-distro-ubuntu2004
  • /test pull-kops-aws-distro-ubuntu2004arm64
  • /test pull-kops-aws-distro-ubuntu2204
  • /test pull-kops-aws-distro-ubuntu2204arm64
  • /test pull-kops-build
  • /test pull-kops-e2e-addon-resource-tracking
  • /test pull-kops-e2e-arm64
  • /test pull-kops-e2e-aws-apiserver-nodes
  • /test pull-kops-e2e-aws-cloud-controller-manager
  • /test pull-kops-e2e-aws-dns-none
  • /test pull-kops-e2e-aws-external-dns
  • /test pull-kops-e2e-aws-ipv6-external-dns
  • /test pull-kops-e2e-aws-ipv6-karpenter
  • /test pull-kops-e2e-aws-ipv6-terraform
  • /test pull-kops-e2e-aws-karpenter
  • /test pull-kops-e2e-aws-load-balancer-controller
  • /test pull-kops-e2e-aws-nlb
  • /test pull-kops-e2e-aws-node-local-dns
  • /test pull-kops-e2e-aws-terraform
  • /test pull-kops-e2e-cni-calico
  • /test pull-kops-e2e-cni-canal
  • /test pull-kops-e2e-cni-cilium
  • /test pull-kops-e2e-cni-cilium-etcd
  • /test pull-kops-e2e-cni-flannel
  • /test pull-kops-e2e-gce-dns-none
  • /test pull-kops-e2e-k8s-aws-calico
  • /test pull-kops-e2e-k8s-ci
  • /test pull-kops-e2e-k8s-ci-ha
  • /test pull-kops-e2e-k8s-gce-calico-u2004-k22-containerd
  • /test pull-kops-e2e-k8s-gce-ci
  • /test pull-kops-e2e-k8s-gce-cilium
  • /test pull-kops-e2e-k8s-gce-cilium-etcd
  • /test pull-kops-e2e-k8s-gce-ipalias
  • /test pull-kops-e2e-k8s-gce-long-cluster-name
  • /test pull-kops-e2e-kubernetes-do-dns-none
  • /test pull-kops-e2e-kubernetes-do-fqdn
  • /test pull-kops-e2e-kubernetes-do-gossip
  • /test pull-kops-e2e-metrics-server
  • /test pull-kops-e2e-pod-identity-webhook
  • /test pull-kops-kubernetes-e2e-ubuntu-gce-build
  • /test pull-kops-test
  • /test pull-kops-verify-boilerplate
  • /test pull-kops-verify-generated
  • /test pull-kops-verify-gofmt
  • /test pull-kops-verify-golangci-lint
  • /test pull-kops-verify-gomod
  • /test pull-kops-verify-govet
  • /test pull-kops-verify-hashes
  • /test pull-kops-verify-terraform

The following commands are available to trigger optional jobs:

  • /test pull-kops-e2e-aws-upgrade-k124-ko124-to-k125-kolatest
  • /test pull-kops-e2e-aws-upgrade-k125-kolatest-to-k126-kolatest
  • /test pull-kops-e2e-aws-upgrade-k127-ko127-to-k128-kolatest-karpenter
  • /test pull-kops-e2e-aws-upgrade-k127-ko127-to-klatest-kolatest-many-addons
  • /test pull-kops-e2e-cni-amazonvpc
  • /test pull-kops-e2e-cni-amazonvpc-ipv6
  • /test pull-kops-e2e-cni-calico-ipv6
  • /test pull-kops-e2e-cni-cilium-eni
  • /test pull-kops-e2e-cni-cilium-ipv6
  • /test pull-kops-e2e-cni-kuberouter
  • /test pull-kops-kubernetes-e2e-cos-gce
  • /test pull-kops-kubernetes-e2e-cos-gce-serial
  • /test pull-kops-kubernetes-e2e-cos-gce-slow

Use /test all to run the following jobs that were automatically triggered:

  • pull-kops-build
  • pull-kops-e2e-k8s-aws-calico
  • pull-kops-e2e-k8s-gce-cilium
  • pull-kops-e2e-k8s-gce-ipalias
  • pull-kops-kubernetes-e2e-ubuntu-gce-build
  • pull-kops-test
  • pull-kops-verify-boilerplate
  • pull-kops-verify-generated
  • pull-kops-verify-gofmt
  • pull-kops-verify-golangci-lint
  • pull-kops-verify-gomod
  • pull-kops-verify-govet

In response to this:

/test foo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

2 similar comments
@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

@hakman
Copy link
Member

hakman commented Dec 20, 2023

/cc @dims @hakuna-matatah

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Dec 20, 2023
@upodroid
Copy link
Member Author

/test presubmit-kops-gce-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

@rifelpet
Copy link
Member

I see the cloud-controller-manager is logging:

E1220 20:26:59.428334 1 route_controller.go:329] Could not create route 210655d5-c5e4-4b69-9cc3-7a7e060502e6 100.97.198.0/24 for node nodes-us-east1-c-hdrg: googleapi: Error 403: QUOTA_EXCEEDED - Quota 'STATIC_ROUTES_PER_NETWORK' exceeded. Limit: 200.0 globally.

Probably need some quotas increased in this project?

@upodroid
Copy link
Member Author

Probably need some quotas increased in this project?

Switching to gce networking to avoid that issue.

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

3 similar comments
@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-kubenet-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-calico-using-cl2

1 similar comment
@upodroid
Copy link
Member Author

/test presubmit-kops-gce-small-scale-calico-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
/test presubmit-kops-gce-small-scale-ipalias-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
/test presubmit-kops-gce-small-scale-ipalias-using-cl2

@upodroid
Copy link
Member Author

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
/test presubmit-kops-gce-small-scale-ipalias-using-cl2

@upodroid
Copy link
Member Author

This PR is ready to merged

@hakman

@upodroid
Copy link
Member Author

/retest

@upodroid
Copy link
Member Author

bump

@upodroid upodroid mentioned this pull request Jan 31, 2024
Open
hakuna-matatah
hakuna-matatah reviewed Jan 31, 2024
View reviewed changes
tests/e2e/scenarios/scalability/run-test.sh
create_args+=("--zones=us-east1-b,us-east1-c,us-east1-d")
create_args+=("--node-size=e2-standard-2")
create_args+=("--node-volume-size=30")
create_args+=("--master-volume-size=1000")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting to see the volume size to be 1000, out of curiosity, why would it need 1000 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bigger disks have more iops on GCE

@hakuna-matatah
Copy link
Contributor

Just before this PR is about to be merged, could we run aws test at 5k scale presubmits to make sure everything is working as expected @upodroid ?

@upodroid
Copy link
Member Author

upodroid commented Feb 1, 2024

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
/test presubmit-kops-gce-small-scale-ipalias-using-cl2

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Feb 1, 2024 •
edited
Loading

@upodroid: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
presubmit-kops-gce-scale-kubenet-using-cl2 1842008 link true /test presubmit-kops-gce-scale-kubenet-using-cl2
presubmit-kops-gce-small-scale-kubenet-using-cl2 59668e4 link true /test presubmit-kops-gce-small-scale-kubenet-using-cl2
presubmit-kops-gce-scale-calico-using-cl2 eda14ff link true /test presubmit-kops-gce-scale-calico-using-cl2
pull-kops-verify-hashes eda14ff link true /test pull-kops-verify-hashes

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@upodroid
Copy link
Member Author

upodroid commented Feb 1, 2024

/test presubmit-kops-aws-small-scale-amazonvpc-using-cl2
/retest

@upodroid
Copy link
Member Author

upodroid commented Feb 1, 2024

/test presubmit-kops-aws-scale-amazonvpc-using-cl2

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2024
@k8s-ci-robot k8s-ci-robot merged commit 8a5b56b into kubernetes:master Feb 2, 2024
25 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Feb 2, 2024
@BrewTestBot BrewTestBot mentioned this pull request May 16, 2024
Merged
@upodroid upodroid mentioned this pull request Nov 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hakuna-matatah hakuna-matatah hakuna-matatah left review comments

@hakman hakman hakman approved these changes

@dims dims Awaiting requested review from dims

Assignees

@hakman hakman

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/addons area/provider/gcp Issues or PRs related to gcp provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.29
Development

Successfully merging this pull request may close these issues.

Run the 5000 node scale test using kops on GCE
6 participants
@upodroid @k8s-ci-robot @hakman @rifelpet @marseel @hakuna-matatah