diff --git a/content/en/docs/userguide/circuit_breaker.md b/content/en/docs/userguide/circuit_breaker.md index 1359d51..ea883df 100644 --- a/content/en/docs/userguide/circuit_breaker.md +++ b/content/en/docs/userguide/circuit_breaker.md @@ -1,253 +1,216 @@ --- draft: false -linktitle: Circuit Breaker +linktitle: Circuit Breaker menu: docs: parent: user guide weight: 22 -title: Use Circuit Breaker +title: Circuit Breaker toc: true type: docs --- -### Preparation +This task shows you how to configure circuit breakers in KMesh using Fortio for load testing. -1. Ensure KMesh is installed in your Kubernetes cluster (see [Quick Start Guide](https://kmesh.net/en/docs/setup/quickstart/)) +### Before you begin -2. Deploy a sample microservice application +- Install KMesh + Please refer [quickstart](https://kmesh.net/en/docs/setup/quickstart/) and change into ads mode -3. Verify the default namespace is managed by KMesh -### Circuit Breaker Configuration +### Deploy the test applications -##### Deploy a Sample Application - -Let's use a simple microservice setup to demonstrate circuit breaking: - -```bash -kubectl apply -f - <active_connections >= cbs->max_connections) { - // Reject connection - return -1; - } - // Bind socket to cluster - return 0; -} -``` +#### 4.2 Circuit Breaker Test +```bash +# Heavy load to trigger circuit breaker +kubectl exec -it deploy/fortio -- \ +fortio load -c 5 -qps 100 -t 30s http://test-service -### Advanced Configuration Example +# Verify circuit breaker status +kubectl get destinationrule test-circuit-breaker -o yaml -```yaml -apiVersion: kmesh.net/v1alpha1 -kind: CircuitBreaker -metadata: - name: complex-service-circuit-breaker -spec: - services: - - name: service-a - rules: - - priority: HIGH - maxConnections: 50 - - name: service-b - rules: - - priority: MEDIUM - maxConnections: 100 +# Simulate service failure +kubectl scale deployment test-service --replicas=0 ``` +#### 4.3 Recovery Test +```bash +# Restore service +kubectl scale deployment test-service --replicas=1 -### Troubleshooting - -##### Common Issues -- Unexpected connection rejections -- High error rates -- Performance degradation +# Test recovery +kubectl exec -it deploy/fortio -- \ +fortio load -c 2 -qps 20 -t 30s http://test-service +``` -##### Debugging Steps +### Analyzing Results +#### 6.1 Fortio Results ```bash -# Check circuit breaker configuration -kubectl describe circuitbreaker sample-service-circuit-breaker +# View test results +kubectl logs deploy/fortio -# View detailed logs -kubectl logs -n kmesh -l app=kmesh circuit-breaker -c circuit-breaker - -# Check cluster status -kmesh get clusters +# Get detailed metrics +kubectl exec -it deploy/fortio -- /usr/bin/fortio report ``` -### Best Practices +#### 6.2 System Metrics +```bash +# Check pod status +kubectl get pods -w -1. Start with conservative limits -2. Gradually adjust based on service performance -3. Monitor circuit breaker metrics -4. Use priority-based configurations +# View circuit breaker configuration +kubectl get destinationrule test-circuit-breaker -o yaml +``` -### Performance Considerations +### Understanding what happened -- Kernel-native implementation -- Minimal overhead -- Lock-free atomic updates -- eBPF map-based tracking +The circuit breaker configuration: +- Limits concurrent HTTP requests +- Ejects hosts after 3 consecutive errors +- Keeps circuit open for 30 seconds +- Monitors service health every 5 seconds -### Cleanup +When the service is overloaded: +1. Circuit breaker trips after threshold breach +2. Subsequent requests are blocked +3. Service recovers after baseEjectionTime +4. Normal traffic flow resumes -Remove the circuit breaker and sample application: +### Clean up +Remove test components: ```bash -kubectl delete circuitbreaker sample-service-circuit-breaker -kubectl delete deployment sample-service -kubectl delete service sample-service +kubectl delete -f sample-app.yaml +kubectl delete -f fortio.yaml +kubectl delete -f circuit-breaker.yaml ``` -### Limitations +### Troubleshooting -- Currently focuses on TCP connections -- Kernel version dependencies -- Per-cluster granularity +If you encounter issues: +1. Check pod status: +```bash +kubectl get pods +kubectl describe pod +``` -### Sample Code Snippet +2. Check circuit breaker configuration: +```bash +kubectl get destinationrule +kubectl describe destinationrule test-circuit-breaker +``` -```go -// Circuit Breaker Configuration Example -circuitBreaker := &CircuitBreakers{ - Priority: RoutingPriority_HIGH, - MaxConnections: 100, - MaxPendingRequests: 50, - MaxRequests: 200, - MaxRetries: 3, -} +3. View application logs: +```bash +kubectl logs deploy/test-service +kubectl logs deploy/fortio ``` \ No newline at end of file