Tcp long connections metrics #1249

yp969803 · 2025-02-24T13:01:24Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
The pr introduces new feature of tcp_long_conn metrics
Which issue(s) this PR fixes:

Fixes #1211

Special notes for your reviewer:

Does this PR introduce a user-facing change?:
Yes

Tcp long connections metrics

kmesh-bot · 2025-02-24T13:01:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nlgwcy for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yp969803 · 2025-02-24T13:26:39Z

@nlgwcy @LemmyHuang @xiangxinyong can you review the pr, have i writen correct ebpf code, i have done the changes which has been told in previous comments in the proposal.

LiZhenCheng9527

Please present the results of the long connection metrics
The original destination address problem is still unhandle. If you plan to optimize later, you can create a sub-issue under the existing lfx issue to track the task.

LiZhenCheng9527 · 2025-02-26T06:22:34Z

pkg/controller/telemetry/metric.go

@@ -55,6 +55,7 @@ type MetricController struct {
 	EnableAccesslog      atomic.Bool
 	EnableMonitoring     atomic.Bool
 	EnableWorkloadMetric atomic.Bool
+	EnableLongTCPMetric  atomic.Bool


I don't think it's necessary to distinguish between metrics for long connections and short connections.
As long as metrics is turned on, we should handle all types of connections.

LiZhenCheng9527 · 2025-02-26T06:29:28Z

bpf/kmesh/workload/include/config.h

+#define kmesh_perf_map           km_perf_map
+#define kmesh_perf_info          km_perf_info
+#define map_of_long_tcp_conns    km_longtcpconns_map
+#define long_tcp_conns_events    km_longtcpconns_events


BPF_OBJ_NAME_LEN = 16
So I think, this name of map is too long.

Signed-off-by: Yash Patel <[email protected]>

…userspace Signed-off-by: Yash Patel <[email protected]>

Signed-off-by: Yash Patel <[email protected]>

…km_longconn_ev ringbuff Signed-off-by: Yash Patel <[email protected]>

Signed-off-by: Yash Patel <[email protected]>

yp969803 · 2025-03-02T10:22:09Z

@nlgwcy @LiZhenCheng9527 i have updated the accesslogs for realtime tcp conn monitoring

yp969803 · 2025-03-02T10:32:02Z

I have added realtime monitoring for workload and service metrics (means metrics are reported periodically not after close), do we need serperate metrics for long connections ?
@nlgwcy @LiZhenCheng9527

Signed-off-by: Yash Patel <[email protected]>

LiZhenCheng9527 · 2025-03-03T06:22:44Z

how can i see access-logs https://kmesh.net/en/docs/userguide/use_kmeshctl/ kmeshctl accesslog is not a command @hzxuzhonghu

`# Enable/Disable Kmesh's accesslog:
kmeshctl monitoring --accesslog enable/disable

LiZhenCheng9527 · 2025-03-03T06:29:26Z

The BPF Compatability Test failing with error
    bpf_map.go:60: bpf init failed: bpf Load failed: load program: permission denied:
        	; int sockops_prog(struct bpf_sock_ops *skops)
        	0: (bf) r6 = r1
@nlgwcy @LiZhenCheng9527 what u think about it?

This one is probably the current ebpf program, which exceeds the number of instruction sets allowed in the kernel

LiZhenCheng9527 · 2025-03-03T06:30:16Z

which Makefile is compiling ebpf code, i have added a new file ebpf/kmesh/workload/tc.c, how to compile it @nlgwcy @LiZhenCheng9527

make build can compile ebpf code

LiZhenCheng9527 · 2025-03-03T06:40:07Z

pkg/controller/telemetry/metric.go

@@ -407,21 +440,34 @@ func buildV4Metric(buf *bytes.Buffer) (requestMetric, error) {
 		data.origDstPort = connectData.OriginalPort
 	}

-	data.sentBytes = connectData.SentBytes
-	data.receivedBytes = connectData.ReceivedBytes
+	data.sentBytes = connectData.SentBytes - tcp_conns[connectData.ConnId].sentBytes


When are you going to clean up tcp_conns?

LiZhenCheng9527 · 2025-03-03T06:57:12Z

bpf/kmesh/workload/tc.c

+
+        __u64 now = bpf_ktime_get_ns();
+        // Check if connection duration exceeds threshold
+        if ((now - conn->start_ns) > LONG_CONN_THRESHOLD_TIME) {


This is stage reporting.
So what happens if the conn is closed at less than threshold?

hzxuzhonghu · 2025-03-03T07:06:17Z

bpf/kmesh/workload/tc.c

+        BPF_LOG(ERR, TIMER, "Failed to lookup tcp timer\n");
+    } else {
+        // Initialize and start timer
+        bpf_timer_init(timer, &tcp_conn_flush_timer, 1);


These API are only available since 5.15, but kmesh support 5.10 from beginning

yp969803 · 2025-03-03T07:18:19Z

do we need serperate metrics for long connections ?
@hzxuzhonghu @nlgwcy @LiZhenCheng9527

yp969803 · 2025-03-03T07:21:28Z

#1249 (comment), do i have to change the Makefile so that the new c file i have added is also compiled with correct header file
@LiZhenCheng9527

LiZhenCheng9527 · 2025-03-03T07:27:14Z

do we need serperate metrics for long connections ? @hzxuzhonghu @nlgwcy @LiZhenCheng9527

Do you mean handle the metrics for long and short connections separately? Can you tell us your reasons?

LiZhenCheng9527 · 2025-03-03T07:28:28Z

#1249 (comment), do i have to change the Makefile so that the new c file i have added is also compiled with correct header file @LiZhenCheng9527

I am not sure what's your means. Did you have any trouble compiling it?

hzxuzhonghu · 2025-03-03T07:37:28Z

bpf/kmesh/probes/tcp_probe.h

+    }
+    __u64 now = bpf_ktime_get_ns();
+    info_vals->duration = now - info_vals->start_ns;
+    get_tcp_probe_info(tcp_sock, info_vals);


Nit: this is very confusing now, maybe rename to set_xxx

hzxuzhonghu · 2025-03-03T07:39:23Z

bpf/kmesh/probes/probe.h

-    tcp_report(sk, tcp_sock, storage, BPF_TCP_ESTABLISHED);
+    update_tcp_conn_info_on_state_change(tcp_sock, storage, state);
+    if (state == BPF_TCP_CLOSE) {
+        bpf_sk_storage_delete(&map_of_sock_storage, sk);


This kind of map does not need to delete @nlgwcy to confirm

hzxuzhonghu · 2025-03-03T07:39:57Z

bpf/kmesh/probes/tcp_probe.h

+static inline void
+update_tcp_conn_info_on_state_change(struct bpf_tcp_sock *tcp_sock, struct sock_storage_data *storage, __u32 state)
+{
+    struct tcp_probe_info *info = NULL;


hzxuzhonghu · 2025-03-03T07:43:19Z

bpf/kmesh/workload/tc.c

+
+// Also trigger's on icmp packets (hence can be used for monitor packet loss)
+SEC("tc")
+int tc_prog(struct __sk_buff *skb)


can we do this in the existing hook

yp969803 · 2025-03-03T07:47:27Z

Metrics we currently have:

Workload Metrics:  
kmesh_tcp_workload_received_bytes	The size of the  number of bytes received in to a workload 
kmesh_tcp_workload_sent_bytes	The size of the number of bytes sent  to a workload 
etc..
these metrics are workload specific

same way we have serviceMetrics , which are service specififc

What i am proposing was for connection specific metrics (we can show it for the conn whose duration exceeded 1 min), not for short conns because prometheus scrape metrics in 5-15 sec interval conns having less duration is missed by prometheus
ex:

kmesh_long_tcp_conn_received_bytes   The size of recieved bytes by a specific conn,

etc...

these metrics contains label including conn_id which helps us to differentiate conns having src ip, port and destination ip port

Have i made my point clear, sorry for poor english 😂 @LiZhenCheng9527

Signed-off-by: Yash Patel <[email protected]>

Signed-off-by: Yash Patel <[email protected]> rfac: removed callback funcs in tc.c Signed-off-by: Yash Patel <[email protected]>

Signed-off-by: Yash Patel <[email protected]>

Signed-off-by: Yash Patel <[email protected]> rfac: bpf2go.go Signed-off-by: Yash Patel <[email protected]>

kmesh-bot added the kind/feature label Feb 24, 2025

kmesh-bot requested review from hzxuzhonghu and nlgwcy February 24, 2025 13:01

kmesh-bot added the size/L label Feb 24, 2025

LiZhenCheng9527 reviewed Feb 26, 2025

View reviewed changes

yp969803 added 17 commits February 27, 2025 14:21

feat: added tcp_long_conn_maps

d16e491

Signed-off-by: Yash Patel <[email protected]>

feat: added obeserve_long_conn_tcp func in probe.h

13021d5

Signed-off-by: Yash Patel <[email protected]>

feat: added perf/event-flush to get conn-metrics fron kernelspace to …

0b112e3

…userspace Signed-off-by: Yash Patel <[email protected]>

rfac: added now param to observe and report funcs

f6c4ef5

Signed-off-by: Yash Patel <[email protected]>

rfac: flush_long_conns

ac45d37

Signed-off-by: Yash Patel <[email protected]>

rfac: added bounded loop to flush_long_conns func

5493d52

Signed-off-by: Yash Patel <[email protected]>

feat: added timer_func at trafic_control to flush long_conn

38e820a

Signed-off-by: Yash Patel <[email protected]>

rfac: init_tcp_long_conn_flush_timer

bf49aca

Signed-off-by: Yash Patel <[email protected]>

rfac: removed inline from timer-callback func

52a3d58

Signed-off-by: Yash Patel <[email protected]>

rfac: added long_conn_metrics ringbuf reader in userspace

60491cb

Signed-off-by: Yash Patel <[email protected]>

rfac: shorten the map names

04a4405

Signed-off-by: Yash Patel <[email protected]>

rfac: removed enableLongTcp metric from metric controller

2b483ad

Signed-off-by: Yash Patel <[email protected]>

feat: created longConnectionDataV4 longConnectionDataV6 to read from …

63215bb

…km_longconn_ev ringbuff Signed-off-by: Yash Patel <[email protected]>

rfac: added start_ns to long_conn_tcp_event

b48c50e

Signed-off-by: Yash Patel <[email protected]>

rfac: added start_ns and last_report_ns in tcp_probe_info

e19f57d

Signed-off-by: Yash Patel <[email protected]>

rfac: added startNs and lastReportNs in requestMetric

4355de8

Signed-off-by: Yash Patel <[email protected]>

rfac: run func in workload_controller

a698cac

Signed-off-by: Yash Patel <[email protected]>

yp969803 force-pushed the tcp_long_connection_code branch from b79a168 to a698cac Compare February 27, 2025 08:53

yp969803 added 5 commits February 27, 2025 14:37

rfac: removed tcp_long_conn_env ringbuff and tcp_probe ringbuff instead

c2a46fd

Signed-off-by: Yash Patel <[email protected]>

rfac: tcp_report report_tcp_conn

66d26fd

Signed-off-by: Yash Patel <[email protected]>

rfac: added funcs of tcp_long_conn ile in tcp_probe.h

b92b6c6

Signed-off-by: Yash Patel <[email protected]>

rfac: changed maps name in conn_report_time.c

b550bf6

Signed-off-by: Yash Patel <[email protected]>

chore: added some ebpf logs for tcp report func

1f9cb92

Signed-off-by: Yash Patel <[email protected]>

yp969803 added 2 commits March 2, 2025 14:48

rfac: removed clos_ns from tcp_prbe_info as lastReportNs will do the job

7d9dd21

Signed-off-by: Yash Patel <[email protected]>

feat: realtime accesslogs

cfb60a8

Signed-off-by: Yash Patel <[email protected]>

chore: run make-gen check

396d932

Signed-off-by: Yash Patel <[email protected]>

kmesh-bot added size/XL and removed size/L labels Mar 2, 2025

LiZhenCheng9527 reviewed Mar 3, 2025

View reviewed changes

hzxuzhonghu reviewed Mar 3, 2025

View reviewed changes

yp969803 mentioned this pull request Mar 3, 2025

Proposal for tcp_long_connection_metrics #1224

Open

yp969803 marked this pull request as draft March 11, 2025 04:08

kmesh-bot added the do-not-merge/work-in-progress label Mar 11, 2025

yp969803 added 3 commits March 11, 2025 10:29

rfac: split sockops func and creted maual trimer

b9e5092

Signed-off-by: Yash Patel <[email protected]>

feat: added is_managed_by_kmesh in tc.c

aa91f96

Signed-off-by: Yash Patel <[email protected]>

rfac: added is_managed_by_kmesh in cases in workload/sockops.c

e80fc80

Signed-off-by: Yash Patel <[email protected]> rfac: removed callback funcs in tc.c Signed-off-by: Yash Patel <[email protected]>

yp969803 force-pushed the tcp_long_connection_code branch from dd779fd to e80fc80 Compare March 12, 2025 04:10

rfac: iteration for tcp_conns

ec2f364

Signed-off-by: Yash Patel <[email protected]>

kmesh-bot added size/XXL and removed size/XL labels Mar 12, 2025

yp969803 added 2 commits March 12, 2025 12:42

rfac: ebpf2go for tc.c

dcfba26

Signed-off-by: Yash Patel <[email protected]>

rfac: generated go files for tc.c

a3aa24e

Signed-off-by: Yash Patel <[email protected]> rfac: bpf2go.go Signed-off-by: Yash Patel <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tcp long connections metrics #1249

Tcp long connections metrics #1249

yp969803 commented Feb 24, 2025 •

edited

Loading

kmesh-bot commented Feb 24, 2025

yp969803 commented Feb 24, 2025 •

edited

Loading

LiZhenCheng9527 left a comment

LiZhenCheng9527 Feb 26, 2025

LiZhenCheng9527 Feb 26, 2025

yp969803 commented Mar 2, 2025

yp969803 commented Mar 2, 2025 •

edited

Loading

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025 •

edited

Loading

LiZhenCheng9527 Mar 3, 2025

LiZhenCheng9527 Mar 3, 2025

hzxuzhonghu Mar 3, 2025

yp969803 commented Mar 3, 2025

yp969803 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

hzxuzhonghu Mar 3, 2025

hzxuzhonghu Mar 3, 2025

hzxuzhonghu Mar 3, 2025

hzxuzhonghu Mar 3, 2025

yp969803 commented Mar 3, 2025 •

edited

Loading

Tcp long connections metrics #1249

Are you sure you want to change the base?

Tcp long connections metrics #1249

Conversation

yp969803 commented Feb 24, 2025 • edited Loading

kmesh-bot commented Feb 24, 2025

yp969803 commented Feb 24, 2025 • edited Loading

LiZhenCheng9527 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yp969803 commented Mar 2, 2025

yp969803 commented Mar 2, 2025 • edited Loading

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yp969803 commented Mar 3, 2025

yp969803 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

LiZhenCheng9527 commented Mar 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yp969803 commented Mar 3, 2025 • edited Loading

yp969803 commented Feb 24, 2025 •

edited

Loading

yp969803 commented Feb 24, 2025 •

edited

Loading

yp969803 commented Mar 2, 2025 •

edited

Loading

LiZhenCheng9527 commented Mar 3, 2025 •

edited

Loading

yp969803 commented Mar 3, 2025 •

edited

Loading