The start of supporting freebsd as a target

[jszwedko]

This is a WIP.

Currently requires these changes to the cross build image:

• freebsd: Include libdevstat in build image cross-rs/cross#620
• freebsd: include libkvm in build image cross-rs/cross#613

Signed-off-by: Jesse Szwedko [email protected]

[netlify]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	752f334
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62b9ebfc3df4c20008fd0551
😎 Deploy Preview	https://deploy-preview-10614--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

Soak Test Results

Baseline: 11b7f73
Comparison: 4ee0a5a
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Test units below are bytes/second/CPU, except for "skewness". The further "skewness" is from 0.0 the more indication that vector lacks consistency in behavior, making predictions of fitness in the field challenging.

The abbreviated table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. The abbreviated table will be omitted if no statistically interesting changes are observed.

No statistically interesting changes with confidence 90.00%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	baseline mean	baseline stdev	baseline outlier percentage	comparison mean	comparison stdev	comparison outlier percentage	t-statistic	p-value	erratic
datadog_agent_remap_blackhole	525.25KiB	1.79	28.58MiB	496.14KiB	1.57068	29.09MiB	752.52KiB	0.26178	-11.3895	1.46401e-27	False
datadog_agent_remap_datadog_logs_acks	376.07KiB	0.97	37.9MiB	870.06KiB	0	38.26MiB	509.45KiB	1.38122	-6.93949	1.1276e-11	False
splunk_hec_route_s3	114.92KiB	1.37	8.2MiB	1.82MiB	0	8.32MiB	2.14MiB	0.828729	-0.761222	0.446779	False
syslog_loki	113.21KiB	1.58	6.99MiB	88.45KiB	0	7.1MiB	177.65KiB	0	-10.8873	4.50489e-25	True
splunk_hec_to_splunk_hec_logs_acks	63.34KiB	0.33	18.76MiB	1.29MiB	2.76243	18.82MiB	1.42MiB	0.824176	-0.614484	0.53909	False
http_pipelines_blackhole_acks	42.36KiB	4.94	856.87KiB	784.77KiB	3.82353	899.24KiB	712.12KiB	1.65746	-0.74749	0.455025	False
http_pipelines_blackhole	22.32KiB	2.49	898.0KiB	777.51KiB	0.552486	920.32KiB	701.73KiB	0.274725	-0.405929	0.684916	False
syslog_splunk_hec_logs	14.23KiB	0.14	9.82MiB	147.76KiB	1.92837	9.83MiB	101.78KiB	1.09589	-1.51265	0.13086	False
http_datadog_filter_blackhole	8.83KiB	0.38	2.26MiB	1.1MiB	1.59236	2.27MiB	1.24MiB	1.9337	-0.0956434	0.923832	False
http_pipelines_no_grok_blackhole	6.54KiB	0.3	2.11MiB	1.02MiB	0.828729	2.11MiB	1.03MiB	0	-0.0835259	0.933457	False
datadog_agent_remap_blackhole_acks	-1.97KiB	-0.01	30.0MiB	1.37MiB	0	30.0MiB	564.41KiB	1.05541	0.0253405	0.979794	False
datadog_agent_remap_datadog_logs	-49.35KiB	-0.13	38.26MiB	1.45MiB	0	38.21MiB	1.02MiB	0	0.511016	0.609522	False
splunk_transforms_splunk3	-61.01KiB	-1.28	4.66MiB	1.49MiB	0.552486	4.6MiB	1.37MiB	0	0.542495	0.587656	False
syslog_regex_logs2metric_ddmetrics	-63.48KiB	-0.79	7.83MiB	598.14KiB	0	7.77MiB	544.85KiB	0	1.49363	0.135712	False
syslog_log2metric_humio_metrics	-93.5KiB	-1.09	8.39MiB	73.05KiB	0	8.3MiB	196.9KiB	2.05279	8.2215	2.37565e-15	False
syslog_log2metric_splunk_hec_metrics	-190.81KiB	-1.99	9.37MiB	383.98KiB	0	9.18MiB	74.29KiB	3.0303	9.27064	1.31953e-18	False
splunk_hec_indexer_ack_blackhole	-211.55KiB	-0.9	22.94MiB	1.1MiB	0	22.73MiB	1.37MiB	0.277778	2.23643	0.0256445	False
splunk_hec_to_splunk_hec_logs_noack	-369.74KiB	-1.84	19.62MiB	1.28MiB	0.274725	19.25MiB	1.36MiB	1.16279	3.62903	0.000305297	False
http_to_http_acks	-378.31KiB	-8.07	4.58MiB	2.03MiB	0	4.21MiB	2.12MiB	0.550964	2.28283	0.0227653	True
http_to_http_noack	-429.24KiB	-1.89	22.17MiB	1.58MiB	0.314465	21.75MiB	1.51MiB	0.292398	3.48642	0.00052255	False
syslog_humio_logs	-670.6KiB	-6.65	9.85MiB	113.0KiB	0	9.2MiB	64.7KiB	0.552486	97.9878	0	False
fluent_elasticsearch	-1.48MiB	-2.14	68.95MiB	661.82KiB	2.20386	67.47MiB	1.57MiB	0	16.7938	1.91277e-50	False
fluent_remap_aws_firehose	-2.51MiB	-3.45	72.71MiB	1.07MiB	0	70.2MiB	2.44MiB	0	17.9176	1.04421e-55	False

Fine details of each soak run.

(experiment, variant)	total samples	mean	std	min	average	p90	p95	p99	max	skewness
('fluent_remap_aws_firehose', 'baseline')	364	72.71MiB	1.07MiB	70.08MiB	72.82MiB	73.96MiB	74.21MiB	74.59MiB	75.05MiB	-0.210118
('fluent_remap_aws_firehose', 'comparison')	362	70.2MiB	2.44MiB	65.07MiB	69.93MiB	73.11MiB	73.45MiB	73.8MiB	74.15MiB	-0.0448647
('fluent_elasticsearch', 'baseline')	363	68.95MiB	661.82KiB	66.87MiB	68.99MiB	69.74MiB	69.93MiB	70.19MiB	70.34MiB	-0.513482
('fluent_elasticsearch', 'comparison')	372	67.47MiB	1.57MiB	64.97MiB	66.95MiB	69.82MiB	70.07MiB	70.72MiB	71.17MiB	0.432041
('datadog_agent_remap_datadog_logs_acks', 'comparison')	362	38.26MiB	509.45KiB	36.9MiB	38.23MiB	38.93MiB	39.14MiB	39.45MiB	39.71MiB	0.280012
('datadog_agent_remap_datadog_logs', 'baseline')	347	38.26MiB	1.45MiB	35.63MiB	39.04MiB	39.87MiB	40.01MiB	40.26MiB	40.42MiB	-0.162663
('datadog_agent_remap_datadog_logs', 'comparison')	363	38.21MiB	1.02MiB	36.22MiB	38.16MiB	39.52MiB	39.68MiB	39.87MiB	40.21MiB	0.0497395
('datadog_agent_remap_datadog_logs_acks', 'baseline')	341	37.9MiB	870.06KiB	35.95MiB	37.7MiB	38.93MiB	39.21MiB	39.48MiB	39.75MiB	0.0581507
('datadog_agent_remap_blackhole_acks', 'baseline')	379	30.0MiB	1.37MiB	27.11MiB	30.08MiB	31.7MiB	31.89MiB	32.19MiB	32.42MiB	-0.0525026
('datadog_agent_remap_blackhole_acks', 'comparison')	379	30.0MiB	564.41KiB	28.27MiB	30.03MiB	30.68MiB	31.0MiB	31.35MiB	31.57MiB	-0.0274747
('datadog_agent_remap_blackhole', 'comparison')	382	29.09MiB	752.52KiB	26.57MiB	29.15MiB	29.99MiB	30.12MiB	30.47MiB	30.98MiB	-0.288074
('datadog_agent_remap_blackhole', 'baseline')	382	28.58MiB	496.14KiB	27.17MiB	28.6MiB	29.21MiB	29.31MiB	29.62MiB	29.92MiB	-0.34117
('splunk_hec_indexer_ack_blackhole', 'baseline')	360	22.94MiB	1.1MiB	20.11MiB	22.91MiB	24.43MiB	24.85MiB	25.4MiB	25.71MiB	0.064111
('splunk_hec_indexer_ack_blackhole', 'comparison')	360	22.73MiB	1.37MiB	19.4MiB	22.64MiB	24.65MiB	25.09MiB	25.88MiB	26.16MiB	0.169439
('http_to_http_noack', 'baseline')	318	22.17MiB	1.58MiB	18.17MiB	22.24MiB	24.1MiB	24.68MiB	25.83MiB	26.69MiB	0.0654049
('http_to_http_noack', 'comparison')	342	21.75MiB	1.51MiB	18.38MiB	21.78MiB	23.56MiB	24.07MiB	25.25MiB	26.84MiB	0.13246
('splunk_hec_to_splunk_hec_logs_noack', 'baseline')	364	19.62MiB	1.28MiB	16.72MiB	19.55MiB	21.33MiB	21.9MiB	22.47MiB	23.57MiB	0.194699
('splunk_hec_to_splunk_hec_logs_noack', 'comparison')	344	19.25MiB	1.36MiB	14.27MiB	19.28MiB	20.96MiB	21.27MiB	22.11MiB	22.23MiB	-0.387741
('splunk_hec_to_splunk_hec_logs_acks', 'comparison')	364	18.82MiB	1.42MiB	14.69MiB	18.89MiB	20.57MiB	20.89MiB	21.66MiB	23.09MiB	-0.290088
('splunk_hec_to_splunk_hec_logs_acks', 'baseline')	362	18.76MiB	1.29MiB	14.81MiB	18.82MiB	20.38MiB	20.71MiB	21.22MiB	22.62MiB	-0.364376
('syslog_humio_logs', 'baseline')	362	9.85MiB	113.0KiB	9.63MiB	9.85MiB	9.99MiB	10.01MiB	10.05MiB	10.08MiB	-0.0512628
('syslog_splunk_hec_logs', 'comparison')	365	9.83MiB	101.78KiB	9.29MiB	9.84MiB	9.96MiB	9.98MiB	10.01MiB	10.06MiB	-0.752817
('syslog_splunk_hec_logs', 'baseline')	363	9.82MiB	147.76KiB	9.34MiB	9.87MiB	9.96MiB	9.98MiB	10.03MiB	10.1MiB	-1.03355
('syslog_log2metric_splunk_hec_metrics', 'baseline')	361	9.37MiB	383.98KiB	8.8MiB	9.56MiB	9.78MiB	9.81MiB	9.83MiB	9.86MiB	-0.0274745
('syslog_humio_logs', 'comparison')	362	9.2MiB	64.7KiB	8.98MiB	9.2MiB	9.28MiB	9.3MiB	9.34MiB	9.35MiB	-0.11421
('syslog_log2metric_splunk_hec_metrics', 'comparison')	363	9.18MiB	74.29KiB	8.87MiB	9.19MiB	9.28MiB	9.29MiB	9.32MiB	9.33MiB	-0.826109
('syslog_log2metric_humio_metrics', 'baseline')	341	8.39MiB	73.05KiB	8.21MiB	8.39MiB	8.48MiB	8.51MiB	8.55MiB	8.58MiB	0.125744
('splunk_hec_route_s3', 'comparison')	362	8.32MiB	2.14MiB	2.11MiB	8.25MiB	11.08MiB	12.03MiB	13.35MiB	15.61MiB	0.234192
('syslog_log2metric_humio_metrics', 'comparison')	341	8.3MiB	196.9KiB	7.64MiB	8.37MiB	8.49MiB	8.5MiB	8.54MiB	8.57MiB	-1.12532
('splunk_hec_route_s3', 'baseline')	364	8.2MiB	1.82MiB	3.28MiB	8.24MiB	10.54MiB	11.11MiB	12.47MiB	12.9MiB	0.0439564
('syslog_regex_logs2metric_ddmetrics', 'baseline')	362	7.83MiB	598.14KiB	6.56MiB	8.07MiB	8.48MiB	8.5MiB	8.55MiB	8.58MiB	-0.594628
('syslog_regex_logs2metric_ddmetrics', 'comparison')	363	7.77MiB	544.85KiB	6.45MiB	8.02MiB	8.28MiB	8.32MiB	8.36MiB	8.42MiB	-0.662495
('syslog_loki', 'comparison')	365	7.1MiB	177.65KiB	6.77MiB	7.06MiB	7.32MiB	7.34MiB	7.39MiB	7.4MiB	0.0407658
('syslog_loki', 'baseline')	361	6.99MiB	88.45KiB	6.77MiB	6.99MiB	7.1MiB	7.12MiB	7.15MiB	7.19MiB	-0.231062
('splunk_transforms_splunk3', 'baseline')	362	4.66MiB	1.49MiB	1.13MiB	4.62MiB	6.73MiB	7.21MiB	8.5MiB	8.67MiB	0.360861
('splunk_transforms_splunk3', 'comparison')	319	4.6MiB	1.37MiB	853.92KiB	4.5MiB	6.3MiB	6.85MiB	7.65MiB	8.34MiB	0.0695196
('http_to_http_acks', 'baseline')	299	4.58MiB	2.03MiB	943.81KiB	4.5MiB	7.28MiB	7.54MiB	9.0MiB	10.93MiB	0.250649
('http_to_http_acks', 'comparison')	363	4.21MiB	2.12MiB	0B	4.14MiB	7.18MiB	7.86MiB	9.47MiB	10.28MiB	0.554055
('http_datadog_filter_blackhole', 'comparison')	362	2.27MiB	1.24MiB	307.82KiB	2.04MiB	3.94MiB	4.59MiB	5.83MiB	6.69MiB	0.867309
('http_datadog_filter_blackhole', 'baseline')	314	2.26MiB	1.1MiB	0B	2.19MiB	3.86MiB	4.11MiB	5.33MiB	5.92MiB	0.515928
('http_pipelines_no_grok_blackhole', 'comparison')	361	2.11MiB	1.03MiB	156.69KiB	2.06MiB	3.65MiB	3.98MiB	4.49MiB	4.88MiB	0.35854
('http_pipelines_no_grok_blackhole', 'baseline')	362	2.11MiB	1.02MiB	156.69KiB	2.0MiB	3.49MiB	3.89MiB	4.59MiB	5.4MiB	0.463327
('http_pipelines_blackhole', 'comparison')	364	920.32KiB	701.73KiB	0B	831.44KiB	1.91MiB	2.14MiB	2.66MiB	3.1MiB	0.654032
('http_pipelines_blackhole_acks', 'comparison')	362	899.24KiB	712.12KiB	0B	768.02KiB	1.79MiB	2.15MiB	2.98MiB	3.53MiB	0.906729
('http_pipelines_blackhole', 'baseline')	362	898.0KiB	777.51KiB	0B	693.53KiB	1.97MiB	2.41MiB	2.94MiB	3.85MiB	0.98497
('http_pipelines_blackhole_acks', 'baseline')	340	856.87KiB	784.77KiB	0B	676.5KiB	1.86MiB	2.45MiB	3.34MiB	3.82MiB	1.29684

[onedr0p]

@jszwedko looks like those changed were merged in

[jszwedko]

@jszwedko looks like those changed were merged in

👍 it does, but it doesn't look like they've released the changes as a new image yet.

[onedr0p]

Good point, their image is 7 months old, I wonder if they will ever release it 😆

cross-rs/cross#542

[jszwedko]

Good point, their image is 7 months old, I wonder if they will ever release it 😆

cross-rs/cross#542

😄 true, though there has been movement on the repository in recent months so we could see a new image soon 🤞

[jszwedko]

Got further with the new cross images, now it is failing due to:

  = note: /usr/local/lib/gcc/x86_64-unknown-freebsd12/6.4.0/../../../../x86_64-unknown-freebsd12/bin/ld: warning: libcxxrt.so.1, needed by /usr/local/lib/gcc/x86_64-unknown-freebsd12/6.4.0/../../../../x86_64-unknown-freebsd12/lib/libc++.so, not found (try using -rpath or -rpath-link)

[jszwedko]

May I ask if there's a schedule for undertaking this? Could an offer to cooperate on the associated (code, dog-fooding) work move that forward?

Hey @kellytk ! This isn't on our roadmap at the moment, but we'd be happy to support a pull request .

[onedr0p]

Vector can be built without hiem-rs, do you actually need the features it provides?

[onedr0p]

Personally I use Prometheus and node-exporter for insights into host metrics. I only use Vector for shipping logs, not host/container metrics.

[jszwedko]

👍 building without heim could be a good start to at least getting something on FreeBSD.

[abh]

@jszwedko Yeah, it's sad that the "bonus features" are blocking shipping the core competencies of Vector on FreeBSD. :-)

[jszwedko]

@jszwedko Yeah, it's sad that the "bonus features" are blocking shipping the core competencies of Vector on FreeBSD. :-)

Unfortunately I haven't been able to push this forward, but I'd be excited to see someone else pick up the torch. You can see the platform specific feature flags that we have in the Cargo.toml. These could be used to exclude components that don't build on FreeBSD.

[onedr0p]

I'd take a crack at it more but unfortunately I've moved away from all BSDs which also solves the problem 😅