dnsdist handle more than 1 million concurrent connection #14845
Replies: 9 comments 5 replies
-
|
Hi, You are not giving us a lot to work with, we are going to need much more information to be able to help you:
10k QPS is very, very low, to be clear, dnsdist can handle a lot more than that on decent hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
Can you rule out the Powerdns Auth and it's (Database?) backend as a bottleneck?
Winfried
Am 14. November 2024 11:02:02 MEZ schrieb ahmedhkhalil ***@***.***>:
…Problem:
We are experiencing over 1 million concurrent requests across all VMs, but despite this high query load:
The CPU and memory utilization remain very low.
The maximum QPS (Queries Per Second) for dnsdist is around 10k.
Question:
How can we:
Increase the QPS (queries per second) handled by dnsdist?
Improve the utilization of CPU resources to handle more requests faster and ensure the system can handle this high traffic load effectively?
Environment:
dnsdist setup for load balancing across multiple VMs.
PowerDNS configured as the backend.
Public DNS authoritative server.
We are seeing low CPU/memory utilization, but the DNS query rate is capped at 10k QPS.
What We've Tried:
We have checked and optimized configurations like buffer sizes, worker threads, and query timeouts.
The system scales well vertically in terms of available resources, but QPS still remains limited to ~10k.
Tried adjusting various dnsdist performance and load-balancing settings.
What We're Looking For:
Suggestions for tuning dnsdist to handle more QPS.
CPU optimizations to ensure that we are fully utilizing available CPU cores.
Best practices for configuring dnsdist in high-traffic environments to ensure scalability and improved throughput.
--
Reply to this email directly or view it on GitHub:
#14845
You are receiving this because you are subscribed to this thread.
Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Database is scale out and not bottleneck it's mysql |
Beta Was this translation helpful? Give feedback.
-
|
Would you mind running a |
Beta Was this translation helpful? Give feedback.
-
|
ov 14 14:12:12 Lookup for 'ANY' of '1908609430.powerdns.com' within zoneID -1 |
Beta Was this translation helpful? Give feedback.
-
While not bad, it means that this backend can only do 645 fully uncached lookups per second. In practice it will be better than that thanks to the record and packet caches, but you need to be aware of this potential bottleneck. |
Beta Was this translation helpful? Give feedback.
-
|
To put this into perspective, this is what I get from a LMDB backend:
which roughly translates to ~170k fully uncached lookups per second. |
Beta Was this translation helpful? Give feedback.
-
|
how to enhance , should i increase mysql specs or use other backend , or apply some kind of indexing |
Beta Was this translation helpful? Give feedback.
-
|
could i migrate to LMDB from mysql |
Beta Was this translation helpful? Give feedback.
-
|
I don't know enough about your use-case to tell you for sure, but if the LMDB backend supports the features you need (see https://doc.powerdns.com/authoritative/backends/index.html) then yes: https://doc.powerdns.com/authoritative/migration.html#migrating-data-from-one-backend-to-another-backend |
Beta Was this translation helpful? Give feedback.
-
|
how can i increase number of threads , as i suspect issue not only in database because i test database from several servers simultaneals and give me same response time |
Beta Was this translation helpful? Give feedback.
-
|
First, you need to debug what kind of traffic is your problem. If it is high volume of "normal" traffic, then your system should be fast due to caching. If the queries are random subdomain attacks, the caches will not help and every single query will trigger one or more DB queries. And this is the bottleneck. |
Beta Was this translation helpful? Give feedback.
-
|
For customers which have 24x7 random subdomain attacks we opted for both: We replaced dnsdist with Knot. Knot gets the zone data (AXFR) from PowerDNS. So we can still use PowerDNS for zone provisioning and doing plenty of tricks, but on the public side we have Knot answering the queries. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks alot Dears, |
Beta Was this translation helpful? Give feedback.
-
Problem:
We are experiencing over 1 million concurrent requests across all VMs, but despite this high query load:
The CPU and memory utilization remain very low.
The maximum QPS (Queries Per Second) for dnsdist is around 10k.
Question:
How can we:
Increase the QPS (queries per second) handled by dnsdist?
Improve the utilization of CPU resources to handle more requests faster and ensure the system can handle this high traffic load effectively?
Environment:
dnsdist setup for load balancing across multiple VMs.
PowerDNS configured as the backend.
Public DNS authoritative server.
We are seeing low CPU/memory utilization, but the DNS query rate is capped at 10k QPS.
What We've Tried:
We have checked and optimized configurations like buffer sizes, worker threads, and query timeouts.
The system scales well vertically in terms of available resources, but QPS still remains limited to ~10k.
Tried adjusting various dnsdist performance and load-balancing settings.
What We're Looking For:
Suggestions for tuning dnsdist to handle more QPS.
CPU optimizations to ensure that we are fully utilizing available CPU cores.
Best practices for configuring dnsdist in high-traffic environments to ensure scalability and improved throughput.
Beta Was this translation helpful? Give feedback.
All reactions