Hi folks, I've got a Dell R320 running CentOS 7, and a 10G NIC. I'm running a DNS server on it, for testing. As part of my testing, I'm attempting to capture all the DNS queries arriving on the server, using tcpdump. However, tcpdump's performance is abysmal, and it loses lots of the packets. Here's the output when I stop the capture: # tcpdump -nn -i p1p1 -w /dev/shm/queries.pcap -B 65536 dst host 193.0.14.129 tcpdump: listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes ^C4096558 packets captured 6882162 packets received by filter 2785604 packets dropped by kernel The number 6882162 is exactly the number of DNS queries I am sending from another server (the source). The filter is seeing them. However, not all of them make it into the pcap file. First I was writing to the disk, and thought it might be a bottleneck, so I tried to write to /dev/shm. You'll see I've also increased the buffer size with -B, but none of this helps. The packet rate is also not that high. From the sending side, this is what I have: # tcpreplay -i qtx:p1p1 5min.pcap Actual: 6882162 packets (620533334 bytes) sent in 317.08 seconds. Rated: 1951900.0 Bps, 15.61 Mbps, 21648.97 pps Flows: 4254416 flows, 13382.96 fps, 6882162 flow packets, 0 non-flow This is a very modest rate. Any ideas why tcpdump loses so many packets? And how to make it do better? Regards, Anand
On 08/14/2016 03:20 AM, Anand Buddhdev wrote:> The number 6882162 is exactly the number of DNS queries I am sending > from another server (the source). The filter is seeing them. However, > not all of them make it into the pcap file.Try specifying "ether host <src mac>" and compare the pcap files. How are you counting the number of packets in the pcap file?
Capturing 10 gigabit traffic with no packet loss at line speed is difficult at best. Make sure that you've configured the IRQ affinity properly on the sending and receiving sides to start. On Aug 14, 2016 11:52 AM, "Gordon Messmer" <gordon.messmer at gmail.com> wrote:> On 08/14/2016 03:20 AM, Anand Buddhdev wrote: > >> The number 6882162 is exactly the number of DNS queries I am sending >> from another server (the source). The filter is seeing them. However, >> not all of them make it into the pcap file. >> > > Try specifying "ether host <src mac>" and compare the pcap files. How are > you counting the number of packets in the pcap file? > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >
On 14/08/16 12:20, Anand Buddhdev wrote: Hi folks, I've discovered something. See below:> The packet rate is also not that high. From the sending side, this is > what I have: > > # tcpreplay -i qtx:p1p1 5min.pcapIf I send packets without qtx, like this: tcpreplay -i p1p1 5min.pcap then tcpdump on the receiving box has no problem, and keeps up happily with the queries, and receives all of them into the pcap file. It seems like the qtx module is somehow interfering with the packet capture, but I don't know how or why yet. Anyway, for low packet rates, such as 20,000 q/s, qtx isn't necessary, so I will not use it when I want to do packet captures. Regards, Anand
Hello, I've found it is helpful to limit the length of the packet you are capturing by using something like -s 256. On 08/14/2016 06:04 PM, Anand Buddhdev wrote:> On 14/08/16 12:20, Anand Buddhdev wrote: > > Hi folks, > > I've discovered something. See below: > >> The packet rate is also not that high. From the sending side, this is >> what I have: >> >> # tcpreplay -i qtx:p1p1 5min.pcap > If I send packets without qtx, like this: > > tcpreplay -i p1p1 5min.pcap > > then tcpdump on the receiving box has no problem, and keeps up happily > with the queries, and receives all of them into the pcap file. It seems > like the qtx module is somehow interfering with the packet capture, but > I don't know how or why yet. > > Anyway, for low packet rates, such as 20,000 q/s, qtx isn't necessary, > so I will not use it when I want to do packet captures. > > Regards, > Anand > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >-- Stephen Clark *NetWolves Managed Services, LLC.* Director of Technology Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.clark at netwolves.com http://www.netwolves.com
> On Aug 14, 2016, at 6:20 AM, Anand Buddhdev <anandb at ripe.net> wrote: > > Any ideas why tcpdump loses so many packets?Saw your nanog posts... How many RX queues are configured? What does 'ethtool -S p1p1' show? Any discarded packets in the RX queue(s)?