TAP interface misses packets (mostly ICMP)

[acsplayon]

I have been fighting/investigating this issue for about 3 days and finally figured it out. I found that either using the 'dispatcher' call, or through viface::receive() many packets were missed. They were always visible using tcpdump, wireshark, and iptables logging, but my user-space application would only receive mostly multicast and/or broadcast messages (such as ARP, multicast, and UDP broadcast messages) sent from the host. Trying to 'ping' another virtual device attached to the userspace program (with it's own MAC and IP), I would receive the ARP asking for the mac, and the return would be sent and accepted by the host (device was shown in the arp table). However, the ping command would then start sending ICMP echo request packets to the interface, and they would not be received using either receive() or dispatch functionality. After days of pulling out my hair, messing with iptables rules, rp_filters, the 'sniffer' from libtins, etc. I finally decided to modifiy 'receive' to read from both rx and tx queues and return any packet pending. I now receive all packets.

receive() was modified to this:

int e;
    // Read packet into our buffer
    int nread = read(this->queues.rx, &(this->pktbuff[0]), this->mtu);
    e=errno;

// viface issue.  Kernel writes incoming packets to BOTH queues, so we
// need to read from both
    if (((nread<0) && (e==EAGAIN)) || (nread==0))
    {
        nread = read(this->queues.tx, &(this->pktbuff[0]), this->mtu);
        e=errno;
    }
    // Handle errors
    if (nread == -1) {

I will have to also modify the 'dispatch' reader to add the tx queue file descriptor to the select.
I am not a 'git' guy, so I don't want to have to figure out how to merge my changes in, just wanted to let you know.

I believe this effects ICMP messages the most because it seems the kernel tries to add out-of-band (high priority) messages to the tx queue and not the standard read queue.

[carlos-jenkins]

Thank you very much for this debug, it is indeed very peculiar. I would like to investigate and understand better why this is happening, and if changing to reading from both queues would have other consequences. Again thank you.

[mwiniarski]

I experienced the exact same bug but on TUN interfaces. Half of different types of packets were missing.

After changing to only one queue, as @acsplayon suggested (for both sending and receiving) everything works well.