Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: Mystery RX packet drops on SLES11 SP2 every 30 sec

Hybrid View

  1. #1

    Mystery RX packet drops on SLES11 SP2 every 30 sec

    We have deployed several SP2 servers for testing, and are finding this annoying issue.
    Approx every 30 seconds, the RX dropped counter ticks up by 1. This is only happening on SLES11 SP2 systems.
    Usually when this counter ticks, it lines up with rx_fw_discards in ethtool statistics. I cannot find a matching stat for this drop.

    eth0 Link encap:Ethernet HWaddr 00:21:9B:A0:07:7E
    inet addr:192.168.69.247 Bcast:192.168.69.255 Mask:255.255.255.0
    inet6 addr: fe80::221:9bff:fea0:77e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:61591 errors:0 dropped:276 overruns:0 frame:0
    TX packets:38438099 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:39053090 (37.2 Mb) TX bytes:49909260932 (47597.1 Mb)
    Interrupt:36 Memory:d2000000-d2012800

    NIC statistics:
    rx_bytes: 42672588
    rx_error_bytes: 0
    tx_bytes: 55266311872
    tx_error_bytes: 0
    rx_ucast_packets: 32153
    rx_mcast_packets: 31988
    rx_bcast_packets: 4239
    tx_ucast_packets: 37456
    tx_mcast_packets: 42720893
    tx_bcast_packets: 4
    tx_mac_errors: 0
    tx_carrier_errors: 0
    rx_crc_errors: 0
    rx_align_errors: 0
    tx_single_collisions: 0
    tx_multi_collisions: 0
    tx_deferred: 0
    tx_excess_collisions: 0
    tx_late_collisions: 0
    tx_total_collisions: 0
    rx_fragments: 0
    rx_jabbers: 0
    rx_undersize_packets: 0
    rx_oversize_packets: 0
    rx_64_byte_packets: 3667
    rx_65_to_127_byte_packets: 22572
    rx_128_to_255_byte_packets: 16761
    rx_256_to_511_byte_packets: 353
    rx_512_to_1023_byte_packets: 70
    rx_1024_to_1522_byte_packets: 24957
    rx_1523_to_9022_byte_packets: 0
    tx_64_byte_packets: 56728
    tx_65_to_127_byte_packets: 34705
    tx_128_to_255_byte_packets: 1478329
    tx_256_to_511_byte_packets: 2166944
    tx_512_to_1023_byte_packets: 3321491
    tx_1024_to_1522_byte_packets: 35700156
    tx_1523_to_9022_byte_packets: 0
    rx_xon_frames: 0
    rx_xoff_frames: 0
    tx_xon_frames: 0
    tx_xoff_frames: 0
    rx_mac_ctrl_frames: 0
    rx_filtered_packets: 45678
    rx_ftq_discards: 0
    rx_discards: 0
    rx_fw_discards: 0

    driver: bnx2
    version: 2.1.11
    firmware-version: 6.4.5 bc 5.2.3 NCSI 2.0.11
    bus-info: 0000:01:00.0
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes

    I have tried the latest driver 2.2.11j from broadcom with same results. Any ideas what might tick this counter?

  2. #2

    Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    I can confirm this behaviour... also within ESXi5 VMs, so it does not depend on the NIC (we use vmxnet3 NICs).
    Anyway, one dropped packet every ~30 seconds seems not to bother anything.

  3. Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by enovaklbank View Post
    I can confirm this behaviour... also within ESXi5 VMs, so it does not depend on the NIC (we use vmxnet3 NICs).
    Anyway, one dropped packet every ~30 seconds seems not to bother anything.
    Well it is a pain because we monitor these counts for legitimate drops. There has to be a reason why this is happening.

  4. #4

    Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by kevins7189_5 View Post
    Well it is a pain because we monitor these counts for legitimate drops. There has to be a reason why this is happening.
    I've been seeing this too with a couple of SLES 11 SP2 Xen setups. I was planning on setting up a small (network) isolated test server to see if it happens there too.

    Trying to orientate how to find out what the server is dropping, I came across tool called dropwatch (http://linux.die.net/man/1/dropwatch). Not sure if it will run on SLES 11, but there are some packages for it here : http://pkgs.org/download/dropwatch

    How are you guys planning to go about finding whats getting dropped?

    Cheers,
    Willem
    Knowledge Partner (voluntary sysop)
    ---
    If you find a post helpful and are logged into the web interface,
    please show your appreciation and click on the star below it. Thanks!

  5. #5

    Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by Magic31 View Post
    Trying to orientate how to find out what the server is dropping, I came across tool called dropwatch (http://linux.die.net/man/1/dropwatch). Not sure if it will run on SLES 11, but there are some packages for it here : http://pkgs.org/download/dropwatch

    Ah, even better - an OBS build for SLES 11 SP2 : https://build.opensuse.org/package/r...er%3Adropwatch (as I also don't know how trustworthy pkgs.org downloads are)
    Knowledge Partner (voluntary sysop)
    ---
    If you find a post helpful and are logged into the web interface,
    please show your appreciation and click on the star below it. Thanks!

  6. Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by Magic31 View Post
    Ah, even better - an OBS build for SLES 11 SP2 : https://build.opensuse.org/package/r...er%3Adropwatch (as I also don't know how trustworthy pkgs.org downloads are)
    Interesting utility.
    I was able to get this every time I saw the counter move up about every 30 sec. No idea what it means

    1 drops at __netif_receive_skb+1fe (0xffffffff8138388e)

  7. #7

    Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by kevins7189_5 View Post
    Interesting utility.
    I was able to get this every time I saw the counter move up about every 30 sec. No idea what it means

    1 drops at __netif_receive_skb+1fe (0xffffffff8138388e)
    Interesting indeed

    I haven't been able to put more time into this that trying to simulate it in a (very small) test environment. Funny enough, I did not witness any dropped packets there.

    My next move is to do this at the two sites where I am seeing this, but I have not had a window to do so yet (time and other priorities).


    If you can open an SR, that would be best. It could well be certain type packets are intentionally for some security reason or other. I don't know, by far, enough about the workings of the kernel and modules... but this snip talken from a Google search did catch my interest:

    * Add a packet_type handler and see if we can prevent
    * other packet_type's from handling an skb
    * Specifically, we will register our packet_type to be
    * the first handler invoked by netif_receive_skb()
    * If the packet received meets certain conditions, then,
    * drop it, i.e, prevent subsequent ptype_all and ptype_base
    * handlers in netif_receive_skb() from processing the packet


    Sorry I can't be of more help here. I will pass on this thread to my Novell contact to see if this might be something Novell is aware of.

    -Willem
    Knowledge Partner (voluntary sysop)
    ---
    If you find a post helpful and are logged into the web interface,
    please show your appreciation and click on the star below it. Thanks!

  8. #8

    Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by Magic31 View Post
    ...I will pass on this thread to my Novell contact to see if this might be something Novell is aware of.
    I should have asked earlier! : http://www.novell.com/support/kb/doc.php?id=7007165

    There you go (and me too).

    Cheers,
    Willem
    Knowledge Partner (voluntary sysop)
    ---
    If you find a post helpful and are logged into the web interface,
    please show your appreciation and click on the star below it. Thanks!

  9. Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    Quote Originally Posted by Magic31 View Post
    I should have asked earlier! : http://www.novell.com/support/kb/doc.php?id=7007165

    There you go (and me too).

    Cheers,
    Willem
    that reply seems kinda cop-out ish. Seems like this would be an easy to find problem if this has been happening since 2.32.37, but can't find any so easy.

    Have no drops here, so that one is out
    cat /proc/net/softnet_stat
    03619a59 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    035d90a8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    03609a2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    035dce37 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    Last edited by kevins7189_5; 13-Aug-2012 at 17:24.

  10. Re: Mystery RX packet drops on SLES11 SP2 every 30 sec

    the other issue is this goes away if I run tcpdump on the machine and try to catch the packets. I can go hours an hours with tcpdump running and the rx counter won't move, but soon after shutting it down the rx counters start incrementing again.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •