Results 1 to 9 of 9

Thread: SLES 12 network issue

Hybrid View

  1. #1

    Unhappy SLES 12 network issue

    I am having a problem with a Linux guest that I upgraded in place from SLES 11 SP4 to SLES 12 SP3. The problem is that I cannot PuTTY into the server from my network (using a NAT address) nor from my clients' network using the real address. The PuTTY sessions time out. I believe that the request is getting to the SLES 12 system and is being rejected. SUSE Support (I have opened to SR's for this) says that the OS is working fine as I can SSH into this server from a server running on the same z/VM system on the same 10.17.1.156/26 network.

    In order to minimize the application outage time for the PROD server, I decided to upgrade a clone of PROD's boot volume using a new, temporary, server. The upgrade was performed by creating a new virtual server, TEST, and restoring the PROD boot volume from tape (from a clean backup taken while PROD was shut down) to a new UCB address. I used the SLES12 REXX exec and specified a profile called TEST where the only difference from PROD is the IP address. The upgrade went smoothly and took about two hours to complete, with most of the time needed to modify about 25 config files where the upgrade created .rpmnew files.

    When the starter linux system completed and rebooted the server, I was unable to PuTTY into the server using the new IP address. After lots of trial and error I opened an SR with support. They said that since ssh worked from another server, that there was nothing wrong with the SLES 12 system itself, that my network is the problem. After more testing I still thought the problem was with SLES 12 and opened a new SR and referred to the previous one. Support still says that SLES 12 isn't the problem.

    As a test, I shut down a SLES 12 server that I am able PuTTY into (this server was upgraded to SLES 12 in mid-September and is at a different package level than TEST). I then modified /etc/sysconfig/netowrk/ifgcfg-eth0 on TEST and made it look exactly like the file on the server I just shut down. I shut down TEST and IPL'ed the guest (as opposed to rebooting the server). I verified from the console that TEST was using the IP address from the SLES 12 server that I was able to access via PuTTY. I cannot access TEST with PuTTY (using the real IP address from the clients' network, nor from my network using the NAT address).

    I ran a tracert command from a Windows jump server on the client network. After the first hop was displayed by tracert I started seeing the following messages on the console for TEST:
    Code:
    2017-12-04T13:18:49.874194-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:18:49.874220-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:18:53.795374-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:18:53.795398-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:18:57.804186-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:18:57.804212-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:19:01.794178-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:19:01.794194-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:19:05.794200-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:19:05.794226-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:19:09.804197-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:19:09.804220-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..                                                                                                                         
    2017-12-04T13:19:14.114174-06:00 wcs-mf-winxs-db2p kernel: IPv4: martian source 10.17.1.156 from 172.24.8.30, on dev eth0           
    2017-12-04T13:19:14.114201-06:00 wcs-mf-winxs-db2p kernel: ll header: 00000000: 02 00 00 00 02 d6 00 26 51 cc f9 41 08 00        ...
    ....&Q..A..
    The jump server's address is 172.24.8.30 and TEST's IP is 10.17.1.156. This is what leads me to believe that the problem is with the SLES 12 system installed on TEST. Linux is displaying messages showing that the request is reaching the IP stack. Note that this problem exists whether or not SuSEFirewall2 is running. There isn't anything about this request being logged in /var/log/firewall.

    Any ideas on what I need to change to resolve this?


    Harley

  2. #2

    Re: SLES 12 network issue

    Disclaimer: I do not have a Z system, so I'm going to ignore everything
    about it other than reboot/IPL which, to my limited mind, mean similar
    things enough that I'll just probably say "reboot" and hope you can help
    me cover the gap there. :-)

    First, nice reporting of the issue. I appreciate the time it took you to
    write all of that, and think through it, and fight through SRs, etc.

    On to the problem, I think I agree with each of you a bit; clearly some
    packets are reaching your system, so the network outside of the system is
    working correct to get the packets there. To get more information
    regarding what is happening on the wire from this machine's perspective I
    would recommend using tcpdump:

    Code:
    sudo zypper install tcpdump  #in case it is not installed already
    sudo /usr/sbin/tcpdump -n -s 0 -i any port 22
    While that is running, go ahead and try to SSH in from some other system
    that does not work. If you can do it from a system that does work, that
    may be useful too. What we should see are packets going back and forth,
    rather than just going one-way. tcpdump is neat because it works before
    the Linux firewall (Netfilter) interferes, so even if it is blocked by the
    host's firewall (you said it is not) the packets should show up.

    Another test we may be able to do is get more output from SSH during the
    connection. If you have a Linux box that CANNOT SSH in, then use that
    one; just add '-vvv' (three 'v' characters) to get a ton of debugging
    information from the client (passwords and keys are never shown as I recall):

    Code:
    ssh -vvv username@remote.box.goes.here
    Some other useful information for us may be ip and routing information,
    which is simple to get:

    Code:
    ip a  #ip address info
    ip r  #routing information
    ip neigh  #arp/neighbor information
    In this case I m mostly interested in the address and routing information,
    as I wonder if there may be a route missing to allow packets to get back
    to the source properly. Do you have other systems, SLES 11 maybe, that
    can be SSH'd-info, from which we can get the same commands' output? It
    could be useful for comparison purposes if nothing else.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.

  3. #3

    Re: SLES 12 network issue

    Ab,

    For the most part, IPL = boot. Us mainframers still think the old way .

    Here is the tcpdump output from the PuTTY session (fails):
    Code:
    reading from file from-putty.pcap, link-type LINUX_SLL (Linux cooked)
    10:56:57.250880 IP 10.17.1.156.ssh > 10.17.1.163.36438: Flags [P.], seq 3141535555:3141535691, ack 2392637200, win 301, options [nop,nop,TS val 893702 ecr 156803946], length 136
    10:56:57.250909 IP 10.17.1.163.36438 > 10.17.1.156.ssh: Flags [.], ack 136, win 661, options [nop,nop,TS val 156803947 ecr 893702], length 0
    10:57:04.769580 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    10:57:07.773934 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    10:57:13.779932 IP 172.24.8.30.52930 > 10.17.1.156.ssh: Flags [S], seq 3987265288, win 8192, options [mss 1460,nop,nop,sackOK], length 0
    IP 10.17.1.156 is the server that is having the issue.
    IP 10.17.1.163 is the server where I issued 'ssh -X root@10.17.1.156' to get to Linux.
    IP 172.24.8.30 is the Windows server where I attempted to connect via PuTTY.

    Here is the tcpdump output form an SSH connection from another server running on the same VM as the server having the issue"
    Code:
    reading from file from-ssh.pcap, link-type LINUX_SLL (Linux cooked)
    11:12:58.514541 IP 10.17.1.156.ssh > 10.17.1.163.36458: Flags [P.], seq 2483141602:2483141738, ack 1440035107, win 317, options [nop,nop,TS val 989828 ecr 156900073], length 136
    11:12:58.514577 IP 10.17.1.163.36458 > 10.17.1.156.ssh: Flags [.], ack 136, win 471, options [nop,nop,TS val 156900073 ecr 989828], length 0
    11:13:01.403400 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [S], seq 1421146552, win 14520, options [mss 1452,sackOK,TS val 975507037 ecr 0,nop,wscale 3], length 0
    11:13:01.403484 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [S.], seq 3247949623, ack 1421146553, win 28960, options [mss 1460,sackOK,TS val 990117 ecr 975507037,nop,wscale 7], length 0
    11:13:01.403805 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1, win 1815, options [nop,nop,TS val 975507038 ecr 990117], length 0
    11:13:01.404014 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1:24, ack 1, win 1815, options [nop,nop,TS val 975507038 ecr 990117], length 23
    11:13:01.404022 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 24, win 227, options [nop,nop,TS val 990117 ecr 975507038], length 0
    11:13:01.423467 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1:22, ack 24, win 227, options [nop,nop,TS val 990119 ecr 975507038], length 21
    11:13:01.423707 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 0
    11:13:01.423869 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], seq 24:1464, ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 1440
    11:13:01.423876 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1464:1896, ack 22, win 1815, options [nop,nop,TS val 975507040 ecr 990119], length 432
    11:13:01.423878 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 1896, win 272, options [nop,nop,TS val 990119 ecr 975507040], length 0
    11:13:01.424021 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 22:1006, ack 1896, win 272, options [nop,nop,TS val 990119 ecr 975507040], length 984
    11:13:01.432116 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1896:1944, ack 1006, win 2175, options [nop,nop,TS val 975507040 ecr 990119], length 48
    11:13:01.443100 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1006:1286, ack 1944, win 272, options [nop,nop,TS val 990121 ecr 975507040], length 280
    11:13:01.486581 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1286, win 2421, options [nop,nop,TS val 975507046 ecr 990121], length 0
    11:13:03.071856 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1944:1960, ack 1286, win 2421, options [nop,nop,TS val 975507204 ecr 990121], length 16
    11:13:03.111578 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 1960, win 272, options [nop,nop,TS val 990288 ecr 975507204], length 0
    11:13:03.111664 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 1960:2016, ack 1286, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 56
    11:13:03.111672 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 2016, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 0
    11:13:03.111731 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1286:1342, ack 2016, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 56
    11:13:03.111879 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1342, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 0
    11:13:03.111980 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2016:2088, ack 1342, win 2421, options [nop,nop,TS val 975507208 ecr 990288], length 72
    11:13:03.112225 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1342:1414, ack 2088, win 272, options [nop,nop,TS val 990288 ecr 975507208], length 72
    11:13:03.114994 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2088:2176, ack 1414, win 2421, options [nop,nop,TS val 975507209 ecr 990288], length 88
    11:13:03.115578 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1414:1486, ack 2176, win 272, options [nop,nop,TS val 990288 ecr 975507209], length 72
    11:13:03.152534 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1486, win 2421, options [nop,nop,TS val 975507213 ecr 990288], length 0
    11:13:05.543461 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2176:2264, ack 1486, win 2421, options [nop,nop,TS val 975507452 ecr 990288], length 88
    11:13:05.545216 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1486:1542, ack 2264, win 272, options [nop,nop,TS val 990531 ecr 975507452], length 56
    11:13:05.545250 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 1542, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 0
    11:13:05.545268 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2264:2352, ack 1542, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 88
    11:13:05.545415 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1542:1582, ack 2352, win 272, options [nop,nop,TS val 990531 ecr 975507452], length 40
    11:13:05.545547 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2352:2480, ack 1582, win 2421, options [nop,nop,TS val 975507452 ecr 990531], length 128
    11:13:05.549783 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 1582:2534, ack 2480, win 294, options [nop,nop,TS val 990531 ecr 975507452], length 952
    11:13:05.582522 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2534, win 2781, options [nop,nop,TS val 975507456 ecr 990531], length 0
    11:13:05.582544 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2534:2590, ack 2480, win 294, options [nop,nop,TS val 990535 ecr 975507456], length 56
    11:13:05.582695 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2590, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 0
    11:13:05.582742 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 2480:3024, ack 2590, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 544
    11:13:05.583165 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2590:2710, ack 3024, win 317, options [nop,nop,TS val 990535 ecr 975507456], length 120
    11:13:05.584766 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2710:2814, ack 3024, win 317, options [nop,nop,TS val 990535 ecr 975507456], length 104
    11:13:05.584790 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2814, win 2781, options [nop,nop,TS val 975507456 ecr 990535], length 0
    11:13:05.639840 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2814:2902, ack 3024, win 317, options [nop,nop,TS val 990540 ecr 975507456], length 88
    11:13:05.672520 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2902, win 2781, options [nop,nop,TS val 975507465 ecr 990540], length 0
    11:13:06.239160 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3024:3064, ack 2902, win 2781, options [nop,nop,TS val 975507521 ecr 990540], length 40
    11:13:06.239453 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2902:2942, ack 3064, win 317, options [nop,nop,TS val 990600 ecr 975507521], length 40
    11:13:06.239565 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2942, win 2781, options [nop,nop,TS val 975507521 ecr 990600], length 0
    11:13:06.374758 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3064:3104, ack 2942, win 2781, options [nop,nop,TS val 975507535 ecr 990600], length 40
    11:13:06.374877 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2942:2982, ack 3104, win 317, options [nop,nop,TS val 990614 ecr 975507535], length 40
    11:13:06.374996 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 2982, win 2781, options [nop,nop,TS val 975507535 ecr 990614], length 0
    11:13:06.505230 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3104:3144, ack 2982, win 2781, options [nop,nop,TS val 975507548 ecr 990614], length 40
    11:13:06.505344 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 2982:3022, ack 3144, win 317, options [nop,nop,TS val 990627 ecr 975507548], length 40
    11:13:06.505401 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3022, win 2781, options [nop,nop,TS val 975507548 ecr 990627], length 0
    11:13:06.671049 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3144:3184, ack 3022, win 2781, options [nop,nop,TS val 975507564 ecr 990627], length 40
    11:13:06.671151 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3022:3062, ack 3184, win 317, options [nop,nop,TS val 990644 ecr 975507564], length 40
    11:13:06.671205 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3062, win 2781, options [nop,nop,TS val 975507564 ecr 990644], length 0
    11:13:06.799167 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3184:3224, ack 3062, win 2781, options [nop,nop,TS val 975507577 ecr 990644], length 40
    11:13:06.800044 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3062:3230, ack 3224, win 317, options [nop,nop,TS val 990656 ecr 975507577], length 168
    11:13:06.800063 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [P.], seq 3230:3310, ack 3224, win 317, options [nop,nop,TS val 990656 ecr 975507577], length 80
    11:13:06.803578 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3230, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0
    11:13:06.803590 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0
    11:13:06.803616 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3224:3264, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 40
    11:13:06.803629 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [P.], seq 3264:3336, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 72
    11:13:06.803643 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [.], ack 3336, win 317, options [nop,nop,TS val 990657 ecr 975507578], length 0
    11:13:06.803653 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [F.], seq 3336, ack 3310, win 3027, options [nop,nop,TS val 975507578 ecr 990656], length 0
    11:13:06.805795 IP 10.17.1.156.ssh > 10.17.1.139.42251: Flags [F.], seq 3310, ack 3337, win 317, options [nop,nop,TS val 990657 ecr 975507578], length 0
    11:13:06.815075 IP 10.17.1.139.42251 > 10.17.1.156.ssh: Flags [.], ack 3311, win 3027, options [nop,nop,TS val 975507579 ecr 990657], length 0
    IP 10.17.1.156 is the server that is having the issue.
    IP 10.17.1.139 is the server where I issued 'ssh -X root@10.17.1.156' to get to Linux.


    Here is the output from 'ssh -vvv -X root@10.17.1.156'.
    Code:
    OpenSSH_6.6.1, OpenSSL 0.9.8j-fips 07 Jan 2009
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 20: Applying options for *
    debug2: ssh_connect: needpriv 0
    debug1: Connecting to 10.17.1.156 [10.17.1.156] port 22.
    debug1: Connection established.
    debug1: permanently_set_uid: 0/0
    debug1: identity file /root/.ssh/id_rsa type -1
    debug1: identity file /root/.ssh/id_rsa-cert type -1
    debug1: identity file /root/.ssh/id_dsa type -1
    debug1: identity file /root/.ssh/id_dsa-cert type -1
    debug1: identity file /root/.ssh/id_ecdsa type -1
    debug1: identity file /root/.ssh/id_ecdsa-cert type -1
    debug1: identity file /root/.ssh/id_ed25519 type -1
    debug1: identity file /root/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.6.1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2
    debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
    debug2: fd 3 setting O_NONBLOCK
    debug3: load_hostkeys: loading entries for host "10.17.1.156" from file "/root/.ssh/known_hosts"
    debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:1
    debug3: load_hostkeys: loaded 1 keys
    debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
    debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit: first_kex_follows 0
    debug2: kex_parse_kexinit: reserved 0
    debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
    debug2: kex_parse_kexinit: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
    debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
    debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
    debug2: kex_parse_kexinit: none,zlib@openssh.com
    debug2: kex_parse_kexinit: none,zlib@openssh.com
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit: first_kex_follows 0
    debug2: kex_parse_kexinit: reserved 0
    debug2: mac_setup: setup hmac-sha1-etm@openssh.com
    debug1: kex: server->client aes128-ctr hmac-sha1-etm@openssh.com none
    debug2: mac_setup: setup hmac-sha1-etm@openssh.com
    debug1: kex: client->server aes128-ctr hmac-sha1-etm@openssh.com none
    debug1: sending SSH2_MSG_KEX_ECDH_INIT
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ECDSA 97:05:65:80:55:a8:c5:23:de:c5:57:16:f0:ea:28:ed [MD5]
    debug3: load_hostkeys: loading entries for host "10.17.1.156" from file "/root/.ssh/known_hosts"
    debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:1
    debug3: load_hostkeys: loaded 1 keys
    debug1: Host '10.17.1.156' is known and matches the ECDSA host key.
    debug1: Found key in /root/.ssh/known_hosts:1
    debug1: ssh_ecdsa_verify: signature correct
    debug2: kex_derive_keys
    debug2: set_newkeys: mode 1
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug2: set_newkeys: mode 0
    debug1: SSH2_MSG_NEWKEYS received
    debug1: Roaming not allowed by server
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug2: key: /root/.ssh/id_rsa ((nil)),
    debug2: key: /root/.ssh/id_dsa ((nil)),
    debug2: key: /root/.ssh/id_ecdsa ((nil)),
    debug2: key: /root/.ssh/id_ed25519 ((nil)),
    debug1: Authentications that can continue: publickey,keyboard-interactive
    debug3: start over, passed a different list publickey,keyboard-interactive
    debug3: preferred publickey,keyboard-interactive,password
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive,password
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: /root/.ssh/id_rsa
    debug3: no such identity: /root/.ssh/id_rsa: No such file or directory
    debug1: Trying private key: /root/.ssh/id_dsa
    debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ecdsa
    debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
    debug1: Trying private key: /root/.ssh/id_ed25519
    debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
    debug2: we did not send a packet, disable method
    debug3: authmethod_lookup keyboard-interactive
    debug3: remaining preferred: password
    debug3: authmethod_is_enabled keyboard-interactive
    debug1: Next authentication method: keyboard-interactive
    debug2: userauth_kbdint
    debug2: we sent a keyboard-interactive packet, wait for reply
    debug2: input_userauth_info_req
    debug2: input_userauth_info_req: num_prompts 1
    Password:
    debug3: packet_send2: adding 32 (len 18 padlen 14 extra_pad 64)
    debug2: input_userauth_info_req
    debug2: input_userauth_info_req: num_prompts 0
    debug3: packet_send2: adding 48 (len 6 padlen 10 extra_pad 64)
    debug1: Authentication succeeded (keyboard-interactive).
    Authenticated to 10.17.1.156 ([10.17.1.156]:22).
    debug1: channel 0: new [client-session]
    debug3: ssh_session2_open: channel_new: 0
    debug2: channel 0: send open
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug2: callback start
    debug2: fd 3 setting TCP_NODELAY
    debug3: packet_set_tos: set IP_TOS 0x10
    debug2: client_session2_setup: id 0
    debug2: channel 0: request pty-req confirm 1
    debug1: Sending environment.
    debug3: Ignored env LESSKEY
    debug3: Ignored env NNTPSERVER
    debug3: Ignored env INFODIR
    debug3: Ignored env MANPATH
    debug3: Ignored env HOSTNAME
    debug3: Ignored env XKEYSYMDB
    debug3: Ignored env HOST
    debug3: Ignored env SHELL
    debug3: Ignored env TERM
    debug3: Ignored env PROFILEREAD
    debug3: Ignored env HISTSIZE
    debug3: Ignored env MORE
    debug3: Ignored env USER
    debug3: Ignored env LS_COLORS
    debug3: Ignored env XNLSPATH
    debug3: Ignored env ENV
    debug3: Ignored env HOSTTYPE
    debug3: Ignored env FROM_HEADER
    debug3: Ignored env PAGER
    debug3: Ignored env CSHEDIT
    debug3: Ignored env XDG_CONFIG_DIRS
    debug3: Ignored env MINICOM
    debug3: Ignored env MAIL
    debug3: Ignored env PATH
    debug3: Ignored env CPU
    debug3: Ignored env INPUTRC
    debug3: Ignored env PWD
    debug1: Sending env LANG = POSIX
    debug2: channel 0: request env confirm 0
    debug3: Ignored env PYTHONSTARTUP
    debug3: Ignored env QT_SYSTEM_DIR
    debug3: Ignored env SHLVL
    debug3: Ignored env HOME
    debug3: Ignored env LESS_ADVANCED_PREPROCESSOR
    debug3: Ignored env OSTYPE
    debug3: Ignored env LS_OPTIONS
    debug3: Ignored env XCURSOR_THEME
    debug3: Ignored env WINDOWMANAGER
    debug3: Ignored env LESS
    debug3: Ignored env MACHTYPE
    debug3: Ignored env LOGNAME
    debug3: Ignored env XDG_DATA_DIRS
    debug1: Sending env LC_CTYPE = en_US.UTF-8
    debug2: channel 0: request env confirm 0
    debug3: Ignored env LESSOPEN
    debug3: Ignored env INFOPATH
    debug3: Ignored env LESSCLOSE
    debug3: Ignored env G_BROKEN_FILENAMES
    debug3: Ignored env COLORTERM
    debug3: Ignored env _
    debug2: channel 0: request shell confirm 1
    debug2: callback done
    debug2: channel 0: open confirm rwindow 0 rmax 32768
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: PTY allocation request accepted on channel 0
    debug2: channel 0: rcvd adjust 2097152
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: shell request accepted on channel 0
    Last login: Tue Dec  5 11:16:48 2017 from 10.17.1.139
    wcs-mf-winxs-db2p:~ # exitdebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
    debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
    debug2: channel 0: rcvd eow
    debug2: channel 0: close_read
    debug2: channel 0: input open -> closed
    debug2: channel 0: rcvd eof
    debug2: channel 0: output open -> drain
    debug2: channel 0: rcvd close
    debug3: channel 0: will not send data after close
    
    logout
    debug3: channel 0: will not send data after close
    debug2: channel 0: obuf empty
    debug2: channel 0: close_write
    debug2: channel 0: output drain -> closed
    debug2: channel 0: almost dead
    debug2: channel 0: gc: notify user
    debug2: channel 0: gc: user detached
    debug2: channel 0: send close
    debug2: channel 0: is dead
    debug2: channel 0: garbage collecting
    debug1: channel 0: free: client-session, nchannels 1
    debug3: channel 0: status: The following connections are open:
      #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
    
    Connection to 10.17.1.156 closed.
    Transferred: sent 2952, received 2868 bytes, in 3.4 seconds
    Bytes per second: sent 870.7, received 845.9
    debug1: Exit status 0
    From the server having the issue:
    Code:
    wcs-mf-winxs-db2p:/tmp # ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 02:00:00:00:02:d9 brd ff:ff:ff:ff:ff:ff
        inet 10.17.1.156/26 brd 10.17.1.191 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::ff:fe00:2d9/64 scope link
           valid_lft forever preferred_lft forever
    wcs-mf-winxs-db2p:/tmp # ip r
    10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.156
    wcs-mf-winxs-db2p:/tmp # ip neigh
    10.17.1.139 dev eth0 lladdr 02:00:00:00:00:24 STALE
    10.17.1.189 dev eth0 lladdr 00:22:55:7a:12:c1 STALE
    10.17.1.190 dev eth0 lladdr 00:26:51:cc:f9:41 STALE
    10.17.1.163 dev eth0 lladdr 02:00:00:00:02:74 DELAY
    From a server on the same hardware and same network that does not have the issue"
    Code:
    wcs-mf-smt:~ # ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
        inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN qlen 1000
        link/ether 02:00:00:00:00:24 brd ff:ff:ff:ff:ff:ff
        inet 10.17.1.139/26 brd 10.17.1.191 scope global eth0
        inet6 fe80::ff:fe00:24/64 scope link
           valid_lft forever preferred_lft forever
    wcs-mf-smt:~ # ip r
    default via 10.17.1.129 dev eth0
    10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.139
    127.0.0.0/8 dev lo  scope link
    169.254.0.0/16 dev eth0  scope link
    wcs-mf-smt:~ # ip neigh
    10.17.1.129 dev eth0 lladdr 00:00:0c:9f:f2:00 REACHABLE

    Harley

  4. #4

    Re: SLES 12 network issue

    Unless I am missing something, your broken box does not have a default
    route. Your 'ip r' information shows one route, for the local network,
    but that does not help at all when you want to get packets from here to
    any other box off the local network, including back to where packets
    arriving from there originated.

    Compare the output from your broken and working boxes, respectively:

    Broken:

    Code:
    wcs-mf-winxs-db2p:/tmp # ip r
    10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.156
    Working:

    Code:
    wcs-mf-smt:~ # ip r
    default via 10.17.1.129 dev eth0
    10.17.1.128/26 dev eth0  proto kernel  scope link  src 10.17.1.139
    127.0.0.0/8 dev lo  scope link
    169.254.0.0/16 dev eth0  scope link
    Try adding the default route of 100.17.1.129 to your broken box. On non-Z
    systems I do this with 'yast lan', like this:

    Code:
    sudo /sbin/yast lan
    In there you should see a tab for routing, and in there is a spot for an
    IPv4 default gateway. Put in the IP address, save/Finish, and then test
    again.


    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.

  5. #5

    Re: SLES 12 network issue

    I think that you mis-typed as there isn't a 100.17.1.129 it is 10.17.1.129.

    /etc/sysconfig/network/routes contains
    Code:
    default 10.17.1.129 - -
    On the Routing tab of 'yast lan':
    Default IPv4 Gateway is set to 10.17.1.129.
    Default IPv6 Gateway is blank.
    The Routing Table box is empty.
    Enable IPv4 Forwarding is unchecked.
    Enable IPv6 Forwarding is unchecked.

    Not sure where to make the change you suggested.

  6. #6

    Re: SLES 12 network issue

    Ab,

    I was also working this issue with the Network team here (the router and firewall folks). One of them found that the default gateway was not being picked up from /etc/sysconfig/network/routes. He had me run 'netstat -rnv' and it displayed:

    Code:
    wcs-mf-winxs-db2p:/tmp # netstat -rnv
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    10.17.1.128     0.0.0.0         255.255.255.192 U         0 0          0 eth0
    He then had me temporarily add a gateway by issuing 'route add default gw 10.17.1.129'. I was then able to PuTTY into the server.

    Any idea how to make this permanent (i.e. without coding a script to issue the command at boot time)? I will continue to research how to accomplish that.


    Harley

  7. #7

    Re: SLES 12 network issue

    Yes, exactly; I'm glad the same conclusion was reached.

    It should already be permanent, per the configuration files, but it seems
    the networking configuration or stack is a bit off. I have seen routes
    NOT-add properly when the required NIC was not yet fully up, but I presume
    you are using static IP addresses, so that should be unlikely. I've also
    seen odd things happen when there was a duplicate IP on the network
    somewhere, as the system may try to avoid conflicts for you since those
    are generally bad; any chance of that situation happening lately?

    Do you see anything useful when trying to get the status of the network
    service?

    Code:
    sudo systemctl status network
    For fun, while on the box physically-ish, could you restart your
    networking to see if it comes up properly now that the rest of the system
    is already working/booted?

    Code:
    sudo systemctl restart network
    As a note, while it is probably generally fine to do, I've seldom worked
    with the files under /etc/sysconfig/network/ directly, and would generally
    recommend using Yast to configure things when doing so manually. Yast is
    pretty robust, particularly around networking, and uses those files
    itself; since I have had great luck using Yast, and seldom any problems
    like this, it makes me wonder if some of the magical bits may be unhappy
    with your changes for some pesky reason. To be fair, you already know I
    am not a Z-series expert, so maybe issues there happen more-often in these
    cases and all of my experience is for naught.

    The route command basically shows the same thing as the 'ip' command, so
    that's good, and adding a route manually via ip (or route) does what the
    system should be doing behind the scenes. Hopefully we can find something
    in the logs (above) that will help us figure things out.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.

  8. #8

    Re: SLES 12 network issue

    Ab,

    Your asking for info from 'sudo systemctl status network' showed me what my problem is.

    In order to clone the disk for the server I needed to have it come up with a new IP address as the 'real' server is up an running. I saved a copy of ifcfg-eth0 in /etc/sysconfig/network and called it ifcfg-eth0.save.
    Code:
    wcs-mf-winxs-db2p:~ # systemctl status network
     wicked.service - wicked managed network interfaces
       Loaded: loaded (/usr/lib/systemd/system/wicked.service; enabled; vendor preset: disabled)
       Active: active (exited) since Wed 2017-12-06 07:53:35 CST; 14s ago
      Process: 1884 ExecStart=/usr/sbin/wicked --systemd ifup all (code=exited, status=0/SUCCESS)
     Main PID: 1884 (code=exited, status=0/SUCCESS)
        Tasks: 0 (limit: 512)
       CGroup: /system.slice/wicked.service
    
    Dec 06 07:53:04 wcs-mf-winxs-db2p systemd[1]: Starting wicked managed network interfaces...
    Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: lo              up
    Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: eth0            up
    Dec 06 07:53:35 wcs-mf-winxs-db2p wicked[1884]: eth0.save       no-device
    Dec 06 07:53:35 wcs-mf-winxs-db2p systemd[1]: Started wicked managed network interfaces.
    Seeing the 'eth0.save' in the list rang a bell where (about 10 years ago) we had weird network issues upgrading to SLES 10 when we named a saved copy as filename.save. I renamed ifcfg-eth0.save to save.ifcfg-eth0, rebooted, and tested. I was able to PuTTY in without issuing the temporary command to define the gateway.

    Thank you very much for your assistance with this.


    Harley

  9. #9

    Re: SLES 12 network issue

    Well that's just great! Thank-you for posting back how you found the
    issue as that is one worth remembering, and since memories are imperfect,
    this is worth indexing via Google.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •