PDA

View Full Version : Unknown Outage



carnold6
21-May-2014, 00:18
SLES11 SP2. How can i find out if there is some "rogue" piece on my SLES11 install. Heres what we have happening:

With the server plugged in, it appears as the server is flooding the network where we lose intranet and internet. I have narrowed it down to the server. I also installed a different PCI NIC and the same thing happens. I installed clamAV but unable to update the definitions due to not being able to plug in to the network. How can i tell if there is something "rogue" in the SLES install?

jmozdzen
21-May-2014, 09:50
Hi carnold6,

SLES11 SP2. How can i find out if there is some "rogue" piece on my SLES11 install. Heres what we have happening:

With the server plugged in, it appears as the server is flooding the network where we lose intranet and internet. I have narrowed it down to the server. I also installed a different PCI NIC and the same thing happens. I installed clamAV but unable to update the definitions due to not being able to plug in to the network. How can i tell if there is something "rogue" in the SLES install?

to see if anything was tampered with, run "rpm -V" to verify all installed files. But beware, I've seen reports of malicious software that updates the RPM database with new checksums so *those* modifications will not be caught that way.

Have you had a look at what packets are floodiing the network? You might want to run a short "tcpdump" to get an impression of what is going on, that way you'll might get a hint at what software to look at.

Next thing then, if you suspect malicious or ill-behaving software, is a look at "lsof -i4" and "lsof -i6" to see which applications have open ports via IPv4 and IPv6, which may be helpful, too.

Further steps then depend on what damage had been done.

With regards,
Jens

carnold6
22-May-2014, 02:11
Hi carnold6,


to see if anything was tampered with, run "rpm -V" to verify all installed files. But beware, I've seen reports of malicious software that updates the RPM database with new checksums so *those* modifications will not be caught that way.

I am not sure how to parse this but here is the output:
rpm -V -a
S.5....T c /etc/modprobe.conf
..5....T c /etc/modprobe.d/unsupported-modules
S.5....T c /etc/ssl/openssl.cnf
.......T c /etc/pki/nssdb/pkcs11.txt
S.5....T c /etc/postfix/main.cf
S.5....T c /etc/postfix/master.cf
S.5....T c /etc/rsyncd.conf
S.5....T c /etc/rsyncd.secrets
.......T c /usr/share/fonts/100dpi/fonts.dir
.......T c /usr/share/fonts/100dpi/fonts.scale
S.5....T c /usr/share/fonts/Speedo/fonts.dir
S.5....T c /usr/share/fonts/Speedo/fonts.scale
S.5....T c /usr/share/fonts/Type1/fonts.dir
S.5....T c /usr/share/fonts/Type1/fonts.scale
.......T c /usr/share/fonts/cyrillic/fonts.dir
.......T c /usr/share/fonts/cyrillic/fonts.scale
S.5....T c /usr/share/fonts/truetype/fonts.dir
S.5....T c /usr/share/fonts/truetype/fonts.scale
..5....T c /etc/inittab
S.5....T c /usr/share/sax/api/data/cdb/Monitors
.......T /usr/lib64/xulrunner-1.9.1.19/.autoreg
.......T c /usr/lib64/gconv/gconv-modules.cache
S.5....T c /etc/clamd.conf
S.5....T c /etc/sysctl.conf
S.5....T c /etc/ipsec.conf
.M...... /etc/ipsec.d/private
S.5....T c /etc/ipsec.secrets
S.5....T c /etc/strongswan.conf
S.5....T c /var/lib/sfcb/registration/providerRegister
S.5....T c /etc/pam.d/login
S.5....T c /etc/xinetd.d/swat
S.5....T c /usr/share/fonts/encodings/encodings.dir
S.5....T c /usr/share/fonts/misc/fonts.dir
.......T c /usr/share/fonts/misc/fonts.scale
missing /usr/bin/chattr
missing /usr/bin/lsattr
S.5....T c /etc/X11/xdm/xdm-config
.......T /var/lib/misc/PolicyKit.reload
S.5....T c /etc/crontab
/etc/crontab should be root:root 0644. (wrong permissions 0755)
SM5...GT c /etc/samba/smb.conf
S.5....T c /etc/fonts/suse-font-dirs.conf
S.5....T c /etc/insserv.conf
....L... c /etc/pam.d/common-account
....L... c /etc/pam.d/common-auth
....L... c /etc/pam.d/common-password
....L... c /etc/pam.d/common-session
S.5....T c /etc/xinetd.d/vnc
S.5...GT /etc/VRTSralus/ralus.cfg
......G. /opt/VRTSralus/bin/AgentConfig
......G. /opt/VRTSralus/bin/VRTSralus.init
......G. /opt/VRTSralus/bin/bediag
......G. /opt/VRTSralus/bin/begather
......G. /opt/VRTSralus/bin/beremote
......G. /opt/VRTSralus/bin/crbclnt
......G. /opt/VRTSralus/bin/hct
......G. /opt/VRTSralus/bin/libDeviceIo.so
......G. /opt/VRTSralus/bin/libbe_util.so
......G. /opt/VRTSralus/bin/libbebsdu.so
......G. /opt/VRTSralus/bin/libbecluster.so
......G. /opt/VRTSralus/bin/libbedscomn.so
......G. /opt/VRTSralus/bin/libbedsedir.so
......G. /opt/VRTSralus/bin/libbedsrman.so
......G. /opt/VRTSralus/bin/libbedssms.so
......G. /opt/VRTSralus/bin/libbedssmsp.so
......G. /opt/VRTSralus/bin/libbedsvx.so
......G. /opt/VRTSralus/bin/libbeengsvr.so
......G. /opt/VRTSralus/bin/libbenetapi.so
......G. /opt/VRTSralus/bin/libbenetns.so
......G. /opt/VRTSralus/bin/libbenettcp.so
......G. /opt/VRTSralus/bin/libbenetutl.so
......G. /opt/VRTSralus/bin/libbeobk.so
......G. /opt/VRTSralus/bin/libbescsicap.so
......G. /opt/VRTSralus/bin/libbesocket.so
......G. /opt/VRTSralus/bin/libbestdutl.so
......G. /opt/VRTSralus/bin/libbetools.so
......G. /opt/VRTSralus/bin/libbsafs.so
......G. /opt/VRTSralus/bin/libcrbproxy.so
......G. /opt/VRTSralus/bin/libdbsb.so
......G. /opt/VRTSralus/bin/libdbsbora.so
......G. /opt/VRTSralus/bin/libeng_dsss.so
......G. /opt/VRTSralus/bin/libndmp_dsss.so
......G. /opt/VRTSralus/bin/libndmp_loops.so
......G. /opt/VRTSralus/bin/libndmp_tpfmt.so
......G. /opt/VRTSralus/bin/libndmpclient.so
......G. /opt/VRTSralus/bin/libndmpcomm.so
......G. /opt/VRTSralus/bin/libndmpsrvr.so
......G. /opt/VRTSralus/bin/libnrdscommon.so
......G. /opt/VRTSralus/bin/libnrdsmessage.so
......G. /opt/VRTSralus/bin/libnrdsnet.so
......G. /opt/VRTSralus/bin/libnrdsparameter.so
......G. /opt/VRTSralus/bin/libobk.so
......G. /opt/VRTSralus/bin/libsmstools.so
......G. /opt/VRTSralus/bin/libsth.so
......G. /opt/VRTSralus/bin/libsthapi.so
......G. /opt/VRTSralus/bin/libsthtar.so
......G. /opt/VRTSralus/bin/libsts.so
......G. /opt/VRTSralus/bin/libstsapi.so
......G. /opt/VRTSralus/bin/libstsimage.so
......G. /opt/VRTSralus/bin/libstspisimdisk.so
......G. /opt/VRTSralus/bin/libtapesrvr.so
......G. /opt/VRTSralus/bin/libvxACEI.so.3
......G. /opt/VRTSralus/bin/libvxTAOI.so.3
......G. /opt/VRTSralus/bin/libvxbsa.so
......G. /opt/VRTSralus/bin/libvxcrypto.so
......G. /opt/VRTSralus/bin/libvxicudata.so
......G. /opt/VRTSralus/bin/libvxicui18n.so
......G. /opt/VRTSralus/bin/libvxicuuc.so
......G. /opt/VRTSralus/bin/libvxssl.so
......G. /opt/VRTSralus/bin/libvxustdio.so
......G. /opt/VRTSralus/bin/mktls
......G. /opt/VRTSralus/bin/rptool
......G. /opt/VRTSralus/bin/svr_begather.cfg
......G. /opt/VRTSralus/bin/tracercli
......G. /var/VRTSralus/ralus.ver
S.5....T c /etc/sysconfig/scripts/SuSEfirewall2-custom
.......T /usr/lib64/xulrunner-1.9.2.27/.autoreg
S.5....T c /etc/pure-ftpd/pure-ftpd.conf
S.5....T c /etc/xinetd.d/pure-ftpd
S.5....T c /etc/ntp.conf
S.5....T c /etc/openldap/ldap.conf


Have you had a look at what packets are floodiing the network? You might want to run a short "tcpdump" to get an impression of what is going on, that way you'll might get a hint at what software to look at.

This first output was while trying unsuccessfully to serf suse.org with firefox:
tcpdump
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:45:30.337581 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 514
20:45:30.665165 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 500
20:45:30.758781 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 498
20:45:31.070835 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 434
20:45:31.492300 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 443
20:45:31.601572 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:31.601645 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:32.599510 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:32.599519 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:32.818168 IP 192.168.124.191.ssdp > 239.255.255.250.ssdp: UDP, length 486
20:45:33.613540 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:33.613568 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:35.034717 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1d:09:2a:12:39 (oui Unknown), length 300
20:45:35.049206 arp who-has 192.168.124.194 tell 192.168.124.3
20:45:36.048386 arp who-has 192.168.124.194 tell 192.168.124.3
20:45:36.050442 IP 192.168.124.3.bootps > 192.168.124.194.bootpc: BOOTP/DHCP, Reply, length 300
20:45:36.074362 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:1d:09:2a:12:39 (oui Unknown), length 313
20:45:36.076281 IP 192.168.124.3.bootps > 192.168.124.194.bootpc: BOOTP/DHCP, Reply, length 301
20:45:36.081083 arp who-has 192.168.124.194 tell 0.0.0.0
20:45:36.281307 arp who-has 192.168.124.194 tell 0.0.0.0
20:45:36.453058 arp who-has 192.168.124.2 tell 192.168.124.191
20:45:36.481533 arp who-has 192.168.124.194 tell 0.0.0.0
20:45:36.681760 arp who-has 192.168.124.194 (00:1d:09:2a:12:39 (oui Unknown)) tell 192.168.124.194
20:45:36.690360 IP 192.168.124.3 > 192.168.124.194: ICMP echo request, id 46792, seq 0, length 28
20:45:36.881997 arp who-has 192.168.124.194 (00:1d:09:2a:12:39 (oui Unknown)) tell 192.168.124.194
20:45:42.163014 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:42.163160 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:42.325005 arp who-has 192.168.124.2 tell 192.168.124.231
20:45:43.114560 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:43.114569 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:44.113216 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:44.113225 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:44.960704 arp who-has 192.168.124.8 tell 192.168.124.2
20:45:45.004130 arp who-has 192.168.124.8 tell 192.168.124.2
20:45:45.054129 arp who-has 192.168.124.8 tell 192.168.124.2
20:45:52.303608 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:52.303632 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:53.114762 arp who-has 192.168.124.7 tell 192.168.124.191
20:45:53.114771 arp who-has 192.168.124.8 tell 192.168.124.191
20:45:53.505726 arp who-has 192.168.124.3 tell 192.168.124.191

41 packets captured
199 packets received by filter
0 packets dropped by kernel

This 2nd output with -v was while trying to ping google.com:
tcpdump -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:49:39.126346 arp who-has 192.168.124.8 tell 192.168.124.191
20:49:39.266010 IP (tos 0x0, ttl 128, id 33, offset 0, flags [none], proto UDP (17), length 96) 192.168.124.231.netbios-ns > 192.168.124.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
20:49:39.281597 IP (tos 0x0, ttl 128, id 34, offset 0, flags [none], proto UDP (17), length 96) 192.168.124.231.netbios-ns > 192.168.124.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
20:49:39.281653 IP (tos 0x0, ttl 128, id 35, offset 0, flags [none], proto UDP (17), length 96) 192.168.124.231.netbios-ns > 192.168.124.255.netbios-ns: NBT UDP PACKET(137): REGISTRATION; REQUEST; BROADCAST
20:49:39.349369 arp who-has 192.168.124.8 tell 192.168.124.2
20:49:39.386259 arp who-has 192.168.124.8 tell 192.168.124.2
20:49:39.436260 arp who-has 192.168.124.8 tell 192.168.124.2
20:49:39.578390 arp who-has 192.168.124.231 tell 192.168.124.231
20:49:39.598137 arp who-has 192.168.124.2 tell 192.168.124.231

10 packets captured
175 packets received by filter
0 packets dropped by kernel


Next thing then, if you suspect malicious or ill-behaving software, is a look at "lsof -i4" and "lsof -i6" to see which applications have open ports via IPv4 and IPv6, which may be helpful, too.

lsof -i4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rpcbind 4834 root 6u IPv4 10787 0t0 UDP *:sunrpc
rpcbind 4834 root 7u IPv4 11357 0t0 UDP *:cycleserv
rpcbind 4834 root 8u IPv4 10791 0t0 TCP *:sunrpc (LISTEN)
sshd 5385 root 3u IPv4 11112 0t0 TCP *:ssh (LISTEN)
cupsd 5428 root 1u IPv4 11696 0t0 TCP localhost:ipp (LISTEN)
cupsd 5428 root 4u IPv4 11698 0t0 UDP *:ipp
pluto 5450 root 10u IPv4 11859 0t0 UDP hostname:isakmp
pluto 5450 root 11u IPv4 11860 0t0 UDP hostname:ipsec-nat-t
pluto 5450 root 12u IPv4 11861 0t0 UDP 127.0.0.2:isakmp
pluto 5450 root 13u IPv4 11862 0t0 UDP 127.0.0.2:ipsec-nat-t
pluto 5450 root 14u IPv4 11863 0t0 UDP localhost:isakmp
pluto 5450 root 15u IPv4 11864 0t0 UDP localhost:ipsec-nat-t
charon 5456 root 9u IPv4 11775 0t0 UDP *:isakmp
charon 5456 root 10u IPv4 11776 0t0 UDP *:ipsec-nat-t
rsyncd 5465 root 4u IPv4 11189 0t0 TCP *:rsync (LISTEN)
master 5785 root 12u IPv4 12756 0t0 TCP localhost:smtp (LISTEN)
smbd 5814 root 27u IPv4 12950 0t0 TCP *:microsoft-ds (LISTEN)
smbd 5814 root 28u IPv4 12952 0t0 TCP *:netbios-ssn (LISTEN)
kdm 5838 root 9u IPv4 13173 0t0 UDP *:xdmcp
xinetd 5899 root 5u IPv4 13204 0t0 TCP *:ftp (LISTEN)
xinetd 5899 root 8u IPv4 13205 0t0 TCP *:swat (LISTEN)
xinetd 5899 root 9u IPv4 13206 0t0 TCP *:5901 (LISTEN)
xinetd 5899 root 10u IPv4 13207 0t0 TCP *:5801 (LISTEN)
beremote 5911 root 3u IPv4 13237 0t0 TCP *:ndmp (LISTEN)
ntpd 7276 ntp 16u IPv4 17153 0t0 UDP *:ntp
ntpd 7276 ntp 17u IPv4 17157 0t0 UDP localhost:ntp
ntpd 7276 ntp 18u IPv4 17158 0t0 UDP 127.0.0.2:ntp
ntpd 7276 ntp 20u IPv4 17160 0t0 UDP hostname:ntp
dhcpcd 7658 root 7u IPv4 156290 0t0 UDP *:bootpc

Pluto and charon are strongswan.

jmozdzen
22-May-2014, 12:58
Hi carnold6,

I am not sure how to parse this but here is the output:
rpm -V -a
S.5....T c /etc/modprobe.conf
..5....T c /etc/modprobe.d/unsupported-modules
S.5....T c /etc/ssl/openssl.cnf
[...]
the status bits are described in detail in the man page of RPM. As a general guide, any configuration file will likely have been modified on purpose, although a quick glipse over that list doesn't hurt.

What catches the eye are two missing files from /usr/bin - but you might have an explanation for those.


This first output was while trying unsuccessfully to serf suse.org with firefox:
tcpdump
tcpdump: WARNING: eth0: no IPv4 address assigned

Hm, interestingly the interface you're tracing doesn't have an IPv4 address... are you listening on the right interface? And still, it is connected to the network and sees traffic.

Is there anything "interesting/strange" that catches the eye?


This 2nd output with -v was while trying to ping google.com:
tcpdump -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:49:39.126346 arp who-has 192.168.124.8 tell 192.168.124.191

During the second trace, the interface seems to have an IP address... do you have an explanation for yourself, or is this something you will have to look at?

If your machine is responsible for flooding the network, you'll have to filter for sender IP address of your machine - else it'll be hard to judge.


lsof -i4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[...]
Pluto and charon are strongswan.

Anything you believe that shouldn't be there?

I've had cases where Strongswan does send quite some traffic while trying to re-establish a connection - maybe that is a source of the problems... you might try to simply stop it and have a look. If this is indeed the source of the excess traffic, you'll have to tune your IPsec configuration.

Regards,
Jens

carnold6
28-May-2014, 20:49
OK, i ran maldet and it found 1 infection: {hex}qzbase64.15. I used maldet to quarantine it and now i have 0 packet loss. However, if i run netstat -tpn i see this suspicious entry:


tcp 0 0 192.168.124.194:44953 119.145.148.76:666 ESTABLISHED 32074/.IptabLex
tcp 0 0 192.168.124.194:60096 119.145.148.105:666 ESTABLISHED 32558/.IptabLes


I am a noob at linux so be gentle..... We are not using iptables and not using the SLES11 firewall (we have a hardware firewall). I have searched for these files but do not see them. It appears to me that something somewhere is/has opened a tcp connection. How do i find it and close it?

carnold6
28-May-2014, 21:09
OK, i ran maldet and it found 1 infection: {hex}qzbase64.15. I used maldet to quarantine it and now i have 0 packet loss. However, if i run netstat -tpn i see this suspicious entry:



I am a noob at linux so be gentle..... We are not using iptables and not using the SLES11 firewall (we have a hardware firewall). I have searched for these files but do not see them. It appears to me that something somewhere is/has opened a tcp connection. How do i find it and close it?

See i told you i was a noob :)
Found the files in question in /etc/init.d and /boot. Deleted the files and run netsta -tpn and now no entries of the Asia Pacific IP!

But i have another question: how does someone gain access to put the files in /etc/init.d and /boot?
This server does use strongswan for VPN with certificates. They just has setup secure FTP and chroot was used. Thats the only "remote" access this server was using. If i was a betting man, i would bet it was via secure FTP.