PDA

View Full Version : SLES11sp1 asymmetric routing



Simeonof
01-Nov-2012, 15:08
I'm having the following strange issue since I've updated one of our routers to sles11sp1. On the below picture you can see the network architecture:

http://i45.tinypic.com/95xxjd.jpg

Router 1 - SLES11sp1
default gateway – ISP Router public IP@
ip route add 192.168.131.0/24 via 192.168.0.2 dev eth0

Router 2 - SLES10sp2
default gateway – 192.168.0.1
ip forwarding enabled

PC1
default gateway – 192.168.0.1

PC2
default gateway – 192.168.131.1


When PC1 pings PC2 – all is fine.

When PC2 pings PC1, then several things happen:
the icmp request reaches PC1
PC1 sends icmp reply to its default gateway – Router 1
Router 1 receives the reply, but doesn't forward it to Router 2, the reply gets lost somehow (can't see it anywhere in any log)
PC2 ping times out

If on PC1 I manually do „route add 192.168.131.0/24 via 192.168.0.2 dev eth0“ then all is fine, but this solution is not applicable in our case.

When Router 1 was SLES10sp2 – all was fine. Since we updated Router 1 to SLES11sp1, we've got the above issue. I suspect some parameter changed for the kernel, but I can't find which one it is. Is there something wrong with this setup?

I've tried the TID 7007649 http://www.novell.com/support/kb/doc.php?id=7007649 but setting the rp_filter to 0 or to 2 on Router 1 and Router 2 didn't make any difference.

Any help will be greatly appreciated!

Cheers

KBOYLE
01-Nov-2012, 18:25
Simeonof wrote:

>
> I'm having the following strange issue since I've updated one of our
> routers to sles11sp1.

> When PC2 pings PC1, then several things happen:
> the icmp request reaches PC1
> PC1 sends icmp reply to its default gateway Router 1
> Router 1 receives the reply, but doesn't forward it to Router 2, the
> reply gets lost somehow (can't see it anywhere in any log)
> PC2 ping times out

I may be able to help...

It seems that even though IP Forwarding is on, packets are only
forwarded between two separate interfaces.

I had a similar situation and got it to work this way.

Edit /etc/sysconfig/SuSEfirewall2

Scroll down past the EXPERT OPTIONS to item 33.

Add (in your case) eth0.

That should do it.


> #
> #---------------------------------------------------------------------
> #
> # # EXPERT OPTIONS - all others please don't change these!
> #
> #---------------------------------------------------------------------
> #

> # 33.)
> # Bridge interfaces without IP address
> #
> # Traffic on bridge interfaces like the one used by xen appears to
> # enter and leave on the same interface. Add such interfaces here in
> # order to install special permitting rules for them.
> #
> # Format: list of interface names separated by space
> #
> # Note: this option is deprecated, use FW_FORWARD_ALLOW_BRIDGING
> instead #
> # Example:
> # FW_FORWARD_ALWAYS_INOUT_DEV="xenbr0"
> #
> FW_FORWARD_ALWAYS_INOUT_DEV="eth0"

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below...

Simeonof
02-Nov-2012, 09:02
Thank you Kevin for the reply, however I'm not using SuSEfirewall - it's stopped. I'm using iptables (through firewallbuilder interface). I see option number 33 in the file you mentioned :

# 33.)
# Bridge interfaces without IP address
#
# Traffic on bridge interfaces like the one used by xen appears to
# enter and leave on the same interface. Add such interfaces here in
# order to install special permitting rules for them.
#
# Format: list of interface names separated by space
#
# Note: this option is deprecated, use FW_FORWARD_ALLOW_BRIDGING instead
#
# Example:
# FW_FORWARD_ALWAYS_INOUT_DEV="xenbr0"
#
FW_FORWARD_ALWAYS_INOUT_DEV=""

Do you know exactly which kernel settings this option modifies, so I can give it a try?

Regards,
Milko Simeonov




Simeonof wrote:

>
> I'm having the following strange issue since I've updated one of our
> routers to sles11sp1.

> When PC2 pings PC1, then several things happen:
> the icmp request reaches PC1
> PC1 sends icmp reply to its default gateway Router 1
> Router 1 receives the reply, but doesn't forward it to Router 2, the
> reply gets lost somehow (can't see it anywhere in any log)
> PC2 ping times out

I may be able to help...

It seems that even though IP Forwarding is on, packets are only
forwarded between two separate interfaces.

I had a similar situation and got it to work this way.

Edit /etc/sysconfig/SuSEfirewall2

Scroll down past the EXPERT OPTIONS to item 33.

Add (in your case) eth0.

That should do it.


> #
> #---------------------------------------------------------------------
> #
> # # EXPERT OPTIONS - all others please don't change these!
> #
> #---------------------------------------------------------------------
> #

> # 33.)
> # Bridge interfaces without IP address
> #
> # Traffic on bridge interfaces like the one used by xen appears to
> # enter and leave on the same interface. Add such interfaces here in
> # order to install special permitting rules for them.
> #
> # Format: list of interface names separated by space
> #
> # Note: this option is deprecated, use FW_FORWARD_ALLOW_BRIDGING
> instead #
> # Example:
> # FW_FORWARD_ALWAYS_INOUT_DEV="xenbr0"
> #
> FW_FORWARD_ALWAYS_INOUT_DEV="eth0"

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below...

KBOYLE
02-Nov-2012, 14:57
Simeonof wrote:

> Do you know exactly which kernel settings this option modifies, so I
> can give it a try?

I'm sorry, I don't. I haven't worked with iptables. Perhaps someone
else has and can offer a suggestion or perhaps you could do some
research and find out what needs to be changed to allow forwarding to
occur "on the same interface".

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below...

Simeonof
03-Nov-2012, 10:07
The strange thing is that "forwarding on the same interface" works fine if the "initiator" is PC1, and doesn't work at all if the "initiator" is PC2...


Simeonof wrote:

> Do you know exactly which kernel settings this option modifies, so I
> can give it a try?

I'm sorry, I don't. I haven't worked with iptables. Perhaps someone
else has and can offer a suggestion or perhaps you could do some
research and find out what needs to be changed to allow forwarding to
occur "on the same interface".

--
Kevin Boyle - Knowledge Partner
If you find this post helpful and are using the web interface,
show your appreciation and click on the star below...

jmozdzen
05-Nov-2012, 11:17
Hi Simeonof,

> Do you know exactly which kernel settings this option modifies, so I
> can give it a try?

that's most probably not affecting kernel settings, but simply adding an appropriate forwarding rule (the comment reads "in order to install special permitting rules for them").

> Router 1 receives the reply, but doesn't forward it to Router 2, the reply gets lost somehow (can't see it anywhere in any log)

This may be something totally different than rules... your routing setup will result in ICMP redirects sent by router1 and may be affected by a bug in recent Linux kernels/IP stack (on PC1, in this case). This bug's results are affected by timing, so it's independent of "ICMP echo request" vs. "echo reply":

- PC1 sends ICMP echo request to PC2. The packet is sent to router1, as no other option is available to the IP stack on PC1
- router1 notices that the packet will leave via the same interface as it entered the router, so router1 sends an "ICMP redirect" to PC1 ("check your routing setup... and use router2, not me!")
- router1 forwards the packet to router2
- ... (router2 forwards to PC2, response PC2 to router2 to PC1)

This can happen for some time, until PC1 actually reacts to the ICMP redirect (this can take several seconds) and sends the request packets to router2 directly, making it PC1-router2-PC2.

At some point in time, the redirect information stored in PC1 times out. The correct way would be that PC1 sends further packets to router1 again... but due to the bug, nothing will be sent from PC1.

Now, this can be with *any* packet from PC1 to PC2 - an ICMP request or an ICMP response, a first one or starting somewhere in the middle of the packet chain.

> Router 1 receives the reply, but doesn't forward it to Router 2, the reply gets lost somehow (can't see it anywhere in any log)

Have you actually verified this using i.e. tcpdump? And did you mean "router1 receives the replies"? If yes, the problems you describe should not be caused by the mentioned Linux bug.

Regards,
Jens

Simeonof
05-Nov-2012, 11:52
Problem solved.

Turned out that for whatever reason, when PC2 sends the ping "request" to PC1, the ping "reply" of PC1 reaches Router 1 in state "INVALID" . No idea why, but that's the fact. Of course, on Router 1 I have iptables rule that looks like this:


$IPTABLES -A OUTPUT -s 192.168.0.0/24 -d 192.168.131.0/24 -m state --state NEW -j ACCEPT
$IPTABLES -A FORWARD -s 192.168.0.0/24 -d 192.168.131.0/24 -m state --state NEW -j ACCEPT

As you can see, it won't allow packets in "INVALID" state. So I made it stateless:


$IPTABLES -A OUTPUT -s 192.168.0.0/24 -d 192.168.131.0/24 -j ACCEPT
$IPTABLES -A FORWARD -s 192.168.0.0/24 -d 192.168.131.0/24 -j ACCEPT

The real question now is why those packets are marked "INVALID" on their way from PC1 to PC2 ...

enovaklbank
09-Nov-2012, 09:45
The forum engine is still perfect, so let's write my post once again...

Please check dmesg for packets from "martian source".

I think the problem is some sort of routing problem, and such messages would reveal it.

Also, see /etc/sysconfig/SuSEfirewall2 :

# 17.)
# Do you want to enable additional kernel TCP/IP security features?
# If set to yes, some obscure kernel options are set.
# (icmp_ignore_bogus_error_responses, icmp_echoreply_rate,
# icmp_destunreach_rate, icmp_paramprob_rate, icmp_timeexeed_rate,
# ip_local_port_range, log_martians, rp_filter, routing flush,
# bootp_relay, proxy_arp, secure_redirects, accept_source_route
# icmp_echo_ignore_broadcasts, ipfrag_time)
#
# Tip: Set this to "no" until you have verified that you have got a
# configuration which works for you. Then set this to "yes" and keep it
# if everything still works. (It should!) ;-) <-- try this as well.
#
# Choice: "yes" or "no", if not set defaults to "yes"
#
FW_KERNEL_SECURITY="yes"

If it's related to some sysctl settings, the fastest way to test it is this.

Simeonof
10-Nov-2012, 09:46
I also thought it was some routing problem, but it turned out to be what I've explained in my previous post, INVALID packets dropped by the firewall.

P.S. I'm not using SuSEfirewall2, I'm using iptables with FWBuilder GUI.



The forum engine is still perfect, so let's write my post once again...

Please check dmesg for packets from "martian source".

I think the problem is some sort of routing problem, and such messages would reveal it.

Also, see /etc/sysconfig/SuSEfirewall2 :

# 17.)
# Do you want to enable additional kernel TCP/IP security features?
# If set to yes, some obscure kernel options are set.
# (icmp_ignore_bogus_error_responses, icmp_echoreply_rate,
# icmp_destunreach_rate, icmp_paramprob_rate, icmp_timeexeed_rate,
# ip_local_port_range, log_martians, rp_filter, routing flush,
# bootp_relay, proxy_arp, secure_redirects, accept_source_route
# icmp_echo_ignore_broadcasts, ipfrag_time)
#
# Tip: Set this to "no" until you have verified that you have got a
# configuration which works for you. Then set this to "yes" and keep it
# if everything still works. (It should!) ;-) <-- try this as well.
#
# Choice: "yes" or "no", if not set defaults to "yes"
#
FW_KERNEL_SECURITY="yes"

If it's related to some sysctl settings, the fastest way to test it is this.