SLES11 SP2 with 2 NIC's. Each NIC has its own (private) ip and each NIC is physically attached to a different network. A quick network diagram:
|sles11 server|--nic 1 192.168.123.3---192.168.123.x network switch (unmanaged)--trusted port of sonicwall\
|___________|--nic 2 192.168.124.3---192.168.124.x network switch (unmanaged)--opt port of sonicwall----/
This server has a default gateway of 192.168.124.2 which is the sonicwall OPT interface.
I spent days (and money) making sure it was not the sonicwall. From a device on the LAN, you can access other devices on the OPT side and OPT to LAN works fine. When it comes to this multihomed server, the problems arise.
Using the (rough) diagram above, when (specifically) 192.168.123.3 sends a requests to 192.168.124.3, it fails (whether a ping, telnet to port 25, https or anything). So i added a route on the sles11 server: route add -net 192.168.123.0 netmask 255.255.255.0 gw 192.168.124.2 (i know this is not persistent and will address this at a later time when the correct fix is found). This route add now allow a device on the LAN to access the 192.168.124.3 via ping, telnet port 25 and https. But a server on the LAN, 192.168.123.3, can not access 192.168.124.3. This server is not multihomed and is also a SLES11 server.
What i believe is happening is when a request is sent to 192.168.124.3, the request makes it to 192.168.124.3 server but comes out from 192.168.123.x and the requestor is not looking for an ACK from 192.168.123.x but from 192.168.124.x and therefore drops the packet. First, does that sound right to you? Then if that is right, how do i keep ACK's (request sent back to the requestor) so it returns from the right NIC?
In an effort to give as much needed info as possible, here is the bigger sketch of what is going on:
When you go to https://mail.domain.tld (i will give you this info privately if you want it) it goes to the sonicwall which sends 443 requests to a private ip of 192.168.123.3. That sles11 server(LAN) is running apache and via proxypass statements will send the request to 192.168.124.3(OPT) which is the multihomed sles server. I am sure the proxypass statement works fine as i have had apache guru's verify its correct. If you would like to look at anything related to this, i will be happy to share.