PDA

View Full Version : SMT/curl problems with proxy server



imoore
04-Dec-2014, 03:04
I'm having trouble getting our SMT server to work with the new proxy server (McAfee Web Gateway) which our education dept has installed. Unfortunately, there is no way for us to bypass the proxy to get to the net and the only access we have to the server is to block/unblock web sites. They have agreed to allow unauthenticated access from our server ip address range, however I still can't get SMT to work.

The proxy server uses a self signed CA certificate which I have saved in /etc/ssl/certs/ - when you access an SSL site, the proxy forwards it's own cert instead of the original, so you need it's CA on every host.
EG importing the CA cert into a web browser's CA cert list allows you to access https sites without being prompted to accept the certificates.

For some reason, just adding the CA cert to /etc/ssl/certs/ doesn't make curl work (this is just running curl directly, not via smt-ncc-sync). Instead, I have to use the --cacert option to specify the CA certificate. You can either add it to the command line or put it in /root/.curlrc. Still, that gets curl working.

Now back to smt-ncc-sync. This normally runs as the user smt, so I copied the working .curlrc file from /root to smt's homedir /var/lib/smt. Unfortunately, that doesn't work - curl completely ignores the .curlrc file when run from the smt binary. Same story runinng it from the command line as root or when it's run via cron as smt.

I've even gone to the extent of replacing curl with a shell script pointing to the 'real' curl:
#!/bin/sh
/usr/bin/curl.bin --cacert /etc/ssl/certs/0823-MWG-CA.pem $@

(curl.bin is the "real" curl renamed)

Even doing that doesn't fix smt-ncc-sync, it still gives the same errors: Invalid response:500 CURL ERROR(60) Peer certificate cannot be authenticated with known CA certificates
That leads me to think that smt binaries must have curl embedded within them and set not to use ~/.curlrc

I've run out of ideas for getting around this problem, so hoping somebody here might have a suggestion.

malcolmlewis
04-Dec-2014, 04:53
On Thu 04 Dec 2014 02:14:02 AM CST, imoore wrote:


I'm having trouble getting our SMT server to work with the new proxy
server (McAfee Web Gateway) which our education dept has installed.
Unfortunately, there is no way for us to bypass the proxy to get to the
net and the only access we have to the server is to block/unblock web
sites. They have agreed to allow unauthenticated access from our server
ip address range, however I still can't get SMT to work.

The proxy server uses a self signed CA certificate which I have saved in
/etc/ssl/certs/ - when you access an SSL site, the proxy forwards it's
own cert instead of the original, so you need it's CA on every host.
EG importing the CA cert into a web browser's CA cert list allows you to
access https sites without being prompted to accept the certificates.

For some reason, just adding the CA cert to /etc/ssl/certs/ doesn't make
curl work (this is just running curl directly, not via smt-ncc-sync).
Instead, I have to use the --cacert option to specify the CA
certificate. You can either add it to the command line or put it in
/root/.curlrc. Still, that gets curl working.

Now back to smt-ncc-sync. This normally runs as the user smt, so I
copied the working .curlrc file from /root to smt's homedir
/var/lib/smt. Unfortunately, that doesn't work - curl completely ignores
the .curlrc file when run from the smt binary. Same story runinng it
from the command line as root or when it's run via cron as smt.

I've even gone to the extent of replacing curl with a shell script
pointing to the 'real' curl:
#!/bin/sh
/usr/bin/curl.bin --cacert /etc/ssl/certs/0823-MWG-CA.pem $@

(curl.bin is the "real" curl renamed)

Even doing that doesn't fix smt-ncc-sync, it still gives the same
errors: *Invalid response:500 CURL ERROR(60) Peer certificate cannot be
authenticated with known CA certificates*
That leads me to think that smt binaries must have curl embedded within
them and set not to use ~/.curlrc

I've run out of ideas for getting around this problem, so hoping
somebody here might have a suggestion.




Hi
Your proxy should should accept Novell domain (or at least selected
sites used like nu.novell.com and secure.novell.com) without
authentication.

You need to populate proxy setting in YAST and to add two lines
with your proxy information to /etc/profile, for example:

export http_proxy="http://10.10.1.1:8081/"
export https_proxy="https://10.10.1.1:8081/"

As well as the /root/.curlrc

Now, is this also for SLE 12 (if so, then that's a new server instance)?

You might also want to peruse;
https://www.suse.com/support/kb/doc.php?id=7005002

--
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.28-4-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

imoore
04-Dec-2014, 06:21
Hi,

Your proxy should should accept Novell domain (or at least selected
sites used like nu.novell.com and secure.novell.com) without
authentication.
Yes, the SMT server ip is set to bypass proxy authentication, so can access any site without a username & password (confirmed from firefox on the SMT server).

The proxy is configured in Yast - I forgot to mention that.

I've added the proxy config to /etc/profile.local (/etc/profile suggests custom settings be put there as /etc/profile itself may get overwritten by upgrades)
I've rebooted just to be sure the changes take effect & can see the proxy settings when I run env, but still getting the same errors from smt-ncc-sync.



Now, is this also for SLE 12 (if so, then that's a new server instance)?
No, I haven't done any SLE12 installs yet, so just SLES10/11 for now.

imoore
04-Dec-2014, 06:27
Also, just tried adding CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt to /etc/profile.local
I've put the proxy server's CA certificate (in PEM format) in ca-certificates.crt.
That also hasn't worked :-(

malcolmlewis
04-Dec-2014, 06:37
On Thu 04 Dec 2014 05:24:02 AM CST, imoore wrote:


Hi,
> Your proxy should should accept Novell domain (or at least selected
> sites used like nu.novell.com and secure.novell.com) without
> authentication.
Yes, the SMT server ip is set to bypass proxy authentication, so can
access any site without a username & password (confirmed from firefox on
the SMT server).

The proxy is configured in Yast - I forgot to mention that.

I've added the proxy config to /etc/profile.local (/etc/profile suggests
custom settings be put there as /etc/profile itself may get overwritten
by upgrades)
I've rebooted just to be sure the changes take effect & can see the
proxy settings when I run env, but still getting the same errors from
smt-ncc-sync.


> Now, is this also for SLE 12 (if so, then that's a new server
> instance)?
No, I haven't done any SLE12 installs yet, so just SLES10/11 for now.




Hi
Makes me wonder if it may be related to the current infrastructure
changes (NCC and SCC). Let me ask my SUSE Contacts to see if there have
been any issues.

--
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.28-4-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

imoore
04-Dec-2014, 08:17
Ok thanks - the new proxy was installed about tge same time as the NCC and SCC updates so it's possible that is the problem.
Cheers, Ian

malcolmlewis
04-Dec-2014, 14:21
On Thu 04 Dec 2014 07:24:02 AM CST, imoore wrote:


Ok thanks - the new proxy was installed about tge same time as the NCC
and SCC updates so it's possible that is the problem.
Cheers, Ian




Hi
Heard back, there are no known issues. Is it possible to do a packet
capture (tcpdump or wireshark) to see what is happening on the wire?

--
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.28-4-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

malcolmlewis
04-Dec-2014, 14:37
Hi
One other thought (see https://forums.suse.com/showthread.php?5699-Where-are-mirror-credentials-now&p=25250#post25250) is to consider moving to SLE 12 SMT setup and using SCC which you will need to do?

imoore
05-Dec-2014, 03:07
Hi
Heard back, there are no known issues. Is it possible to do a packet
capture (tcpdump or wireshark) to see what is happening on the wire?

Ok, have done a few traces with wireshark.
I can see the SMT server connecting to nu.novell.com:443
Next, the proxy server sends it's fake SSL cert for novell.com to the SMT server, then there's a server key exchange
Then the SMT server sends a TLSv1 Alert (Fatal, Unknown CA).

I still think that's the underlying problem.

Next are a series of curl requests from the SMT server - the odd thing here is that all of those curl requests result in either 302: Temporarily Moved or 404: Not found replies - except for one or two.
The failed ones are for SLES10 < SP4 or SLE11 < SP3.
The one that seems to work is the SLES10 SP4 ATI repo (which I'm not mirroring). The trace for that request is:

HEAD http://www2.ati.com/suse/sle10sp4/repodata/repomd.xml HTTP/1.1
User-Agent: libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Host: www2.ati.com
Accept: */*
Proxy-Connection: Keep-Alive

Then there's just some TCP AKs etc back & forth, followed by another curl request for the next repo.

I don't know why there are no curl requests for my mirrored repos in the trace - current SLES & OES versions.


One other thought (see https://forums.suse.com/showthread.p...5250#post25250) is to consider moving to SLE 12 SMT setup and using SCC which you will need to do?

I gave that a go - I assume I'm correct in thinking that I just need to switch the existing server using cmd line or Yast as per the readme?

The NCC credentials test button in the Yast smt-server module gives an Invalid Credentials error when I click on it now - switching to SCC and entering the new Org credentials from the SCC gives the same error! I guess that's a proxy problem. So I suspect. If I try to save the changes, the back end migration fails. I can switch back to the NCC, though of course the connectivity test fails with the usual certificate errors.

malcolmlewis
05-Dec-2014, 03:44
On Fri 05 Dec 2014 02:14:01 AM CST, imoore wrote:


malcolmlewis;25259 Wrote:
> Hi
> Heard back, there are no known issues. Is it possible to do a packet
> capture (tcpdump or wireshark) to see what is happening on the wire?

Ok, have done a few traces with wireshark.
I can see the SMT server connecting to nu.novell.com:443
Next, the proxy server sends it's fake SSL cert for novell.com to the
SMT server, then there's a server key exchange
Then the SMT server sends a TLSv1 Alert (Fatal, Unknown CA).

I still think that's the underlying problem.

Next are a series of curl requests from the SMT server - the odd thing
here is that all of those curl requests result in either *302:
Temporarily Moved* or *404: Not found* replies - except for one or
two.
The failed ones are for SLES10 < SP4 or SLE11 < SP3.
The one that seems to work is the SLES10 SP4 ATI repo (which I'm not
mirroring). The trace for that request is:

HEAD http://www2.ati.com/suse/sle10sp4/repodata/repomd.xml
HTTP/1.1
User-Agent: libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Host: www2.ati.com
Accept: */*
Proxy-Connection: Keep-Alive

Then there's just some TCP AKs etc back & forth, followed by another
curl request for the next repo.

I don't know why there are no curl requests for my mirrored repos in the
trace - current SLES & OES versions.

> One other thought (see
> https://forums.suse.com/showthread.p...5250#post25250) is to consider
> moving to SLE 12 SMT setup and using SCC which you will need to do?

I gave that a go - I assume I'm correct in thinking that I just need to
switch the existing server using cmd line or Yast as per the readme?

The NCC credentials test button in the Yast smt-server module gives an
Invalid Credentials error when I click on it now - switching to SCC and
entering the new Org credentials from the SCC gives the same error! I
guess that's a proxy problem. So I suspect. If I try to save the
changes, the back end migration fails. I can switch back to the NCC,
though of course the connectivity test fails with the usual certificate
errors.




Hi
So your using both SLE and OES, if that's the case then you need one
SMT server for OES and one SMT server for SLE (updated to 12 and
pointing at SCC).

--
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.28-4-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

imoore
05-Dec-2014, 04:13
Hi
So your using both SLE and OES, if that's the case then you need one
SMT server for OES and one SMT server for SLE (updated to 12 and
pointing at SCC).
Right, well I'll have to setup a new server to do one of the two. Maybe a fresh start will help with the proxy problem (though somehow I doubt it :( ) In any case, I'll do that and post the results next week sometime.

Thanks heaps for your help!

smflood
05-Dec-2014, 11:46
On 05/12/2014 02:44, malcolmlewis wrote:

> So your using both SLE and OES, if that's the case then you need one
> SMT server for OES and one SMT server for SLE (updated to 12 and
> pointing at SCC).

Not necessarily.

You only need an (updated) SMT server pointing at SCC for SLE12 repos -
SLE11 and earlier repos are still available via NCC (as are OES repos).

HTH.
--
Simon
SUSE Knowledge Partner

------------------------------------------------------------------------
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below. Thanks.
------------------------------------------------------------------------

imoore
09-Dec-2014, 01:58
Well, I've built a new SLES11SP3 vm. I haven't installed SMT yet, but just trying to register it (directly with NCC/SCC rather than our SMT server) using the Yast NCC module gives the same error:
Error: Peer certificate cannot be authenticated with known CA certificates: (60) message.

I did my trick with renaming /usr/bin/curl to curl.bin and substituted curl with a script to run curl --cacert /etc/ssl/certs/0823-MWG-CA.pem $@
Then ran the registration again and did a ps ax|grep curl and could see that this time my script really was working to call curl with the --cacert parameter:


root 9141 0.0 0.2 12760 1456 ? S 09:05 0:00 /bin/sh /usr/bin/curl --silent --fail --connect-timeout 1 -I http://169.254.169.254/1.0/meta-data/instance-id
root 9142 0.0 0.4 46932 2672 ? S 09:05 0:00 /usr/bin/curl.bin --cacert /etc/ssl/certs/0823-MWG-CA.pem --silent --fail --connect-timeout 1 -I http://169.254.169.254/1.0/meta-data/instance-id
The first line is my curl script and the second is the "real" curl binary with the appropriate ca cert parameter.

Despite that, it still fails with the "Peer certificate" error.

As far as I can tell, the -cacert parameter works to make curl accept the proxy's certificate because I can run
curl https://www.google.com
and I get the response I would expect:


<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com.au/?gfe_rd=cr&amp;ei=nC-GVPGoCaqN8Qeo2oCwCQ">here</A>.
</BODY></HTML>

If I run curl --silent --fail --connect-timeout 1 -I http://169.254.169.254/1.0/meta-data/instance-id as per the output of the ps command above, it just comes straight back to the command prompt with no output and no errors.
So I guess the yast module runs that and then does something else that results in the error message. Unfortunately, it fails too quickly to catch it with a ps.

I also tried registering with /usr/bin/suse_register but get the same response:

All services have been refreshed.
Repository 'SUSE-Linux-Enterprise-Server-11-SP3 11.3.3-1.138' is up to date.
All repositories have been refreshed.
ERROR: HTTP/1.0 200 Connection established


(2)
ERROR: Peer certificate cannot be authenticated with known CA certificates: (60)
(2)
ERROR: Peer certificate cannot be authenticated with known CA certificates: (60)
(2)

It's a perl script, but I can't figure out what it's doing to troubleshoot any further.

smflood
10-Dec-2014, 14:02
On 09/12/2014 01:04, imoore wrote:

> Well, I've built a new SLES11SP3 vm. I haven't installed SMT yet, but
> just trying to register it (directly with NCC/SCC rather than our SMT
> server) using the Yast NCC module gives the same error:
> -Error: Peer certificate cannot be authenticated with known CA
> certificates: (60)- message.
>
> I did my trick with renaming /usr/bin/curl to curl.bin and substituted
> curl with a script to run *curl --cacert /etc/ssl/certs/0823-MWG-CA.pem
> $@*
> Then ran the registration again and did a ps ax|grep curl and could see
> that this time my script really was working to call curl with the
> --cacert parameter:
>
>
> ROOT 9141 0.0 0.2 12760 1456 ? S 09:05 0:00
> /BIN/SH /USR/BIN/CURL --SILENT --FAIL --CONNECT-TIMEOUT 1 -I
> HTTP://169.254.169.254/1.0/META-DATA/INSTANCE-ID
> ROOT 9142 0.0 0.4 46932 2672 ? S 09:05 0:00
> /USR/BIN/CURL.BIN --CACERT /ETC/SSL/CERTS/0823-MWG-CA.PEM --SILENT
> --FAIL --CONNECT-TIMEOUT 1 -I
> HTTP://169.254.169.254/1.0/META-DATA/INSTANCE-ID
> The first line is my curl script and the second is the "real" curl
> binary with the appropriate ca cert parameter.
>
> Despite that, it still fails with the "Peer certificate" error.
>
> As far as I can tell, the -cacert parameter works to make curl accept
> the proxy's certificate because I can run
> curl https://www.google.com
> and I get the response I would expect:
>
>
> *<HTML><HEAD><meta http-equiv="content-type"
> content="text/html;charset=utf-8">
> <TITLE>302 Moved</TITLE></HEAD><BODY>
> <H1>302 Moved</H1>
> The document has moved
> <A
> HREF="https://www.google.com.au/?gfe_rd=cr&amp;ei=nC-GVPGoCaqN8Qeo2oCwCQ">here</A>.
> </BODY></HTML>*
>
> If I run *curl --silent --fail --connect-timeout 1 -I
> http://169.254.169.254/1.0/meta-data/instance-id* as per the output of
> the ps command above, it just comes straight back to the command prompt
> with no output and no errors.
> So I guess the yast module runs that and then does something else that
> results in the error message. Unfortunately, it fails too quickly to
> catch it with a ps.
>
> I also tried registering with /usr/bin/suse_register but get the same
> response:
>
> All services have been refreshed.
> Repository 'SUSE-Linux-Enterprise-Server-11-SP3 11.3.3-1.138' is up to
> date.
> All repositories have been refreshed.
> ERROR: HTTP/1.0 200 Connection established
>
>
> (2)
> ERROR: Peer certificate cannot be authenticated with known CA
> certificates: (60)
> (2)
> ERROR: Peer certificate cannot be authenticated with known CA
> certificates: (60)
> (2)
>
> It's a perl script, but I can't figure out what it's doing to
> troubleshoot any further.

I seem to recall that there was previously a curl error 60 issue which
was fixed by installing an updated openssl-certs package though I can't
remember the exact details.

Checking download.suse.com/patch/finder I see that there are updated
openssl-certs packages for SLES11 SP3 so perhaps try that?

I'd provide a direct link to a download but don't know which
architecture or flavour (regular or "for VMware") you're using.

HTH.
--
Simon
SUSE Knowledge Partner

------------------------------------------------------------------------
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below. Thanks.
------------------------------------------------------------------------

imoore
11-Dec-2014, 01:31
Yes!! That fixed the registration problem. There were 3 of those patches, so I installed the latest one and ran /usr/bin/suse_register and it succeeded this time.
Tried installing it on the original SMT server and it said already installed, so that's presumably not the issue with SMT. However, I'll now install SMT on the new server and see how I go with that.

Cheers,
Ian

imoore
11-Dec-2014, 03:46
More good news - I've installed SMT on the new server and it can authenticate with the NCC and mirror repos. Evidently I must have stuffed up some setting on the old server in trying to get it to talk to the new proxy and now I can't seem to undo that.
So I'll re-register all our servers with the new SMT server and should be able to patch them over the Xmas break after all :)

Thanks for all your help Malcolm and Simon!

Cheers,
Ian

smflood
11-Dec-2014, 12:24
On 11/12/2014 02:54, imoore wrote:

> More good news - I've installed SMT on the new server and it can
> authenticate with the NCC and mirror repos. Evidently I must have
> stuffed up some setting on the old server in trying to get it to talk to
> the new proxy and now I can't seem to undo that.
> So I'll re-register all our servers with the new SMT server and should
> be able to patch them over the Xmas break after all :)
>
> Thanks for all your help Malcolm and Simon!

Glad to hear you're sorted. It seems curl "error 60" is one of those
generic catch-all errors which has multiple causes.

Thanks for reporting back.
--
Simon
SUSE Knowledge Partner

------------------------------------------------------------------------
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below. Thanks.
------------------------------------------------------------------------

Magic31
12-Dec-2014, 12:23
..I seem to recall that there was previously a curl error 60 issue which
was fixed by installing an updated openssl-certs package though I can't
remember the exact details.

IIRC that was with SLES10 (and maybe SLES11 GM/SP1) and had to do with the previous CA being expired that handled repo signing.

Interesting that updating the openssl-cert package was the remedy again. Good catch!

Cheers,
Willem