Jump to content

VIPS / port polling / SSL failures


BruceHartley

Recommended Posts

I am having an issue where we (meaning not me) have set up a number of VIPS (one VIP per port) and i have been told they have to be able to poll the port on the receiving end to make sure it's up.  If that is true, how does one set up this "polling" to use HTTPS for HTTPS destination ports, because right now whatever method it's using is causing all sorts of errors to show up when I turn on SSL debugging.  Furthermore, it's doing this polling every 5 seconds thus making it super hard to see what is a true SSL error and what is being caused by Netscaler.

The target ports are on a webMethods box, they are set up as HTTPS, and here is what I see when I tell webMethods to turn on SSL debugging (I'm having an issue with the senders and their SSL, thus this issue)  - I purposely masked the IP address in the errors for obvious security reasons.

Each port only supports TLS 1.2 and above and only known secure ciphers - the system is running Java 11 which has by default significantly improved security by only allowing known secure ciphers.  Here is the errors I get every 5 seconds

INFO   | jvm 1    | 2024/01/28 02:17:35 | javax.net.ssl|DEBUG|04 12|HTTP Handler xxx.xxx.xxx.xxx|2024-01-28 02:17:35.695 EST|HandshakeContext.java:296|Ignore unsupported cipher suite: TLS_AES_256_GCM_SHA384 for TLSv1.2
INFO   | jvm 1    | 2024/01/28 02:17:35 | javax.net.ssl|DEBUG|04 12|HTTP Handler xxx.xxx.xxx.xxx|2024-01-28 02:17:35.698 EST|HandshakeContext.java:296|Ignore unsupported cipher suite: TLS_AES_128_GCM_SHA256 for TLSv1.2
INFO   | jvm 1    | 2024/01/28 02:17:35 | javax.net.ssl|DEBUG|04 12|HTTP Handler xxx.xxx.xxx.xxx|2024-01-28 02:17:35.698 EST|HandshakeContext.java:296|Ignore unsupported cipher suite: TLS_CHACHA20_POLY1305_SHA256 for TLSv1.2
INFO   | jvm 1    | 2024/01/28 02:17:35 | javax.net.ssl|DEBUG|04 12|HTTP Handler xxx.xxx.xxx.xxx|2024-01-28 02:17:35.699 EST|SSLSocketInputRecord.java:487|Raw read: EOF
INFO   | jvm 1    | 2024/01/28 02:17:35 | javax.net.ssl|ERROR|04 12|HTTP Handler xxx.xxx.xxx.xxx|2024-01-28 02:17:35.699 EST|TransportContext.java:352|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
INFO   | jvm 1    | 2024/01/28 02:17:35 | "throwable" : {
INFO   | jvm 1    | 2024/01/28 02:17:35 |   javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1697)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1515)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1013)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.net.HttpInputStream.readHeaderUntil(HttpInputStream.java:473)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.net.HttpInputStream.readRequest(HttpInputStream.java:340)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.net.HttpInputStream.readHeader(HttpInputStream.java:213)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.net.HttpHeader.read(HttpHeader.java:605)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.app.b2b.server.HTTPRequest.<init>(HTTPRequest.java:27)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.app.b2b.server.HTTPState.readHeader(HTTPState.java:151)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:307)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at com.wm.util.pool.PooledThread.run(PooledThread.java:127)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/java.lang.Thread.run(Thread.java:829)
INFO   | jvm 1    | 2024/01/28 02:17:35 |   Caused by: java.io.EOFException: SSL peer shut down incorrectly
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)
INFO   | jvm 1    | 2024/01/28 02:17:35 |       ... 15 more}
INFO   | jvm 1    | 2024/01/28 02:17:35 | 
INFO   | jvm 1    | 2024/01/28 02:17:35 | )

 

Not knowing this product very well, I don't know if it sees the port as HTTPS and is trying those ciphers listed above which Java 11 is rejecting or that those are just FYI messages saying those TLS 1.2 ciphers are not supported.

Setting the polling interval to a longer value is somewhat of a workaround, but what I'm really hoping for is to understand the root cause of this and how to prevent it so that I can turn on debugging and not have the logs be full of this every 5 seconds and preferably not at all.

If necessary, we can easily write a custom SOAP or REST v2 service that would tell Netscaler the port is up, my biggest problem is I don't know how to help my network person to eliminate this issue or mitigate it better.  I have also reached out to the vendor of webMethods as well.  I saw this type of error in my logs years ago too in prior versions of webMethods that ran java 8 as well and was told by a network person I was working with at the time that it was due to how Citrix Netscaler polls the ports and that it's literally a raw HTTP call.

Any help would be appreciated.

Link to comment
Share on other sites

Hi Bruce. 

You are talking very highlevel, we need to bring it a few stories down to understand what you are trying achieve. from reading your post, i have not grasped whats wrong, or what you are trying to do.

However, there are some tools/buttons in NetScaler you should know about, based on what you are writing, however its guesstimated from my side, so please bare with me 🙂

- SSL Profiles

Here you can control pr service/servicegroup which kind of SSL/TLS Settings (including Chiper list) you want pr. service or service groups (you can also use the same profile multiple places)

- Monitors

Its possible to create custom monitors pr. service/servicegroup, so you change your "polling" to the backend server.

You can combine multiple monitors for 1 service, and check for specific HTTP requests if you use HTTP-ECV

 

You should be able to get more help, if you can specify the question are bit more, like which client is connecting to what, or what you have configured in netscaler - and whats not working.

Link to comment
Share on other sites

  • 2 weeks later...

Morten and everyone else;

What I'm trying to accomplish is simple - I don't want Netscaler (orwhatever else it is) polling a port on a middleware system that has defined ports every 5 seconds to see if it's "awake" which is what is happening right now.  I don't even know if it's Netscaler doing it, because the person who is in charge of setting it up has told me it's only supposed to be doing it every 5 minutes.  My gut tells it it's this because there are no logs of anything hitting the firewall and nothing I know internally hits the ports in question so the only thing left is Netscaler and an unknown system to me.

Every time I turn on SSL debugging on the middleware, I see something hitting it every 5 seconds like clockwork.  It's gotten to the point I'm going to install Wireshark and run it for 30 seconds to capture what is doing it.  I know it's not the interfaces going to it that are doing it and it may be something besides Netscaler - I'm not trying to accuse the tool so to speak.

So at a high level I was asking a couple of questions because I don't know the product and what it requires - The biggest question so far is this ...

Is this "polling" required - in other words, does Netscaler absolutely need to do it or is this just so someone can look and see by what I call a "light now" what ports are working and what ones aren't.  I get the status is important, but when one connects via raw TCP to an HTTPS port that has SSL debugging on it, all sorts of errors go off and makes it hard and almost impossible to see what the issues are - it's like trying to find a single tree in a 100 acre forest.

It it's not required, I'd rather for now they shut it off until we figure out what is going on.  I solved one interface that was having issues this weekend, I might be closer to the other and it also looks like a firewall tool is the issue and not SSL.

I am the middleware person here by the way, I can control every last little aspect of it, but I don't control and have very limited visibility and no control over the firewalls and Netscaler as those are run by others.

My follow-up question is then - if the polling is required, is there a way to tell it to use SSL to do it instead of whatever else it's doing so it doesn't throw a ton of errors.

Link to comment
Share on other sites

Just to let you all know where this is.

Installed and ran Wireshark.  Some particular node is hitting the HTTPS ports every 5 seconds (except the one we made that was 5 minutes).  That tells me that the IP it is coming from is Netscaler because otherwise it would have been doing the one port we made 5 minutes as well.

So bask to the questions:

1 - Does this product require port polling to see if it's awake

2 - Can the polling be adjusted to do HTTPS things as opposed to HTTP or raw TCP things to do it's poll

Link to comment
Share on other sites

Hi AGain, you can disable monitoring on a service and servicegroup, this will make the service be always "green" and not do any polling, just forwarding packets.

 

and yes, you can customized your monitoring requirements, anything is possible.

Edited by Morten Kallesøe
adding monitorting suggestion
Link to comment
Share on other sites

On 2/14/2024 at 8:31 AM, BruceHartley said:

Just to let you all know where this is.

Installed and ran Wireshark.  Some particular node is hitting the HTTPS ports every 5 seconds (except the one we made that was 5 minutes).  That tells me that the IP it is coming from is Netscaler because otherwise it would have been doing the one port we made 5 minutes as well.

So bask to the questions:

1 - Does this product require port polling to see if it's awake

2 - Can the polling be adjusted to do HTTPS things as opposed to HTTP or raw TCP things to do it's poll

Hi Bruce

Sorry for my bad english 😉

With netscaler you can do what you want... 

Polling backend server is mandatory, netscaler must know server condition.

You can configure a simple TCP Port monitoring (with your problems)

Or you can configure a HTTPS \HTTPS-ECV probe with specific path (like "GET /healt.html") and resp code (like 200) (or string in response like "UP", in HTTP-ECV monitor)

https://docs.netscaler.com/en-us/citrix-adc/13-1/load-balancing/load-balancing-builtin-monitors/monitor-ssl-services

Also you can set probe paramiters like interval polling (on adv configurations of monitor)

https://docs.netscaler.com/en-us/citrix-adc/13-1/load-balancing/load-balancing-configure-monitors/configuring-monitoring-parameters

 

image.png.98fccd5ce73484335d569abc859c6f8f.png

Edited by Nicola Campaci
Link to comment
Share on other sites

3 hours ago, Morten Kallesøe said:

Sorry to say Nicola, its not mandatory🤷‍♂️

health mon.png

Hi Morten
Mandatory... if you want know server health like on a normal load balancing

Your example bypass monitoring functions. It's a workaround, not a solution. In case of servers malfunctions, you have client side problem on lb service (it's always up)

Link to comment
Share on other sites

Hi Nicola, i guess that's a choice you make, and choice are not mandatory.

You could configure an ICMP probe instead of monitoring the TCP port of the service, i would call that a workaround, but disabling monitoring completely is definitely a solution. (Bruce have not asked for monitoring, he actually asked how to disable it....)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...