Hi Dev connect team,

Greetings!

One of the customer reported the below issue. Appreciate your inputs and guidance to resolve it. DD 5.1
======
Below are findings from java core and logs , it looks to be - ICConnectorServer$CommandProcessorThread is not getting expected response.

1. Can see that almost every threads ( 200 + ) are waiting for
com/avaya/sce/icconnector/web/servlet/server/ICConnectorServer$CommandProcessorThread.run(ICConnectorServer.java:914)

2. All most every web container threads ( 46) are having issue at below class

at java/net/SocketInputStream.socketRead0(Native Method)
at java/net/SocketInputStream.read(SocketInputStream.java:140(Compiled Code))
at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
at java/io/BufferedInputStream.read1(BufferedInputStream.java:269(Compiled Code))
at java/io/BufferedInputStream.read(BufferedInputStream.java:328(Compiled Code))
at sun/net/www/http/HttpClient.parseHTTPHeader(HttpClient.java:722(Compiled Code))
at sun/net/www/http/HttpClient.parseHTTP(HttpClient.java:665(Compiled Code))
at sun/net/www/http/HttpClient.parseHTTPHeader(HttpClient.java:803(Compiled Code))
at sun/net/www/http/HttpClient.parseHTTP(HttpClient.java:665(Compiled Code))
at sun/net/www/protocol/http/HttpURLConnection.getInputStream(HttpURLConnection.java:1230(Compiled Code))
at com/avaya/sce/runtime/connectivity/ic/web/ICCommand.executeCommand(ICCommand.java:115(Compiled Code))
at com/avaya/sce/runtime/VDUVariableField.updateValueForGet(VDUVariableField.java:228(Compiled Code))
at com/avaya/sce/runtime/VariableField.getStringValue(VariableField.java:330(Compiled Code))
at com/avaya/sce/runtime/Condition.getStringValue(Condition.java:118(Compiled Code))
at com/avaya/sce/runtime/Condition.evaluate(Condition.java:578(Compiled Code))
at flow/CheckAICFlags.executeDataActions(CheckAICFlags.java:218(Compiled Code))
at com/avaya/sce/runtime/Data.handleRequest(Data.java:81(Compiled Code))
at com/avaya/sce/runtime/AppServlet.processRequest(AppServlet.java:78(Compiled Code))
at com/avaya/sce/runtime/SCEServlet.requestHandler(SCEServlet.java:266(Compiled Code))
at com/avaya/sce/runtime/SCEServlet.doPost(SCEServlet.java:184(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:738(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:831(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1661(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:944(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:507(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:374(Compiled Code))
at com/avaya/sce/runtime/SCEServlet.forward(SCEServlet.java:1272(Compiled Code))
at com/avaya/sce/runtime/Data.handleRequest(Data.java:128(Compiled Code))
at com/avaya/sce/runtime/AppServlet.processRequest(AppServlet.java:78(Compiled Code))
at com/avaya/sce/runtime/SCEServlet.requestHandler(SCEServlet.java:266(Compiled Code))
at com/avaya/sce/runtime/SCEServlet.doPost(SCEServlet.java:184(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:738(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:831(Compiled Code))

• Thread Status Analysis
Status Number of Threads : 358 Percentage
Deadlock
0 0 (%)
Runnable
52 15 (%)
Waiting on condition
300 84 (%)
Waiting on monitor
0 0 (%)
Suspended
0 0 (%)
Object.wait()
0 0 (%)
Blocked
4 1 (%)
Parked
2 1 (%)

Hung thread details:
Java Stack at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167)
at com/avaya/sce/icconnector/web/servlet/server/ICConnectorServer$CommandProcessorThread.run(ICConnectorServer.java:914)

Please find below findings for the date 11/25/14.
1) @19:28 we found an Runtime exception on loading PluggableConnectorInitialize class caused StringIndexOutOfBoundsException (see below logs)
2) @19:37 first thread hung occurred & application starts to get slow response.
3) Once the sessionid gets invalid the hung threads are closed. (Please refer below logs)

[11/25/14 19:28:14:512 GST] 0000002d SystemOut O com.avaya.sce.runtimecommon.SCERuntimeException: Error loading PluggableConnectorInitialize class [com.avaya.sce.runtime.connectivity.ic.pdc.ICConnectorInit] for initialVariablesPosted java.lang.StringIndexOutOfBoundsException
**** Root Cause
java.lang.StringIndexOutOfBoundsException
at java.lang.String.substring(String.java:1111)
at com.avaya.sce.runtime.connectivity.ic.web.NewCall.parseResponse(NewCall.java:41)
at com.avaya.sce.runtime.connectivity.ic.web.ICCommand.executeCommand(ICCommand.java:205)
at com.avaya.sce.runtime.connectivity.ic.pdc.ICConnectorInit.initialVariablesPosted(ICConnectorInit.java:206)
:
:
:
[11/25/14 19:35:48:797 GST] 0000003e SystemOut O Request is invalid sessionid [0uqeDZQigZ-aRXGOeign2Oe:/ADCB] class [flow.srv_InitalNode] and saved class name [flow.subflow.TransferCall.PlayTransferMessage is -> false
[11/25/14 19:35:48:797 GST] 0000003e SystemOut O *** Abandoning thread sessionid [0uqeDZQigZ-aRXGOeign2Oe:/ADCB], request invalid AppServlet, ignore trace above
[11/25/14 19:35:48:797 GST] 00000034 SystemOut O ExitReason :::
[11/25/14 19:35:48:906 GST] 00000016 SystemOut O ExitReason :::
[11/25/14 19:35:51:964 GST] 00000016 SystemOut O ExitReason :::
[11/25/14 19:35:57:346 GST] 0000003e SystemOut O ExitReason :::
[11/25/14 19:36:16:784 GST] 00000016 SystemOut O ExitReason :::
[11/25/14 19:37:02:056 GST] 00000016 SystemOut O ExitReason :::
[11/25/14 19:37:30:667 GST] 00000016 SystemOut O ExitReason :::
[11/25/14 19:37:54:536 GST] 00000014 ThreadMonitor W WSVR0605W: Thread "WebContainer : 23" (00000036) has been active for 606045 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:140)
:
:
[11/25/14 19:44:01:301 GST] 0000007f SystemOut O Request is invalid sessionid [67qR2Phr3cV2SZmu2CRjAG2:/ADCB] class [flow.CheckAICFlags] and saved class name [flow.subflow.TransferCall.PlayTransferMessage is -> false
[11/25/14 19:44:01:301 GST] 0000007f SystemOut O *** Abandoning thread sessionid [67qR2Phr3cV2SZmu2CRjAG2:/ADCB], request invalid AppServlet, ignore trace above
[11/25/14 19:44:04:577 GST] 0000007f ThreadMonitor W WSVR0606W: Thread "WebContainer : 44" (0000007f) was previously reported to be hung but has completed. It was active for approximately 977225 milliseconds. There is/are 27 thread(s) in total in the server that still may be hung.
==============

Thanks in advance.

Reagrds,
Sandeep.

Can you have them upgrade to a newer version of DD/OD? Even if they go to DD 5.1 SP3, that will pick up a number of changes that were put in place around parsing the response from HTTPVox. It would be better to move to OD 7.0 though, but if that isn't possible at least a Service Pack upgrade.

After the upgrade, if problems still persist, I need to see an ICC log as well as logs from IC side:HTTPVox and HTTPC logs. Just run one or two calls and send logs.

thanks

Can you post the application trace and IC connector log?

Hi NeilGoldsmith,

Can you please tell me why IBM WAS is getting hung. IBM WAS is getting hung freaquently like once in a week from past one month.

We have configured HTTP VOX recently in the month of October 2014. Is this because of IC connector HTTP VOX?

If it is because of HTTP VOX, can we solve this issue in DD 5.1 sp2.

And the tread configuration which we have done in HTTP VOX is 240 as the number of channels are 240. is it enough or it will be solved if we increase the threads in HTTP VOX.

Thanks in advance.

There are parse errors which could be causing the issues with the threads. I can't be sure, but definitely need to clean up those errors. I would suggest using DD 5.1 SP3 at least and possibly upgrading to OD 7.1 if that is possible.

You don't need to increase threads. If you are referring to the ICC thread pool, you do not need one thread per port. On a busy system 50 or so threads should be fine.

Hi NeilGoldsmith,


Now, We can not ask customer to upgrade to DD 5.1 SP3 or DD 7.0.
Is there any possibility that we can clean up those errors in DD 5.1 SP2.

Please advice as it is critical issue and impacting business.

No, sorry, we cannot patch DD 5.1. That release has been out of support for nearly 2 years. Updating to SP3 would be the same thing as applying a patch.

However, I still need to see an ICC log as well as logs from IC side:HTTPVox and HTTPC logs to see what is actually happening with the parse issue.

We found that there was HTTP Connector (webchannel) got crashed at time of web sphere hung.

We are suspecting that it is because of crashing of HTTPConnector.

The implementation in IC is that we are using same work flow for both Voice and EDU values updation.

May be work flow was over loaded with the calls and requests from IVR for EDU values updation. It happened in the peak hours only.

It could be a resource issue during peak traffic and the data back to the ICC is incorrect causing the parse error. The SP3 changes will guard against corrupt data, however the bigger issue is the cause of the HTTPC failing.

Hi Neil,

will you please assist in reviewing the logs delivered.
Please refer to the email I've sent you.

Thanks,
Flora

Hi,

Can any one please suggest us how many threads we can configure in IBM web sphere.

We have licences for 240 calls

We are using SOAP calls in the application.
HTTPVOX configuration is enabled
DD 5.1.0.1101
VP 5.1
AIC 7.3

Please suggest us how many threads we can configure in the IBM web sphere console or please share us the document if you have any thing.

Thanks in advance.

Configuration of the app server is going to be different for every customer. You need to analyze the peak traffic that server will handle and allocate resources according to that. If this is a dedicated server, then you should allocate (threads/memory) as much as the system can handle on a very busy system. You also need to analyze what else is running on the server. You should also consider offloading to multiple servers if one is unable to handle the load.

We have only one web server that is web sphere which is serving only one IVR.

IVR application has been configured with ICConnector and AIC 7.3.

Considering IVR is handling SOAP requests for all the calls in the IVR.
Approximately how many treads we can configure to handle 240 simultaneous calls.

Thanks.

There's no thread count I can give you. It's going to depend alot on how your app uses those threads per call. Are the webservice calls short? Do you use IC efficiently? How many simultaneous threads do all the 240 calls use at one time? If this is a dedicated server, then allocate as much of the systems resources as you can. If the system is still not able to keep up, you need to add more servers.

Do you have a staging system to test before going to a production environment?

Hi NeilGoldSmith,

There are 50 threads configured in web sphere and we can not increase the threads in web sphere as it is windows 2003 32 bit.
It is dedicated server. We do not have staging system.
We have configured 1000 threads in ICC thread pool size.