We are seeing a lot of exception in application detail report(as attached), this causes the call unable to reach the IVR call flow and keep ringing. Is it causing java heap memory increased tremendously?

EXCEPTION>

java.util.EmptyStackException

at com.avaya.sce.runtimecommon.StackUnsynchronized.peek(StackUnsynchronized.java:69)

at com.avaya.sce.runtimecommon.SCESession.getVariableTOSAppName(SCESession.java:949)

at com.avaya.sce.runtime.SCEServlet.requestHandler(SCEServlet.java:227)

at com.avaya.sce.runtime.SCEServlet.doPost(SCEServlet.java:190)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:751)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)

at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:243)

at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3432)

at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3402)

at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)

at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2285)

at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2201)

at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)

at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1572)

at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)

at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)



AEP version 7.0.1
WebLogic version 12.1.3

Is there anyone from Avaya able to advise?

The error you are getting shows the variable stack is gone. Can you show the whole app log for this call? Make sure all the options in ddrt.properties are enabled.

Hi, Thanks for your response. I have attached the the trace log..

Is there any update on this?There is a case in BBE,we can see this issue will cause an outage.When the call cannot remove the session due to this error,and eventually make session full,cannot answer new call.

Can you post a trace.log that has vxml also enabled? Also what version of OD and EP you are using?

Seow Yan already posted the trace log , please refer the attachment.
We are using OD7 and application running weblogic 12c on windows and MS sql server 2008 is database

EP version is Avaya Aura® Experience Portal 7.0.1 (ExperiencePortal)

It is intermittent issue and went to outage on 15 th and 16th due to same error.
Can you please let us know if you have any solution on this.

Thank you,
Swathi

we have not forgotten you. I had to go through the log and by hand extract a single session worth of data was too hard to read with all the sessions interleaved.

HI,

I have attached 15th Sept Log when the First call got Exception (SCE Session - HPbP_3j1gyVICdgyYh6Q2ZeR8eKjPM97v60s0v6_MRk_Y_SpTzl1) and the session id - MOMCCMPP1-2015258075430-28.

Please let me know if you need further logs on this issue.

thank you
Regards,
Swathi

I can sort of explain what I see in the logs but have no idea why

In an successful case we see this:


08/09/2015 08:26:29:238 DEBUG - teaqWFuOx0s5ovP8JYtoPdFMkg5CqzWtmEk6p3Nq2LBFECCPoc6F!310480730!1441671961486:/MOM_Main : Subflow return node "SpecialAnn-Disc" forwarding to "Disconnect"
08/09/2015 08:26:29:425 INFO - teaqWFuOx0s5ovP8JYtoPdFMkg5CqzWtmEk6p3Nq2LBFECCPoc6F!310480730!1441671961486:/MOM_Main : Using SCESession teaqWFuOx0s5ovP8JYtoPdFMkg5CqzWtmEk6p3Nq2LBFECCPoc6F!310480730!1441671961486:/MOM_Main servlet : Disconnect
08/09/2015 08:26:29:644 DEBUG - teaqWFuOx0s5ovP8JYtoPdFMkg5CqzWtmEk6p3Nq2LBFECCPoc6F!310480730!1441671961486:/MOM_Main : *** Reply for [/MOM_Main/Disconnect] ***
08/09/2015 08:26:29:831 DEBUG - teaqWFuOx0s5ovP8JYtoPdFMkg5CqzWtmEk6p3Nq2LBFECCPoc6F!310480730!1441671961486:/MOM_Main : 0:<?xml version="1.0" encoding="UTF-8"?>
1:<vxml version="2.1" xmlns="http://www.w3.org/2001/vxml" xml:lang="en-us">
2:<meta name="author" content="Avaya Aura Orchestration Designer"/>
3:<meta name="runtime-version" content="07.01.08.04"/>
4:<meta name="runtimecommon-version" content="07.01.08.04"/>
5:<meta name="copyright" content="Copyright (c) 2002-2011, Avaya"/>
6:<form id="Disconnect">


However in the failure case we see:


08/09/2015 08:27:04:510 DEBUG - ZlmqWNFJEhQSpRQTHJMLpx9QJtysMvDIV-NIuPAyQKQYnh3XxaAt!310480730!1441671991625:/MOM_Main : Subflow return node "SpecialAnn-Disc" forwarding to "Disconnect"
08/09/2015 08:27:04:557 INFO - ZlmqWNFJEhQSpRQTHJMLpx9QJtysMvDIV-NIuPAyQKQYnh3XxaAt!310480730!1441671991625:/MOM_Main : Using SCESession ZlmqWNFJEhQSpRQTHJMLpx9QJtysMvDIV-NIuPAyQKQYnh3XxaAt!310480730!1441671991625:/MOM_Main servlet : setEndReason
08/09/2015 08:27:04:604 INFO - ZlmqWNFJEhQSpRQTHJMLpx9QJtysMvDIV-NIuPAyQKQYnh3XxaAt!310480730!1441671991625:/MOM_Main : ** Popped Stack Frame [/MOM_Main]


Even though we are forwarding the http request to the Disconnect servlet, the "setEndReason" get activated.

        LoggerUtil.writeTrace(mySession, ITraceInfo.TRACE_LEVEL_DEBUG, "Subflow return node \"" + getSubflowReturnName() + "\" forwarding to \"" + nextServlet + "\"");

		forward(mySession, nextServlet);



. . .

. . . And forward just does this

. . .



		HttpServletRequest request = mySession.getRequest();

		HttpServletResponse response = mySession.getResponse();

		if (request == null) {

			ReportInfo.alarm(IReportInfo.ALARM_LEVEL_MINOR, "Unable to forward to " + location

					+ " request is null, application stoping sessionid[" + mySession.getSessionId() + "],", mySession);

			return;

		}

		if (response == null) {

			ReportInfo.alarm(IReportInfo.ALARM_LEVEL_MINOR, "Unable to forward to " + location

					+ " response is null, application stoping sessionid[" + mySession.getSessionId() + "],", mySession);

			return;

		}

		RequestDispatcher dispatcher = request.getRequestDispatcher(location);

		if (dispatcher == null) {

			ReportInfo.alarm(IReportInfo.ALARM_LEVEL_MINOR, "Unable to forward to " + location

					+ " dispatcher is null, application stoping sessionid[" + mySession.getSessionId() + "],", mySession);

			return;

		}

		mySession.setForwardRedirect(true);

		dispatcher.forward(request, response);

Questions.

Have you seen this "*** Abandoning thread sessionid " in the weblogic equivalent of catalina.out (where ever System.out goes on weblogic)?

Could I given you a new scert.jar to try if I think of something?

This only seems to happen when there is a disconnect in the Contingency Module. The sequence is local call to the contingency module, then a disconnect, return from contingency module to the main, then return from a subflow.

Hi,

Thank you.

Do we need to update scert.jar file? Is there any patch (windows/server)cause to this update required?

Just to share with you, This application is working since past 2 months and all of suddenly its happening since september.

I havent seen any abandon session id in weblogic logs but i will look over again and share with you the logs.

Please let me know if we need to update anything in the OD and server Level. Thank you

Regards,
Swathi

Hi,

Thank you.

Do we need to update scert.jar file? Is there any patch (windows/server)cause to this update required?

Just to share with you, This application is working since past 2 months and all of suddenly its happening since september.

I havent seen any abandon session id in weblogic logs but i will look over again and share with you the logs.

Please let me know if we need to update anything in the OD and server Level. Thank you

Regards,
Swathi

At present there are no patches, just checking how agreeable you would be to a test version of the jar IFF I could think of something to change. Given that this was working fine for 2 months, something must have changed. Do you have any thoughts as to what that might be?

Hi,

We encountered the same error again this morning, 1st Oct, 1026am in production AEP: channel:unknown looping.
The session eventually stopped on its own after around 35 seconds.

Attached is the trace logs of this call (StackExceptionErr.1026.log).
<br>
The incidents last 15 Sep shows the channel:unknown error looped forever until the whole weblogic service has been stopped 20 mins from the start of the incident. (See attached err-ch-un-***.txt logs of 4 different calls).

Our question: Is there a way for us to handle such channel:unknown exception error in our OD application so we can prevent it from looping and end the call/session?

Please advise. Thanks.

Regards,

RUEL