Receiving this exception as calls flow through a custom snap-in. Would you be able to advise where we can investigate this issue?

call.servlet.InitialInviteListenerHelperImpl ERROR - SNAPIN-1.0.0.0.89 - addCallDataInformationToNewCall client e=
java.lang.IllegalStateException: Connection is still allocated
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.getConnection(BasicHttpClientConnectionManager.java:266)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager$1.get(BasicHttpClientConnectionManager.java:217)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.comcast.nco.XMPortErrors.MyCallListener.getToken(MyCallListener.java:477)
at com.comcast.nco.XMPortErrors.MyCallListener.callIntercepted(MyCallListener.java:146)
at com.avaya.collaboration.call.servlet.InitialInviteListenerHelperImpl.addCallDataInformationToNewCall(InitialInviteListenerHelperImpl.java:112)
at com.avaya.collaboration.call.async.message.processor.RunnableMessageProcessorSsal.run(RunnableMessageProcessorSsal.java:108)
at com.avaya.collaboration.call.async.message.processor.AsyncMessageProcessorAgentImpl.create(AsyncMessageProcessorAgentImpl.java:73)
at com.avaya.collaboration.call.servlet.InitialInviteListener.initialInviteReceived(InitialInviteListener.java:105)
at com.avaya.service.ipt.invite_connection.statemachine.state.InviteConnectionInitialState.handleInitialInvite(InviteConnectionInitialState.java:79)
at com.avaya.service.ipt.invite_connection.statemachine.InviteConnectionStateMachineImpl.handleInitialInvite(InviteConnectionStateMachineImpl.java:660)
at sun.reflect.GeneratedMethodAccessor418.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.avaya.service.ipt.invite_connection.statemachine.async.AsyncWorkImpl.doAsyncTask(AsyncWorkImpl.java:73)
at com.avaya.service.ipt.invite_connection.statemachine.async.AsyncGenericComponentImpl.executeWork(AsyncGenericComponentImpl.java:86)
at com.avaya.service.ipt.invite_connection.statemachine.async.AsyncGenericComponentImpl.processAsynchronously(AsyncGenericComponentImpl.java:68)
at com.avaya.service.ipt.invite_connection.statemachine.async.AsyncInviteConnectionStateMachineImpl.processOperationAsynchronously(AsyncInviteConnectionStateMachineImpl.java:121)
at com.avaya.service.ipt.invite_connection.statemachine.async.AsyncInviteConnectionStateMachineImpl.handleInitialInvite(AsyncInviteConnectionStateMachineImpl.java:298)
at com.avaya.service.ipt.invite_connection.InviteConnectionImpl.doInitialInvite(InviteConnectionImpl.java:411)
at com.avaya.service.ipt.invite_connection.DefaultInviteConnectionSipServlet.handleInvite(DefaultInviteConnectionSipServlet.java:489)
at com.avaya.service.ipt.invite_connection.DefaultInviteConnectionSipServlet.doInvite(DefaultInviteConnectionSipServlet.java:389)
at com.avaya.ssal.b2bua.sip.WellBehavedB2bSipServlet.doInvite(WellBehavedB2bSipServlet.java:136)
at javax.servlet.sip.SipServlet.doRequest(SipServlet.java:474)
at javax.servlet.sip.SipServlet.doRequestWrapper(SipServlet.java:544)
at javax.servlet.sip.SipServlet.service(SipServlet.java:899)
at com.avaya.service.ipt.invite_connection.DefaultInviteConnectionSipServlet.processService(DefaultInviteConnectionSipServlet.java:219)
at com.avaya.service.ipt.invite_connection.DefaultInviteConnectionSipServlet.service(DefaultInviteConnectionSipServlet.java:209)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1232)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:781)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:480)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.ws.sip.container.was.SipFilter.doFilter(SipFilter.java:100)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:967)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1107)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:947)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.wsspi.webcontainer.WebContainer.handleRequest(WebContainer.java:95)
at com.ibm.ws.sip.container.was.SipMessage.dispatch(SipMessage.java:1407)
at com.ibm.ws.sip.container.was.SipMessage.run(SipMessage.java:1378)
at com.ibm.ws.sip.container.was.WebsphereInvoker.invokeSipServlet(WebsphereInvoker.java:140)
at com.ibm.ws.sip.container.router.SipRouter.invokeSipServlet(SipRouter.java:1740)
at com.ibm.ws.sip.container.tu.TransactionUserImpl.handleUASRequest(TransactionUserImpl.java:1915)
at com.ibm.ws.sip.container.tu.TransactionUserImpl.processRequestUASMode(TransactionUserImpl.java:2007)
at com.ibm.ws.sip.container.tu.TransactionUserImpl.processRequest(TransactionUserImpl.java:1738)
at com.ibm.ws.sip.container.tu.TransactionUserWrapper.processRequest(TransactionUserWrapper.java:915)
at com.ibm.ws.sip.container.transaction.ServerTransaction.processRequest(ServerTransaction.java:105)
at com.ibm.ws.sip.container.router.tasks.RequestRoutedTask.doTask(RequestRoutedTask.java:99)
at com.ibm.ws.sip.container.router.tasks.InitialRequestRoutedTask.doTask(InitialRequestRoutedTask.java:63)
at com.ibm.ws.sip.container.router.tasks.RoutedTask.run(RoutedTask.java:103)
at com.ibm.ws.sip.container.was.SignalingEndOfTask.run(SignalingEndOfTask.java:80)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)

Also seeing several of these throughout the asm log.

statemachine.nonblocking.NonBlockingStateMachineHelperImpl ERROR - run Exception caught:
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor659.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.avaya.zephyr.helper.statemachine.nonblocking.StateEventMethodImpl.invoke(StateEventMethodImpl.java:64)
at com.avaya.zephyr.helper.statemachine.nonblocking.StateEventImpl.invoke(StateEventImpl.java:57)
at com.avaya.zephyr.helper.statemachine.nonblocking.NonBlockingStateMachineHelperImpl.run(NonBlockingStateMachineHelperImpl.java:161)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
at com.ibm.ws.asynchbeans.ExecutorServiceImpl$1.run(ExecutorServiceImpl.java:81)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:811)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)
at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:206)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Caused by: java.lang.NullPointerException
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateClosed.enterState(P2pAamsMediaServerEndpointStateClosed.java:74)
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateMachineImpl.updateState(P2pAamsMediaServerEndpointStateMachineImpl.java:342)
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateClosing.enterState(P2pAamsMediaServerEndpointStateClosing.java:50)
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateMachineImpl.updateState(P2pAamsMediaServerEndpointStateMachineImpl.java:342)
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.helper.P2pAamsMediaServerEndpointStateMachineBehaviorsImpl.closeTypical(P2pAamsMediaServerEndpointStateMachineBehaviorsImpl.java:512)
at com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateFailed.close(P2pAamsMediaServerEndpointStateFailed.java:38)

I believe that these errors are related to an AMS issue that we were having. For some reason our AMS was in a "Platform Locked" state. We were able to unlock it. However, my question is when I am doing a media operation, how can I catch events related to an AMS not being available when the operation is invoked? From what I'm being told, during this outage our callers just heard dead air and the calls were not able to pass through the snap in.

We are just using a basic promptAndCollect, and I don't see that it throws any exceptions.

myId = mediaService.promptAndCollect(participant, playItem,
digitOptions, mediaListener);

Hi Michael,

I am looking at the stacktrace. It shows a new call arriving into Breeze which the advises the snap-in's MyCallListener.callIntercepted() about the call.

It looks like the snap-in then is attempting to retrieve an HTTP connection, which fails.

I suspect this might be related to attempting to re-use an existing connection. Look at http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html. In there it states:


====================================================
2.3.2. Simple connection manager
BasicHttpClientConnectionManager is a simple connection manager that maintains only one connection at a time. Even though this class is thread-safe it ought to be used by one execution thread only. BasicHttpClientConnectionManager will make an effort to reuse the connection for subsequent requests with the same route. It will, however, close the existing connection and re-open it for the given route, if the route of the persistent connection does not match that of the connection request. If the connection has been already been allocated, then java.lang.IllegalStateException is thrown.

This connection manager implementation should be used inside an EJB container.
====================================================

Another issue you might be running into is that the client logic isn't consuming the entity completely before attempting to re-use the connection.

Please let me know what you find out.

Thanks,
Thor

You could be right on the first stack trace. However the second seems directly to point to AMS issue. Can you advise on my question of how to catch an error related to the fact that an AMS might not be in a state that is able to process a connection? In our case it was "Platform Locked"

com.avaya.media.p2p.aams.widget.endpoint.state.machine.P2pAamsMediaServerEndpointStateMachineImpl.updateState(P2pAamsMediaServerEndpointStateMachineImpl.java:342

In response to your comment regarding BasicHttpConnection, this is coming from the Avaya samples in the Breeze developers guide under the section labelled "Considerations about sending outbound HTTP requests" on page 80. So unless the sample needs to be corrected/adjusted I would assume this is not the culprit.

For the second part of your question regarding not consuming the entity fully, I am using

EntityUtils.consume(response.getEntity());

.

However I did add a finally block, to ensure it gets called should there be an earlier exception before we reach that point. I will monitor for a while and see if anything new occurs.

I would still like to get a response on how to handle the AMS not being available exceptions. Thanks.

Hi,

What version of Breeze is being used here?

--Rob

3.4.0.0.12340003

I believe we may have made changes in a later version of Breeze related to the ability for a snap-in to have a way to detect that the AAMS is in a state where it cannot fulfill requests. I need to investigate further.

There is a bug fix in 3.4 service pack 1 plus patch (3.4.0.1.340120 plus patch 3.4.0.1.09340120) that provides an indication to a snap-in when all media servers -- along the lines you've indicated here -- are unable to provide service. The precise behavior varies a bit depending on the precise scenario. But based on what you've described, here's what I think you would see under 3.4.0.1.09340120. Your snap-in's callTerminated method would get invoked with a cause of MEDIA_SERVER_FAILURE. Additionally, a SIP 503 message would be sent to the caller's phone, which will probably result in most phones playing a busy signal to the caller. In some other scenarios, the add participant failed callback will be invoked, too.

There is a bit of an irony in a situation like this. One nice thing about being able to play announcements to callers is that your snap-in can "play out" announcements with useful information for certain kinds of error conditions that a caller might encounter. Of course, the ability to play an announcement from a snap-in requires use of a media server. In your scenario here, with no media server available, the options available to indicate an error to a calling party are pretty limited. I do think typically a busy signal is a better experience for a caller than silence, so I think the fix in the patch is a good improvement.

Does that mean that the call would be physically disconnected in the new patch, or would I be able to call.allow to have the call proceed gracefully to its original destination? I would have hoped there was a way to allow the call to continue in this scenario.

The call would be disconnected.

I've checked with a few colleagues, and none of us know of a way to allow the call to proceed gracefully to its original destination. You might consider working with your account team to open a GRIP request for an enhancement along the following lines. Allow a call to proceed if a snap-in encounters a non-critical media operation and/or allow the setting of a policy to indicate desired behavior on media server failure. Please be sure to reference this forum posting in the request, so that we have adequate context for the request.

Then, here's an interim solution that you might consider. It would allow a large majority of your calls to proceed without failing under these circumstances. When your snap-in gets the notification of a call termination due to media server failure, set an internal flag making note of this error. From this point forward, bypass media operations for x minutes. After x minutes, return to normal media operations. If you get the media server failure again, repeat the process. While this approach is admittedly not optimal, it would allow a high percentage of the calls to proceed to the original destination. And when the media server service is restored, normal call treatment will resume automatically.

Thank you for following up. What we did to mitigate was to just allocate a second AMS so that if one does have an issue and become unavailable, the other would still be functional and process calls. We tested this internally, and it seems to work for our purposes.