Hi.
We have a system with 2 WebLogic Servers and several VPMS systems. Recently we had a failure where a disruption between the network and one of the MPPs caused for the weblogic to become unresponsive for everybody.
Upon investigating we discovered that whenever a thread encountered a network issue upon starting it would acquire a lock for an object that would not be released. This lock on an object means that the Start servlet would not complete and all users would be left hearing dead air.
We are aware that people affected by the network outage should receive dead air, but there should be no locks on the web logic.
We have discussed the issue with WebLogic support and they have told us that this is an application issue.
The web logic log servers' logs stated the stuck threads were always at the same point:
<Dec 6, 2012 11:29:48 AM COT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '338' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "603" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@15e6b879[
GET /Elefante/Start?session___sessionid=VPMSMPPCAL-2012341161941-11&__VPapploggingurl=https%3a%2f%2f10%2e73%2e229%2e32%2faxis%2fservices%2fVPReport4&__VPvpms=10%2e73%2e229%2e32&__VPapplog=%2faxis2%2fservices%2fVPAppLogService&__VPappvars=%2faxis2%2fservices%2fVPAppVarsService&__VPVarAppDate=0&__VPVarAppURL=https%3a%2f%2f10%2e73%2e229%2e32%2faxis%2fservices%2fVPAppRuntimeVars&__VPVarGlobalDate=1315577256919&__VPbreadcrumb=disabled&__VPmaxbackuplogfiles=10&__VPlogname=%25default%257YzntLLOwasXexH4&__VPlogpassword=ZpaXs4dOgzBf9AlVv%2fctCOeM3qgmcL2q4e9v%2bw6NiNJoKTF4tXomkquLHUJVW%2bsc&__VPloglevel=Info&__VPappname=Servilinea HTTP/1.1
Accept: */*
User-Agent: AvayaVXI/2.0
AVB-Version: 2.1.4
Cache-Control: max-age=0
Cache-Control: max-stale=0
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-415 "[STUCK] ExecuteThread: '338' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {
-- Blocked trying to get lock: com.avaya.runtimecommon.platforms.vp.PlatformParams@1882cec2[fat lock]
com.avaya.runtimecommon.platforms.vp.PlatformParams.checkForUpdate(PlatformParams.java:30)
com.avaya.sce.runtimecommon.SCESession.getSession(SCESession.java:240)
com.avaya.sce.runtimecommon.SCESession.getSession(SCESession.java:525)
com.avaya.sce.runtime.SCEServlet.requestHandler(SCEServlet.java:191)
com.avaya.sce.runtime.SCEServlet.doGet(SCEServlet.java:176)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3687)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:116)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2213)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2135)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1420)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
weblogic.work.ExecuteThread.run(ExecuteThread.java:168)
}
>
