|
Hi,
We have recently deployed to production a new version of an existing Speech Application. Within the next 24 hours the Weblogic server went down three times due a large number (e.g. 300) Log4j blocked threads. Stacktrace sample of a blocked thread here below.
The changes made to the IVR application are rather small and there is nothing special related to logging that has changed. Logging is intense due to the high call traffic but it has been always that way, and the IVR application has been performing normally for 2 years, until now.
Versions used are OD 6.0.0.1103, Weblogic 11g, Java 1.7.0_80 and log4j-1.2.16
Any clue or help would be much appreciated. Thanks!!!
"[STUCK] ExecuteThread: '112' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x000000002207a800 nid=0xa82 waiting for monitor entry [0x000000006b356000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:444)
- waiting to lock <0x000000070240c690> (a java.util.Hashtable)
at org.apache.log4j.LogManager.shutdown(LogManager.java:238)
at com.avaya.runtimecommon.platforms.desktop.tracking.ReportWriter.close(ReportWriter.java:39)
at com.avaya.sce.runtimecommon.SCESession.doRemoveSession(SCESession.java:679)
at com.avaya.sce.runtimecommon.SCESession.valueUnbound(SCESession.java:1931)
at weblogic.servlet.internal.session.SessionData.removeAttribute(SessionData.java:743)
at weblogic.servlet.internal.session.SessionData.removeAttribute(SessionData.java:708)
at weblogic.servlet.internal.session.SessionData.remove(SessionData.java:993)
- locked <0x00000007ae3933e8> (a java.lang.Object)
at weblogic.servlet.internal.session.MemorySessionContext.invalidateSession(MemorySessionContext.java:69)
at weblogic.servlet.internal.session.SessionData.invalidate(SessionData.java:869)
at weblogic.servlet.internal.session.SessionData.invalidate(SessionData.java:887)
at com.avaya.sce.runtime.AvayaSessionTerminationServlet.requestHandler(AvayaSessionTerminationServlet.java:62)
at com.avaya.sce.runtime.AvayaSessionTerminationServlet.doGet(AvayaSessionTerminationServlet.java:33)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3732)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
|
