Hi

We have an app developed with OD7.00.18.01, deployed on Tomcat 7.0.54 on Linux CentOs 6.4. The app we have works for a few days but then, when under load, the processor usage shoots up and the app becomes unresponsive.

We have put monitoring in place and have figured out that a thread becomes blocked when logging and that causes the outage (attached an image of such a blocked thread, notice the high number for processor usage).

First, we saw that the blocking part was the AESConnector (hence the DistributeCSTA event) and unloaded it from our Tomcat. This means that the problem is not as pervasive, but still happens.

We have identified that the piece that blocks is always org.apache.log4j.Category.callAppenders, and on this page https://issues.apache.org/bugzilla/show_bug.cgi?id=51047&action=View they state that the code might lock under some circunstances.

Is there anything we can do to prevent the system from locking up?

You can disable logging in your ddrt.properties file.

I have not seen logging as a bottleneck in the past. What type of load increase are you getting when the issue shows up? Is it possible you need multiple servers?

We have disabled the logging and the issue, so far, seems to have disappeared. Thanks!

However, we did not have that much logging to begin with (20Mb/ day) and few calls: About 5 concurrent normally, and about 10 when disaster strikes.
This is however our first project using Tomcat 7 and OD7, we've had projects with lots more logging on other versions of OD and Tomcat with no issues.