Hi.

We have an app with a Weblogic server, the app works fine normally but under very heavy load we start getting threads stuck with the following log:

Thread-89 "[STUCK] ExecuteThread: '74' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {

    -- Blocked trying to get lock: com.avaya.runtimecommon.platforms.vp.PlatformParams@1063b5b8[fat lock]

    com.avaya.runtimecommon.platforms.vp.PlatformParams.checkForUpdate(PlatformParams.java:30)

    com.avaya.sce.runtimecommon.SCESession.getSession(SCESession.java:240)

From what we know this is because the thread starts waiting on a synchronized block (hence the fat lock). Once this happens all threads start blocking behind this one, and the system becomes deadlocked.

This is with DD5.00.17. We have spoken with Webogic support and they say that we are causing these threads to get stuck. The particular app we have has no VP configurable variables, could we use this to our advantage to bypass this checkForUpdate?

Is there anything that we can do to avoid this lock?
Could this be a file locking issue?

Thanks

Have you tried first upgrading to DD 5.1 or above? I can't guarantee that will solve the issue, but the method that's showing the error has had some changes which would help it synchronize multiple threads. It's possible that is enough, though it's difficult to tell what the issue is at all.

Sorry about the mismatch on the version number. The full name of the version we are using is Service Pack 3 (Release 5.1.0.17.02). According to https://www.devconnectprogram.com/site/global/products_resources/avaya_aura_orchestration_designer/releases/dialog_designer_5_1/index.gsp that is the latest version.

Have you tried restarting the app server and VPMS to see if the problem goes away?

Upon restarting the WAS the problem goes away, but comes back a few days later, on peak traffic day.

Can you attach the app's trace.log as well as the vpappclient.log file after the error happens? The latter is located in the main "log" directory for the app server. For example, Tomcat uses:

<Tomcat_Base>\logs

WAS has a longer directory path:

<WebSphere_Base>\AppServer\profiles\AppSrv01\installedApps\<Machine_Nme>Node01Cell\logs

Or something to that effect. There should be a subdirectory within the log directory that has the file.

The PlatformParams.checkForUpdate method has nothing to do with the configurable variables. It is for getting some general platform parameters such as the license url on VPMS. Only the first call will take a little longer since it calls the VPMS web service. The values are cached on the app server after the first call. The subsequent calls shouldn't take any time. I am wonder why the thread is blocked. There was a bug in 5.1 sp2 where call to the VPMS web service happened on every call but you said you are running sp3. Can you double check? What is the version on the scertcommon<version>.jar? Can you also send me some app trace log so I can see what the behavior is like?

We have double checked and it is SP3. The trace log is turned off, attached the stdout.

I can't make anything out of this log. It only tells a thread is locked. Can you post the trace.log file of the application instead? You would need to change the data/ddrt.properties (as follow) to capture the debug logs:

localapptrace=enabled