We have run into a problem where a simple addAddressListener is taking upwards of 50 to 60 seconds or more during peak times. We also see delayed messaging coming through JTAPI also appears to be worse during peak times. We have confirmed through MST and AES tracing that the messaging itself from AES - CM - JTAPI is not delayed because there is constant trace logging occurring during the listener addition process , but the actual completion does not occur for a very long time.

Can you help us understand what might be going on during this listener addition process that is taking so long?

We are using just a simple test app to reproduce --

            ACDAddress a = (ACDAddress) provider.getAddress(skillExt);

            a.addAddressListener(skillListener);

I see several events like below in the trace for many different stations occurring just during this listener addition for this one skill extension process but it's not clear why all this would be happening during the registration of the listener.

CSTAQueryDeviceInfo
CSTAQueryDeviceInfoConfEvent

what release of the JTAPI client are you using?
I recommend retesting with the most recent version of the version of AES you are using. I can think of a recent problem that may be contributing to this.

10.1
https://www.devconnectprogram.com/site/global/products_resources/avaya_aura_application_enablement_services/releases/10_1/index.gsp#download-8

8.1.3
https://www.devconnectprogram.com/site/global/products_resources/avaya_aura_application_enablement_services/releases/8_1_3/index.gsp#download-8

We were on 8.1 and also upgraded to latest 10 release and have same experience.

And I just retested again, 8.1.3 performs even worse. Right now it's taking about 15s. If I move to 10.1 it's 8s, which is still horrible. AES is SW Version: 8.1.3.4.0.2-0

please enable tsapi tracing and capture the csta-trace.log and then open a support ticket through the portal and I will have someone take a look. You can also see what the client is doing during the interval before the application gets the completion indication. We are very short staffed at the moment due to the holidays, so please be patient with our response.

https://www.devconnectprogram.com/site/global/products_resources/avaya_aura_application_enablement_services/support/faq/tsapi/index.gsp?tab=general&accordion=faq-10

Looks like my last message never posted. We have an Avaya incident already on SR 1-19410460842 , and the engineer we spoke with today Jeff Little (little2@avaya.com) was going to try to find some jtapi support but sounded like he wasn’t sure about where to go.

If you could follow up with him I would greatly appreciate it. All the csta traces and mst are captured there. Appreciate your quick responses and help. Thanks.

Also, check how your application handles threading. If a thread is blocked, waiting for something to happen, it is not available to process other messages. Never use the JTAPI threads to do anything complex - the listeners should only be used to pass incoming messages to your own internal queue and processing threads. NEVER make API calls using the JTAPI threads - that can lead to madness.

Martin

At this point we are using a test application which does nothing except add the listener. Even using Avaya default listener implementation it just takes a really long time to complete. There is no custom code at all, just a simple a.addAddressListener(skillListener);

Is it normal for jtapi to query all the agents in the skill when adding a listener to a skill?

The JTAPI SDK does a lot of activity in order to build its object model. A simple operation in your view can trigger a lot of work by the library. From one version of the library to the next changes are made, and sometimes those changes can insert undesirable behaviors. The issue I allueded to above was one such problem. I see no great reason it needs to query for information about all the agents in a skill before returning a success to the application, but from your comment, it is. If the skill is of significant size that can take a while. In our environment most of our skills only contain a few agents, and rarely is more than one logged in.

Is it querying ALL agents, or all LOGGED IN agents? What version of the SDK are you working with at this point? 10.1?

There was a bug in some releases of the JTAPI SDK which could lead to delays when processing a lot of events. This may be causing your problem.

The bug is marked as fixed in versions 8.1.3.6.0 and 10.1.0.2.0 of the JTAPI SDK. 10.1.0.2.0 is available now on the Devconnect website and 8.1.3.6.0 will be available soon (probably in the New Year). Hopefully, upgrading to one of these will solve your problem.

Martin

Sorry all, I'm not sure why my messages aren't posting. So we did confirm that the adding of the listener seems to be performing as intended. We did tests to comparative skills on other systems, and the listener takes a while to complete if you have a large number of agents.

The next part of this (which admittedly I did not query above because we were focused on a test with adding of the listener) is that we seem to be hit with the performance bottlenecks on receiving events as well. Again when I compare this to other systems in our environment we see 200-300ms constantly without change. However on this system that we are troubleshooting we see high fluctuations into the many SECONDS regularly. Sometimes even MINUTES. We have Avaya support looking at this as well, but any ideas here?

[INFO ][2022-12-21 08:34:41,006][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 219
[INFO ][2022-12-21 08:34:41,008][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 221
[INFO ][2022-12-21 08:34:41,009][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 221
[INFO ][2022-12-21 08:34:41,037][DistributeCSTAEvent] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 2121
[INFO ][2022-12-21 08:34:41,339][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 2841
[INFO ][2022-12-21 08:34:41,340][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 2835
[INFO ][2022-12-21 08:34:41,793][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 1531
[INFO ][2022-12-21 08:34:42,645][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 1603
[INFO ][2022-12-21 08:34:42,646][GetEventThread ] - PerfStatisticsCollector: - Service request turnaround time threshold(200) value crossed: 1605

More entries along these same lines --
FO ][2022-12-21 08:34:51,870][DistributeCSTAEvent] - PerfStatisticsCollector: ---- - Handler latency threshold(100) value crossed: 10833
[INFO ][2022-12-21 08:34:51,870][DistributeCSTAEvent] - PerfStatisticsCollector: ------ Queue length threshold(50) value crossed: 75
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: ---- - : Message latency threshold(100) value crossed: 10829
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#---#CSTA#----AES-2 - : Message latency threshold(100) value crossed: 10828
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#----#CSTA#-------AES-2 - : Message latency threshold(100) value crossed: 10828
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#----#CSTA#-----AES-2 - : Message latency threshold(100) value crossed: 10828
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#-----#CSTA#----AES-2 - : Message latency threshold(100) value crossed: 10780
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#----#CSTA#-------AES-2 - : Message latency threshold(100) value crossed: 10780
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#---#CSTA#---AES-2 - : Message latency threshold(100) value crossed: 10779
[INFO ][2022-12-21 08:34:51,871][DistributeCSTAEvent] - PerfStatisticsCollector: AVAYA#----#CSTA#-------AES-2 - : Message latency threshold(100) value crossed: 10773

1) there is currently a problem where the session timers for the forums are not aligned with the session timers on the portal. If the forum session timer has expired, and the portal timer has not, it seems you can begin to post into a forum, but when you actually post you will find the page updates to the main forums index. If this happens use the browser back, re-access the post you had created and re-submit it. This overcomes that problem.

2) what release of the JTAPI client are you running. Till you get on a very recent version of 8.1.3 from the support site, we are viewing this as a known issue in the JTAPI client. As I said if you watch the traffic between the client and AES you will typically see an overabundance of traffic immediately after the event arrives at the client.

If this is not what is occurring
a) open a devconnect ticket, the forums are not where we want to be working this type of problem
b) provide AES csta logs and JTAPI client logs.