Ok thanks, we will try turning on AES tracing to see if it gives more info. So far the only maintenance we've found running is backups running during about half of the instances of the past month where this has happened.

I found this info in the logs on one of the servers that blipped on 8/15 if this helps narrow it down

EMC XML log
Aug 15 04:06:59 EMC-XML-10 AXMLServer[1868]: -05:00 2019 901 1 .cce | 0 Major [3472] CTID3472 caught ACSUnSolicitedClass(error 0:StreamFailed) on acsHandle 18089152 and 1 licenses (clients) were released.


AES log.log
04:03:41.753 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to TCP Down(207)
04:03:41.753 ERROR:WARNING:trans_serv:PreserveSession:Session, 5ce51740, for switch, swlink20, has been preserved.
04:03:42.702 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Restart(206)
04:03:42.710 ERROR:WARNING:trans_serv:processConnAcceptData:Preserved session, 5ce51740, for switch, swlink20, MID, 1, has been restored.
04:03:42.710 ERROR:FYI:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Talking(202)
04:04:54.179 ERROR:WARNING:trans_serv:ReadMsg:Error notification received switch: swlink20 dest: 172.23.70.25 cause: 5 data: <none>
04:04:54.179 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to No Reply To HB(5)
04:04:54.179 ERROR:WARNING:trans_serv:PreserveSession:Session, 5ce51740, for switch, swlink20, has been preserved.
04:04:56.898 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Restart(206)
04:04:57.228 ERROR:WARNING:trans_serv:processConnAcceptData:Preserved session, 5ce51740, for switch, swlink20, MID, 1, has been restored.
04:04:57.230 ERROR:FYI:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Talking(202)
04:05:14.850 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to TCP Down(207)
04:05:14.850 ERROR:WARNING:trans_serv:PreserveSession:Session, 5ce51740, for switch, swlink20, has been preserved.
04:05:15.720 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Restart(206)
04:05:15.768 ERROR:WARNING:trans_serv:processConnAcceptData:Preserved session, 5ce51740, for switch, swlink20, MID, 1, has been restored.
04:05:15.768 ERROR:FYI:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Talking(202)
04:05:29.522 ERROR:WARNING:trans_serv:SendMsg:SendMsg() failed Error= 5 switch= swlink20 IP= 172.23.70.25
04:05:30.134 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to TCP Down(207)
04:05:30.135 ERROR:WARNING:trans_serv:PreserveSession:Session, 5ce51740, for switch, swlink20, has been preserved.
04:05:30.819 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Restart(206)
04:05:30.889 ERROR:WARNING:trans_serv:processConnAcceptData:Preserved session, 5ce51740, for switch, swlink20, MID, 1, has been restored.
04:05:31.075 ERROR:FYI:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Talking(202)
04:05:38.155 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to TCP Down(207)
04:05:38.155 ERROR:WARNING:trans_serv:PreserveSession:Session, 5ce51740, for switch, swlink20, has been preserved.
04:05:39.119 ERROR:WARNING:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Restart(206)
04:05:39.144 ERROR:WARNING:trans_serv:processConnAcceptData:Preserved session, 5ce51740, for switch, swlink20, MID, 1, has been restored.
04:05:39.144 ERROR:FYI:trans_serv:SetStatus:Connection state for link 172.23.70.25 on switch swlink20 changed to Talking(202)


AES sec.log
04:05:47.258 SECURITY:FYI:TSAPI:ClientHandler::streamClosed():user emcxmluser10 with IP Address 172.23.73.40 disconnected from driver service AVAYA#SWLINK20#CSTA-S#EMC-AES-10
04:05:47.524 SECURITY:FYI:TSAPI:ClientHandler::streamClosed():user emcicruser10 with IP Address 172.23.73.40 disconnected from driver service AVAYA#SWLINK20#CSTA-S#EMC-AES-10
04:06:05.762 SECURITY:FYI:TSAPI:tsviAuthenticateSession:user emcxmluser10 logged in to AVAYA#SWLINK20#CSTA-S#EMC-AES-10 from 172.23.73.40 successfully
04:06:09.743 SECURITY:FYI:TSAPI:tsviAuthenticateSession:user emcicruser10 logged in to AVAYA#SWLINK20#CSTA-S#EMC-AES-10 from 172.23.73.40 successfully

Thanks I will try turning on tracing. I checked the log.YYYYMMDD file and it only has 3 entries for the entire day, none around the time it dropped connection. I looked at some of the previous days and most files have very few entries, below is the most common one.

08:27:18.060 ERROR:WARNING:TSAPI:terminates:Unexpected termination for primitive 61

We have a set of EMC XML Servers we're using to monitor phones and we're getting TServerNotActive errors every couple days causing the EMC servers to lose connection to AES. Is there any logging we can setup in AES to troubleshoot why its losing connection?

So if this application makes a CSTAMakeCall request on a AES thats not monitoring the device, how long does it hold onto the Domain Control? Is there a way in the CM to find out which AES servers have domain control of a device?

We have an application using EMC XML servers to do screen pops and call control and we're getting outstandingRequestLimitExceeded errors with CSTAMakeCall. Below is a sample of the request and error in the TSAPI logs on one of the AES servers. What does the oustanding request limit exceeded mean? If theres a limit around number of concurrent requests it can process do you know what that limit is? Is the limit at the CM level or AES level because we have 11 AES boxes.

CSTA Log:
06/02/2017 10:23:49.444:TSAPI:Thread 0x977d0b90: TSERVER Src: CLNT
06/02/2017 10:23:49.444:TSAPI:Thread 0x977d0b90: Login: "emcxmluser1" App Name: "AXMLServer" SessionID: 40 Transport ID: 172.23.73.25
06/02/2017 10:23:49.444:TSAPI:Thread 0x977d0b90: InvokeID: 100017960
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: Driver: AVAYA#SWLINK11#CSTA-S#EMC-AES-1
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: Message:
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: value CSTAMakeCall ::=
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: {
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: callingDevice "55504",
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: calledDevice "917087616308"
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: }
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90:
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: Private Data:
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: value ATTMakeCall ::=
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: {
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: destRoute "",
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: priorityCalling FALSE,
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: userInfo
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: {
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: type uuiIa5Ascii,
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: data ''H
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: }
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90: }
06/02/2017 10:23:49.445:TSAPI:Thread 0x977d0b90:
06/02/2017 10:23:49.530:TSAPI:Thread 0x9b3d6b90: TSERVER Src: DRVR
06/02/2017 10:23:49.530:TSAPI:Thread 0x9b3d6b90: Login: "emcxmluser1" App Name: "AXMLServer" SessionID: 40 Transport ID: 172.23.73.25
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: InvokeID: 100017960
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: Driver: AVAYA#SWLINK11#CSTA-S#EMC-AES-1
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: Message:
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: value UniversalFailureConfEvent ::=
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: {
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: error outstandingRequestLimitExceeded
06/02/2017 10:23:49.531:TSAPI:Thread 0x9b3d6b90: }


G3trace:
06/02/2017 10:23:49.454:TSAPI:swlink11: Received CSTA Msg (CSTAREQUEST)(3) (CSTA_MAKE_CALL)(23)
06/02/2017 10:23:49.454:TSAPI:swlink11: msgoff(76)msglen(128)privoff(24)privlen(52)
06/02/2017 10:23:49.454:TSAPI:swlink11: invokeID(20503)monXref(0)sessionID(40)clsOfServ(0)
06/02/2017 10:23:49.454:TSAPI:swlink11: callingDevice(55504)
06/02/2017 10:23:49.454:TSAPI:swlink11: calledDevice(917087616308)
06/02/2017 10:23:49.454:TSAPI:swlink11: eventType(ATT_MAKE_CALL)(110)
06/02/2017 10:23:49.454:TSAPI:swlink11: priorityCalling(F)
06/02/2017 10:23:49.454:TSAPI:swlink11: destRoute()
06/02/2017 10:23:49.454:TSAPI:swlink11: userInfotype(UUI_IA5_ASCII)(4)
06/02/2017 10:23:49.454:TSAPI:swlink11: .length(0)
06/02/2017 10:23:49.456:TSAPI:swlink11: Sent ASAI Message A3PDCInfo
06/02/2017 10:23:49.456:TSAPI:swlink11: cap,prim,sao_id C_3PDC_REQ,C_REQUEST,42182968
06/02/2017 10:23:49.456:TSAPI:swlink11: extension 55504
06/02/2017 10:23:49.456:TSAPI:swlink11: domain_type 3

06/02/2017 10:23:49.528:TSAPI:LinkThread01: Received ASAI Message A3PDCNak
06/02/2017 10:23:49.528:TSAPI:LinkThread01: cap,prim,sao_id C_3PDC_CONF,C_NEG_ACK,42182968
06/02/2017 10:23:49.528:TSAPI:LinkThread01: cause_value CS3/63
06/02/2017 10:23:49.529:TSAPI:LinkThread01: Sent CSTA Msg (CSTACONFIRMATION)(5) (CSTA_UNIVERSAL_FAILURE_CONF)(53)
06/02/2017 10:23:49.529:TSAPI:LinkThread01: msgoff(24)msglen(4)privoff(0)privlen(0)
06/02/2017 10:23:49.529:TSAPI:LinkThread01: invokeID(20503)monXref(2989)sessionID(40)clsOfServ(0)
06/02/2017 10:23:49.529:TSAPI:LinkThread01: error(OUTSTANDING_REQUEST_LIMIT_EXCEEDED)(44)

I'm trying to implement single step transfer with 6.3 and getting an unsupported error. I found in some documentation that it requires private data level 8 or higher and the phone events are saying version 1. How do I change the privatedata version using the EMC XML c# libraries?

Is there a way of blocking outbound calls via EMC? Use case would be to block people from calling do not call numbers. Only solution i've come up with so far is on CSTAOriginated do the data dip to see if that number should be blocked then call CSTAClearCall but the call does ring for a split second.

I dont see a DMCC test utility but the TSAPI test utility worked

I forgot to add that we do have the tracing levels set to finest so if it would help to provide the entire dmcc-trace file, I can do that.

The switch name is correct and we tried restarting both the TSAPI and DMCC services

We're receiving this error when trying to register a device through DMCC. We've compared the switch connection settings on this AES box with another AES box that works fine and don't see any differences. We don't even have a H323 GK defined on the AES box having this problem or one of our other AES boxes that are working fine. The AES box with the error below is 6.2, the working AES box is 6.1.

2013-10-02 16.45.20,707 :T-76: com.avaya.platform.broker.impl.AsyncServiceMethodImpl invoke
WARNING: Exception when calling method.invoke
java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at com.avaya.platform.broker.impl.AsyncServiceMethodImpl.invoke(AsyncServiceMethodImpl.java:164)
   at com.avaya.workflow.impl.RouterImpl.routeRequest(RouterImpl.java:97)
   at com.avaya.mvcs.proxy.CstaRouter.routeRequest(CstaRouter.java:173)
   at com.avaya.mvcs.proxy.CstaRouterService.routeRequest(CstaRouterService.java:141)
   at com.avaya.mvcs.proxy.CstaRouterNode.processPacket(CstaRouterNode.java:349)
   at com.avaya.mvcs.proxy.AbstractPipelineNode.process(AbstractPipelineNode.java:130)
   at com.avaya.mvcs.proxy.Pipeline$PipelineSubscriber.inform(Pipeline.java:427)
   at com.avaya.common.eventservice.UnfilteredSubscription.notify(UnfilteredSubscription.java:70)
   at com.avaya.common.multicaster.Multicaster.notify(Multicaster.java:386)
   at com.avaya.common.channel.Channel.publish(Channel.java:115)
   at com.avaya.common.eventservice.EventService.publish(EventService.java:123)
   at com.avaya.common.eventservice.EventServiceManager.publish(EventServiceManager.java:156)
   at com.avaya.common.eventservice.Publisher.publish(Publisher.java:110)
   at com.avaya.mvcs.proxy.CstaUnmarshallerNode$CstaUnmarshallerProcessorThread.run(CstaUnmarshallerNode.java:255)
   at com.avaya.common.util.concurrent.impl.RunnableWrapper.run(RunnableWrapper.java:56)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:722)
Caused by: ch.ecma.csta.errors.InvalidParameterValueException: java.lang.IllegalArgumentException: [swlink1:0.0.0.0:52323:0] No GK IP address is available for switch=swlink1
   at com.avaya.cmapi.intsvc.CstaTerminalImpl.register(CstaTerminalImpl.java:451)
   at com.avaya.cmapi.intsvc.CstaTerminalImpl.registerTerminal(CstaTerminalImpl.java:652)
   at com.avaya.cmapi.extsvc.H323RegistrationServices.registerTerminal(H323RegistrationServices.java:100)
   ... 22 more
Caused by: java.lang.IllegalArgumentException: [swlink1:0.0.0.0:52323:0] No GK IP address is available for switch=swlink1
   at com.avaya.mvcs.station.h323.ras.DefaultTerminalDevice.privateRegisterDevice(DefaultTerminalDevice.java:686)
   at com.avaya.mvcs.station.h323.ras.DefaultTerminalDevice.registerDevice(DefaultTerminalDevice.java:634)
   at com.avaya.mvcs.station.h323.H323Station.privateRegisterDevice(H323Station.java:307)
   at com.avaya.mvcs.station.h323.H323Station.registerDevice(H323Station.java:285)
   at com.avaya.mvcs.station.h323.H323Station.registerSharedDevice(H323Station.java:251)
   at com.avaya.mvcs.terminal.h323.TerminalH323.privateRegisterDevice(TerminalH323.java:387)
   at com.avaya.mvcs.terminal.h323.TerminalH323.registerDevice(TerminalH323.java:288)
   at com.avaya.mvcs.terminal.h323.TerminalH323.registerSharedDevice(TerminalH323.java:234)
   at com.avaya.cmapi.intsvc.TerminalSession.registerDevice(TerminalSession.java:174)
   at com.avaya.cmapi.intsvc.CstaTerminalImpl.register(CstaTerminalImpl.java:437)
   ... 24 more

Martin,
I did some quick testing and believe this will work. Its giving me a "redirected" event clause for the phone dialed and "keyoperation" on all the others. I do get the "newCall" value on internal calls. Would you be able to provide all the possible event clauses so I can handle them all in our code?

Thanks!

Vectors and VDN's within the CM

our automated message that answers the call saying if you know your parties extension dial it now, otherwise press 1 for XX, press 2 for XX, etc