Hi,

AEP 7.2.3
POM 3.1.3.2.9.7

All POM services are up except Agent SDK Service.

Receiving the following error in my client. Would anyone have a clue. Using the Default zone, so I wouldn't think would be an issue. Much appreciated if someone can point me in the right direction.,

Received cert File path: D:\_dev\POM-AgentDesktop\Workspace\SamplePOMDesktopJavaAPI\pomTrustStore
Agent Login failed.9012

------------------------------------------------------------------------------------
AgentId Agent State Call State Nail State
11973 NotReady NoState NotNailedUp
------------------------------------------------------------------------------------
Agent login successful. Sending agent state change : Ready.
Agent state change failed.
Error info

Logs - I think the problem might be here. The client can not find the zone for the Agent.

04 Apr 2022 12:02:42,871 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:791 - POMAgentFactory::getSocketClientForZone() for zone : Default exitIfCannotConnectFlag = false
04 Apr 2022 12:02:42,871 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:803 - POMAgentFactory::getSocketClientForZone() zoneSocketMap does not contain : Default
04 Apr 2022 12:02:42,871 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:835 - POMAgentFactory::getSocketClientForZone() zoneIPMap does not contain : Default
04 Apr 2022 12:02:42,871 [main] FINEST - InitPOMAgentFactory.getMgmtSocketClient:913 - POMAgentFactory::GetMgmtSocketClient()
04 Apr 2022 12:02:42,872 [main] FINEST - POMAgent.GetPAMForZone:2136 - POMAgent::GetPAMForZone() - -1
04 Apr 2022 12:02:42,872 [main] FINEST - SocketClient.send:575 - SocketClient::send() - Waiting for lock ...
04 Apr 2022 12:02:42,872 [main] FINEST - SocketClient.send:579 - SocketClient::send() - adding ID: 999 packet - GetPAMForZone
04 Apr 2022 12:02:42,872 [main] FINEST - SocketClient.send:580 - SocketClient::send() - queue size is 1 adding agentCommand = com.avaya.sdk.Internal.AgentCommand@1d29cf23
04 Apr 2022 12:02:42,873 [main] FINE - POMAgent.initialize:374 - POMAgent::AGTLogon() - PAM Zone manager where zoneName = Default for agent 11973 not found.
04 Apr 2022 12:02:42,873 [main] FINE - POMAgent.initialize:375 - POMAgent::AGTLogon() - Sleeping for 300 ms.
...
04 Apr 2022 12:02:43,174 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:791 - POMAgentFactory::getSocketClientForZone() for zone : Default exitIfCannotConnectFlag = false
04 Apr 2022 12:02:43,175 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:803 - POMAgentFactory::getSocketClientForZone() zoneSocketMap does not contain : Default
04 Apr 2022 12:02:43,175 [main] FINEST - InitPOMAgentFactory.getSocketClientForZone:835 - POMAgentFactory::getSocketClientForZone() zoneIPMap does not contain : Default
04 Apr 2022 12:02:43,175 [main] FINEST - InitPOMAgentFactory.getMgmtSocketClient:913 - POMAgentFactory::GetMgmtSocketClient()
04 Apr 2022 12:02:43,176 [main] FINEST - POMAgent.GetPAMForZone:2136 - POMAgent::GetPAMForZone() - -1
04 Apr 2022 12:02:43,176 [main] FINEST - SocketClient.send:575 - SocketClient::send() - Waiting for lock ...
04 Apr 2022 12:02:43,176 [main] FINEST - SocketClient.send:579 - SocketClient::send() - adding ID: 999 packet - GetPAMForZone
04 Apr 2022 12:02:43,176 [main] FINEST - SocketClient.send:580 - SocketClient::send() - queue size is 0 adding agentCommand = com.avaya.sdk.Internal.AgentCommand@4d95d2a2
04 Apr 2022 12:02:43,176 [main] FINE - POMAgent.initialize:374 - POMAgent::AGTLogon() - PAM Zone manager where zoneName = Default for agent 11973 not found.
...
*********************************************************************************
POM_METRICS CLIENT (d1ptaeppri.lan.creditcorp.com.au:9971): Sending : AGTLogon C11973 999 8 5 159735 119734 true5 en-AU16 Australia/Sydney7 Default6 Agent126 271;POM-Outbound;1;###@###
*********************************************************************************

04 Apr 2022 12:02:45,358 [SocketClient-21] FINEST - SocketClient.run:553 - SocketClient::ProcessThread() bytesRead from socket - (d1ptaeppri.lan.creditcorp.com.au:9971) packet - [65, 103, 101, 110, 116, 83, 68, 75, 67, 111, 110, 110, 101, 99, 116, 101, 100, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 82, 45, 49, 68, 101, 102, 97, 117, 108, 116, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 55, 54, 55, 32, 32, 32, 32, 32, 51, 32, 32, 32, 48, 32, 32, 32, 32, 49, 52, 32, 32, 32, 32, 32, 32, 83, 101, 115, 115, 105, 111, 110, 73, 68, 61, 78, 85, 76, 76, 52, 55, 32, 32, 32, 32, 32, 32, 65, 100, 101, 108, 97, 105, 100, 101, 59, 100, 49, 112, 116, 97, 101, 112, 112, 114, 105, 46, 108, 97, 110, 46, 99, 114, 101, 100, 105, 116, 99, 111, 114, 112, 46, 99, 111, 109, 46, 97, 117, 59, 57, 57, 55, 50, 59, 52, 54, 32, 32, 32, 32, 32, 32, 68, 101, 102, 97, 117, 108, 116, 59, 100, 49, 112, 116, 97, 101, 112, 112, 114, 105, 46, 108, 97, 110, 46, 99, 114, 101, 100, 105, 116, 99, 111, 114, 112, 46, 99, 111, 109, 46, 97, 117, 59, 57, 57, 55, 49, 59, 3] size = 210
04 Apr 2022 12:02:45,358 [SocketClient-21] FINEST - IncomingPacketProcessor.addPacketToQueue:67 - IncomingPacketProcessor::addPacketToQueue
04 Apr 2022 12:02:45,359 [Thread-9] FINEST - SocketUtil.processPacketReceived:250 - SocketUtil::processPacketReceived()
******************************************************************************************
POM_METRICS CLIENT : Received header = AgentSDKConnected R-1Default 767 0 packet = AgentSDKConnected R-1Default 767 0 0 params = SessionID=NULLAdelaide;d1ptaeppri.lan.creditcorp.com.au;9972;Default;d1ptaeppri.lan.creditcorp.com.au;9971;
******************************************************************************************
04 Apr 2022 12:02:45,361 [AgentEventListener16] FINEST - AgentEventListener.processEvent:189 - AgentEventListener::processEvent() receiverQ size = 1
04 Apr 2022 12:02:45,361 [AgentEventListener16] FINEST - AgentEventListener.processEvent:200 - AgentEventListener::processEvent() socketFeedback header = AgentSDKConnected:R:767 packet = AgentSDKConnected R-1Default 767 0 0 params = SessionID=NULLAdelaide;d1ptaeppri.lan.creditcorp.com.au;9972;Default;d1ptaeppri.lan.creditcorp.com.au;9971;
04 Apr 2022 12:02:45,361 [AgentEventListener16] FINEST - AgentEventListener.processSocketFeedback:243 - AgentEventListener::processSocketFeedback() Mon Apr 04 12:02:45 AEST 2022RECEIVED FROM PAM: AgentSDKConnected WITH ID:767
04 Apr 2022 12:02:45,362 [AgentEventListener16] FINEST - InitPOMAgentFactory.setZonePAMMap:1197 - POMAgentFactory::SetZonePAMMap() - pair is Adelaide : d1ptaeppri.lan.creditcorp.com.au:9972
04 Apr 2022 12:02:45,362 [AgentEventListener16] FINEST - InitPOMAgentFactory.setZonePAMMap:1212 - POMAgentFactory::SetZonePAMMap() - Setting new value - d1ptaeppri.lan.creditcorp.com.au:9972
04 Apr 2022 12:02:45,362 [AgentEventListener16] FINEST - InitPOMAgentFactory.setZonePAMMap:1197 - POMAgentFactory::SetZonePAMMap() - pair is Default : d1ptaeppri.lan.creditcorp.com.au:9971
04 Apr 2022 12:02:45,362 [AgentEventListener16] FINEST - InitPOMAgentFactory.setZonePAMMap:1201 - POMAgentFactory::SetZonePAMMap() - Found existing value - d1ptaeppri.lan.creditcorp.com.au:9971
04 Apr 2022 12:02:45,362 [AgentEventListener16] FINEST - InitPOMAgentFactory.setZonePAMMap:1212 - POMAgentFactory::SetZonePAMMap() - Setting new value - d1ptaeppri.lan.creditcorp.com.au:9971
04 Apr 2022 12:02:45,363 [AgentEventListener16] FINEST - AgentEventListener.executeAgentTasksInSequence:1838 - In executeAgentTasksInSequence.
04 Apr 2022 12:02:45,363 [AgentEventListener16] FINEST - InitPOMAgentFactory.checkOrCreateLockForAgent:1388 - POMAgentFactory::checkOrCreateLockForAgent() Creating lock for -1Default at Mon Apr 04 12:02:45 AEST 2022 TID (16)
04 Apr 2022 12:02:45,363 [pool-1-thread-3] FINEST - InitPOMAgentFactory.getLock:1372 - POMAgentFactory::getLock() Lock found for -1Default at Mon Apr 04 12:02:45 AEST 2022 TID (23)
04 Apr 2022 12:02:45,364 [pool-1-thread-3] FINEST - AgentFeedbackExecutor.run:61 - AgentFeedbackExecutor::run() Trying to lock for -1Default
04 Apr 2022 12:02:45,364 [pool-1-thread-3] FINEST - AgentFeedbackExecutor.run:70 - AgentFeedbackExecutor::run() Got lock for -1Default for AgentSDKConnectedRESP
04 Apr 2022 12:02:45,365 [POMSDKAgentFeedbackExecutor--1Default-23] FINEST - AgentFeedbackExecutor.invoke:196 - DONE INVOKE FOR : class com.avaya.sdk.Internal.InternalPOMAgentHandler WITH ID:767
04 Apr 2022 12:02:45,365 [POMSDKAgentFeedbackExecutor--1Default-23] FINEST - InitPOMAgentFactory.removeIfLockDone:1410 - POMAgentFactory::removeIfLockDone() Lock unlocked for -1Default at Mon Apr 04 12:02:45 AEST 2022 TID (23)
04 Apr 2022 12:02:45,365 [POMSDKAgentFeedbackExecutor--1Default-23] FINEST - InitPOMAgentFactory.removeIfLockDone:1415 - POMAgentFactory::removeIfLockDone() Removed lock for -1Default at Mon Apr 04 12:02:45 AEST 2022 TID (23)
04 Apr 2022 12:02:45,366 [POMSDKAgentFeedbackExecutor--1Default-23] FINEST - AgentFeedbackExecutor.run:97 - AgentFeedbackExecutor::run() Task done for -1Default for AgentSDKConnectedRESP
04 Apr 2022 12:02:45,639 [SocketClient-21] FINEST - SocketClient.run:553 - SocketClient::ProcessThread() bytesRead from socket - (d1ptaeppri.lan.creditcorp.com.au:9971) packet - [65, 71, 84, 76, 111, 103, 111, 110, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 82, 49, 49, 57, 55, 51, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 57, 57, 57, 32, 32, 32, 32, 32, 48, 32, 32, 32, 57, 48, 49, 50, 32, 3] size = 79
04 Apr 2022 12:02:45,639 [SocketClient-21] FINEST - IncomingPacketProcessor.addPacketToQueue:67 - IncomingPacketProcessor::addPacketToQueue
04 Apr 2022 12:02:45,639 [Thread-9] FINEST - SocketUtil.processPacketReceived:250 - SocketUtil::processPacketReceived()
******************************************************************************************
POM_METRICS CLIENT : Received header = AGTLogon R11973 999 0 packet = AGTLogon R11973 999 0 9012 params =
******************************************************************************************

Further I found these logs.

Which raises the questions. Does the Agent SDK API requires licenses? I don't recall this being a requirement and what is

SocketServer.packetArrived:255 - Server is not initialized! sending error response

All POM services are up and running as far as I can see, including agtmgr.

04 Apr 2022 16:26:32,338 [AM-Adelaide] FINEST - AgentLicenseManager.printLicense:875 - LICENSES: -1 : Preview: 0 Predictive: 100 Actual Preview : 0 Actual Predictive : 100 Diff Preview : 0 Diff Predictive : 0 Inprogress preview : 0 Inprogress predictive : 0 agentSDKApiLicenses: 0 Actual agentSDKApiLicenses : 0 Diff agentSDKApiLicenses : 0 Inprogress agentSDKApiLicenses : 0
...
04 Apr 2022 16:51:47,905 [SSLChannel-10.0.21.35:56090] FINEST - SocketUtil.processPacketReceived:196 - Cmd:AGTLogon Type:C min param:6 actual:6
04 Apr 2022 16:51:47,905 [SSLChannel-10.0.21.35:56090] FINE - SocketUtil.processPacketReceived:226 - -----------------------------------------------------------------------
04 Apr 2022 16:51:47,905 POM_METRICS Received Data 21 : AGTLogon C11973 999 6 5 159735 119734 true5 en-AU16 Australia/Sydney7 Default^C
-----------------------------------------------------------------------
Packet byte length : [140]
Packet String length : [140]
packet ID:[999]
Data Offset : [73]
Data : [5 159735 119734 true5 en-AU16 Australia/Sydney7 Default^C]

04 Apr 2022 16:51:47,905 [SSLChannel-10.0.21.35:56090] FINEST - SocketServer.packetArrived:217 - Message :AGTLogon C11973 999 0
04 Apr 2022 16:51:47,905 [SSLChannel-10.0.21.35:56090] FINEST - SocketServer.packetArrived:255 - Server is not initialized! sending error response
04 Apr 2022 16:51:47,905 [SSLChannel-10.0.21.35:56090] FINE - SSLChannel.flushData:381 - flushData

Here are a few things to consider

Common error/notification on agent desktop for error 9012 : Agent manager is not initialized properly. Common reason for this is that certificates on server side are not exchanged properly. Check the Appserver keystore as well as the server.xml file on the appserver.

Remember that a password is required to be set on the Agent login (not a security code)

The Agent Web API Service Licenses are required for the POMAgentSDK service. This SDK is available only for Workspace Elite Integration. POM agent SDK microservice acts as proxy between Web desktop (workspace) and Agent manager service.

Thanks Mike.

This is CCElite install. Currently using the SampleApp to test so Workspace is not involved. Currently don't have licenses for Agent web service but working on it.

I've been looking at certificates as I feel they may be the cause of my pain.

App Server, have the following certificates in my TrustStore;
- EPM axis certificate
- EPM Root certificate
- POM certificate

For EPM I have the certificates;
- App Server certificate
- POM certificate
- EPM axis certifcate

In POM, I have certificates;
- EPM certificate
- EPM root certificate
- App server certificate

are there any other certificates?

I'm not the Certificate expert, but your list looks OK to me.

Check out this Avaya KB, it has some info on the 9012 error
https://support.avaya.com/ext/index?page=content&id=SOLN315394

You mentioned that the POM services are all up.
I'll assume you've had success running a test agentless campaign. It's best to use the AvayaPOMAnnouncement application in your test campaign since that will verify that your OD license is accessible by the app server (as well as verifying all the certs).

Thanks Mike.

This is now resolved.

It was indeed the certificates. The App server had outdated EPM certificates. For anyone who is searching;

1. There are two certificates for EPM which you can grab from urls;
- https://<epm_server>;
- https://<epm_server>/axis2

2. Make sure your MPPs can connect to the app server. This was my second gotcha. The App server was on a Windows machine and I was obsessed with certificates that I did not check MPP <--> App Server connection. I did check on EPM Admin with the validate and it worked but this not always enough, so it is always good to logon to the MPPs and use curl to check connectivity to application urls on the App Server.

3. Certificates can be confusing, so if you want to be safe, just add all the following certificates to EPM, POM and App Server;
- EPM certificates
- EPM Axis 2 certificates
- EPM Root CA certificate
- POM certificates
- App Server certificates

4. Use fqdn names and make sure each server can resolve the fqdn of every other server.