Additional Logs

Hi Team,

We are using AES connector in OD8.1.2 with Websphere.
And JTAPI(tsapi) tracelog is working output without any issue.
However this tracelog output is suddenly stopped without any configuration change.
This issue has occurred in multiple AesConnectors.
I collected some tracelog files and tsapi.pro file.

How should we do to resolve this issue. Please advise.

Description in tsapi.pro :
altTraceFile=(specified directory)/tsapi_trace.txt.CCSWebServer11
traceFileCount=31
traceFileSize=100MB

<Product>
HW?ASP130(PowerEdge R640)
SW?EP/OD 8.1.2

?EPM?
<OVA for deploy>
ExperiencePortal-Primary-EPM-8.1.2.0.0202.ova
<Patches>
epavl-8.x.x.0.2210.tar.gz
EPM_8.1.2.0.0347.tar.gz
?MPP?
<OVA for deploy>
ExperiencePortal-MPP-8.1.2.0.0202.ova
<Patches>
epavl-8.x.x.0.2210.tar.gz
8.1.2.0.0346.tar.gz
?Web Application Server?
Orchestration Designer?08.12.11.01
JAVA?IBM Java V1.8.0
WAS?WebSphere Application Server V9.0.2.x
OS?RHEL8.1

Thank you, K.Yamahara

Hi all.

I got below suggestion. After follow it, I could resolved this issue.
===
Following two files located in “<aesconnector install directory>\WEB-INF\lib” needs to be move/copy to “lib\ext” directory.

ecsjtapia.jar (Move to "C:\Program Files\IBM\WebSphere\AppServer\lib\ext")
avayaprca.jks (Copy to "C:\Program Files\IBM\WebSphere\AppServer\lib\ext")

Logs are generated as defined in “<aesconnector install directory>\WEB-INF\classes\ log4j2.xml”
===

Thank you, Kenji

Hi Team,

We are facing "AES Connector Not Logging" issue on OD 8.1.2.1101 on Websphere.
It seems "AES Connector" is working well without any error.
However only logs were not outputted after below logged message.

30/05/2023 18:06:31:654 DEBUG - CTIConnectorManager.setTServerWait: TServer wait time is 4
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.init: initing AES connector. Verbosity set at 3
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.init: AES Connector Version is 08.12.11.01
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.CheckObserverTimeoutSet: Observer timeout interval found: 100
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.CheckObserverTimeoutSet: Provider startup delay found: 4000
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.getConnectionString: No entry for TServer in configuration file
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.initProvider: Look for TServer config in DD application configuration (new method to 4.0)
30/05/2023 18:06:31:661 DEBUG - CTIConnectorManager.ConfigureFromConfigurationFile: getting provider:AVAYA#CM#CSTA-S#TPBKCCS0PA001F0

On the other hand, I found below as known issue in release note.
In this case of workaround, it is for Tomcat.
If we are using OD with Websphere, please advise how to fix this issue as workaround.
I'm asking same question to OD team as well.
However this is issue on AES connector side.

Avaya Orchestration Designer 8.1.2.1101 release note
4 Known Issues
• AES Connector Not Logging
Due to the upgrade of JTAPI to support Log4j v2, AES Connector might stop logging at the following statement (in italic) in early stage of setting up the AES connection.
CTIConnectorManager.ConfigureFromConfigurationFile: getting provider:xxxxxxxxxxxxx
*Note: by default, a version of log4j.xml has also been placed in the WEB-INF/classes directory. This could fix the issue without the following workaround.
When this happens, it is necessary to take the following steps:
a. Go to the TOMCAT_HOME/webapps/aesconnector/WEB-INF/ directory
b. Copy the aeslog4j2.xml file into TOMCAT_HOME/lib as log4j.xml
c. Edit this line to tell aesconnector where to create the log file: <Property name="log-path"> ${log4j:configParentLocation}/../data/log</Property>
Or it will just create the trace.log file in the TOMCAT_HOME/data/log directory as the default

Best regards, K.Yamahara

Hi Massimo-san,

We confirmed "xml-apis.jar" was described in MANIFEST.M of below jar files.
- serializer-2.7.2.jar
- xalan-2.7.2.jar

Let me confirm do we still require both files for OD?
If we need it, we should install "xml-apis.jar" on customer's system.
If we don't need it, we will removed both file from class loader for shared library

- serializer-2.7.2.jar / META-INF / MANIFEST.MF
Manifest-Version: 1.0
Created-By: 1.7.0_51 (Oracle Corporation)
Main-Class: org.apache.xml.serializer.Version
Class-Path: xml-apis.jar

- xalan-2.7.2.jar / META-INF / MANIFEST.MF
Manifest-Version: 1.0
Created-By: 1.7.0_51 (Oracle Corporation)
Main-Class: org.apache.xalan.xslt.Process
Class-Path: xercesImpl.jar xml-apis.jar serializer.jar

Best regards, K.Yamahara

Hi Team,

We are facing "AES Connector Not Logging" issue on OD 8.1.2.1101 on Websphere.
It seems "AES Connector" is working well.
However only logs were not outputted after below logged message.

30/05/2023 18:06:31:654 DEBUG - CTIConnectorManager.setTServerWait: TServer wait time is 4
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.init: initing AES connector. Verbosity set at 3
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.init: AES Connector Version is 08.12.11.01
30/05/2023 18:06:31:657 DEBUG - CTIConnectorManager.CheckObserverTimeoutSet: Observer timeout interval found: 100
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.CheckObserverTimeoutSet: Provider startup delay found: 4000
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.getConnectionString: No entry for TServer in configuration file
30/05/2023 18:06:31:658 DEBUG - CTIConnectorManager.initProvider: Look for TServer config in DD application configuration (new method to 4.0)
30/05/2023 18:06:31:661 DEBUG - CTIConnectorManager.ConfigureFromConfigurationFile: getting provider:AVAYA#CM#CSTA-S#TPBKCCS0PA001F0

On the other hand, I found below as known issue in release note.
In this case of workaround, it is for Tomcat.
If we are using OD with Websphere, please advise how to fix this issue as workaround.

Avaya Orchestration Designer 8.1.2.1101 release note
4 Known Issues
• AES Connector Not Logging
Due to the upgrade of JTAPI to support Log4j v2, AES Connector might stop logging at the following statement (in italic) in early stage of setting up the AES connection.
CTIConnectorManager.ConfigureFromConfigurationFile: getting provider:xxxxxxxxxxxxx
*Note: by default, a version of log4j.xml has also been placed in the WEB-INF/classes directory. This could fix the issue without the following workaround.
When this happens, it is necessary to take the following steps:
a. Go to the TOMCAT_HOME/webapps/aesconnector/WEB-INF/ directory
b. Copy the aeslog4j2.xml file into TOMCAT_HOME/lib as log4j.xml
c. Edit this line to tell aesconnector where to create the log file: <Property name="log-path"> ${log4j:configParentLocation}/../data/log</Property>
Or it will just create the trace.log file in the TOMCAT_HOME/data/log directory as the default

Best regards, K.Yamahara

Hi Massimo-san,

Thank you for your update.

>What exactly does it mean "the Voice Browser hangs up" ? Do you see any error message ? The OD 7.2.3 and 8.1.2 are using the same AVB/VXI 7.2.1.5
[Kenji]
In our call flow, the Voice Browser was looks like hangs up. Then call was disconnected.
And we confirmed below message in logs of getmpplogs.sh.
Therefore we understood VXMLMGR were stopped or restarted.
---
@2023-05-11 10:01:11,972|PSESM00002|WARN|SessMgr|13420|FileName=AppCtrl.cpp,LineNumber=233|Network error occurred on VXML message channel, error = 107|vmmpp81287####
---
@2023-05-11 10:01:11,973||FINE|SessMgr|1904||CSessionMgr::Monitor - App process error has occurred: VXMLMGR, with error 107|vmmpp81287####
----
Best regards, K.Yamahara

Hi Team,

We were used EPM 7.0.2/OD 7.0.
Then we deployed new EPM 8.1.2/OD 8.1.2 with same flow on previous version.
As a results, voice browser hangs when executing certain call flows.

1. Run the application created by OD on EP
2. Execute some menu selection on the call flow
3. Phone goes silent and disconnects after menu selection

At first step of my investigation, we created a test flow using the grammar we used previous version.

The test flow has 4 Prompt&Collect, and each Prompt&Collect calls the provided grammar as follows.
"TestCustomGrammar" below is the OD flow name.

Prompt&Collect node:
1st: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_ContractS.grxml
2nd: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_Service2.grxml#main_rule
3rd: /TestCustomGrammar/data/Japanese/grammars/grammar_FcSelService__Dealings.grxml#main_rule
4th: /TestCustomGrammar/data/Japanese/grammars/grammar_RefDetail_KamokuInput.grxml#main_rule

Next, we tried to reproduce this issue with combination of EPM and OD version using sample flow with same grammar file.
As a results, we can say that this issue were occurred when we used EP8.1.

Case1: EP7.0 / OD7.0.1 - OK
Case2: EP8.1.2 / OD8.1.2 - NG
Case3: EP8.1.2 / OD7.0.1 - NG
Case4: EP7.0 / OD8.1.2 - OK

In NG case, 1st and 2nd Prompt&Collect node were working well. Then we confirmed same issue was occurred on 3rd Prompt&Collect node.
If we tried to add “#main_rule” id after 1st uri as below, we confirmed same issue was occurred on 1st Prompt&Collect node.

1st: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_ContractS.grxml#main_rule
2nd: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_Service2.grxml#main_rule
3rd: /TestCustomGrammar/data/Japanese/grammars/grammar_FcSelService__Dealings.grxml#main_rule
4th: /TestCustomGrammar/data/Japanese/grammars/grammar_RefDetail_KamokuInput.grxml#main_rule

Furthermore we confirmed all of Prompt&Collect node can be work without “#main_rule” id after uri as below.

1st: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_ContractS.grxml
2nd: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_Service2.grxml
3rd: /TestCustomGrammar/data/Japanese/grammars/grammar_FcSelService__Dealings.grxml
4th: /TestCustomGrammar/data/Japanese/grammars/grammar_RefDetail_KamokuInput.grxml

As a results, we can say that this issue was only occurred when we use “EP8.1.2” or “ #main_rule” after each uri on Avaya Japan lab.
Please advise whether the specification change or the supported node requirements change for the Voice Browser between EP/OD7.0 and EP/OD8.1.2 are there or not?

In our investigation, if grammar file has multiple rule id like below, if we add rule id after uri, Voice Browser didn't hung up.
-----
1st: /TestCustomGrammar/data/Japanese/grammars/grammar_Input_ContractS.grxml#main_rule

-- grammar_Input_ContractS.grxml --
<?xml version="1.0" encoding="UTF-8"?>
<grammar version="1.0" xml:lang="ja-JP" root="main_rule" mode="dtmf" tag-format="semantics/1.0" xmlns="http://www.w3.org/2001/06/grammar">

<rule id="main_rule" scope="public">
<item><ruleref uri="#main_sub_rule"/>#</item>
</rule>

<rule id="main_sub_rule" scope="public">
<one-of>
<item repeat="8"><ruleref uri="#number"/></item>
<item repeat="10"><ruleref uri="#number"/></item>
</one-of>
</rule>

<rule id="number" scope="public">
<one-of>
<item>0</item>
<item>1</item>
<item>2</item>
<item>3</item>
<item>4</item>
<item>5</item>
<item>6</item>
<item>7</item>
<item>8</item>
<item>9</item>
</one-of>
</rule>

</grammar>
-----

If grammar file has single "rule id" like below, if we add rule id after uri, Voice Browser will be hung up.
Is this estimated results?
-----
4th: /TestCustomGrammar/data/Japanese/grammars/grammar_RefDetail_KamokuInput.grxml#main_rule

-- grammar_RefDetail_KamokuInput.grxml --
<?xml version="1.0" encoding="UTF-8"?>
<grammar version="1.0" xmlns="http://www.w3.org/2001/06/grammar" tag-format="semantics/1.0" mode="dtmf" root="main_rule">

<rule id="main_rule" scope="public">
<one-of>
<item>1#</item>
<item>2#</item>
<item>3#</item>
<item>4#</item>
<item>6#</item>
<item>7#</item>
</one-of>
</rule>

</grammar>
-----
Best regards, K.Yamahara

Hi Massimo-san,

Thank you for your support.

I tried to made the config guide for OD Runtime License Server Secure Access Configuration for Websphere.
Could you review and update it?

And could you accept to handle this ticket by escalated SR?

Best regards, K.Yamahara

Hi Massimo-san,

Thank you for your reply.

I understood what did you said in your last comments.
We need to modify/configure the OD runtimeconfig for Websphere with some config commands to use secure communication with IBMX509.
Then we are trying to do that. However it is very difficult to complete it.
It is almost same as new development for OD runtimeconfig with Websphere.
So, could you completely provide how to configure the OD runtimeconfig for Websphere with secure communication(IBMX509) again?

Anyway, I believe AVAYA should provide the developer guide or application note how to modify/configure the OD runtimeconfig for Websphere. And It should be guaranteed as an official configuration method by AVAYA.
Because AVAYA said OD is supporting websphere.

Best regards, Kenji

Hi Massimo-san,

Thank you for your support.

As you said below, we need to configure name for our configuration to use "IbmX509" to comminicate with Weblm with secure link.
So, please advise how to configure the Name for "IbmX509"?
If there are some recommended procedure in Dev guide or other Devconn thread, please advise.
Futhermore if we need to configure some of setting related secure TLS communication with WebLM server except Name, please advise what setting should we perform.

> with IBM, the JVM should be "IbmX509". "SunX509" is the default value used by JBoss. Did you explicitly configure that name ?

If those configuration defficults to change by ourselves, could you please release other runtimeconfig for secure link with Websphere.

Best regards, K.Yamahara

Hi Massimo-san,

Thank you for your reply.
We will remove the entry that reference below file.
The logs say: /AP/IHS/htdocs/CCSWebServer11/runtimeSupport/xml-apis.jar

Anyway, please remove the entry from original file in OD which will release in the future version.

So, let's close this forum.
Best regards, Kenji

Hi Massimo-san,

Thank you for your support.

If you can provide all of required .jar or any files to use OD with Websphere, please provide them to us once.
Then we will try to confirm whether OD can be work with them or not.

After that please release them formally as MR.

Best regards, K.Yamahara

Hi Massimo-san.

Thank you for your support.

We confirmed there is no similar logs in our system.
trace.out log file:
17/12/2018 14:13:37:785 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : Remaining grace period 2236206022
17/12/2018 14:13:37:785 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : License is bad, allowing grace period
17/12/2018 14:13:37:793 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : License refresh timer started
vpAppLogClient.log:
2018-12-17 14:13:37,776 9485 [WebContainer : 0] DEBUG - ReportWriter:report() - level: Warning ||type=In Progress ||transactionName=- ||reason=Runtime License Error :com.avaya.weblm.LicenseException: Problem with connection to server: Received fatal alert: handshake_failure ||variableName=- ||variableData=- ||userLogInfo=- ||moduleIdNodeId=PruMVP:Start
SystemOut_date.log:
[12/20/18 15:30:48:724 EST] 00000096 SystemOut O WebContainer : 0, READ: TLSv1 Alert, length = 2
[12/20/18 15:30:48:724 EST] 00000096 SystemOut O WebContainer : 0, RECV TLSv1.2 ALERT: fatal, handshake_failure
[12/20/18 15:30:48:725 EST] 00000096 SystemOut O WebContainer : 0, called closeSocket()
[12/20/18 15:30:48:725 EST] 00000096 SystemOut O WebContainer : 0, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

If my understanding correct, this issue was resolve java1.8 which used our site.
By the way , please advise which file or parameter should we change if we need to add below Generic JVM arguments.
-Dhttps.protocols=TLSv1.2 -Djdk.tls.client.protocols=TLSv1.2 -Dcom.ibm.jsse2.overrideDefaultTLS=true

Again, please let you advise below my question.
>With IBM, the JVM should be "IbmX509". "SunX509" is the default value used by JBoss. Did you explicitly configure that name ?
[Kenji]
I believe answer is "No".
So, please advise how to configure the Name for "IbmX509"?
If there are some recommended procedure in Dev guide or other Devconn thread, please advise.

Or if we need to configure some of setting related secure TLS communication with WebLM server, please advise what setting should we perform.
Best regards, Kenji

Hi Massimo-san.

Thank you for your support.

>So the Tomcat is working with the same OD same WebLM, the issue rises up when using the WS , right ?
[Kenji]
You are right. Only Websphere case is NOT working.

>I see WebSphere 9 with Java 8.
>I faced a similar issue in Java 1.8 that was trying to connect to the WebLM using SSL v3 when WebSphere was configured to only accept TLS v1.2.
>Might you confirm the WS configuration is fine ?
[Kenji]
Yes. We are using WebSphere 9 with Java 1.8.
And we configured WebSphere to use "TLS v1.2".

As you said, I find below AVAYA knowledge base. It sees it is fit for our configuration. We will try to configure this solution.

Orchestrational Designer: Unable to get license from WebLM on WebSphere
https://support.avaya.com/ext/index?page=content&id=SOLN335073
Doc ID : SOLN335073

Details :
OD runtime cannot get valid license under WebSphere 9.0.0.6 with Java 8

trace.out log file:
17/12/2018 14:13:37:785 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : Remaining grace period 2236206022
17/12/2018 14:13:37:785 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : License is bad, allowing grace period
17/12/2018 14:13:37:793 DEBUG - F1KBXv93DfMULQ8sD4bgpRJ:/PruMVP : License refresh timer started
vpAppLogClient.log
2018-12-17 14:13:37,776 9485 [WebContainer : 0] DEBUG - ReportWriter:report() - level: Warning ||type=In Progress ||transactionName=- ||reason=Runtime License Error :com.avaya.weblm.LicenseException: Problem with connection to server: Received fatal alert: handshake_failure ||variableName=- ||variableData=- ||userLogInfo=- ||moduleIdNodeId=PruMVP:Start
SystemOut_date.log
[12/20/18 15:30:48:724 EST] 00000096 SystemOut O WebContainer : 0, READ: TLSv1 Alert, length = 2
[12/20/18 15:30:48:724 EST] 00000096 SystemOut O WebContainer : 0, RECV TLSv1.2 ALERT: fatal, handshake_failure
[12/20/18 15:30:48:725 EST] 00000096 SystemOut O WebContainer : 0, called closeSocket()
[12/20/18 15:30:48:725 EST] 00000096 SystemOut O WebContainer : 0, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Problem Clarification :
Issue with OD, WebSphere 9.x, and Java 8. OD is unable to obtain license file from the WebLM when it is hosted on a Websphere server.

Cause :
Issue in Java that is trying to connect to the WebLM using SSLv3 when WebSphere is configured to only accept TLSv1.2

Solution :
You may be able to fix this problem by adding the following Generic JVM arguments
-Dhttps.protocols=TLSv1.2 -Djdk.tls.client.protocols=TLSv1.2 -Dcom.ibm.jsse2.overrideDefaultTLS=true

Best regards, K.Yamahara