Hi,

In attachment you can find an archive with logs during one hour.

Please, do not pay attention to other errors... I know about them (3 types) and they will be fixed after I provide solution for the current problem.

I can say that the load on the server (and application itself) is quite high.

Thanks in advance for your help!

So you get 1000 lines of

"VPAppLogClientBase:getSaveReportPath: savereportLogPath: /app/tomcat/webapps/Dillers614/../../logs/"

in an hour?

Well, it is about ~5000 lines...
That is why I am a little bit worry about the problem.

These logs are not normal. Do you see any error in the vpapplogclient.log? It looks like it is constantly saving report items in in the savereport file instead of sending them to EPM.

Hi,
Now I see that something is wrong with configuration.
Yes, I see a lot of errors.

I did small classification of errors and can divide them in several groups:
1. Unexpected character.
Please check Unexpected_character_error.txt

2. I see a huge amount of warnings
2018-06-20 19:51:29,527 49335928 [VPBatchLogWriterThread: https://<IP_Addr>/axis2/services/VPAppLogService] WARN - BatchLogWriter:run() - BatchLogWriter thread is EXITING!!!

3. Read timeout error.
Please check Read_timeout_error.txt

4. Also a lot of errors like the following...
2018-06-20 06:01:26,856 2868520895 [VPAppRuntimeVarsMonitorThread: https://<IP_Addr>/axis2/services/VPAppVarsService] ERROR - VPAppRuntimeVars:EPMWebServicesClientBaseImpl:getAppVarsAxis2() sent failed, add to monitor list: https://<IP_Addr>/axis2/services/VPAppVarsService
2018-06-20 06:01:26,856 2868520895 [VPAppRuntimeVarsMonitorThread: https://<IP_Addr>/axis2/services/VPAppVarsService] ERROR - AppRuntimeVarsMonitor:callAppRuntimeVarsWS: VPAppRuntimeVarsMonitorThread: https://<IP_Addr>/axis2/services/VPAppVarsService Failed to get App Runtime Vars

5. And the following
2018-06-20 05:57:35,882 2868289921 [VPBatchLogWriterThread: https://<IP_Addr>/axis2/services/VPAppLogService] ERROR - BatchLogWriter:run() - logBatch failed with error: null
java.lang.NullPointerException

6. protocol_version error
protocol_version_error.txt

7. A lot of errors like
2018-06-26 17:03:46,010 1062648131 [http-bio-8080-exec-4033] ERROR - VPAppLogClientBase:getUserId() - User ID is EMPTY
2018-06-26 17:03:46,010 1062648131 [http-bio-8080-exec-4033] ERROR - VPAppLogClientBase:getPassword() - Password is EMPTY
2018-06-26 17:03:46,010 1062648131 [http-bio-8080-exec-4033] ERROR - VPAppRuntimeVars:VPAppRuntimeVarsFromVPMS() - DO NOT SEND request. Either userid or password is empty!!!

8.
2018-06-19 09:33:23,523 430825644 [VPBatchLogWriterThread: https://<IP_Addr>/axis2/services/VPAppLogService] ERROR - BatchLogWriter:EPMWebServicesClientBaseImpl:logBatchAxis2() - ReportBatch FAILED with status = Error initializing database, log entry size: 555 to URL: https://10.77.224.181/axis2/services/VPAppLogService from AE server: epapp02rst First SID: mpp01rst-2018170063307-26 Last SID: mpp02rst-2018170063227-34
2018-06-19 09:33:23,524 430825645 [VPBatchLogWriterThread: https://<IP_Addr>/axis2/services/VPAppLogService] ERROR - BatchLogWriter:logBatch() - Send batch FAILED, size: 555, ADD https://<IP_Addr>/axis2/services/VPAppLogService to the lastErrorTimeMap

Is "<IP_Addr>" something you put in to mask the IP? These errors indicate communication problem with EPM. If there are lot of "2018-06-26 17:03:46,010 1062648131 [http-bio-8080-exec-4033] ERROR - VPAppLogClientBase:getUserId() - User ID is EMPTY", you need to check the user/password settings for the reporting in EPM. By default, there is one setup already. You may need to restart EPM and get some new logs to see anything change at this point.

Hi Wilson!
Yes, <IP_Addr> - is a mask. So, in general no problem with IP address itself.

Thanks for a piece of advise.
I will check and come back soon with the results.

Hi Wilson,

We (me and VP engineer) did re-check of logs.
I can say that I did not put my attention to the timestamps in vpAppLogClient log. Most of the problems were in the past.
Well, a little bit more details about our configuration: we have 4 different application servers and all errors in vpAppLogClient.logs on each server were in the past.
Only one log from one web server contains actual errors:

2018-07-24 19:29:56,798 1928440569 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getReportUrl() - report URL is EMPTY

2018-07-24 19:29:56,798 1928440569 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getReportUrl() - report URL is EMPTY

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getUserId() - User ID is EMPTY

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - ReportWriter:setUserID, useris is NOT set

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getPassword() - Password is EMPTY

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getUserId() - User ID is EMPTY

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getPassword() - Password is EMPTY

2018-07-24 19:29:56,806 1928440577 [http-bio-8080-exec-31342] ERROR  - VPAppRuntimeVars:VPAppRuntimeVarsFromVPMS() - DO NOT SEND request. Either userid or password is empty!!!

2018-07-24 19:29:56,835 1928440606 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getUserId() - User ID is EMPTY

2018-07-24 19:29:56,835 1928440606 [http-bio-8080-exec-31342] ERROR  - VPAppLogClientBase:getPassword() - Password is EMPTY

2018-07-24 19:29:56,835 1928440606 [http-bio-8080-exec-31342] ERROR  - VPAppRuntimeVars:VPAppRuntimeVarsFromVPMS() - DO NOT SEND request. Either userid or password is empty!!!

All other vpAppLogClient.logs files do not contain actual errors.
But still I see a lot of output like

VPAppLogClientBase:getSaveReportPath: savereportLogPath: /app/tomcat/webapps/Dillers614/../../logs/

in all 4 web servers in catalina.out file.
Depending on the server I see from ~2000 to ~5000 lines of savereportLogPath per hour.

So, my view of the situation is the following: either it is not enough debug level and we do not see something in vpAppLogClient.logs OR may be we see such message on each application invocation (?).
What do you think?
Could you please check when the method which produces the line below is invoked:

VPAppLogClientBase:getSaveReportPath: savereportLogPath: /app/tomcat/webapps/Dillers614/../../logs/

?
I really want to know what (which application) is the trigger for producing this line...

May be we need to try to set a DEBUG level in VPAppLogClient.log4j.properties (The current level is WARN)?

So, would be appreciate for any help!

What is triggering producing this line is the report writing to EPM is not working so that it's writing to a file called savedreport. I don't have the exact line in the code where it's happening since I don't have access to the code. I need to ask some folks here to find out more. Setting debug in vpapplogclient.properties will allow you to see more logs.

Surprisingly, after my colleague checked the code, he told me this was a problem in earlier release (7.0 or older). For 7.1, it's been fixed and moved to the vpapplogclient debug mode. You would need to upgrade to OD 7.1 and give it a try.

Hi Wilson!
Thanks for a good news.

But my doubts are based on the fact that we have such output only for 3 application and they are different on each web server.
For example, on each server we have 3 different applications running: Dillers614, PhoneIVR, CallBackIVR. They are different.
The point here is that in catalina.out on server1 I see the lines for Dillers614. For server2 - PhoneIVR, For server3 - CallBackIVR. Sometimes it could be a mix of such lines (so, two applications provide the same lines).
But web servers contains other applications as well and they do not provide such output at all.
That is why I really wanted to know what\how the output line is triggered. And most probably something could be wrong in configuration.

It is good to know that this printout is moved to vpapplogclient debug mode. I did check of libs versions and see that the customer has
VPAppLogClient_7.0.0.jar
VPAppLogClientWS_7.0.0.jar
VPWebServiceClient-07.00.00.01.jar
I am afraid that it will not be upgraded in nearest future.
So, the question: is it possible to provide a patch for these (or just one) library to avoid such output?

This line should be produced by every call when the session is setting up the log writer. Not sure why it‘s not showing on some of your systems.
I am afraid there is no patch for 7.0. However this is not handled by the OD team. You may need to contact Avaya Support.

Hi Wilson,

Thanks a lot for your support.

I think I have caught one very important but very simple thing...
It may seems very stupid but I always thought that there is a strict connection between EP version and OD version. And if we need a newer version of OD than we need a newer version of EP (and vice-cerssa).
But now I see in OD Release Notes that compatibility is much wider than I supposed...
Is there any compatibility matrix between OD and EP? Or do I need to read carefully all OD Release Notes?

Currently customer has two version of EP (different locations):
7.0.2.0.0304
7.2.0.0.1201
So, what OD version can be successfully used with both EP versions?
OD 7.1.0.0.1202 or OD 7.2.0.0.0904?
I am going to propose to do an upgrade for OD (and tomcat accordingly) to the customer.
(The strange point right now is that currently customer uses OD 7.0.0.1801 to build all applications for both EP versions).

I am not sure where that compatibility doc is. Support might have one. But I can say that OD 7.2 is compatibility with EP 7.0 or up. You best bet is upgrading. We don't have a patch.

Hi Wilson,

Sorry for a long reply (I was on a vacation)...
Thank you very much for you support.

Do I correctly understand that an upgrade of OD will automatically require an upgrade of Tomcat on each EP related server?
Currently we have apache-tomcat-7.0.50 and I am thinking if we need just only libs update or the whole tomcat must be upgraded?