Hello,
Starting on the 18th of June, one of our production application servers (which hasn't had any modifications in months, and has otherwise had no issues) ran out of disk space, an event which happened very quickly, due to a 30+ GB catalina.out file, which showed the following type of log statement printed 1000+ times a second:
2018-06-21 17:01:41,079 24418653 [VPBatchLogWriterThread:
https://<remote-AEP-EPM-IP-Address>/axis2/services/VPAppLogService] ERROR - BatchLogWriter:logFromBackupFile, Failed to send last block, stop sending more from the save log
Looking a little further, I also found the following stack traces being printed:
12501666 [VPBatchLogWriterThread:
https://10.180.95.1/axis2/services/VPAppLogService] ERROR com.avaya.vp.applog.client - BatchLogWriter:logBatchAxis2() - Caught Exception FAILED with AxisFault faultCode: null error: Invalid white space character (0x2) in text to output to log entry size: 9 to URL:
https://<remote-AEP-EPM-IP-Address>/axis2/services/VPAppLogService from AE server: <local app server hostname> First SID: <remote-AEP-MPP-Hostname>-2018170052929-14 Last SID: <remote-AEP-MPP-Hostname>-2018170055955-14 NOTE: It could be caused by this VPMS is not online or some other network issues.
org.apache.axis2.AxisFault: Invalid white space character (0x2) in text to output
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.SOAPMessageFormatter.getBytes(SOAPMessageFormatter.java:118)
at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:87)
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub.reportBatch(VPAppLogServiceStub.java:606)
at com.avaya.vp.applog.client.AppLogAxis2.logBatchAxis2(AppLogAxis2.java:72)
at com.avaya.vp.applog.client.BatchLogWriter.logBatch(BatchLogWriter.java:270)
at com.avaya.vp.applog.client.BatchLogWriter.doLog(BatchLogWriter.java:589)
at com.avaya.vp.applog.client.BatchLogWriter.sendLastBlock(BatchLogWriter.java:940)
at com.avaya.vp.applog.client.BatchLogWriter.logFromBackupFile(BatchLogWriter.java:452)
at com.avaya.vp.applog.client.BatchLogWriterBase.checkForFailuresAndLog(BatchLogWriterBase.java:301)
at com.avaya.vp.applog.client.BatchLogWriter.run(BatchLogWriter.java:144)
Caused by: com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x2) in text to output
at com.ctc.wstx.sw.BaseStreamWriter.writeCharacters(BaseStreamWriter.java:511)
at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.writeCharacters(XMLStreamWriterWrapper.java:100)
at org.apache.axiom.om.impl.MTOMXMLStreamWriter.writeCharacters(MTOMXMLStreamWriter.java:289)
at org.apache.axis2.databinding.utils.writer.MTOMAwareXMLSerializer.writeCharacters(MTOMAwareXMLSerializer.java:139)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub$VPReportEntry4.serialize(VPAppLogServiceStub.java:2728)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub$VPReportEntry4.serialize(VPAppLogServiceStub.java:2482)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub$ReportBatch.serialize(VPAppLogServiceStub.java:4462)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub$ReportBatch.serialize(VPAppLogServiceStub.java:4406)
at com.avaya.xml.ws.vpapplog.voiceportal.VPAppLogServiceStub$ReportBatch$1.serialize(VPAppLogServiceStub.java:4394)
at org.apache.axis2.databinding.ADBDataSource.serialize(ADBDataSource.java:93)
at org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerialize(OMSourcedElementImpl.java:695)
at org.apache.axiom.om.impl.util.OMSerializerUtil.serializeChildren(OMSerializerUtil.java:563)
at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:874)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.serializeInternally(SOAPEnvelopeImpl.java:283)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:245)
at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:193)
at org.apache.axis2.transport.http.SOAPMessageFormatter.getBytes(SOAPMessageFormatter.java:110)
... 25 more
Caused by: java.io.IOException: Invalid white space character (0x2) in text to output
at com.ctc.wstx.sw.XmlWriter.throwInvalidChar(XmlWriter.java:545)
at com.ctc.wstx.sw.BufferingXmlWriter.writeCharacters(BufferingXmlWriter.java:531)
at com.ctc.wstx.sw.BaseStreamWriter.writeCharacters(BaseStreamWriter.java:509)
... 41 more
I understand from the following thread that the first set of repeating messages can be due to a login missing for the report user on the remote EPM:
https://www.devconnectprogram.com/forums/posts/list/1189.page
However, while I am waiting for access into the remote EPM to address this, I question why this would cause the above message to be logged so many times. This type of behavior isn't acceptable and is there a way to prevent this logging from happening should there be a problem with a remote AEP instance?
These are AEP 6 runtimes. I can get specific version infos if requested.
