Hi,
In 8-10 percent of calls, an error occurs - “Failed to load grammar from URL.”
This error does not appear when testing in OD.
The grammar is simple static DTMF. Neither ARS nor TTS are used in the application.
Here are the error logs from the application server:

[22/Nov/2023:09:35:35 +0600] "GET /UIVR_all/Start?session___sessionid=kz-aepmpp02-2023326033530-137&__VPapploggingurl=https%3a%2f%2f172%2e28%2e24%2e171%2faxis%2fservices%2fVPReport4&__VPapploggingurl_2=https%3a%2f%2f172%2e28%2e24%2e172%2faxis%2fservices%2fVPReport4&__VPapplog=%2faxis2%2fservices%2fVPAppLogService&__VPvpms=172%2e28%2e24%2e171%7c172%2e28%2e24%2e172&__VPappvars=%2faxis2%2fservices%2fVPAppVarsService&__VPVarAppDate=1697108647271&__VPVarAppURL=https%3a%2f%2f172%2e28%2e24%2e171%2faxis%2fservices%2fVPAppRuntimeVars&__VPVarAppURL2=https%3a%2f%2f172%2e28%2e24%2e172%2faxis%2fservices%2fVPReport4&__VPVarGlobalDate=1692086047280&__VPbreadcrumb=disabled&__VPmaxbackuplogfiles=10&__VPlogname=%25default%25jx1GOozCAVbbOc6C&__VPlogpassword=r0yFasKwSgqu2RAY7TMKufnsIIbKG%2fzT5JZGXYFM8rk4S4amSGqzhas%2f2IV72WCQ&__VPloglevel=Info&__VPappname=0%3aUIVR%5fall HTTP/1.1" 200 10595

[22/Nov/2023:09:35:36 +0600] "POST /UIVR_all/getSOC?___DDSESSIONID=195FED6346A3E7B58F71E8ED4B6466C2%3A%2FUIVR_all HTTP/1.1" 200 9125

[22/Nov/2023:09:35:50 +0600] "GET /UIVR_all/data/english/grammars/oneKey-srgs.grxml HTTP/1.1" 200 986

[22/Nov/2023:09:36:26 +0600] "GET /UIVR_all/data/ddfetchAudio.wav HTTP/1.1" 200 2264

[22/Nov/2023:09:36:36 +0600] "POST /UIVR_all/error?___DDSESSIONID=195FED6346A3E7B58F71E8ED4B6466C2%3A%2FUIVR_all HTTP/1.1" 200 1064

[22/Nov/2023:09:36:46 +0600] "GET /UIVR_all/AvayaSessionTerminationServlet?___DDSESSIONID=195FED6346A3E7B58F71E8ED4B6466C2%3A%2FUIVR_all HTTP/1.1" 200 124

22/11/2023 09:35:36:364 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Admin SimpleVariable:getSimpleVariable(): return admin simple variable - wavpath

22/11/2023 09:35:36:364 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Admin VariableField:getStringValue(): return string value for wavpath

22/11/2023 09:35:36:364 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : get param for app: 0:UIVR_all

22/11/2023 09:35:36:364  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Use file ---> /UIVR_WAV/PVL_632_eng.wav

22/11/2023 09:35:36:364  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Next step ---> PlayPrompt

22/11/2023 09:35:36:364  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Using SCESession 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all servlet : PlayPrompt

22/11/2023 09:35:36:365  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : [condition] Expression [firstPlay] Is True

22/11/2023 09:35:36:365 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Admin SimpleVariable:getSimpleVariable(): return admin simple variable - wavpath

22/11/2023 09:35:36:365 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Admin VariableField:getStringValue(): return string value for wavpath

22/11/2023 09:35:36:365 DEBUG - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : get param for app: 0:UIVR_all

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Using SCESession 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all servlet : error

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Storing :statistic___message to complex: statistic:message  as [Failed to load grammar from a URL]

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Storing :statistic___link to complex: statistic:link  as [undefined]

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Storing :statistic___event to complex: statistic:event  as [error.badfetch]

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Assigning [1] to [statistic:error]

22/11/2023 09:36:36:034  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Using SCESession 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all servlet : trace

22/11/2023 09:36:36:073  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Using SCESession 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all servlet : Exit

22/11/2023 09:36:46:046  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Removing session from AvayaSessionTermination Servlet

22/11/2023 09:36:46:046  INFO - 195FED6346A3E7B58F71E8ED4B6466C2:/UIVR_all : Termination URL reply :

How to deal with this?

Hi. Try to search for the exact error message on the Voice Browser logs.

You can collect the MPP VB logs from the MPP web interface (attached picture below) or

- enable trace (attached picture below)

- trace viewer selecting the proper MPP server / "Voice Browser" as 'Components' and ''All levels" as 'Trace Level' (attached picture below).

MPP log:

@2023-11-22 09:35:39,904||FINE|VB|1555415|Session=kz-aepmpp02-2023326033530-137|successfully sent message VXML_PAGE_LOADED|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:39,905||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023326033530-137|AVBrecLoadGrammarFromURI() - type =application/srgs+xml, uri=http://kz-aepapp07.kar-tel.local:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml, bDTMFNode=1|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|SBinetHttpStream.cpp:1409:SBinetHttpStream::getHeaderInfo: Waited 1ms for HTTP response|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|SBinetHttpStream.cpp:1415:SBinetHttpStream::getHeaderInfo: rc=0, HttpStatus=200|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|Fetching fresh copy of http://kz-aepapp07.kar-tel.local:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml from server|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|SBinetHttpCacheStream::openCacheWrite called url=http://kz-aepapp07.kar-tel.local:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:09,521|PAVB_00165|INFO|VB|2613|Session=kz-aepmpp02-2023326033530-137|Failed opening cache entry for URL http://kz-aepapp07.kar-tel.local:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023326033530-137|osbrec_utils.cpp:888:GenerateGrammarStringFromURI: Unable to Open URL to fetch grammar|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536|PAVB_06003|INFO|VB|2613|Session=kz-aepmpp02-2023326033530-137|Failed to create grammar from URI: http://kz-aepapp07.kar-tel.local:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml Return code = CHANNEL_RESULT_FETCH_ERROR|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023326033530-137|osbrec.cpp:1012:AVBrecLoadGrammarFromURI() returning rc=51|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Interpreter|2613|Session=kz-aepmpp02-2023326033530-137|GrammarManager.cpp:744:Grammar loading has seen a pending event to throw. Skipping grammar loading.|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Interpreter|2613|Session=kz-aepmpp02-2023326033530-137|GrammarManager.cpp:744:Grammar loading has seen a pending event to throw. Skipping grammar loading.|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Interpreter|2613|Session=kz-aepmpp02-2023326033530-137|GrammarManager.cpp:744:Grammar loading has seen a pending event to throw. Skipping grammar loading.|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,538|PAVB_03335|INFO|VB|2613|Session=kz-aepmpp02-2023326033530-137|Got VoiceXML exception: error.badfetch in http://kz-aepapp07.kar-tel.local:8080/UIVR_all/getSOC?___DDSESSIONID=195FED6346A3E7B58F71E8ED4B6466C2%3A%2FUIVR_all|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,538||FINE|CommonVXI.Interpreter|2613|Session=kz-aepmpp02-2023326033530-137|VXI::DoEvent - Enter|kz-aepmpp02.kar-tel.local####

I am attaching the full session log below.

Here you are:

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|Fetching fresh copy of http://kz-aep.xxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml from server|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:35:54,501||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023326033530-137|SBinetHttpCacheStream::openCacheWrite called url=http://kz-aep.xxxxxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:09,521|PAVB_00165|INFO|VB|2613|Session=kz-aepmpp02-2023326033530-137|Failed opening cache entry for URLhttp://kz-aep.xxxxxxxxxxxxxxxxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023326033530-137|osbrec_utils.cpp:888:GenerateGrammarStringFromURI: Unable to Open URL to fetch grammar|kz-aepmpp02.kar-tel.local####

@2023-11-22 09:36:19,536|PAVB_06003|INFO|VB|2613|Session=kz-aepmpp02-2023326033530-137|Failed to create grammar from URI:http://kz-aepxxxxxxxxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml Return code = CHANNEL_RESULT_FETCH_ERROR|kz-aepmpp02.kar-tel.local####


- The problem seems the cache entry for URL 'http://kz-aep.xxxxxxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml' (your external grammar).
I suppose the URL is fine as you stated the issue happens 1 out of 10 calls (discontinuous). The root cause is not present. Do you see anything else on the trace or log viewer ?

- Might you check if permissions are fine ?

- Might you try to clear the MPP Cache and Application service cache files then test ? Does the issue still persist ?

I would also suggest packet sniffing the connection between the MPP and http://kz-aep.xxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml to understand if the fetch is failing for some network reason.

I also think that the problems are in the network infrastructure. I'll try to figure it out.
The application server log shows that the file was downloaded successfully, perhaps the delay was too long?
Is it possible to place a grammar file on the MPP itself and provide a link to the file in the application?

Basing on your description of the issue

In 8-10 percent of calls, an error occurs - “Failed to load grammar from URL.”

and the error messages

Fetching fresh copy of http://kz-aep.xxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxml from server|kz-aepmpp02.kar-tel.local####
...
Failed opening cache entry for URL http://kz-aep.xxxxxxxxxxxxxxxxxxxxxx:8080/UIVR_all/data/english/grammars/oneKey-srgs.grxm


the root cause can be a network problem, ex. network instability (connection unstable going up and down even if the ping is working).

Because the issue happens while fetching a copy of the grammar, please double-check also the fetch timeout in the MPP server (see attached pictures for the exact location on the EPM web interface), the default value is 15 secs.

Concerning your last question, are you referring to an external grammar ? It is possible to set an external grammar on the OD app, take a look at

- https://www.devconnectprogram.com/forums/posts/list/20102.page
- The OD Developer's guide (I don't know your release, this is for the latest 8.1.2): https://support.avaya.com/css/public/documents/101083618 on pages 255-259, 265-266,473-474

Overall duration of the download could also be a possibility. The packet trace would help understand if the intervals vary between success and failure cases (so might the logs in looking the timestamps).

Hi,
I put the grammar file on the MPP server. The problem persists. I am attaching the error log. It looks like the problem is not in the network environment, but in MPP itself, perhaps in the cache settings. So far the problem has only occurred on one of the two MPPs, perhaps that matters. The problem is not reproduced in OD.

@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SBinetHttpStream.cpp:1415:SBinetHttpStream::getHeaderInfo: rc=0, HttpStatus=200|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|Fetching fresh copy of http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml from server|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SBinetHttpCacheStream::openCacheWrite called url=http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:55:03,050|PAVB_00165|INFO|VB|2613|Session=kz-aepmpp02-2023341085418-36|Failed opening cache entry for URL http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:55:13,063||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023341085418-36|osbrec_utils.cpp:888:GenerateGrammarStringFromURI: Unable to Open URL to fetch grammar|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:55:13,063|PAVB_06003|INFO|VB|2613|Session=kz-aepmpp02-2023341085418-36|Failed to create grammar from URI: http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml Return code = CHANNEL_RESULT_FETCH_ERROR|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:55:13,063||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023341085418-36|osbrec.cpp:1012:AVBrecLoadGrammarFromURI() returning rc=51|kz-aepmpp02.kar-tel.local####

@2023-12-07 14:55:13,063||FINE|CommonVXI.Interpreter|2613|Session=kz-aepmpp02-2023341085418-36|GrammarManager.cpp:744:Grammar loading has seen a pending event to throw. Skipping grammar loading.|kz-aepmpp02.kar-tel.local####

what releases are you on? Are you up to date with the most recent Experience Portal and MPP releases?

EPM version - 8.1.2.0.0328
MPP version - 8.1.2.0.0328
I know there are updates - 0377 and 0399. Should I try updating?

I dont think that is it, I was thinking maybe you were way out of date. Let's see what Massimo thinks.

On the EP 8.1.2 downloads

https://support.avaya.com/support/en/download/1399855686067?productId=P0407&releaseId=8.1.x

I don't see changes related to this problem, anyway feel free to update the EPM to 8.1.2.0.0377 and the MPP to 8.1.2.0.0399.

The second log ( grammar on the MPP) seems the mirror of the first one using 127.0.0.1 instead of the remote IP Address.

I see the same three events PAVB_00165, PAVB_06003 and PAVB_03335

The VXI (VoiceXML interpreter, a main component of the MPP) by default caches all documents that are fetched.

You can see here:
@2023-12-07 14:54:33,387||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023341085418-36|AVBrecLoadGrammarFromURI() - type =application/srgs+xml, uri=http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml, bDTMFNode=1|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SWIsocket.cpp:329:Established Connection to 16777343|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SBinetHttpStream.cpp:1409:SBinetHttpStream::getHeaderInfo: Waited 0ms for HTTP response|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SBinetHttpStream.cpp:1415:SBinetHttpStream::getHeaderInfo: rc=0, HttpStatus=200|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|Fetching fresh copy of http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml from server|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:54:48,031||FINE|CommonVXI.Cache|2613|Session=kz-aepmpp02-2023341085418-36|SBinetHttpCacheStream::openCacheWrite called url=http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml|kz-aepmpp02.kar-tel.local####


It seems the VXI/MPP is not able to open the cache for fetching the grammar:


@2023-12-07 14:55:03,050|PAVB_00165|INFO|VB|2613|Session=kz-aepmpp02-2023341085418-36|Failed opening cache entry for URL http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:55:13,063||FINE|CommonVXI.Rec|2613|Session=kz-aepmpp02-2023341085418-36|osbrec_utils.cpp:888:GenerateGrammarStringFromURI: Unable to Open URL to fetch grammar|kz-aepmpp02.kar-tel.local####
@2023-12-07 14:55:13,063|PAVB_06003|INFO|VB|2613|Session=kz-aepmpp02-2023341085418-36|Failed to create grammar from URI: http://127.0.0.1/mpp/misc/grammars/oneKeyT-srgs.grxml Return code = CHANNEL_RESULT_FETCH_ERROR|kz-aepmpp02.kar-tel.local####



- Did you check all MPP logs (ex. Session Manager logs) to see if there are error messages ? You can also use the log viewer on the EPM web interface (Home -> System Maintenance -> Log Viewer).

- Did you enable the trace then collect logs ?