How to troubleshoot this type of issue? I don't see anything in the maintenance guide specifically for this. Thanks.

Finally figured out how to find these in the logs (ce logv) but I am no closer in understanding what is wrong. I see lots of errors on the dependencies I have added like the below. Can you advise how we can further troubleshoot?

org.eclipse.jst.j2ee.commonarchivecore.internal.helpers.WARFileAnnotationsDetector locateAnnotations Failed to create class reader for [ io/advantageous/boon/core/Conversions$1.class ] in [ WEB-INF/lib/boon-reflekt-0.6.4.jar ]

Hi Michael, sorry for the delay. I'm working on rounding up somebody that can help with this issue.

A couple of questions / suggestions:

* What version of Breeze are you on?
* How are you installing the new version? Did you:
* Keep the same version number and scp/deploy_service from the CLI?
* Change the version number and install via System Manager?
* If upversion / install via SMGR, is the previous version still installed?

These instances are 3.2. Originally I would do everything through the GUI in SM. I would load the SVAR, then click install to the breeze clusters. I would always go up to a new serviceVersion via the pom.xml in the project.

I realized this stopped working when I had started adding third party dependencies to the pom, and the new version would show "failed to run". I found it difficult to find out where I could look for errors, then realized I could "ce logv" and find them there. Recently while troubleshooting this issue I would just make modifications to the pom keep version the same, scp/deploy_service on one breeze server so that I could watch it in real time attempt to start up. This is much easier for troubleshooting purposes.

While I have been troubleshooting I have found out a few things. The Websphere instance that Breeze is running on runs JDK 1.7 (SystemOut.log snippet below)

************ Start Display Current Environment ************

WebSphere [IBMJAVA7 7.0.8.0 cf041446.01]Platform 8.5.5.4 [BASE 8.5.5.4 cf041446.03] running with process name avaya-asmNode01Cell\avaya-asmNode01\server1 and process id 5077

Host Operating System is Linux, version 2.6.32-431.el6.x86_64

Java version = 1.7.0, Java Runtime Version = pxa6470sr8ifx-20141114_01 (SR8+IV66608+IV66375+IX90155+IV66944 SR8+IV66608+IV66375+IX90155+IV66944), Java Compiler = j9jit26, Java VM name = IBM J9 VM

Some of the third party libraries (swagger-codegen, jersey, etc) are compiled for version 1.8. I would imagine this is why there is an issue. I do know that Websphere contains an option "managesdk" that will allow you to change to version 1.8 if it is installed, but I can't run it so not sure it applies to Breeze installations. I have my counterpart checking with the Avaya resource that installed their system to see if it can be adjusted.

If you have any other feedback or comments feel free to share. Currently I am going to try to scale back the versions of the dependencies to see if I can find one that is 1.7 compliant to check to see if that resolves the issue.

Hi Michael, we believe that your suspicion is correct about the JDK version mismatch. Breeze versions 3.3 and later support JDK 1.8. Is it possible that you could upgrade to 3.4?

I managed to make a lot of headway today. Looks like we were right on the 1.8 dependency. I re-worked the project by including maven enforcer to confirm jar 1.7 compliance, and had to end up recompiling one of the jars that does not have a 1.7 version. Now I'm able to at least get to where the WAR is attempting to initializing.

TestService-1.1.1.22.0 : Using most default ClassLoader settings (exception using "SINGLE" classloader):  likely Parent First

TestService-1.1.1.22.0 : Configuring shared libraries: ce_shared+ce_isolated+ce_gs

TestService-1.1.1.22.0 : Deployed

StartOnInstall= None

TestService-1.1.1.22.0 : Initializing...

TestService-1.1.1.22.0 : com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_fc30986c_18.05.03_15.27.35.8422272275009861442273.txt com.ibm.ws.management.event.ListenerInfo.handleNotification 158

TestService-1.1.1.22.0 :   J2CA0130I: Data Source [engagementdesigner_workflow] has Component-managed Authentication Alias [engagementdesigner_workflow-engagementdesigner] but no corresponding J2C Authentication Data Entry is defined in security.xml. This may have undesirable effects.

TestService-1.1.1.22.0 :   J2CA0130I: Data Source [engagementdesigner_quartz] has Component-managed Authentication Alias [engagementdesigner_quartz-engagementdesigner] but no corresponding J2C Authentication Data Entry is defined in security.xml. This may have undesirable effects.

TestService-1.1.1.22.0 : com.ibm.ws.webcontainer.servlet.ServletWrapper init Uncaught.init.exception.thrown.by.servlet

TestService-1.1.1.22.0 : SipletServletInitiatorImpl  Failed to initialize servlet

I now get some new errors that are actually caused by one of the jars I included, and not Websphere itself (ClassNotFound, etc), which I'm working through. Including the below was what ultimately led me to find the problem dependencies. I wanted to include this for anyone coming across this type of issue.

			<plugin>

		        <groupId>org.apache.maven.plugins</groupId>

		        <artifactId>maven-enforcer-plugin</artifactId>

		        <version>1.4.1</version> <!-- find the latest version at http://maven.apache.org/plugins/maven-enforcer-plugin/ -->

		        <executions>

		          <execution>

		            <id>enforce-bytecode-version</id>

		            <goals>

		              <goal>enforce</goal>

		            </goals>

		            <configuration>

		              <rules>

		                <enforceBytecodeVersion>

		                  <maxJdkVersion>1.7</maxJdkVersion>

		                  <excludes>

		                    <exclude>org.mindrot:jbcrypt</exclude>

		                  </excludes>

		                </enforceBytecodeVersion>

		              </rules>

		              <fail>true</fail>

		            </configuration>

		          </execution>

		        </executions>

		        <dependencies>

		          <dependency>

		            <groupId>org.codehaus.mojo</groupId>

		            <artifactId>extra-enforcer-rules</artifactId>

		            <version>1.0-beta-5</version>

		    	</dependency>

	        </dependencies>

	      </plugin>

Good to hear that you're making progress. Thanks for sharing.

So here is where I am stuck. Fixed all the java issues. I have scp the file over to a breeze server. I run deploy_service, and this is the output--

TestService-1.1.1.22.0 : Using most default ClassLoader settings (exception using "SINGLE" classloader):  likely Parent First

TestService-1.1.1.22.0 : Configuring shared libraries: ce_shared+ce_isolated+ce_gs

TestService-1.1.1.22.0 : Deployed

StartOnInstall= None

TestService-1.1.1.22.0 : Initializing...

TestService-1.1.1.22.0 :   J2CA0130I: Data Source [engagementdesigner_workflow] has Component-managed Authentication Alias [engagementdesigner_workflow-engagementdesigner] but no corresponding J2C A                                                                           uthentication Data Entry is defined in security.xml. This may have undesirable effects.

TestService-1.1.1.22.0 :   J2CA0130I: Data Source [engagementdesigner_quartz] has Component-managed Authentication Alias [engagementdesigner_quartz-engagementdesigner] but no corresponding J2C Authe                                                                           ntication Data Entry is defined in security.xml. This may have undesirable effects.

TestService-1.1.1.22.0 : DEPLOY-ERROR: use 'ce logv' and 'ce dlogv' to see more

So I go check logv TestService, and I can see that it has found my service life cycle and listener, but the code in my life cycle is not being run. I should at least see a logger.fine message inside init--

2018-05-04 13:56:26,966 [SoapConnectorThreadPool : 1] TestService INFO - TestService-1.1.1.22.0 - Listener class "com.comcast.nco.MyService.MyListenerLifeCycle" found with annotation "TheServiceLifeCycle" and implementation of "ServiceLifeCycle"

2018-05-04 13:56:26,967 [SoapConnectorThreadPool : 1] TestService INFO - TestService-1.1.1.22.0 - Listener class "com.comcast.nco.MyService.MyCallListener" found with annotation "TheCallListener" and implementation of "CallListener"

	@Override

	public void init() {

		

		try {

			logger.fine("MyService init");

			configureSEQ();

		} catch (Exception e) {

			// TODO: handle exception

		}

		



	}

Hi Michael, it's good to hear that you're making progress. Quick sanity check, did you do a "ce dlogon MyService"? If not, the default log level is "info" and higher. "fine" logs would not appear. I think that you may need to turn up log levels every time after doing the deploy_service.

Yes it's set to ALL/ALL right now. Interestingly enough I did find that it ran the destroy method so I guess it is finding something there. Is there anything I have to do in order to get the init to fire, or is it just automatic once it is deployed?--

2018-05-04 14:15:00,647 [SoapConnectorThreadPool : 0] TestService FINE - TestService-1.1.1.22.0 - MyService destroy

Should just be automatic. Another question: are you sure that you got the Logger from the Avaya package rather than the standard Java Logger? And that it's set from the correct class rather than copy / pasted incorrectly? Those are things that have bit me in the past.

This is what I have now, if I recall, I had to give it the MyCallListener.class is that appropriate for logging from the LifeCycle implementation?


public class MyListenerLifeCycle implements ServiceLifeCycle {

private final Logger logger = Logger.getLogger(MyCallListener.class);

The thing to check is the import statement for Logger. If you're doing automatic imports from e.g. Eclipse, it's easy to grab the standard java.util Logger instead of our com.avaya.collaboration.util.logger.Logger

If you see a log statement working from the destroy method that's unlikely to be the issue though.

Ok I got it finally. Someone had changed the service profile to not run the service :(

2018-05-04 14:58:31,496 [SoapConnectorThreadPool : 1] TestService INFO - TestService-1.1.1.22.0 - MyService init
2018-05-04 14:58:31,496 [SoapConnectorThreadPool : 1] TestService INFO - TestService-1.1.1.22.0 - Entering Config SEQ
2018-05-04 14:58:31,591 [SoapConnectorThreadPool : 1] TestService INFO - TestService-1.1.1.22.0 - Config SEQ complete

So now to the next step, what I'm via init doing is attempting to post log entries to a remote http endpoint on our internal network. Are there any gotchas as far as outbound connections from a breeze webservice ?

The port is 5341 if that matters:

http://<server>:5341