Hi,

Im trying to get past a specific error (TSERVER_STREAM_FAILED) and need a little help...

I have searched the forums already and the Tsapi prog ref and know that i get this error message either when the Tsapi Client buffer is full, or when the AES Server does not get acknowledgement back from the Tsapi Client which suggests its too busy to respond. Which is when it terminates my link.

Even if my environment has low traffic i can still get this same error message. So when i get one i recover by recreating the link to Tsapi and re-monitor everything. I have a single and multi-threaded version of the same app and still get the same error message. Finally i have experimented with the Receive Queue/Buffer values and these not make a noticable difference either.

Can anything else cause this error message?
What should i be looking for to get to the root cause?

Is it imperative to run in a tight polling (receiving) loop to take from Tsapi Client, or is it fine as long as the end of the buffer is not hit?

I have also found that i can recreate the problem on my Aure BDE if i unmonitor and then remonitor enough devices.

Any advice is welcome right now... thank you,

Rich.

At its lowest level this error is indicating that the TSAPI client detected a socket close on the client application server. Some of these are caused when AE Services is trying to send to the client (e.g. TCP buffer fills). Some are caused by network conditions, or OS issues on the client application server.

The first thing in troubleshooting this problem is to figure out where the socket closes are originating from ... either by using a packet sniffer, or enabling and examining logging in both the client and AE Services. IMHO the packet sniffer approach will orient you in the right direction faster than second guessing things from the log files. Since the log files are from entities fairly high in the application stack, they can be misleading.

There are two buffers in play as it relates to the normal problems with teh link dropping.. A TCP receive buffer in the client side, and a TCP send buffer in AE Services. It is imperative that the client side TCP buffer be serviced often to avoid a situation where AE Services tries to send to the client and there is no room in the client for the message. This causes most problems we see.

Hi John,

Again, thanks for you speedy reply.

When you say client application server, this would be my application?

I have already tried a the MS NetMon sniffer but was not sure what to look for. So from what you are saying, i should see a TCP disconnect message like a FIN message?

When you say TCP buffer, are you referring to the NIC or the Send/Receive Queues for the AES and Tsapi Client?

So im guessing that if i see the disconnect come from the AES, my app is not servicing the Queue enough. But if it come from the Client it would that be an issue on the AES?

I usually try to have a third party server monitor the connection, that way directionality can be determined.

client application server == your application (yes)

The FIN says the TCP connection is finished.. finding who sends that is important to know where to look for the why behind the connection failing.

TCP buffer is typically associated with the TCP stack sitting on top of the NIC in the application. There isa send and a receive buffer... usually the problem (with not receiving fast enough) is seen when the client side application (your application)'s TCP receive buffer fills and stays filled through 3 transmission attempts from AE Services to send a packet from AES to the client side application.

If AE Services sends a FIN, then there is typically a nice error message in the AES logs (/opt/mvap/log.YYYYMMDD) indicating why AEServices closed the socket (which it rarely does). If it comes from the client, then either the TCP stack at the client, or the TSAPI client itself can be viewed as the origin.

Thanks John, this was really helpful. As I found some messages in the AES logs (easy to spot when you know what you are looking for). And it said that the AES was terminating the link after trying to send 5 times.

So with this I will look at my Rx loop.

I did get a sniffer on the machine running my app, but when I experienced the error I only saw Reset messages from TCP, not FIN messages. Is this correct?

Honestly I look at so many failure conditions they all become fuzzy in my head. The data you have from the trace is 'gospel,' my words are hearsay. Use what is in front of you from the logs... they are not subject to (my) personal interpretation/memories.