Bug in v3.50.430 NI-X100 firmware?

PhreaK

Just been debugging a system and noticed some strange behaviour. When I open a telnet connection to the master the internal message queue processing slows to a halt.

It's not to much of a problem if there is only minimal communication happening between devices but if you have any verbose interaction (gating activity, signal levels etc) happening it's enough to freeze up the system.

It only appears to happen when connected via telnet. The system will happilly be processing everything but the second you telnet in you can watch the Interpreter and Java Router buffers (via 'show buffers') start to grow. If you then close the telnet connection and give it a bit of time to clear out the queues it will gradually return to normal.

Anyone else having a similar issue?

HARMAN_shane-a-roo

Are you executing any telnet commands? A telnet connection by itself should have no effect in the master beyond initial session creation. After that, it's simply an open socket pending incoming data (i.e. keystrokes)

PhreaK

Nope. As long as the telnet connection is open the buffers will continue to grow.

The system can be running perfectly, but when you connect response times start to slow right down. I can confirm this by connecting, executing 'show buffers' straight away. It may show around 6 buffered commands by time i've typed and executed and if I continue to execute 'show buffers' I can watch them grow. If I then disconnect and reconnect 30 seconds later I can see that the number of buffered commands has greatly decreased (will return to zero if you give it time) but it will start to grow again whilst connected. Disconnect again and it will work it's way through the buffered messages and return to an operational state.

jazzwyld

Does this happen when viewing in the Diagnostic window as well?

HARMAN_shane-a-roo

The Interpreter and Java tasks run at the lowest priority of the system, so any activity on a Telnet session will have higher priority and preempt the Interpreter from running and processing messages off of it's queue. So if you have a system with lots of incoming event traffic and you execute a command on a telnet session, it is understandable that the Interpreter might backlog some.

The question is if the Interpreter is backlogging when there is a Telnet session that is doing nothing. The only way to determine this is to create a Telnet session and execute the show buffers only once to get a "baseline" read of the queues. Then do nothing on the Telnet session for some time before executing the "show buffers" again. If it's a problem with an idle Telnet session sucking CPU, then your count will have increased dramatically. If not, then the count should be about the same as the first execution.

What is the load like with regards to incoming event data? Is the Input LED on solid?

PhreaK

@jazzwyld
Monitoring device activity via the diagnostics window doesn't seem to affect the buffers.

@shane-a-roo
That's the way I've been testing this. If this system is only under minimal load (basic device control and occasional status updates) you don't really notice it a great amount, the response times slow down but becuase there's not a lot of messgaes passing through the queues it will catch back up within a second or two. Where I really notice it is when I start receiving things like gate status and signal presence feedback from the DSP's. The ClearOne's trigger abour 4 - 6 data events per second on top of other system traffic which quickly start to fill things up.

DHawthorne

I just loaded this update on a fairly simple system, and saw no buffer buildup whatever, except while the master was starting up (which is normal, a lot is going on then). So the complexity of the system and its notifications is a definite factor.

I wonder, however, if it's really new to this version. I have seen device notifications back up the queue like that when turned on, then clear when turned off again. It's entirely possible this has always happened and the new version simply has more going on, so it's more noticeable.

HARMAN_shane-a-roo

My guess is that this has always been marginal on your system. Build 430 does have some new Java runtime efficiencies. If you are running any Duet modules it may change their timing slightly. I don't know if that is affecting the overall execution in your system. i.e. throwing more CPU cycles to Java vs. Interpreter.

PhreaK

I've just been doing a bit of optimisation on this system and low and behold noticed a situation that would cause the master to enter a 'pseudo' infinite loop - a buffer was being updated while it was being processed (damn pass by reference). The extra resources involved in opening a telnet connection must have been just enough to push the system over the edge. All is now good, cheers for the help.