New Runtime Error with latest duest firmware...

hodeyp

Since upgrading to the new formware, my logs have been filling up with the following messages...

(0013711815) CIpEvent::AddInternalEvent - Max Queue Count = 25
(0013711824) CIpEvent::AddInternalEvent - Max Queue Count = 50
(0013711831) CIpEvent::AddInternalEvent - Max Queue Count = 75
(0013711838) CIpEvent::AddInternalEvent - Max Queue Count = 100
(0013711844) CIpEvent::AddInternalEvent - Max Queue Count = 125
(0013711851) CIpEvent::AddInternalEvent - Max Queue Count = 150
(0013711860) CIpEvent::AddInternalEvent - Max Queue Count = 175
(0013711867) CIpEvent::AddInternalEvent - Max Queue Count = 200
(0013711874) CIpEvent::AddInternalEvent - Max Queue Count = 225
(0013711882) CIpEvent::AddInternalEvent - Max Queue Count = 250
(0013711887) CIpEvent::AddInternalEvent - Max Queue Count = 275
(0013711897) CIpEvent::AddInternalEvent - Error Overflow

Any idea what this could be?

champ

I guess the internal queue and threshold sizes have been adjusted

The quick solution is to add Queue_and _Threshold_Sizes.axi to your program.

You should however look at tech notes 476 and 737 and do a search on these forums for "queue and threshold" to see the possible problems with this.

I found this problem occuring when I had a repeating timeline for emptying a serial queue and I set the repeat time below 100, although it was several firmware versions ago.

Also in TN737 I like this line ..."Do not hesitate to consult with AMX Technical Support about any issues, before or after attempting improvements"

hodeyp

thanks, tried your suggestion and sadly no difference.

It is definately something in the new firmware, I have reverted back to the previous version and all is now ok again.

I have managed to identify the module the problem is occuring in but annoyingly this is a compiled tko and I do not have access to the source.

Might pass this onto AMX...

Marc Scheibein

I also have this thing after updating to v3.12. It seems that bus handling tries to operate, but the master is still in boot process and is unable to handle the bus.

If this only occurs after a reboot, I'm ignoring it....

hodeyp

sadly for me it happens all the time and is affecting feedback routines for the AV processor. Looks to me as if the data coming back from the processor is somehow not being captured correctly and filling up a buffer somewhere.

Marc Scheibein

If your master is running 3.12 firmware, please check the memory sizes from terminal or Telnet with the SHOW MEM command.

>show mem
Volatile Free   : 14208004 (largest free block in bytes)
NonVolatile Free:   666792 (bytes free)
Disk Free       : 27443200 (bytes of free space)
Duet Memory Free :   396404 (bytes)
 Partition 1 -    396404 (bytes)
 Partition 2 - <UNKNOWN>

A problem I found was that the v3.12 firmware needs a lot more Duet memory than previous v3 versions.
The above values are of my NI2000, running no application(!!!). If I go to the webserver, Duet memory goes down to 40kByte, which is too less just to operate the webserver properly.
So it may help to size up the Duet mem to 2 or 3 MegaBytes:

>set duet mem size

Duet Memory: 1M
Enter new Duet Memory size (between 1M and 16M):  3M
Duet Memory set to 3M
>reboot

Another issue may be the feedbacks. If you do your feedbacks the following way

DEFINE_PROGRAM
[dvPanel,1] = [dvRelais,1]
....

it may help to delay them a little bit. This reduces traffic to the devices and will give back the master some more performance

DEFINE_PROGRAM
WAIT 3
{
[dvPanel,1] = [dvRelais,1]
....
}

DHawthorne

I'm inclined to think this is not a firmware bug per se, but a problem in your code that previous firmwares didn't catch. It is definitely the message queue; now the challenge is finding what device or code segment is causing it to overflow. This is one of those times where it would really be useful to have a more detailed error output - like the ability to output the current queue, so you can see what is backing it up. It would also be good if a backlogged message queue didn't bring the master to it's knees - something like this will eventually lock the system up, after which point there is no hope of figuring out what went wrong, all you can do is reboot and hope to catch it next time before it gets that far.

I hope someone in engineering is paying attention - we are getting more and more circumstances where the available debugging tools are simply not adequate for the job. It makes us look bad, and it makes AMX look bad when a customer has a system that keeps shutting down, and no one can quite figure out why. "Error Overflow" is useless by itself, and a "Max Queue Count" event is not in itself an error. Together they are only a hint at where the problem is, and not really a very good one in a complex system.

hodeyp

thanks, am I being simple or is CIpEvent going to be the result of messaging between the controller and an IP controlled device? If so, I do have a syslog tool writing debug data to a linux server. Will try switching this off and see if it makes an improvement.

DHawthorne

hodeyp wrote:

thanks, am I being simple or is CIpEvent going to be the result of messaging between the controller and an IP controlled device? If so, I do have a syslog tool writing debug data to a linux server. Will try switching this off and see if it makes an improvement.

Yes, that's the kind of event. But since all external (not part of the controller itself) NetLink devices that aren't Axlink are IP, that doesn't really narrow it down much. Turning off your log is a good start if there is any chance of there being a connection problem.

hodeyp

well I switched out all syslog stuff and errors are still there, it is definately something in my AV data_event as I see the errors only when I get feedback from that device.

However, I switched out all of my code and it still throws the error. I am using a 3rd party compiled module for the actual communication with the unit and I do not have access to the source code, looks like I may need to rewrite the comm module myself to fix this which is extreme in the least!!

o well, here goes...

hodeyp

ah, no need, just got a mail from amx tech support who are looking at a potential issue with the firmware itself.

will keep you posted...