Persistent Device Logging Module

DHawthorne

I've seen several solutions for logging events on a NetLinx device that will survive a master reboot, but none of them really did what I needed for my own use. So I'm going to share what I cobbled together and am using in my own projects.

This module requires no external server. It creates a daily log file for every device you want to monitor on the RAM drive of the master itself, which you can then access by FTP'ing into the master. You can specify how many days worth of log files it will maintain, and there are hourly timestamps in the logs to help narrow down problem spots. The module supports 100 strings worth of filtering, to trap out all that useless dreck you don't need in your logs. If your system crashes and burns, the log should be intact up to, or pretty nearly up to, the time of impact .

I have been using this on several system, and have been tweaking it as I go; I've included the full source so anyone using it can do their own tweaking and modifications. I deem it ready for prime time, but you may have a different opinion, so feel free to adjust it as necessary on your own. If I make any significant changes, I'll repost it.

There is a Readme in the ZIP file that should cover everything you need to know to use the module. Don't forget to compile it, only the source is included.


EDIT: I have uploaded a new version below, and deleted this one because it was broken. See the post below...

Thomas Hayes

Great Dave
This could be handy for those hard to troubleshoot problems.

DHawthorne

OK, color me embarrassed.

I made a quick update to that module before uploading it here, and didn't think it was a big enough change to warrant extensive testing. Well, it shouldn't have been, but it seems I broke the filtering feature. If you downloaded it, replace it with the one attached to this post.

DHawthorne

DevTermLogger v. 2.0

I'm putting out a new version of this module that adds two things:

1) Drive space checking. I've found that a runaway error condition can potentially fill up the RAM drive with logged messages, especially on smaller masters (I have one of the original ME cards at home, it's tiny compared to the ME260 or ME260/64's). Since NetLinx does not seem to have any built-in memory checking functions (grumble), I've had to put in a once-a-minute query to the master for it's available drive space. Between queries, it updates the number based on what it knows it has written to the file, but this can only be considered an approximation because it can't know what other modules or processes may be doing to that number. Hence, the once-a-minute check to get the real value. I have also put in a user-settable threshhold at which logging will be disabled. For example, if you tell it to reserve 10,000 bytes, when there are 10,000 or less bytes free on the drive, logging will stop until some space is freed up. It will automatically restart when drive space becomes available. You can't set the threshhold below 2048, and I wouldn't think you would want to in any case .

2) E-mail notifications. The module now uses i!-EquipmentMonitorOut to send an e-mail notification if the available drive space reaches the reserved space threshhold. I would think the last thing you would want on a system you are monitoring is a full drive of error messages, and the logging happily shut down for weeks before you got around to checking it. Chances are, if you have run out of drive space, something dire has happened, and you need to look into it prontissimo. So, if you enable the e-mail, you will be notified and can do so right away.

These changes require some additions to the parameters to the module, so it's not exactly a drop-in replacement for version 1. You'll have to add those new parameters to update an existing project that uses it.

DHawthorne

Having run this version of the module on a local master for several days, I find there is an annoying side effect from having to query the memory status. I'm setting a flag, suspending the log, doing the query, and reinstating the log when I get a response. However, because of the way NetLinx programs run internally, sometimes part of that response actually goes through before the flag gets set, even though I am setting it in code before I actually send the query. Due to the nature of the telnet DATA_EVENT that tracks the responses, all manner of variations of partial "show mem" commands wind up in the log from the repeated queries. THe module, as posted, filters out the most egregious of these partial responses, but after running a few days I see there are still some variants getting through. I'm not sure it's worth chasing them all down, so a few iterations of "show me," and an occasional partial memory listing show up on the log that would otherwise be trapped. I could probably fix this with a delay, but I'm concerned about a real event coming in while I have the log temporarily disabled. So I have decided to live with these little artifacts here and there. The system I am testing on only shows two or three in a 24-hour period, so I guess I can live with it ... until they add a "free disc space" keyword to NetLinx.

DHawthorne

Yet another version. Since I added e-mail notifications to the RAM drive space monitor, I thought, why not expand that? The module now takes another string array as a parameter, and if any of those strings are logged, and you have the e-mail parameters filled in, it will send an e-mail out to that effect. In essence, this can be used to replace existing !i-EquipmentMonitorOut in your code (it will co-exist with those, but I think it better to consolidate). For example, if you are currently using !i-EquipmentMonitorOut to notify you a projector lamp needs replacement, you can replace that with a SEND_STRING 0, "'Projector lamp needs replacement", and then put {'Projector lamp'} as one of the parameters for the e-mail alerts. The need to replace the lamp will both be logged, and an e-mail sent to that effect.

pauld

Wow, thats a great module, and with email no less. Thanks for sharing this useful module with the rest of us.

Paul d

DHawthorne

My thanks to NMarkRoberts - he caught a bug in my Dates.axi include file. It doesn't affect these modules because the goof is in functions I added later, and kind of forgot I added, so I haven't even used them myself . It could very much mess you up if you are using that include in other projects though. I'm going to update the zip files posted in this thread, but if you want to save the download, just edit Dates.axi accordingly:

Line 192 : change
     ACTIVE((dtTest.sGregorian <> "'Invalid'") || (dtRef.sGregorian <> "'Invalid'")) : 	RETURN FALSE ;
to
     ACTIVE((dtTest.sGregorian == "'Invalid'") || (dtRef.sGregorian == "'Invalid'")) : RETURN FALSE ;

Line 211 : change
     ACTIVE((dt1.sGregorian <> "'Invalid'") || (dt2.sGregorian <> "'Invalid'")) : RETURN FALSE ;
to
     ACTIVE((dt1.sGregorian == "'Invalid'") || (dt2.sGregorian == "'Invalid'")) : RETURN FALSE ;

DHawthorne

Just a heads up: the e-mail portion of this module is broken at the moment. I had to put in a flag to suppress e-mails when certain events were logged to prevent and endless loop of e-mails being sent out, but my flag is not resetting properly and suppressing them all the time. I just noticed this on-site at a customers, and I have my hands full with other matters at the moment. I probably won't get a fix up until next week. Just be aware of this if you are using the module ... you can reactive the emails if you connect to your master and do a watch on the variable bHold_Email. When you have it open, set it to 0. When an e-mail goes out, it may reset to 1, so you might have to do this repeatedly after each e-mail until I get it fixed . Sorry about the inconvenience, something just slipped by me in testing it somehow ...

DHawthorne

It seems the nature of the bug is that if the email alert list is left empty, the flag is not reset properly for a manually sent email, so it's not completely broken and should still work if you are watching for something. I should be able to get a revision out Monday.

DHawthorne

OK, here's the fix for the e-mail functions. I'm not altogether sure what I was thinking the way I originally managed that flag . It was one of those odd bugs that didn't show the way I was testing it, it wasn't until I plugged it into a different system than my test setup that I noticed it ...

dthorson

Something Similar?

Dave, this is a great module. I'm hopping to use this one for some troubleshooting.

Do you have a something similar that logs commands to/from an rs-232 port? I have a problematic Sanyo projector that seems to be intermittently locking up.

Thanks.

dthorson

Nevermind, I got it.

USE Send_String 0,

Thanks

travis

has this changed much in the last 5 years?

DHawthorne

I haven't updated it in quite a while, but I use it in every single project I create. In other words, it does everything I need it to do, and if there are still some bugs I simply haven't uncovered them the way I use it.

travis

Thanks. I use it most projects .

adys

Is there any example available for this module?

All I need is to save logs to a local file on my master

thanks

vining

.? It's posted for download earlier in this thread.

adys

vining wrote: »

.? It's posted for download earlier in this thread.

The module is, I don't see any examples?

vining

adys wrote: »

The module is, I don't see any examples?

What type of example are you looking for, how to install and use the module or what the log looks like?

adys

How to use it for a simple write to file... I need to track down something that is happening every few days.

I saw the text file but it a bit complicated, all I need is a simple example, thanks

* maybe this is over kill, but I couldn't find any way to create a log file ( I searched the forum and found this thread)

the8thst

There is plenty of help in the Netlinx Keyword Help file. Just search for "file". You don't need a module to create a simple log file.

I'm feeling generous... this will do what you want:

	define_function integer logToFile (char msg[])
	{
		stack_var slong slDir
		stack_var slong slFile
		stack_var slong slResult
		stack_var char logMsg[200]
		
		slDir = file_createdir('\LOGS\')
		
		logMsg = "'',date,' ',time,'--> ',msg,10"
		
		if (slDir == 0 || slDir == -13) {
			slFile = file_open('\LOGS\log.txt',file_rw_append)
			
			if (slFile > 0) {
				slResult = file_write(slFile,logmsg,length_string(logMsg))
				file_close(slFile)
			} else {
				send_string 0, "'<logToFile> Directory OK: File Error[',getFileError(slFile),']'"
			}
		} else {
			send_string 0,"'<logToFile> File Error[',getFileError(slDir),']'"
		}
	}
define_function char[60] getFileError (slong slErr)
	{
		switch (abs_value(slErr))
		{
			case 0: return '(0) Success'
			case 1: return '(-1) Invalid File Handle'
			case 2: return '(-2) Invalid File Path or Name'
			case 3: return '(-3) Invalid Value Supplied for IO Flag'
			case 4: return '(-4) Invalid Directory Path'
			case 5: return '(-5) Disk I/O Error'
			case 6: return '(-6) Invalid Parameter'
			case 7: return '(-7) File Already Closed'
			case 8: return '(-8) File Name Already Exists'
			case 9: return '(-9) End Of File Reached'
			case 10: return '(-10) size of DirPath buffer too small for dir path name'
			case 11: return '(-11) Disk Full'
			case 13: return '(-13) Directory Already Exists'
			case 14: return '(-14) Max Num Files Already Open (10)'
			case 15: return '(-15) Invalid File Format'
			default: return "'(default) slErr[ ',itoa(slErr),' ]'"
		}
	}

vining

One thing that Dave's module does is check the systems memory and it prevent you from writng to the log if there's not enough left. it also deletes old files that are older then the number of specified days so you minimize the amount of memory used. Then the are filters and triggers that let you tailor what's logged and when the module should send email notications.

adys

Log File

the8thst wrote: »

There is plenty of help in the Netlinx Keyword Help file. Just search for "file". You don't need a module to create a simple log file.

I'm feeling generous... this will do what you want:

	define_function integer logToFile (char msg[])
	{
		stack_var slong slDir
		stack_var slong slFile
		stack_var slong slResult
		stack_var char logMsg[200]
		
		slDir = file_createdir('\LOGS\')
		
		logMsg = "'',date,' ',time,'--> ',msg,10"
		
		if (slDir == 0 || slDir == -13) {
			slFile = file_open('\LOGS\log.txt',file_rw_append)
			
			if (slFile > 0) {
				slResult = file_write(slFile,logmsg,length_string(logMsg))
				file_close(slFile)
			} else {
				send_string 0, "'<logToFile> Directory OK: File Error[',getFileError(slFile),']'"
			}
		} else {
			send_string 0,"'<logToFile> File Error[',getFileError(slDir),']'"
		}
	}
define_function char[60] getFileError (slong slErr)
	{
		switch (abs_value(slErr))
		{
			case 0: return '(0) Success'
			case 1: return '(-1) Invalid File Handle'
			case 2: return '(-2) Invalid File Path or Name'
			case 3: return '(-3) Invalid Value Supplied for IO Flag'
			case 4: return '(-4) Invalid Directory Path'
			case 5: return '(-5) Disk I/O Error'
			case 6: return '(-6) Invalid Parameter'
			case 7: return '(-7) File Already Closed'
			case 8: return '(-8) File Name Already Exists'
			case 9: return '(-9) End Of File Reached'
			case 10: return '(-10) size of DirPath buffer too small for dir path name'
			case 11: return '(-11) Disk Full'
			case 13: return '(-13) Directory Already Exists'
			case 14: return '(-14) Max Num Files Already Open (10)'
			case 15: return '(-15) Invalid File Format'
			default: return "'(default) slErr[ ',itoa(slErr),' ]'"
		}
	}

Many thanks!
I am sure this will help other people as well.

Too bad I can't add a tag, I named my respond as "log file", hope that it will do the work.

Ady

the8thst

adys wrote: »

Many thanks!
I am sure this will help other people as well.

Too bad I can't add a tag, I named my respond as "log file", hope that it will do the work.

Ady

What do you mean?
logToFile("'TAG - Some Message Variable',foo")
of just edit the function.

the8thst

vining wrote: »

One thing that Dave's module does is check the systems memory and it prevent you from writng to the log if there's not enough left. it also deletes old files that are older then the number of specified days so you minimize the amount of memory used. Then the are filters and triggers that let you tailor what's logged and when the module should send email notications.

Yes I know about all of the extra things that Daves module adds, but I was replying to the person asking for a simple way to make a log for a few days to find a bug. The module is over his head and overkill for that scenario.

vining

Agreed, i just wanted to make sure he handles the file properly and doesn't let it use up all available memory. It may take a long time depending on how much is logged but if left unchecked the system may crash when the memory is gone. Could be a couple of years, a month, maybe never but it should be considered.

adys

the8thst wrote: »

What do you mean?
logToFile("'TAG - Some Message Variable',foo")
of just edit the function.

I meant to the forum post

DHawthorne

vining wrote: »

Agreed, i just wanted to make sure he handles the file properly and doesn't let it use up all available memory. It may take a long time depending on how much is logged but if left unchecked the system may crash when the memory is gone. Could be a couple of years, a month, maybe never but it should be considered.

When I was developing that module, and before I put the memory check in, I actually had a runaway bug completely fill the master's memory drive in a matter of hours and lock the whole thing up. It takes surprisingly little. Perhaps it's not so easy nowadays, and the masters have a lot more onboard memory.

That said, for a temporary sort of thing, it might not be ill-advised to connect a laptop, open a telnet connection and set the client to log to a file, and just leave it running. That way, the drive space eaten up is on the laptop, not the master.

nicholasjames

DHawthorne wrote: »

That said, for a temporary sort of thing, it might not be ill-advised to connect a laptop, open a telnet connection and set the client to log to a file, and just leave it running. That way, the drive space eaten up is on the laptop, not the master.

For tracking down specific bugs, that's exactly what I do. For logging and scripting, I prefer SecureCRT. I talked about it some in this thread: Controlling the Master from Batch Files

For logging, each profile can have a different method of logging. For example, I have one session for "daily" logging that creates a log file based on the date and auto sends "msg on 3" so I only get the SEND_STRING 0 messages, but I have a separate session for debugging purposes that initially sends "msg on all" and bases the log file on the current time (down to the millisecond if you need!) so that each time the processor reboots, I get a separate log file so I don't have to wade through hours of logging data to find that last error message.

When you do find that error message, you could use SCRT's great scripting integration to do almost anything with the data. While you can use JScript and Python for scripting, they have a great guide on using VBScript and quite a number of example VB scripts that come with the program. You can control almost every aspect of the program, including creating new sessions, starting/stopping logs, parsing/responding to data, modifying session parameters, and everything else VB is capable of.

-Nick

[size=-2]Fineprint: I swear I'm not being compensated by VanDyke Software. I just really like their program. ;-)[/size]