What's the best way to store text data logs onto a PC?

Brent Simpson

I use NetLinx panels to automatically manage buildings without users having to touch anything. The systems tend to generate large text logs (during the set phase) of which I've been using a local PC to log via Hyperterm but thought why can't I just get NetLinx to create a new file on the local PC each day and just send data to it as required? I've used the File_ commands to create files read and write to the 'disk on chip' so logically I figure it should be able to go to a PC. Could anybody point me in the right direction to do that? Example ot technical doc would be useful....

Many Thanks!

Brent - Am suffering Brain Fatigue!

Joe Hebert

FTP and email are two ways that come to mind.

vining

You can use i!-FTPSender to transfer files to your PC. I use it w/ FileZilla Server (free down load from SourceForege).
http://amx.com/techcenter/downloadConfirm.asp?fn=/assets/applicationFiles/i!-FTPSenderSetup.exe

You can also email a daily log to yourself or any body else you want. DHawthorne wrote a DevTermLogger module http://amxforums.com/showthread.php?t=1187&highlight=devtermlogger
that you can use to create a daily log and then use i!-EquipmentMonitor to email it.

jjames

Why not just run a Syslog daemon on the PC? Such as Kiwi Syslog or something along those lines (I'd like to get the newest Kiwi because you can view the logs remotely via the web.) Essentially, you send UDP commands to the server and it stores them.

I've started toying around with this and it works quite well actually. I just need a reliable PC to run the syslog on.

Brent Simpson

Thanks Guy's

I already use the i!-EquipmentMonitor to email system offlines, faults, errors and temperature irregularities to me. It's very useful - I usually know there's a problem before the owner or site managers!

I started storing the data to disk on chip and FTPing it across but the files tend to get fairly large by necessity during setup stages - 120->400Mb per day depending on the size of the system. it tends to stall the master if it gets out of hand....... Hyperterm across the LAN seems to be the best way but once again it has to be a fast link otherwise the master stalls intermittantly.

I was hoping I could somehow use the File_Create / File_Write_Line commands to create a new file on a PC HD and write to it as required. Could see anything in help files / NetLinx manual - Is this not possible?

Maybe I need to create small files and FTP them out 100k at a time?

I'm just making it up as I go along so it may not be practical?

Brent

Joe Hebert

Brent Simpson wrote: »

I was hoping I could somehow use the File_Create / File_Write_Line commands to create a new file on a PC HD and write to it as required. Could see anything in help files / NetLinx manual - Is this not possible?

Yes, that is not possible.

Brent Simpson wrote: »

120->400Mb per day

Is that a typo? If not, what are you logging?

Brent Simpson

Not a typo

No that's not a typo. The short answer is - it's logging text strings that are created as it processes each event. It's a bit complicated to explain - the system is totally event driven from any input so apart from a requirement to process and update a room status every 10 sec there is next to no main line program code. As in any building, anything can happen at any time and that means a lot of logging needs to be recorded in order to identify and correct errors and unforeseen events. So far the largest residential system we've installed manages 48 rooms, including 68 sensors (movement/door bugs), 270 lighting circuits, Diesel Boiler, 28 x individually monitored and managed UFH arears, Full Pool/Spa control, 60 Roller shutters, 2 x Access Controled doors plus 2 remote gates, 4 Sony IP dome cameras, 3phase power monitoring / UPS monitoring and sewage treatment monitoring. It's managed on one NI-4100 and 2 NI-700's for co-processing plus 1 x NXI for Boiler/Pool/Spa and Heating Management. Everything is displayed and controlled via one MVP-8400. Admittedly there are 160 CBus keypads to overrride the lights but they are only used as a last resort.

The older NI-X000 controllers struggled a bit but the newer NI-X100 cope quite well. It takes several weeks for the system to 'adapt' to the occupiers, after which only error logging is required which reduces the log to 20k - 1M per day depending on what the users are doing, power surges and errors that crop up from time to time.

vining

Joe Hebert wrote:

Quote:
Originally Posted by Brent Simpson
I was hoping I could somehow use the File_Create / File_Write_Line commands to create a new file on a PC HD and write to it as required. Could see anything in help files / NetLinx manual - Is this not possible?

Yes, that is not possible.

Isn't that possible through FTP commands if the FTP server you're running on the PC supports it? It's been a long time since I played with FTP between the master and PC but I think I used to do this.

Joe Hebert

Brent Simpson wrote: »

No that's not a typo.

Wow, it sounds like you?re logging every line of code that is run to get that amount of data. Does someone actually read the logs? That must be a full time job in and of itself.

I assume you must not be able to run the code on a standard NI-4100? According to the AMX web site those ship with only a 128 MB CF card. (I?m wondering if that is an error since the NI-3100 says it comes standard with 256 MB) Even if the NI is upgraded to its fullest capacity (1G) that will give you less than 3 days of run-time before the CF card is full if you?re blasting it with up to 400 MB per day. I can understand why you say the logging tends to stall the master if it gets out of hand.

Brent Simpson

Time consuming

Yep it can be pretty time consuming, but I've developed ways of cutting through the files pretty quickly. Pathway was designed to manage itself (well an honest attempt anyway). Sometimes it has a mind of it's own so logging a lot of data is the only way I can think of to figure out what caused odd things happen. I'm hoping in 10-15 years this type of technology will be standard in most houses - if for no other reason than for energy management. All you need is an NI-700, compatible security system, Lighting system and a Keypad/Touch Screen to manage it.

Thanks for you input guys!

PhreaK

Wow, thats a whole lot of data you're collecting there. Rather than utilizing plain text logs have you considering some form of database based logging, it will make it a whole bunch easier to do any analysis and will allow you a much cleaner way of storing everything you collect.

The other way of approaching it which I have found useful is to set up some debug messaging functions in a nice neat include:

(**
 * Functions and code for assisting with system debugging, emulation and remote
 * monitoring.
 *
 *)
PROGRAM_NAME='Debug'


DEFINE_DEVICE

dvDebug =	0:0:0			// Debug Device


DEFINE_VARIABLE

CONSTANT INTEGER DEBUG_OFF = 0	// Available debug verbosity levels
CONSTANT INTEGER DEBUG_ERROR = 1
CONSTANT INTEGER DEBUG_WARN = 2
CONSTANT INTEGER DEBUG_INFO = 3

CONSTANT CHAR DEBUG_LEVEL_STRINGS[4][16] = {
    'DEBUG_OFF',
    'DEBUG_ERROR',
    'DEBUG_WARN',
    'DEBUG_INFO'
}

PERSISTENT INTEGER nDebugLevel	// Current system debug level


(**
 * Returns a string representing the debug level passed.
 *
 * @param	nLevel		an integer specifying the debug level
 * @return		a string representing the level
 *)
DEFINE_FUNCTION CHAR[16] fnDebugGetLevelString (INTEGER nLevel) {
    RETURN DEBUG_LEVEL_STRINGS[nLevel + 1]
}

(**
 * Sets the current system debugging level for controlling debug message
 * verbosity.
 *
 * @param	nLevel		an integer specifying the debug level to set
 *)
DEFINE_FUNCTION fnDebugSetLevel (INTEGER nLevel) {
    STACK_VAR CHAR sDebugLevelString[16]

    IF (nLevel <> DEBUG_OFF &&
	nLevel <> DEBUG_ERROR &&
	nLevel <> DEBUG_WARN &&
	nLevel <> DEBUG_INFO) {
	fnDebugMsg(DEBUG_ERROR, "'Invalid debug level specified, defaulting to DEBUG_ERROR'")
	fnDebugSetLevel(DEBUG_ERROR)
	RETURN
    }

    sDebugLevelString = DEBUG_LEVEL_STRINGS[nLevel + 1]

    IF (nLevel == nDebugLevel) {
	fnDebugMsg(DEBUG_INFO, "'Debug level already set to ', sDebugLevelString, '. Nothing changed'")
    } ELSE {
	fnDebugOutput("'Debug level set to ', sDebugLevelString")
	nDebugLevel = nLevel
    }
}

(**
 * Sends a string to the debug device. If the ability to dump to a file or
 * netorked logging service is required it can be added here.
 *
 * This should not be used for program debug messages, they should be sent to
 * fnDebugMsg for filtering.
 *
 * @param	sOutput		a string containing the message to send
 *)
DEFINE_FUNCTION fnDebugOutput (CHAR sOutput[1024]) {
    SEND_STRING dvDebug, sOutput
}

(**
 * Voices a debug message if required by the current debug level. All system
 * messages should pass through here.
 *
 * @param	nLevel		an interger specifying the debug level of the message
 * @param	sMessage		a string containing the debug message
 *)
DEFINE_FUNCTION fnDebugMsg (INTEGER nLevel, CHAR sMessage[128]) {
    IF (nLevel <= nDebugLevel) {
	SWITCH (nLevel) {
	    CASE DEBUG_INFO: fnDebugOutput("'INFO: ',sMessage")
	    CASE DEBUG_WARN: fnDebugOutput("'WARN: ',sMessage")
	    CASE DEBUG_ERROR: fnDebugOutput("'ERROR: ',sMessage")
	    DEFAULT: fnDebugMsg(DEBUG_ERROR, "'Invalid debug message level specified (',sMessage,') (', __FILE__, ', line ', ITOA(__LINE__), ')'")
	}
    }
}

This approach lets you permamently place all your debug messaging throughout your code and allow you to change your debug messaging levels and where the messages go. That way you can set up rather thorough debugging (ie a message for each event) that you have the option of turning off when you're not interested in what the system is doing.

Brent Simpson

Logging Levels

Thanks Kim, I already do that to a certain extent but it's based on Function processing, I/O and System types rather than just levels. It's set up so that I can alter the logging remotely while it's running. I suppose I could extend that approach to log individual room numbers and system I/O which would reduce the logs - good thought!

Spire_Jeff

This has been touched on already, but I will toss my 2 cents at storing this in a database. You could setup a MySQL server with Apache and use PHP or some other scripting language to create a submission form to store the data in MySQL. If you really have that much data, you could even consider just opening a connection directly to the MySQL server and send INSERT commands with the data.

I am working on doing something similar, but with hopefully less data being generated I will let you know if I find out anything interesting about the processors and creating that many web page requests quickly.

Jeff