AMX Zip

Joe Hebert

I?m working on an application that could potentially produce several hundred files (only a few k bytes or less each) within a couple of dozen or so subdirectories to be stored on an NI Master. I?d like to do automatic periodic offsite backups but I want to zip the entire file directory structure that the application uses and send the data as one valid zip file attachment.

Has anyone ever tried something like this or have any pointers? I?m not looking to do any compression but I do want the file to be a valid zip so that it can be opened and extracted (keeping the file directory structure in tact) without having to write a custom utility on the PC side to get to the data.

Any zip help is appreciated.

Thanks.

vining

Well I don't know how you would go about compressing files into a zipped folder while on the master but if you stick to straight FTP you can make a master list of file you want to check and periodiclally query those files to determine which files have been modified since the last transfer. This way once this query is complete you just build a file list of the files that have changed and thus need to be transfered.

I've done this on my desktop running the FileZilla server and query using the FTP MDTM command which returns the files date time stamp when it was last modified and compare that to the date time stamp I have on record in the master. Now I've only done this one way so I don't know if the master support this command. It seems all FTP servers have a slightly different lists of supported commands.

This might be a better option considering it may be impossible to zip from inside the master. You could probably do it w/ PHP but you'd be transfering the files to the PC to zip and then putting the file back on the master to transfer out and that makes no sense. At least if the master supports the MDTM command you can transfer just what's needed, normally, unzipped.

GSLogic

Joe Hebert wrote: »

I?d like to do automatic periodic offsite backups but I want to zip the entire file directory structure that the application uses and send the data as one valid zip file attachment.

If you find a way let me know. I email client XML files to my server every month, so if their NI master ever completely dies I can get them back up with all the same user setting... this could have been a big waste of time as four years have gone by and I'm still waiting for the first master to die!

I also found that you can not email an XML file that is not in a directory. I don't know why but I thought I'd share it with you in case you try.

Joe Hebert

vining wrote:

Well I don't know how you would go about compressing files into a zipped folder while on the master

That?s not what I?m looking to do. I want to create a file on the master and place inside that file all the data from all the other files, an uncompressed zip file that retains the directory structure.

vining wrote:

This might be a better option considering it may be impossible to zip from inside the master.

I think it?s very possible to do; after all it?s just a matter of creating a file on the master and dumping a bunch of data inside. The only thing I don?t know is what algorithm WinZip uses to do it. Minor detail.

vining wrote:

I've done this on my desktop running the FileZilla server and query using the FTP MDTM command which returns the files date time stamp when it was last modified and compare that to the date time stamp on have on record in the master.

I suppose I could do a manual differential backup (and may end up just doing that) but I?d prefer to create a hands free operation if at all possible.

GSLogic wrote:

I also found that you can not email an XML file that is not in a directory. I don't know why but I thought I'd share it with you in case you try.

Thanks for the heads up. I normally don?t store anything in the root directory except for keys so I should be okay there.


Thanks for the feedback guys.

If there are any WinZip wizards out there please chime in.

yuri

i thought there was someone who had made some sort of zip duet module. You will most definatly have to switch to Duet

vining

Joe Hebert wrote:

I suppose I could do a manual differential backup (and may end up just doing that) but I?d prefer to create a hands free operation if at all possible.

It wouldn't have to be manual. Just pick a time of day to run the MDTM command query (if it works on the master's ftp server) and create a list of the files that have been modified, then automatically do the transfer. I have code that does this but I query for files on a desktop every five minutes to look for changes, then depending flag state for "auto transfers" I will either automatically retrieved the modified files and parse them or just indicate that files have been modified and wait for a manual button push to initiate the transfers. I originally wrote this so I could have customers make notes on a PC, to do lists, kid's chores, etc and have the system automatically retireve the file and parse it for display on a TP. Kind of like a notepad. for up to 10 files but it could easily do 100s of files and just omit the parsing steps.

jweather

ZIP file format does support uncompressed files, so it's just a matter of creating the appropriate header and file structure. The format is open, and descriptions of it are available online (see Wikipedia, for instance). You may find a different format easier to work with, though -- for example, TAR is designed for collecting uncompressed files into a single file archive, and is understood by most modern "zip" software, such as Winzip and 7-zip.

vining

Well it apprears the master doesn't support the MDTM command anyway.

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\VAV>ftp
ftp> o 192.168.1.60
Connected to 192.168.1.60.
220 VxWorks (VxWorks5.5.1) FTP server ready
User (192.168.1.60:(none)): vining
331 Password required
Password:
230 User logged in
ftp> ?
Commands may be abbreviated.  Commands are:

!               delete          literal         prompt          send
?               debug           ls              put             status
append          dir             mdelete         pwd             trace
ascii           disconnect      mdir            quit            type
bell            get             mget            quote           user
binary          glob            mkdir           recv            verbose
bye             hash            mls             remotehelp
cd              help            mput            rename
close           lcd             open            rmdir
ftp>

Joe Hebert

Attached is the .Zip file format specification (v6.3.2 revised 09.28.2007) if anyone else is interested.

Spire_Jeff

Since you are not compressing the files, you may also want to check the .tar format. I was looking at it briefly yesterday and it didn't seem very difficult to implement. I am pretty sure there are fewer options (which means a little less code), so it may be easier to code and still accomplish the same thing.

Jeff

gusgizmo

Here is the wikipedia for tar, it really shouldn't be too hard to implement, and you could write your own timestamps into the headers for each file.

http://en.wikipedia.org/wiki/Tar_(file_format)

The hardest part to implement will probably be the checksumming, which isn't too bad IMO.

Spire_Jeff

Has anyone made any progress on this? Due to a catastrophic blunder on my part, I have a high motivation to make this happen. As a result, I whipped together the beginnings of a duet module that does this. So far, I have spent about 10 minutes writing code, 90 minutes figuring out how to tell java what I want it to do, and 1 hour playing with it on a master to figure out the intricacies of the netlinx master. The nice thing is: I have a module that WILL create a COMPRESSED zip file.

Here is all of the java code I had to write to make this happen:

	public void handleCommandEvent(Event obj, String cmd) {
		super.handleCommandEvent(obj, cmd);
		// Command syntax: ZIP-(\FolderToZip)-(NameOfZip.zip)
		if(cmd.startsWith("UNZIP<>")){
			System.out.println("UNZIP<> not yet implemented.");
		}else if(cmd.startsWith("ZIP<>")){
			System.out.println(cmd);
			StringBuffer zipMe = new StringBuffer(cmd);
			System.out.println(zipMe.toString());
			zipMe.delete(0,zipMe.indexOf("<>")+2);
			System.out.println(zipMe.toString());
			if(zipMe.indexOf("<>")>=0){
				String folder = zipMe.substring(0,zipMe.indexOf("<>"));
				zipMe.delete(0,zipMe.indexOf("<>")+2);
				String fileName = zipMe.toString();
				//folder = folder.substring(folder.length()-2);
				try {
					zipFolder(folder,fileName);
				} catch (Exception e) {
					System.out.println("Problem writing ZIP<>" + folder + "<>" + fileName);
					System.out.println(e.getMessage());
					e.printStackTrace();
				}
			}else{
				System.out.println("Malformed ZIP command");
			}
		}else{
			System.out.println("Unrecognized command.");
		}
	}


	static public void zipFolder(String srcFolder, String destZipFile) throws Exception {
	    ZipOutputStream zip = null;
	    FileOutputStream fileWriter = null;

	    fileWriter = new FileOutputStream(destZipFile);
	    zip = new ZipOutputStream(fileWriter);

	    addFolderToZip("", srcFolder, zip);
	    zip.flush();
	    zip.close();
	  }

	  static private void addFileToZip(String path, String srcFile, ZipOutputStream zip)
	      throws Exception {

	    File folder = new File(srcFile);
	    if (folder.isDirectory()) {
	      addFolderToZip(path, srcFile, zip);
	    } else {
	      byte[] buf = new byte[1024];
	      int len;
	      FileInputStream in = new FileInputStream(srcFile);
	      zip.putNextEntry(new ZipEntry(path + "/" + folder.getName()));
	      while ((len = in.read(buf)) > 0) {
	        zip.write(buf, 0, len);
	      }
	    }
	  }

	  static private void addFolderToZip(String path, String srcFolder, ZipOutputStream zip)
	      throws Exception {
	    File folder = new File(srcFolder);

	    for (int x=0;x< folder.list().length;x++ ) {
	    	String fileName = folder.list()[x];
	      if (path.equals("")) {
	        addFileToZip(folder.getName(), srcFolder + "/" + fileName, zip);
	      } else {
	        addFileToZip(path + "/" + folder.getName(), srcFolder + "/" + fileName, zip);
	      }
	    }
	  }

(I cheated and found the various zip methods already written online )

This is NOT a complete module as I still need to add the ability to extract from a zip file and a bunch of other nice things (like figuring out the correct path automatically), but it does create a zip file and it is compressed.

If anyone wants to try out the module, (WITH ABSOLUTELY NO WARRANTIES nor GUARANTEES!), let me know and I will send it your way. As soon as I a more polished module, I will post it.

Jeff