case
1Oct/1090

Read/Load files from ZIP in JavaScript

At work we do a lot of JavaScript applications that have little to no HTML files (beside the index), everything is generated by JavaScript and we have tons of .js files that load at runtime and to reduce the size we have to minify them and that is really hard to debug and watch in the browser if something goes wrong. So I've made this JavaScript class that comes to help. What can you do is Zip up all your assets (js, images, css) and use ZipLoader class to read the zip and unpack your resources whenever you need them. It's small (10KB), easy to use and quite fast considering the extraction is made by JavaScript.

1
2
3
4
5
6
7
8
9
10
   // loads and caches the zip file
   var loader = new ZipLoader('files.zip');
   // creates a style node in the document header and appends the style
   loader.LoadCSS('files.zip://style.css');
   // creates a script node in the document header and appends the script
   loader.loadScript('files.zip://jquery.js');
   // returns the file content
   var someFileYouNeed = loader.load("files.zip://myFile.txt");
   // returns the base64 encoded image usable as img source
   $("#logo").attr('src', loader.loadImage('files.zip://images/logo.png'));

The level of compression using zip surpasses the js and css minification in all the ways possible. For example, My current application has 79 files (js, and few css) and they all sum up to 924KB in size. Zipping all those 96 files using Ultra compression level gives me 1 file of 126 KB => 1 server request. That's something.

The best performance is made Opera (strangely), then Firefox, Crome, Safari and last IE (no surprise here, I was expecting it before I even started).

It's a V 1.0 that still needs some work and it doesn't work in IE because that stupid browser overrides the mime type and at first null character in the reading of the zip, the content ends. I will fix this eventually icon smile Read/Load files from ZIP in JavaScript I will also include this as a jClass library soon.

Update: Fixed it so it works in IE as well. There is a performance issue still. Will fix that too.

Till then, HERE are the sources and a demo.

The zip in the demo is made using 7Zip with Ultra Compression level.

Happy codding.



Comments (90) Trackbacks (2)
  1. Hello, this is realy a great work!!!

    I love Greasemonkey, so my question – can you make an smaller Version of it, to simple zip and unzip strings for storing them with GM or with the new BrowserStorageObjekt?

    thx

    • I’ve made few compression algorithms in my jClass project. You can just download the library and copy paste code from there in another class. They are written in jClass way but they all have 2 encode/decode functions so jClass is not necessarily

  2. Hi,

    Can it load zip file over http ?

    I need to load a file from server as fast as possible, and zipping it first would be a good solution.
    Then I would just use this library to unzip and use its data, in javascript.

    Best regards,
    Esko

  3. Hi,

    Could you please advice, Firebug reports this error when trying to use the component:

    CLIP–>
    uncaught exception: [Exception… “Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive.js :: loadBinaryResource :: line 507″ data: no]

    Line 0

    uncaught exception: [Exception… “Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive-min.js :: loadBinaryResource :: line 1″ data: no]

    Line 0
    <–CLIP

    please help:)

    br,
    Esko

  4. Sorry, the comment was imperfect, propably due to CLIPpings having < and >s. It should have been like this:

    Could you please advice, Firebug reports this error when trying to use the component:

    CLIP–*
    uncaught exception: [Exception… “Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive.js :: loadBinaryResource :: line 507″ data: no]

    Line 0
    *–CLIP

    The above comes when I have included both scriptfiles, like so:

    CLIP–*
    *script type=”text/javascript” src=”/suppfiles/archive-min.js”**/script*
    *script type=”text/javascript”*
    *–CLIP

    This below comes when the “archive.js” was not included:

    CLIP–*
    uncaught exception: [Exception… “Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive-min.js :: loadBinaryResource :: line 1″ data: no]

    Line 0
    *–CLIP

    please help:)

    br,
    Esko

  5. Hi,

    I added an exception in popup blocker for metsanergia.test.ebsolut.fi, but bthe error comes still

    br Esko

  6. Hi,

    Problem solved!

    I debugged the archive.js file.
    There is a function loadBinaryResource(url), which has a line

    req.open(‘GET’, url, false);

    Function gets called two times, first to open the zipfile.
    Firstly, the parameter url was
    ‘http://metsaenergia.test.ebsolut.fi/smavimap/liitynta.zip’
    for the zip file.

    At second time, it is called to get the file inside the zip.
    Here I had, erroneously, the parameter url being
    tdelem.innerHTML = loader2.load(‘liitynta.zip://liitynta.xml’);
    and it resulted to “Access denied” error.

    After I changed that to
    tdelem.innerHTML = loader2.load(‘http://metsaenergia.test.ebsolut.fi/smavimap/liitynta.zip://liitynta.xml’);
    it worked OK.

    br Esko

    • Yes, it caches the zip by name internally so next time you want something from it it will check if it needs to load the zip or it’s there already.

  7. Hi again,

    The problem was solved in FireFox.

    But in IE, the page freezes altogether.

    Any suggestions ?

    br Esko

    • I don’t know what to say about IE. It’s strings are null terminated. The zip is filled with null chars, so in IE there’s a hack in place to convert the data to some VBscript array and parsed that way. The conversion alone takes some time on IE’s slow Javascript engine, then the actual decompression must be made. I don’t think it will ever get any faster in IE.

  8. The same code does not work on IE. Error is as following:

    Webpage error details

    User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; InfoPath.1)
    Timestamp: Mon, 14 Mar 2011 11:44:05 UTC

    Message: Permission denied
    Line: 523
    Char: 13
    Code: 0
    URI: http://dg26/test/js/archive/archive.js

  9. Hello,

    First off, thank you for this great functionality….I do have a question though…my co’s website is in ASP.net with a content management system interface built on top for any development….this means that any files we want to use have to be uploaded to the system, and are then assigned an ID to access from the website

    ie. “ABC.com/files.zip” would become “ABC.com/resource.aspx?IDX=123″

    so my dilemma is that the “files.zip://jquery.js” is not applicable anymore because I have to change the path for the zip file to be “resource.aspx?IDX=123″…and “resource.aspx?IDX=123://jquery.zip” does not work to access the contents of the zip file

    any thoughts/ideas??

    Thanks,
    Vin

    • you can add another case statement in the load function and test for a new pattern. the functionality should be exactly like the one for ZIP_ENTRY_REG, but instead of ZIP_ENTRY_REG regular expression write your own that matches the resource.aspx?IDX=123://jquery.zip kind of URL’s.

      • Awesome! I changed the regular expression to match the aspx?IDX URL I send it…I have it working in Firefox and Chrome…any luck getting it to work in IE??

        Thanks,
        Vin

  10. Is there an easy way to grab all the files in the archive if you don’t know what they are? This is an awesome piece of work and I was thinking of using it to create a sequential comic book reader, but I would have no way of knowing the number of images in the archive, or what they are named. Thoughts?

    • yes you can. You have to add one more method to the return object of the ZipLoader (where you have the loadImage, loadCSS, loadScript methods) and have it like this:

      getEntries : function(zipName) {
      return ZIP_CACHE[zipName] && ZIP_CACHE[zipName].entries() || [];
      }

      you have to specify the loaded zip name because it is cached by name and you can have multiple zips loaded by a ziploader

      The way to list all the names after adding that method will be something like this

      var loader = new ZipLoader(‘files.zip’);
      loader.getEntries(‘files.zip’).forEach(function(entry) {
      console.log(“Name: “, entry.name(), ” Size: “, entry.size, ” is Directory: “, entry.isDirectory());
      });

      the entry is a ZipEntry object which you can see in the archive.js what methods and properties it exposes if you need to know more.

      Cheers.

  11. Is there an easy way use that to zip content on the client side and send it’s data to the server (I don’t care if it will just send a string that represents the compressed file)

  12. Nice work! Reading through a zip file does not look trival. A very cool idea.
    Regarding performance.. have you tested the zip loader versus a browser with a primed browser cache? The 79 js/css files would be read from cache, versus being unbundled from the cached zip.
    Also, it would be good to test against what many large sites, and CMSs, do – bundle the js/css separately and have the server automagically gzip them. The 79 js/css files would be reduced to 2, and sent over gzipped.
    Having more performance data would help illuminate the benefits. I bet that a page with 10+ assets, and a cleared browser cache, will see the biggest gain. Thank you for sharing your work!

    • indeed you can merge 50 js/css files and gzip them. you will gain amazing size benefits and cut the requests down to 2, but the downside is that it will have to wait until everything is evaluated by the browser. and that is expensive, both for css and js. you might not need 90% of the code/style in some parts of the application. having them all downloaded and ready to go for you to load whenever you need a certain class or style would be much nicer. this project was a one day fedex project and it can be greatly improved to gain some decoding speed and then you won’t have a problem waiting for the unpacking anymore. i do have few feature improvements ideas but it seems i never get the time to do it.

      oh, and one more nice trick regarding the the css/js merging on the server and gziping them. you can parse the css, look for whatever images are included in there as backgrounds and such and embed them in the css as base64 strings. the gzip will make them the same size they actually are so you don’t save up traffic but you save a lot of requests and when the css arrives, everything does. and if the css is cached correctly then all is golden :)

      cheers

  13. We are writing a data analysis package that produces multiple png images and we use an html page for viewing the data. Since there are potentially thousands of png files produced, we wanted to be able to zip them and load them into the html page from the zip file, so your script seemed perfect. I tried testing it with one of our png files, but it won’t display. Is there a size limit to the image that can be loaded with this script?

    • There are size limitations for inline images. Browsers are only required to support URLs up to 1,024 bytes in length, according to the above RFC. Browsers are more liberal in what they’ll accept, however. Opera limits data URLs to about 4,100 characters. Firefox supports data URLs up to 100K, so this technique is best used for small, decorative images.

      • The images we have range from roughly 3kB to 120kB in size, but now I’m thinking it may have something to do with the png format. I converted the images to jpeg and they seem to work ok, even one 112kB in size. I noticed that your example png file was created with photoshop and is shown as a png-2 and ours shows up as a png-1 so the revision might have something to do with it, but I don’t know what library the analysis package is using to create the images. The file sizes for the pngs and jpegs are nearly identical, so that doesn’t seem like it would be the problem.

  14. Another question, if I may. I was able to load an array of all the image names using your suggested loader.getentries function above. What I’d like to do is pass the array elements for each of the images to the loader.loadImage function from within the html img tag to load each of the images.
    like:

    Beginner that I am, I can’t divine how to do that from your function($) example.

    • For some reason the example tag didn’t display. here’s another try.

      [code]

      [/code]

    • loadImage returns a base64 encode of the image preceded by the image mime-type. if the getEntries returns an array of filenames, you can create image tags at runtime and fill their src attribute with.

      var entries = ziploader.getEntries(), div = $('<div></div>');
      for (var i = 0, len = entries.length; i < len; i++) {
         div.append($('<img src="' + ziploader.loadImage(entries[i]) + '" />'));
      }
      $(document).append(div);

      something like that. i can help more you but i need to see what exactly you need to do to provide you with some valuable code.
      admin recently posted..Steve Wozniak wrote BASIC for the Apple computer in binaryMy Profile

      • Thanks for all your help and patience. I’ll try posting some code again.

         
         
        	   Three Band Thermal Image
         
         
         
                     ROI Pixels
         
         
         
                     F - Library Spectrum
         
         
         
                     F - In-scene Spectrum

        The above excerpt is repeated multiple times in the page depending in the number of ROIs detected in the data,

        I used the following to load the images:

                var pics = new Array();
                var i=0;
         
                loader.getEntries('images.zip').forEach(function(entry) {
                    if (!(entry.isDirectory()) &amp;&amp; (getFileExtension(entry.name())=='jpg')) {
                        pics[i] = entry.name();
                        i++;
                    }
                });
                pics.sort();
         
                var imgarr = new Array();
        	for (i=0;i&lt;55;i++) {
                    imgarr[i] = &quot;#tmp&quot;+i.toString();
         	}
         
                (function($) {
                    $(document).ready(function() {
                        for (i=0;i&lt;55;i++) {
                            $(imgarr[i]).attr(&#039;src&#039;, loader.loadImage(&#039;images.zip://&#039; + pics[i]));
                        }
                    });
                })(jQuery)

        This works in part. Some images load, some don’t. Some of the larger ones do while smaller ones don’t, which confuses me. My lack of experience with javascript, html, and css has definitely been a stumbling block, but I’m slowly learning.

        • Sorry. The html code still didn’t display properly. a preview option would be nice :-)

           

          Anyway, there are multiple “img” tags to display data formatted by a style sheet. The test data I’m working with has 55 images.

  15. Thanks very much for making this available, its helped tremendously. I have a couple questions towards it though. I’m loading web pages from a zipped archive, and then need to be able to show the page and its styles/images. I can alter the image tags to use the loadImage method, but how would I bring in the style sheets /js header files as well?

  16. I’ve been working to adapt this for use in an embedded Webbrowser control.

    Security requirements have forced me to adapt the zip loading.

    You’ll notice you BinToArray VB chunk – to which I’ve added

    Function BinaryToArray(Binary)
    Dim i
    ReDim byteArray(LenB(Binary))
    For i = 1 To LenB(Binary)
    byteArray(i-1) = AscB(MidB(Binary, i, 1))
    Next
    BinaryToArray = byteArray
    End Function

    Function ReadBinFile(path)
    Dim Http
    Set Http = CreateObject(“MSXML2.XMLHTTP.6.0″)
    Http.Open “GET”, path, False
    Http.SetRequestHeader “Accept-Charset”, “x-user-defined”
    Http.Send
    BinaryGetURL = Http.ResponseBody
    ReadBinFile = BinaryToArray(BinaryGetURL)
    End Function

    This allows me to get past the security restrictions in browser. However, the rest of your library doesn’t appear to like the binary that’s getting returned. It’s kicking out invalidZip.

    Any thoughts

    • first try to inspect the length of the binary array. see if it converted everything or it stopped along the way for some reason.
      the invalid zip error is given by findEnd function that tries to identify where the zip header ends. does your zip file contains comments ?
      Is this invalid zip exception for some specific zip file you’re trying to open or nothing works (including the zip from my demo) ?
      admin recently posted..Pixel ProgrammingMy Profile

  17. Hi. Very usefull code! But I have a problem with utf8 encoding js files. firebug tellme:
    illegal character: 
    thanks

  18. Hi! i’ve tried to load a big text file (around 50MB+) onto an array, but the browser keeps loading. Any solution to that?

    Thanks!

  19. First, thanks for this great work you’ve done. Do you have a license you want this distributed with, or is it public domain? Also, on many PNG files I get errors while trying to extract. In firebug I get “stored block length did not match one’s complement”. any ideas?

  20. I am planning to use this work in a open source ePub reader and I think I need to extend this by adding features like updating a particular file in zip. Have you done something similar ? Would you like to direct me about how should I add these features ?

  21. hi, I’m trying to create a mobile application using phonegap, in my module I’m reading some data from a Zip folder.
    the problem is: I’m not allowed to use large memory, and the loader cashed all the zip folder each time i read anything from it, so can I clear the loader each time i read something ?

  22. you can just delete the javascript object and the GC should clear up the memory or you can modify the code to suit your needs. I think the second option is the better one. I’ve made another improved inflater that you can find here (https://github.com/cthackers/adm-zip/blob/master/methods/inflater.js)
    admin recently posted..Setting up a IDE to play with GO on WindowsMy Profile

    • the solution i found,
      There is a specific size for a file to be readable in the mobile app using this library,
      the file i was trying to read is 25kb. I reduced the file size to 10kb and it works, and each time i finish the reading, i turned the variable to null status then deleted it.

      var zippedPages = loader.load(pathZippedBook);
      {function….}
      zippedPages = null;
      delete zippedPages;

  23. Is it also possible to read mp3 file stored on a zip file (as for an image) ?

  24. marvelous work..
    Is there any way to extract a epub file using this ??? Plz Help

    • Gaurav, in a comment up above did so. So i guess so, I don’t really know anything about epub formats but if they are zipped things inside it, then yes. you can extract epub files

  25. can i read epub format by this?

  26. Hi, unable to get the demo to work in either safari or chrome! works however in dreamweaver in live view.

    am I’m missing something obvious here or just being a bit dum?

  27. hi.
    my name is cj and i have a idea.
    i need a portable database to work on a simple html or hta.
    no sql server need
    kind of project is for a web host without php or sql support and for working on local file

    (ex:)

    [database.zip]{
    tbl_clients.csv
    tbl_products.csv
    tbl_costumers.csv
    }
    about security? zip password

    hope you can work on idea
    cheers!

  28. Hi,

    I want to load my zip file by a “http” address and read each files (images, text, …)
    I don’t have any problems with “cross domain”.
    Can I have an little example for how i can do that ?

    Thanks !

    • If you hadn’t figure this out already:
      loader.load(“http://www.somedomain.com/somefile.zip://some_file_inside.txt”);
      There are plenty of comments above that have examples in them.

      Cheers

  29. I don’t want get the inline image but the binary image.
    Because I want to save the image (phonegap).

    How can I have that ?

    Thx

  30. I think the image file have missing data to be showed !

  31. I observed several things :
    – The size of image file from the zip is greater than the original file
    – When I open my image file from the zip into the editor “notepad++” and i compare with the original file i can see character have been added or modify.

    I think the load() method doesn’t decode very well image files from a zip.

    Can you help me ?

    My zip file : http://dl.dropbox.com/u/65515222/update.zip
    Please It’s urgent !

  32. Hello,

    i use your code sample and it works like a charm, after that i implement it in my test project but it will not give proper image(while it gives proper html), do i have to some special setting while zipping my files, you can get my zip here http://69.195.38.148/vatshal/course_app/course6.zip , please help.

    thanks for this wonderful library

  33. Thank you so very much for the above solution…
    I just wanted to know if the very same approach would work for loadting file.epub instead of
    a file.zip format only…

    thing is that I want to load the epub on my page whenever the user clicks a link.

  34. Hello All,

    Anyone tried to male it work in a phonegap environment ?

    Kind regards

  35. Hello,

    Any answer on how to read/extract MP3 ?
    (question was already asked before, but no response)

    Regards

    • You should not archive an mp3. They don’t compress at all and the time and overhead of extracting them will be huge.

      • Our replies just crossed.

        I my particular case, I have small MP3s that I want to package.
        I don’t need the “power” of ZIP compression (even if I save around 20% of the size), but it’s easier for me to deal with only 1 file according to project specifications.

        Thanks for your quick answer

        Regards

    • Pfff, I’m such a lazy guy…:p I’ll answer my own question again :D

      Thanks to the work done before by the author, I was able to edit the source code (archive.js) and add those lines on line 660 :

      loadSound : function(url) {
      var data = this.load(url);
      if (data) {
      var tmp = “data:”;
      switch(getFileExtension(url).toLowerCase()) {
      case ‘mp3′ :
      tmp += “audio/mpeg;base64,”;
      break;
      case ‘ogg’ :
      tmp += “audio/ogg;base64,”;
      break;
      }
      tmp += Base64(data);
      }
      return tmp;
      },

      Now it can read MP3 and OGG files by using the new “loadSound” function.
      Finally, was very easy :D

      So if you’re using jPlayer or something like that, you can just use this code :

      var loader = new ZipLoader(‘sound.zip’);
      (…)
      $(“#jquery_jplayer_1″).jPlayer(“setMedia”, {
      mp3: loader.loadSound(“sound.zip://mysound.mp3″)
      }).jPlayer(“play”);

      In theory, you can do the same for any file type as long as you format the string according to mime type and encoding.

      Congratulations again to the author !

      • Great job. I’ll add it to the code when i get home. Thanks

        • If you want to do something very generic you should add a function that takes 1, even 2 more parameters, the first would be the “mime-type” and the second the “encoding-string”.

          So you would have something like :
          loadFile(“myZipFile.zip://myfile.xxx”, mime-type, encoding);

          Ex :
          loader.loadFile(“myZipFile.zip://mysound.mp3″, “audio/mpeg”, “base64″)
          loader.loadFile(“myZipFile.zip://mysound.ogg″, “audio/ogg”, “base64″)

          For power users only ;)

          After if you want to help users so they don’t have to google their mime type, you can add some matching table with “File Extension VS mime type”.

          Finally, you would have 4 functions : load, loadFile, loadCSS, loadScript, the library stays light but respond to more cases.

          (loadImage is out as it can be done with loadFile – just an idea, maybe you want to keep this for backward compatibility or just to keep it simple for users)

          After if people wants to ZIP anything anyhow, they should know as you said earlier than it’s not always a good thing, especially media files.

  36. I see it has been some time since the last activity here. I hope all is well.
    It was a pleasure implementing your solution. Thanks for that!!

    With that said, I came into a performance issue.
    The following examples loads with FF in about one second but with Chrome it takes more than a minute(!).
    http://www.netone.co.il/Sinapis-Admination/zip

    Can any one shed some light on this issue?

  37. This only works in HTTP protocol. I need to use in file//: protocol. How i do this?

  38. is there possible to load a zip thorugh file upload, read and display the image on the same page?


Leave a comment


*

CommentLuv badge