case
1Oct/1049

Read/Load files from ZIP in JavaScript

At work we do a lot of JavaScript applications that have little to no HTML files (beside the index), everything is generated by JavaScript and we have tons of .js files that load at runtime and to reduce the size we have to minify them and that is really hard to debug and watch in the browser if something goes wrong. So I've made this JavaScript class that comes to help. What can you do is Zip up all your assets (js, images, css) and use ZipLoader class to read the zip and unpack your resources whenever you need them. It's small (10KB), easy to use and quite fast considering the extraction is made by JavaScript.

1
2
3
4
5
6
7
8
9
10

   // loads and caches the zip file
   var loader = new ZipLoader('files.zip');
   // creates a style node in the document header and appends the style
   loader.LoadCSS('files.zip://style.css');
   // creates a script node in the document header and appends the script
   loader.loadScript('files.zip://jquery.js');
   // returns the file content
   var someFileYouNeed = loader.load("files.zip://myFile.txt");
   // returns the base64 encoded image usable as img source
   $("#logo").attr('src', loader.loadImage('files.zip://images/logo.png'));

The level of compression using zip surpasses the js and css minification in all the ways possible. For example, My current application has 79 files (js, and few css) and they all sum up to 924KB in size. Zipping all those 96 files using Ultra compression level gives me 1 file of 126 KB => 1 server request. That's something.

The best performance is made Opera (strangely), then Firefox, Crome, Safari and last IE (no surprise here, I was expecting it before I even started).

It's a V 1.0 that still needs some work and it doesn't work in IE because that stupid browser overrides the mime type and at first null character in the reading of the zip, the content ends. I will fix this eventually icon smile Read/Load files from ZIP in JavaScript I will also include this as a jClass library soon.

Update: Fixed it so it works in IE as well. There is a performance issue still. Will fix that too.

Till then, HERE are the sources and a demo.

The zip in the demo is made using 7Zip with Ultra Compression level.

Happy codding.



Related posts:

  1. Open and Save files to Desktop without going to Server
  2. Read data directly from base64 without decoding
  3. JavaScript Object-Oriented Programming (OOP) – jClass
  4. For Javascript devs
  5. Static files locked by Jetty in Eclipse

Enjoy this article?

Consider subscribing to our rss feed!

Tagged as: , , , , , , , , , Leave a comment
Comments (49) Trackbacks (1)
  1. toc
    November 1st, 2010 - 10:34

    Hello, this is realy a great work!!!

    I love Greasemonkey, so my question – can you make an smaller Version of it, to simple zip and unzip strings for storing them with GM or with the new BrowserStorageObjekt?

    thx

    ( REPLY )
    • admin
      November 3rd, 2010 - 22:08

      I’ve made few compression algorithms in my jClass project. You can just download the library and copy paste code from there in another class. They are written in jClass way but they all have 2 encode/decode functions so jClass is not necessarily

      ( REPLY )
  2. eskomk
    December 1st, 2010 - 18:05

    Hi,

    Can it load zip file over http ?

    I need to load a file from server as fast as possible, and zipping it first would be a good solution.
    Then I would just use this library to unzip and use its data, in javascript.

    Best regards,
    Esko

    ( REPLY )
  3. eskomk
    December 2nd, 2010 - 11:14

    Hi,

    Could you please advice, Firebug reports this error when trying to use the component:

    CLIP–>
    uncaught exception: [Exception... "Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive.js :: loadBinaryResource :: line 507″ data: no]

    Line 0

    uncaught exception: [Exception... "Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive-min.js :: loadBinaryResource :: line 1″ data: no]

    Line 0
    <–CLIP

    please help:)

    br,
    Esko

    ( REPLY )
  4. eskomk
    December 2nd, 2010 - 11:31

    Sorry, the comment was imperfect, propably due to CLIPpings having < and >s. It should have been like this:

    Could you please advice, Firebug reports this error when trying to use the component:

    CLIP–*
    uncaught exception: [Exception... "Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive.js :: loadBinaryResource :: line 507″ data: no]

    Line 0
    *–CLIP

    The above comes when I have included both scriptfiles, like so:

    CLIP–*
    *script type=”text/javascript” src=”/suppfiles/archive-min.js”**/script*
    *script type=”text/javascript”*
    *–CLIP

    This below comes when the “archive.js” was not included:

    CLIP–*
    uncaught exception: [Exception... "Component returned failure code: 0x805e000a [nsIXMLHttpRequest.open]” nsresult: “0x805e000a ()” location: “JS frame :: http://metsaenergia.test.ebsolut.fi/suppfiles/archive-min.js :: loadBinaryResource :: line 1″ data: no]

    Line 0
    *–CLIP

    please help:)

    br,
    Esko

    ( REPLY )
  5. eskomk
    December 2nd, 2010 - 11:39

    Hi,

    I added an exception in popup blocker for metsanergia.test.ebsolut.fi, but bthe error comes still

    br Esko

    ( REPLY )
  6. eskomk
    December 2nd, 2010 - 13:38

    Hi,

    Problem solved!

    I debugged the archive.js file.
    There is a function loadBinaryResource(url), which has a line

    req.open(‘GET’, url, false);

    Function gets called two times, first to open the zipfile.
    Firstly, the parameter url was
    ‘http://metsaenergia.test.ebsolut.fi/smavimap/liitynta.zip‘
    for the zip file.

    At second time, it is called to get the file inside the zip.
    Here I had, erroneously, the parameter url being
    tdelem.innerHTML = loader2.load(‘liitynta.zip://liitynta.xml’);
    and it resulted to “Access denied” error.

    After I changed that to
    tdelem.innerHTML = loader2.load(‘http://metsaenergia.test.ebsolut.fi/smavimap/liitynta.zip://liitynta.xml‘);
    it worked OK.

    br Esko

    ( REPLY )
    • The Brain
      April 7th, 2011 - 10:37

      Yes, it caches the zip by name internally so next time you want something from it it will check if it needs to load the zip or it’s there already.

      ( REPLY )
  7. eskomk
    December 2nd, 2010 - 14:00

    Hi again,

    The problem was solved in FireFox.

    But in IE, the page freezes altogether.

    Any suggestions ?

    br Esko

    ( REPLY )
    • admin
      April 7th, 2011 - 10:43

      I don’t know what to say about IE. It’s strings are null terminated. The zip is filled with null chars, so in IE there’s a hack in place to convert the data to some VBscript array and parsed that way. The conversion alone takes some time on IE’s slow Javascript engine, then the actual decompression must be made. I don’t think it will ever get any faster in IE.

      ( REPLY )
  8. Ved
    March 14th, 2011 - 13:46

    The same code does not work on IE. Error is as following:

    Webpage error details

    User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; InfoPath.1)
    Timestamp: Mon, 14 Mar 2011 11:44:05 UTC

    Message: Permission denied
    Line: 523
    Char: 13
    Code: 0
    URI: http://dg26/test/js/archive/archive.js

    ( REPLY )
  9. Vin
    March 30th, 2011 - 20:43

    Hello,

    First off, thank you for this great functionality….I do have a question though…my co’s website is in ASP.net with a content management system interface built on top for any development….this means that any files we want to use have to be uploaded to the system, and are then assigned an ID to access from the website

    ie. “ABC.com/files.zip” would become “ABC.com/resource.aspx?IDX=123″

    so my dilemma is that the “files.zip://jquery.js” is not applicable anymore because I have to change the path for the zip file to be “resource.aspx?IDX=123″…and “resource.aspx?IDX=123://jquery.zip” does not work to access the contents of the zip file

    any thoughts/ideas??

    Thanks,
    Vin

    ( REPLY )
    • admin
      March 31st, 2011 - 09:39

      you can add another case statement in the load function and test for a new pattern. the functionality should be exactly like the one for ZIP_ENTRY_REG, but instead of ZIP_ENTRY_REG regular expression write your own that matches the resource.aspx?IDX=123://jquery.zip kind of URL’s.

      ( REPLY )
      • Vin
        April 4th, 2011 - 16:11

        Awesome! I changed the regular expression to match the aspx?IDX URL I send it…I have it working in Firefox and Chrome…any luck getting it to work in IE??

        Thanks,
        Vin

        ( REPLY )
  10. Bhavya Kashyap
    April 6th, 2011 - 20:55

    Is there an easy way to grab all the files in the archive if you don’t know what they are? This is an awesome piece of work and I was thinking of using it to create a sequential comic book reader, but I would have no way of knowing the number of images in the archive, or what they are named. Thoughts?

    ( REPLY )
    • admin
      April 6th, 2011 - 21:20

      yes you can. You have to add one more method to the return object of the ZipLoader (where you have the loadImage, loadCSS, loadScript methods) and have it like this:

      getEntries : function(zipName) {
      return ZIP_CACHE[zipName] && ZIP_CACHE[zipName].entries() || [];
      }

      you have to specify the loaded zip name because it is cached by name and you can have multiple zips loaded by a ziploader

      The way to list all the names after adding that method will be something like this

      var loader = new ZipLoader(‘files.zip’);
      loader.getEntries(‘files.zip’).forEach(function(entry) {
      console.log(“Name: “, entry.name(), ” Size: “, entry.size, ” is Directory: “, entry.isDirectory());
      });

      the entry is a ZipEntry object which you can see in the archive.js what methods and properties it exposes if you need to know more.

      Cheers.

      ( REPLY )
  11. Gil
    May 9th, 2011 - 11:03

    Is there an easy way use that to zip content on the client side and send it’s data to the server (I don’t care if it will just send a string that represents the compressed file)

    ( REPLY )
  12. Gary
    May 11th, 2011 - 21:22

    Nice work! Reading through a zip file does not look trival. A very cool idea.
    Regarding performance.. have you tested the zip loader versus a browser with a primed browser cache? The 79 js/css files would be read from cache, versus being unbundled from the cached zip.
    Also, it would be good to test against what many large sites, and CMSs, do – bundle the js/css separately and have the server automagically gzip them. The 79 js/css files would be reduced to 2, and sent over gzipped.
    Having more performance data would help illuminate the benefits. I bet that a page with 10+ assets, and a cleared browser cache, will see the biggest gain. Thank you for sharing your work!

    ( REPLY )
    • admin
      May 12th, 2011 - 09:03

      indeed you can merge 50 js/css files and gzip them. you will gain amazing size benefits and cut the requests down to 2, but the downside is that it will have to wait until everything is evaluated by the browser. and that is expensive, both for css and js. you might not need 90% of the code/style in some parts of the application. having them all downloaded and ready to go for you to load whenever you need a certain class or style would be much nicer. this project was a one day fedex project and it can be greatly improved to gain some decoding speed and then you won’t have a problem waiting for the unpacking anymore. i do have few feature improvements ideas but it seems i never get the time to do it.

      oh, and one more nice trick regarding the the css/js merging on the server and gziping them. you can parse the css, look for whatever images are included in there as backgrounds and such and embed them in the css as base64 strings. the gzip will make them the same size they actually are so you don’t save up traffic but you save a lot of requests and when the css arrives, everything does. and if the css is cached correctly then all is golden :)

      cheers

      ( REPLY )
  13. Thomas Gamble
    June 13th, 2011 - 14:10

    We are writing a data analysis package that produces multiple png images and we use an html page for viewing the data. Since there are potentially thousands of png files produced, we wanted to be able to zip them and load them into the html page from the zip file, so your script seemed perfect. I tried testing it with one of our png files, but it won’t display. Is there a size limit to the image that can be loaded with this script?

    ( REPLY )
    • admin
      June 13th, 2011 - 22:33

      There are size limitations for inline images. Browsers are only required to support URLs up to 1,024 bytes in length, according to the above RFC. Browsers are more liberal in what they’ll accept, however. Opera limits data URLs to about 4,100 characters. Firefox supports data URLs up to 100K, so this technique is best used for small, decorative images.

      ( REPLY )
      • Thomas Gamble
        June 14th, 2011 - 17:18

        The images we have range from roughly 3kB to 120kB in size, but now I’m thinking it may have something to do with the png format. I converted the images to jpeg and they seem to work ok, even one 112kB in size. I noticed that your example png file was created with photoshop and is shown as a png-2 and ours shows up as a png-1 so the revision might have something to do with it, but I don’t know what library the analysis package is using to create the images. The file sizes for the pngs and jpegs are nearly identical, so that doesn’t seem like it would be the problem.

        ( REPLY )
  14. Thomas Gamble
    June 23rd, 2011 - 16:31

    Another question, if I may. I was able to load an array of all the image names using your suggested loader.getentries function above. What I’d like to do is pass the array elements for each of the images to the loader.loadImage function from within the html img tag to load each of the images.
    like:

    Beginner that I am, I can’t divine how to do that from your function($) example.

    ( REPLY )
    • Thomas Gamble
      June 23rd, 2011 - 16:34

      For some reason the example tag didn’t display. here’s another try.

      [code]

      [/code]

      ( REPLY )
    • admin
      June 24th, 2011 - 08:38

      loadImage returns a base64 encode of the image preceded by the image mime-type. if the getEntries returns an array of filenames, you can create image tags at runtime and fill their src attribute with.

      var entries = ziploader.getEntries(), div = $('<div></div>');
for (var i = 0, len = entries.length; i < len; i++) {
   div.append($('<img src="' + ziploader.loadImage(entries[i]) + '" />'));
}
$(document).append(div);

      something like that. i can help more you but i need to see what exactly you need to do to provide you with some valuable code.
      admin recently posted..Steve Wozniak wrote BASIC for the Apple computer in binaryMy Profile

      ( REPLY )
      • Thomas Gamble
        June 27th, 2011 - 15:40

        Thanks for all your help and patience. I’ll try posting some code again.

         
 
	   Three Band Thermal Image
 
 
 
             ROI Pixels
 
 
 
             F - Library Spectrum
 
 
 
             F - In-scene Spectrum

        The above excerpt is repeated multiple times in the page depending in the number of ROIs detected in the data,

        I used the following to load the images:

                var pics = new Array();
        var i=0;
 
        loader.getEntries('images.zip').forEach(function(entry) {
            if (!(entry.isDirectory()) &amp;&amp; (getFileExtension(entry.name())=='jpg')) {
                pics[i] = entry.name();
                i++;
            }
        });
        pics.sort();
 
        var imgarr = new Array();
	for (i=0;i&lt;55;i++) {
            imgarr[i] = &quot;#tmp&quot;+i.toString();
 	}
 
        (function($) {
            $(document).ready(function() {
                for (i=0;i&lt;55;i++) {
                    $(imgarr[i]).attr(&#039;src&#039;, loader.loadImage(&#039;images.zip://&#039; + pics[i]));
                }
            });
        })(jQuery)

        This works in part. Some images load, some don’t. Some of the larger ones do while smaller ones don’t, which confuses me. My lack of experience with javascript, html, and css has definitely been a stumbling block, but I’m slowly learning.

        ( REPLY )
        • Thomas Gamble
          June 27th, 2011 - 15:50

          Sorry. The html code still didn’t display properly. a preview option would be nice :-)

           

          Anyway, there are multiple “img” tags to display data formatted by a style sheet. The test data I’m working with has 55 images.

          ( REPLY )
  15. rothtechs
    June 24th, 2011 - 04:10

    Thanks very much for making this available, its helped tremendously. I have a couple questions towards it though. I’m loading web pages from a zipped archive, and then need to be able to show the page and its styles/images. I can alter the image tags to use the loadImage method, but how would I bring in the style sheets /js header files as well?

    ( REPLY )
  17. dermidgen
    September 2nd, 2011 - 01:16

    I’ve been working to adapt this for use in an embedded Webbrowser control.

    Security requirements have forced me to adapt the zip loading.

    You’ll notice you BinToArray VB chunk – to which I’ve added

    Function BinaryToArray(Binary)
    Dim i
    ReDim byteArray(LenB(Binary))
    For i = 1 To LenB(Binary)
    byteArray(i-1) = AscB(MidB(Binary, i, 1))
    Next
    BinaryToArray = byteArray
    End Function

    Function ReadBinFile(path)
    Dim Http
    Set Http = CreateObject(“MSXML2.XMLHTTP.6.0″)
    Http.Open “GET”, path, False
    Http.SetRequestHeader “Accept-Charset”, “x-user-defined”
    Http.Send
    BinaryGetURL = Http.ResponseBody
    ReadBinFile = BinaryToArray(BinaryGetURL)
    End Function

    This allows me to get past the security restrictions in browser. However, the rest of your library doesn’t appear to like the binary that’s getting returned. It’s kicking out invalidZip.

    Any thoughts

    ( REPLY )
    • admin
      September 2nd, 2011 - 08:56

      first try to inspect the length of the binary array. see if it converted everything or it stopped along the way for some reason.
      the invalid zip error is given by findEnd function that tries to identify where the zip header ends. does your zip file contains comments ?
      Is this invalid zip exception for some specific zip file you’re trying to open or nothing works (including the zip from my demo) ?
      admin recently posted..Pixel ProgrammingMy Profile

      ( REPLY )
  18. Alexey
    September 14th, 2011 - 13:14

    Hi. Very usefull code! But I have a problem with utf8 encoding js files. firebug tellme:
    illegal character: 
    thanks

    ( REPLY )
  20. Jack
    September 20th, 2011 - 13:32

    Hi! i’ve tried to load a big text file (around 50MB+) onto an array, but the browser keeps loading. Any solution to that?

    Thanks!

    ( REPLY )

Leave a comment


CAPTCHA Image
*

CommentLuv badge
Read data directly from base64 without decoding « ADM Blog
« Gnome – multiple monitor taskbar Parse XML in JavaScript with jQuery »

Categories

Help a fellow developer

Tags

actionscript actionscript3 as3 browser C# chrome class code components CSS debug download drive eee eeepc email event files firefox flash flex google hack how-to html ie6 iphone javascript jquery library linux opera pc photoshop programmer programmers Python Software story video virtual window windows windows7 xml

Blogroll