base64 encoding January 13, 2014 at 2:13 am

I needed a “natural” Lua base64 encoding routine for a fun project I was working on so I went looking for something in the cloud. I found a “ton” of examples and quickly became disappointed. Man they reeked of poorly performing code.

Part of the cruft was that many of the routines had been written prior to Lua 5.2. In Lua 5.2 the bit32 library was added. This allows some basic bitwise manipulations. My goal was to have the routines in “pure Lua” because I didn’t want to have to build a C library for each of the three platforms that this code will run on. I also wanted to stick with Lua because a good portion of the project is already in Lua.

The best (fastest) routines I found after a search were the first here. (http://lua-users.org/wiki/BaseSixtyFour) Even these just didn’t “feel” right to me. The base64 command line tool from GNU can convert a 800K file in a tiny fraction of a second (0.055s). The fastest natural Lua routines I found came out to about 2.5 seconds (2.586s). This just seemed way too slow to me and would add a significant amount of “user” annoyance with the larger encoding needed.  

The thought of having to have three different io.popen() routines just seemed like a bad idea.

The above mentioned routines also had a side effect of generating 6.5M of intermediary data during the conversion. Yipes! The result output is only 1.1M! The 6.5M is from an  encoded string that is used during the conversion. I thought the bit32 library might come to the rescue.

My first step was to break the routines out and see what was going on. Just replacing the binary string with bit32 wasn’t enough. The best timings I got only improved the performance by about 200ms. Sigh.

Lua is not an optimized language. By this, I mean that the “compiler” doesn’t sift through the code and remove “redundant” temporaries or have good loop optimizations. My feeling was that these issues were major contributing factors.  I was right.

The net result? The same 800K file:

base64 (GNU) base64.lua (E.E.) Lua wiki version
0.055s 1.093s 2.586s

I can’t make natural Lua work as as well as native code, but I think I made an improvement. Not all of the improvement was because of the bit32 library. A general restructuring of how the encoding and decoding works was required.

Interested?

I have published the library on github. (https://github.com/ErnieE5/lua_base64.git)

Ernie

Comments are closed.