Skip to main content

Lua Workshop 2016

· 7 min read
Asumu Takikawa

Now that I work at Igalia on the networking team, I've started to write code in Lua (specfically with LuaJIT) in order to work with Snabb. Since I'm still a Lua newbie, I decided to attend this year's Lua Workshop in San Francisco to learn what's going on in the community.

The workshop was a lot of fun! Kudos to Mashape for hosting the event. The talks were easy to follow even for a Lua newbie like myself. I took notes on all of the talks, but I won't go over all the details since the talks were recorded and you can just see for yourself. Instead, I wanted to go over some of my personal highlights.

The design of Lua​

The day was kicked off by Roberto Ierusalimschy by a talk on the design of Lua. The talk was focused on how tradeoffs are an important part of designing programming languages. In particular, Roberto criticized the widespread "general-purpose" label for programming languages, since no PL is truly general-purpose. For example, a high-level language chock full of abstractions may have unavoidable performance overheads. A language that comes with the kitchen sink is much harder to port to tiny microcontrollers.



Given that, Lua makes explicit tradeoffs to excel along four particular axes: portability, simplicity, small, and scripting. These design constraints make Lua especially suitable for applications such as embedding in TVs and video game engines. Apparently Lua is even embedded in the NetBSD kernel!

This led to an interesting discussion of the presence and lack of various features in today's Lua. In particular, Roberto expressed some regret about adding the "three dots" varargs feature and also on the addition of goto in Lua 5.2.

As a Lisper, I was also really excited to hear that the Lua devs have been "dreaming" about adding macros into the language. I'm hoping that if they do add macros, they look at Matthew Flatt's hygienic macros with sets of scopes that have subsequently been adopted into sweet.js. Although I imagine it will be difficult to scale it into a full-on phased macro system ala Racket because Lua's modules live entirely at runtime (i.e., require is implemented as a function).

Lua history​

Another fun talk on Lua proper was Daurnimator's talk on the development history of Lua. The biggest shock for me from this talk was that Lua is still developed using RCS! Not even CVS or Subversion. It's also a private repository that isn't (yet) hosted for the public to see.

Of course, that makes it hard for outside people to contribute or even just learn from the code base. That's where this talk comes in. Daurnimator has put in the work to port the Lua repo from RCS to git (via CVS import tools---apparently RCS and CVS share repo formats) in order to set up a public mirror.

My understanding is that the plan is to put up a mirror at https://github.com/lua/ that will host the full history. There was some murmuring at the workshop that Lua may switch over to git-based development in the future, though it sounded like the version on github would remain as just a mirror.

You might be wondering where the git history starts, since apparently the initial versions of Lua weren't even developed in source control. It turns out that the initial commit is recreated from a tarball that was found of one of the early releases from around 1993.

Finally, a fun fact from the talk is that despite the Lua developers' insistence that "Lua" isn't an acronym, there's an early commit message that included the following: "Linguagem para Usuarios de Aplicacao".

Applications of Lua​

There were a number of libraries and applications of Lua that were presented at the workshop. One of the projects that I'd be most likely to use and most excited about was the Rosie Pattern Language presented by Jamie Jennings. The slides for the talk are here.

The basic motivation for the tool is that there are many DevOps or Data Science style tasks that require munging data from some raw data format, such as system logs. Many hackers just reach for regular expressions for this kind of task, but regexps often become software engineering burden because they're hard to understand and abstract. Not only that, many regexp engines are slow.

Rosie offers an alternative approach formulated as a language built over parser combinators. It comes with a big library of pre-defined patterns that make it easier to plug together existing solutions. There's also a REPL for interactively trying patterns too. The connection with Lua is that Lua is the implementation language and also the extension scripting language for Rosie.

Another cool talk was Justin Donaldson's talk on creating a HaXe backend that emits Lua code. HaXe is a cross-platform language that compiles to a number of different target languages. It's intended for use in areas like video games (Papers, Please! was written in it) and mobile apps. It seems like quite a neat language based on its list of features.

According to Justin there were some interesting challenges in constructing a Lua backend. Apparently accommodating Lua's multiple value return facility was tricky, as well as dealing with the fact that the Lua VM has a hard limit of 200 local variables per function due to the maximum stack size.

Optimizing for LuaJIT​

Yichun Zhang gave a talk on optimizing LuaJIT applications. One of the main takeaways for me from the talk was "use Flame Graphs for everything". Brendan Gregg's Flame Graphs are a visualization technique for stack traces, or generally any kind of cost that can be attributed in a stack-like manner. Examples include CPU profiling or I/O.

The I/O one was particularly interesting. Normally, profilers track the time that processes spend executing code. But in the case of I/O, you're interested in off-CPU time in which processes are just waiting. Visualizing that in a flame graph gives you an idea of what processes are creating I/O bottlenecks in your application.

There were also some interesting points about feature uses that often come up in these flame graphs too. For example, creating functions in hot paths is obviously bad for performance, and it does pop up as lj_BC_FNEW calls in the graphs (see #50 in the slides linked in the tweet above). Another example as to avoid table creation and resizing if possible, such as by using table.clear to reuse an already allocated table. It was a fast-paced talk and I didn't catch all the details, so I recommend you watch the recording whenever it comes out.

Putting on my Racket developer hat for a second, this talk made me wonder if it would be feasible to create flame graphs for Racket profiler outputs. I've made some progress on that, as this tweet shows:

https://twitter.com/asumu/status/787340486117761024

I plan to release the tool for hooking Racket up to flame graphs soon.

Thoughts​

Overall, I thought the Lua Workshop was great. Something that I enjoyed about the workshop is that there was a strong international presence. I met people from Australia, Brazil, China, Italy, Japan, the UK, and probably other places that I don't remember. Unfortunately while there were many countries represented, the diversity in gender among the attendees wasn't very high. That's something that many language communities deal with but I do hope that it gets better in the future.

Anyhow, happy Lua hacking!