As what as you like

  • Adventures in Lua stack overflows

    Hammerspoon is heavily dependent on Lua - it’s the true core of the application, so it’s unavoidable that we have to interact with Lua’s C API in a lot of places. If you’ve never used it before, Lua’s C API is designed to be very simple to integrate with other code, but it also places a fairly high burden on developers to integrate it properly.

    One of the ways that Lua remains simple is by being stack based - when you give Lua a C function and make it available to call from Lua code, you have to conform to a particular way of working. The function arguments supplied by the user will be presented to you on a stack, and when your C code has finished its work, the return values must have been pushed onto the stack. Here’s an example:

    static int someUsefulFunction(lua_State *L) {
        // Fetch our first argument from the stack
        int someNumber = lua_tointeger(L, 1);
    
        // Fetch our second argument from the stack
        char *someString = lua_tostring(L, 2);
    
        /* Do some useful work here */
    
        // Push two return values onto the stack and return 2 so Lua knows how many return values we provided
        lua_pushstring(L, "some result text");
        lua_pushinteger(L, 42);
        return 2;
    }
    

    All simple enough.

    In this scenario of calling from Lua→C, Lua creates a pseudo-stack for you, so while it’s good practice to keep the stack neat and tidy (i.e. remove things from it that you don’t need), it’s not critical because apart from the return values, the rest of the stack is thrown away. That pseudo-stack only has 20 slots by default though, so if you’re pushing a lot of return arguments, or using the stack for other things, you may need to use lua_checkstack() to grow it larger, up to the maximum (2048 slots).

    Where things get more interesting, is when you’re interacting with the Lua stack without having crossed a Lua→C boundary. For example, maybe you’re in a callback function that’s been triggered by some event in your C program, and now you need to call a Lua function that the user gave you earlier. This might look something like this:

    int globalLuaFunction;
    void someCallback(int aValue, char* aString) {
        // Fetch a pointer to the shared Lua state object
        lua_State *L = some_shared_lua_state_provider();
    
        // Push onto the stack, the Lua function previously supplied by the user, from Lua's global registry
        lua_rawgeti(L, LUA_REGISTRYINDEX, globalLuaFunction);
    
        // Push the two arguments for the Lua function
        lua_pushinteger(L, aValue);
        lua_pushstring(L, aString);
    
        // Call the Lua function, telling Lua to expect two arguments
        lua_call(L, 2, 0);
    
        return;
    }
    

    Slightly more complex than the last example, but still manageable. Unfortunately in practice this is a fairly suboptimal implementation of a C→Lua call - storing things in the LUA_REGISTRYINDEX table is fine, but it’s often nicer to use multiple tables for different things. The big problem here though is that lua_call() doesn’t trap errors. If the Lua code raises an exception, Lua will longjmp to a panic handler and abort() your app.

    So, writing this a bit more completely, we get:

    int luaCallbackTable;
    int globalLuaFunctionRef;
    void someCallback(int aValue, char* aString) {
        // Fetch a pointer to the shared Lua state object
        lua_State *L = some_shared_lua_state_provider();
    
        // Push onto the stack, the table we keep callback references in, from Lua's global registry
        lua_rawgeti(L, LUA_REGISTRYINDEX, luaCallbackTable);
    
        // Push onto the stack, from our callback reference table, the Lua function previously supplied by the user
        lua_rawgeti(L, -1, globalLuaFunctionRef);
    
        // Push the two arguments for the Lua function
        lua_pushinteger(L, aValue);
        lua_pushstring(L, aString);
    
        // Protected call to the Lua function, telling Lua to expect two arguments
        lua_pcall(L, 2, 0, 0);
    
        return;
    }
    

    Ok so this is looking better, we have our own table for neatly storing function references and we’ll no longer abort() if the Lua function throws an error.

    However, we now have a problem, we’re leaking at least one item onto Lua’s stack and possibly two. Unlike in the Lua→C case, we are not operating within the safe confines of a pseudo-stack, so anything we leak here will stay permanently on the stack, and at some point that’s likely to cause the stack to overflow.

    Now here is the kicker - stack overflows are really hard to find by default, you don’t typically get a nice error, your program will simply leak stack slots until the stack overflows, far from the place where the leak is happening, then segfault, and your backtraces will have very normal looking Lua API calls in them.

    If we were to handle the stack properly, the above could would actually look like this (and note that we’ve gone from four Lua API calls in the first C→Lua example, to eight here):

    int luaCallbackTable;
    int globalLuaFunctionRef;
    void someCallback(int aValue, char* aString) {
        // Fetch a pointer to the shared Lua state object
        lua_State *L = some_shared_lua_state_provider();
    
        // Find luaCallbackTable in the Lua registry, and push it onto the stack
        lua_rawgeti(L, LUA_REGISTRYINDEX, luaCallbackTable);
    
        // Find globalLuaFunctionRef in luaCallbackTable, and push it onto the stack
        lua_rawgeti(L, -1, globalLuaFunctionRef);
    
        // Remove luaCallbackTable from the stack *THIS WAS LEAKED IN THE ABOVE EXAMPLE*
        lua_remove(L, -2);
    
        // Push the two arguments for the Lua function
        lua_pushinteger(L, aValue);
        lua_pushstring(L, aString);
    
        if (lua_pcall(L, 2, 0, 0) == false) {
            // Fetch the Lua error message from the stack
            char *someError = lua_tostring(L, -1);
            printf("ERROR: %s\n", someError);
    
            // Remove the Lua error message from the stack *THIS WAS LEAKED IN THE ABOVE EXAMPLE*
            lua_pop(L, -1);
        }
    
        return;
    }
    

    Hammerspoon has been having problems like this for the last few months - lots of crash reports that on the surface, look like completely valid code was executing. I have to admit that it took me a lot longer than it should have, to realise that these were Lua stack overflows rather than my initial suspicion (C heap corruption), but we figured it out eventually and have hopefully fixed all of the leaks.

    So, how did we discover that the problem was stack overflows, and how did we discover where all of the leaks were without manually auditing all of the places where we make C→Lua transitions (of which there are over 100). The answer to the first question is very simple, by defining LUA_USE_APICHECK when compiling Lua, it will do a little extra work to verify its consistency. Crucially, this includes calling abort() with a helpful message when the stack overflows. We turned this on for developers in March and then released 0.9.61 with it enabled, in early April. It’s not normally recommended to have the API checker enabled in production because it calls abort(), but we felt that it was important to get more information about the crashes we couldn’t reproduce.

    Within a few days we started getting crash reports with the words stack overflow in them (as well as a few other errors, which we were able to fix), but that is only half the battle.

    Having discovered that we did definitely have a stack leak somewhere, how did we discover where it was? This did involve a little brute force effort, but thankfully not a full manual audit of all 107 C→Lua call sites. Instead, I wrote two macros:

    #define _lua_stackguard_entry(L) int __lua_stackguard_entry=lua_gettop(L);
    #define _lua_stackguard_exit(L) assert(__lua_stackguard_entry == lua_gettop(L));
    

    These are very simple to use - you call _lua_stackguard_entry() just after you’ve obtained a pointer to the Lua state object, and then you call _lua_stackguard_exit() at every point where the function can return after that. It records the size of the stack (lua_gettop()) at the entry point and assert()s that it’s the same at the exit point (assert() also calls abort() if something is wrong, so now we would get crash logs with the crash in the actual function where the leak is happening). These entry/exit calls were then added to all 107 call sites 4 days after the 0.9.61 was released and I spent 3 evenings testing or manually verifying every site, before releasing 0.9.65 (0.9.62-0.9.64 fixed some of the other bugs found by the API checker in the mean time).

    At the time of writing we’re only 24 hours past the release of 0.9.65, but so far things are looking good - no strange Lua segfault crash reports as yet. There was one issue found today where I’d placed a _lua_stackguard_exit() call after a C statement that seemed unimportant, but actually caused an important object to be freed, but that is already fixed and will be included in 0.9.66.

    Assuming we have now fixed the problem, after months of head-scratching, and a few weeks of research, testing and coding, it turns out that across the 107 call sites we only had two stack leaks - one was in the code that handles tab completion in Hammerspoon’s Console window, and the other was in hs.notify. Hopefully you’re all enjoying a more stable Hammerspoon experience, but I think we’ll be leaving both the API checker and the stack guard macros enabled since they make it very easy to find/fix these sorts of bugs. I’d rather get a smaller number of crashes sooner, than have more months of head-scratching!

    Discuss on Twitter Discuss on Hacker News
  • Getting battery data from AirPods in macOS

    A recent feature request for Hammerspoon requested that we add support for reading battery information about AirPods (UK US).

    Unfortunately because their battery status is quite complex (two earbuds and the case), this information is not reported via the normal IOKit APIs, but with a bit of poking around in the results of class-dump for macOS High Sierra I was able to find some relevant methods/properties on IOBluetoothDevice that let you read information about the battery level of individual AirPods and the case, plus determine which of the buds are currently in an ear!

    So, the next release of Hammerspoon should include this code to expose all of this information neatly via hs.battery.privateBluetoothBatteryInfo() 😁

  • Happy 10th Birthday Terminator!

    Today marks 10 years since the very first public release of Terminator, a multiplexing terminal emulator project.

    I started working on Terminator as a simple way to get 4 terminals to not overlap on my laptop screen. In the following years it grew many features and attracted a userbase I am very proud of.

    As much as I would like to, I cannot claim most of the credit for Terminator surviving for a decade - I stepped away from the project a few years ago and handed the reigns over to Stephen Boddy at a very crucial time - gtk2 was becoming ever more obsolete and our work on a gtk3 port was very incomplete. Stephen has driven the project forward and it now has a very good gtk3 version :)

    So, thank you to Stephen and everybody else who contributed code/docs/translations/suggestions/bugs/etc over the last 10 years (you can see the most active folk here).

    I’d also like to note that according to Ubuntu’s Popularity Contest data, Terminator is installed on at least 56000 Ubuntu machines. Debian also has PopCon data, but the numbers there are a little less impressive ;)

    This was the first project of mine that reached any kind of significant audience, and is the first project of mine to have achieved a decade of active maintenance, so I am feeling pretty happy today!

  • USB Type C is awful

    Intentionally inflammatory title there, but there are some valid reasons to be annoyed at USB Type C.

    Firstly, I have discovered (the hard way) that although there are many cables on sale, the majority of Type C cables do not support even USB 3.0 speeds (so 5Gb/s) let alone USB 3.1gen2 (so 10Gb/s) speeds. They are actually USB 2.0 (so 480Mb/s) cables.

    I can understand that some devices with Type C connectors may only need USB 2.0 speeds, but for the cables to not all be USB 3.x seems crazy to me. Even Apple is doing this - the charging cables for MacBooks (12” and Pros) with Type C ports, only support USB 2.0 speeds. If I had to guess, I’d say it’s because they wanted the cables to be thin and easily bendable, which full USB 3.1 cables tend not to be.

    Secondly, unlike Type A, which marks USB 3.x cables by having blue plastic inside the connectors, there is no way to tell what speed a Type C cable is by looking at it.

    Thirdly, and perhaps most importantly, USB Type C is a vastly more powerful beast than previous versions - a modern Type C port can be capable of:

    • 40Gb/s Thunderbolt 3
    • DisplayPort
    • 10Gb/s USB 3.1gen2
    • 100W of (actively negotiated) power delivery in either direction

    The DisplayPort “alternate mode” can deliver 4K at 60Hz with USB3.1 at the same time, or 5K with USB2.0 at the same time, or 8K (compressed). When used as Thunderbolt, Type C can carry 5K video as well as the PCI data.

    So while one tiny connector can do a whole bunch of really impressive things, the cables are now expected to do vastly more than even USB3.0 Type A cables, let alone USB 2.0, and it seems like that advanced capability isn’t currently aligned with the history of USB as being an ultra-cheap, mass market affair.

    Various awesome folk have put together a spreadsheet of the chargers/cables they’ve tested, and it seems like a serious chunk of the Type C cables currently on the market, are junk. This is bad for everyone, especially users, who can buy what looks like the right cable, only to discover that their devices either don’t work at all, or work to slowly, or won’t charge properly.

    This post exists because I needed a USB3.0, three metre, Type A to Type C cable, and I bought one on Amazon, only to discover that it only supported USB2.0. After far too much searching, I eventually found an Anker cable which meets my requirements:

  • Home networking like a pro - Part 1 - Network Storage

    Introduction

    This is part one in a series of posts about some hardware I recommend (or otherwise!) for people who want to bring some semi-professional flair to their home network.

    The first topic is storage - specifically, Network Attached Storage.

    Background

    For the last few years, I was running a Mac Mini with two 3TB drives in a RAID1 array in a LaCie 2big Thunderbolt chassis (US UK), with the Mac running macOS Server to provide file sharing (AFP and SMB), and Time Machine backups for the rest of the network.

    This was a very nice solution, in that the Mac was a regular computer, so I could make it do whatever I wanted, but it did have the drawbacks that the Thunderbolt chassis only had two drive bays, and I had trouble getting the Mac to run reliably for months at a time (I ran into GPU related kernel panics, perhaps because it was attached to a TV rather than a monitor).

    Around the time I was selecting the Mac/LaCie, most NAS devices in a similar price range were very underpowered ARM devices, and could do little more than share files, but in 2017 almost all NAS devices are much more powerful x86 devices that often have extensive featuresets (e.g. running containers, VMs, hardware accelerated video transcoding, etc.) so I decided it was time to switch.

    Solution

    I ended up choosing a Synology DS916+ (US UK), popped one of the 3TB drives (Western Digital Red (US UK)) out of the LaCie and into the Synology, and set about migrating my data over. I then moved the other drive, and put in two more 3TB drives, all of which are running as a single Synology Hybrid RAID volume with a BTRFS filesystem (note that the Hybrid RAID really seems to just be RAID5).

    I configured the Synology to serve files over both AFP and SMB, and enabled its support for Time Machine via AFP. I was also able to connect both of its Ethernet ports to my switch (a ZyXEL GS1900 24 port switch, which I will cover in an upcoming post) and enabled LACP on each end to bond the two connections into a single 2Gb link.

    So, how did it work out?

    The AFP file sharing is great, and works flawlessly. SMB is a little more complex, because recent versions of macOS tend to enforce encryption on SMB connections, which makes them go much slower, but this can be disabled. I tested Time Machine over SMB, which is officially supported by Synology, but is a very recent addition, and it proved to be unreliable, so that is staying on AFP for now.

    Something I was particularly keen on, with the Synology, was that it has an “app store” and one of the available applications is Docker. I was running a few UNIX daemons on the Mac Mini which I wanted to keep, and Docker containers would be perfect for them, however, I discovered that the version of Docker provided by Synology is pretty old and I ran into a strange bug that would cause dockerd to consume all available CPU cycles.

    For now, the containers are running on an Intel NUC (which will also be covered in an upcoming post) and the Synology is focussed on file sharing.

    Open Source

    Synology’s NAS products are based on Linux, Samba, netatalk and a variety of other Open Source projects, with their custom management GUI on top. They do publish source, but it’s usually a little slow to arrive on the site, and it’s not particularly easy (or in some cases even possible) to rebuild in a way that lets you actively customise the device.

    Conclusion

    Overall, I like the Synology, but I think if I’d known about the Docker issue, I might have built my own machine and put something like FreeNAS on it. More work, less support, but more flexibility.

    The recent 5-8 drive Synologies now support running VMs, which seems like a very interesting prospect, since it ought to isolate you from Synology’s choices of software versions.