The Great UI Warts - Confession 56

2015.06.11 22:02:19

It's now been about a year since I first started work on Parasol. In the process, I had to learn about UI programming in Common Lisp. It pains me a lot to say this, but it is definitely not one of the great strengths of CL. It certainly wasn't back then, and it still isn't now. Since Parasol started I learned a lot about Qt and in particular the Common Lisp bindings, CommonQt. While using Qt is your best bet at writing a native GUI, it just isn't as pleasant as writing other lisp code. Too many things can break, too many brick walls are laying in wait for you to hit your head against, too many things are simply not there infrastructure wise. However, as Parasol grew, and I grew tired of CommonQt's shortcomings, I started to write more and more systems to work around these problems and make the UI experience for the developer a better one. This is the goal of Qtools.

This library started out as an innocent encapsulation of a few things I'd developed in tandem with Parasol. The first serious issue I had with Parasol was memory leaking. Since we‘re accessing Qt –a C++ library– we need to go back to the old times and deal with our memory by hand. This is a very arduous task and one prone to mistakes. So, a system was developed to alleviate this pain. The result of this is Qtools’ finalizers. At the core of it is a generic function that takes care of cleaning up the object it is passed. So in other words, a destructor function. Using this I could ensure that foreign objects were always properly cleaned up. However, I quickly came to realise that I did one very similar thing all the time: Add a finalizer method for my widget, and call finalize on its slot values. Thanks to the Meta Object Protocol's capabilities, I was able to hide this away completely. Now there's almost never a need to write a finalizer method again. It suffices to just add :finalized T to a widget's slot, and in the case of sub-widgets, the system already does it automatically.

The next issue I had was that writing in CommonQt's style is really uncomfortable. You need to duplicate a lot of information and keep track of the slots, signals, and overrides you define in the class definition. You also need to take care of different type and naming styles that come from C++ and leak into your CL application. This spawned Qtools' widget system. Not only does it take care of mapping naming styles and types, but it also allows a much more normal-looking way of defining your widgets. Instead of having to stuff information into your class definition, you can use multiple, separated forms. Just the way it works in your usual Lisp programs. At the heart of this system lies reinitialize-instance. Thanks to this fantastic function (and the MOP), I was able to separate everything out. What happens in the back when you compile a separate form is that it appends the option that should be in the class definition onto a class property, and calls reinitialize-instance. This call subsequently computes the effective class options and injects them into the class re/initialisation, effectively making it appear as if you had indeed added an option onto the class definition itself.

With this step done, much of the awkwardness was gone. Programs looked much more naturally structured, and things could be specified in a way that felt intuitive. However, one stain remained in the picture: Qt method calls. In order to call Qt methods, CommonQt provides a reader macro: #_. Sadly, in order for this reader macro to work, you need to specify the method name as it is in Qt, including the proper capitalisation. Since it is a foreign call, you also can't inspect it, or get any documentation information out of it. Argument list validity also isn't checked. Getting rid of this and allowing some form of normal-looking function call instead was a rather tricky problem to solve. My first thought was to dynamically analyse the available methods and generate Lisp wrapper functions for them. Those wrapper definitions are dumped to file, and then loaded. Sadly, doing so results in a couple hundred thousand method wrappers and a roughly 50Mb FASL file (on SBCL). The initial compile time also suffered because of this of course. This seemed like a less than stellar solution to me, mostly because the overwhelming majority of the wrappers included would never be called by the GUI anyway. So I sought a different solution. I did find one, albeit it is rather mad.

This solution is called Q+. The first part of it is the aforementioned wrapper compiler that I previously used to generate static wrappers. Modifying it a bit, I could use the same system to generate individual wrappers for any specific method I wanted. The second part is detecting when a supposed call to a wrapper function is made. Since it is not precompiled, the CL host cannot know of it. Thus, we need to somehow intercept when such a form is compiled, dynamically compile the wrapper, and then replace it with a call to the actual wrapper function. That sounds like a macro! And indeed, the q+ macro does that. It takes a method name and an argument list, dynamically compiles the wrapper, and finally emits a call to the new wrapper. The truth is a bit trickier here, since the wrapper needs to be available when a file is merely loaded as well, which wouldn't be the case if it was only generated during macro expansion. So instead, a load-time-value form that generates the wrapper is emitted alongside the wrapper call. That way, methods are always around as needed, with no run-time overhead. The last trick to Q+ is the hiding of the q+ macro call. Using the q+ macro solved most of the problems, but it was essentially the same thing as the #_ reader macro, with a bit nicer method name handling. What I wanted instead was to be able to write the actual wrapper function names. That would also allow slime to show docstrings, arguments, and similar information. In order to make this last trick work, I had to hack into the reader.

One of the greater blemishes of the the Common Lisp standard is the inability to hook into the reader's symbol creation process. This exclusion from the standard makes it impossible to write such things as package local nicknames as a library, or make a case like mine easy. What I had to do instead was to override the ( reader macro. Q+ then reads ahead, to see whether you're trying to reference a symbol from the q+ package. If so, it reads the rest of the form, and emits a call to the q+ macro from above instead. It not, it delegates to the standard reader macro for (. Overriding this reader macro is a dirty trick, and I'd rather not have done it. However, there simply is no other way to accomplish this feat, short of writing a complete reader implementation and demanding that people use that instead of the host implementation's, which is a bit too much to ask for, in my opinion. Still, it works fine, and I haven't run into any obvious issues so far. Now Qtools applications look and read like regular lisp code.

However, how the code looks is only one of the aspects that influence writing GUIs. There's a lot more to it, like for example the initial installation and the binary deployment. Those two things are what I've worked on in the past few weeks now. Out of the first item grew qt-libs, which should ensure that the required libraries like smoke and CommonQt are available easily. This currently works fine for Linux, however I did not get enough time before the Quicklisp release to find testers for Mac OS X. Windows is another problem entirely, one that I can only solve through downloading of precompiled libraries. I've wasted the entire day today with trying to get 64bit versions of the smoke libraries compiled on Windows. Hopefully I can push through with that and allow easy setup of a Qt environment on Windows as well. Qt-libs builds fine on Mac OS X now as well, though there's currently an issue remaining in loading the libraries. I'll get that sorted out before the next Quicklisp release though.

The second part grew into Qtools' new deployment system part. This allows really convenient and easy generation of ready-to-ship binaries of your application. The only thing you have to do is update your system definition a little:

(asdf:defsystem :my-system
  :defsystem-depends-on (:qtools)
  :build-operation "qt-program-op"
  :build-pathname "binary-name"
  :entry-point "my-package:start-function-or-main-class")

Once these four lines are added, you can simply launch your implementation from a shell, invoke (asdf:operate :program-op :my-system) and it'll do all the magic –like closing foreign libraries before dump, restoring the proper library search paths after resume, reloading the foreign libraries again using the new paths, etc.– for you. All you'll get is a bin folder in your project folder that you can zip and ship. I've tried this for Halftone and it Just Works™ on Linux so far.

But, the road ahead is still long and twisted. Once deployment and installation work flawlessly, there's still a lot of code left to be written to make working with Qt itself less painful. Hopefully some day I'll be able to say that writing native GUIs in Lisp is actually a nice experience!

The Qtools documentation is long and extensive. It contains a lot of talk on both how to start using Qtools, as well as what the internals are and how they work. If you're interested, have a read.


Written by shinmera