Qt, Python, AsyncIO. And Quetzalcoatl.

I’ve know I’ve said this before, but I’m at work on the next version of Quetzalcoatl, my Qt-based MPD client for KDE, written in Python.

PyQt and PySide

I did not say “PyQt-based”, because PySide compatibility is on the table. Normalizing PyQt for API-compatibility with PySide is easy. If you’re importing PyQt, you set its API to version 2, and monkey patch Signal and Slot aliases (for pyqtSignal and pyqtSlot) to QtCore.

C++ or Python

C++ or Python? Both are good for writing Qt applications. This is no less a consideration than when I started.

C++ has, of course, advantages. It gives you increased performance and reduced memory usage (over Python) for free; it gives you access to classes, such as QtConcurrent and QSignalSpy, that may not be available in the Python bindings. QtConcurrent, in particular, will likely never be available in Python, because of the architecture of Python’s interpreter. Finally, C++ is easier to package (just distribute the executables and link libraries) for Windows and OS X, than Python is.

To me, these aren’t compelling enough to justify the massive increase in complexity that would come from implementing the application in C++. The performance difference isn’t meaningful for an application that spends close to 100 percent of its time waiting for user or network input. Furthermore, I’m targeting only one platform: KDE. KDE is used only on Linux, which invariably has excellent Python support. KDE is also one of the more “heavyweight” platforms available for Linux anyway.

Application Architecture

When I started Quetzalcoatl, I did not know what a socket was. I went straight to searching for Python MPD libraries, and treated them as black boxes. While this was arguably the correct way to start the first iteration, when I wanted to to get it working, it is not the correct way to start the second iteration, when I want to get it right. The new architecture will be based on an understanding of how both desktop applications and asynchronous networking work.

All desktop applications work on the principle of an event loop. This is often handled by the framework and transparent to the programmer, but raw Windows API is an exception. Events, such as clicks, drags, and mouseovers, are events, and organized as a queue. The application itself is single-threaded loop. Event handlers are registered using some variation of the Observer pattern. On each iteration, events are taken from the queue, and registered event handlers are executed.

Asynchronous networking works similarly. It also has event loop. On each iteration, open sockets are checked for the presence of incoming data. Via some implementation of the Observer pattern, you set up code to be executed when data arrives. Observer variations vary widely: callbacks, coroutines, promises, deferreds, futures and others, have all been tried. The asyncio library introduced with Python 3.4 works this way too; it just has a very, very elegant Observer implementation.

The conclusion, of course, is that the only way to do networking in a desktop application, is the following. You integrate the socket notifier into the event loop, and treat incoming data as events.

Qt’s QtNetwork module is based around this architecture. If you’re doing networking in a Qt application without it, you’re doing it wrong.

Networking in Qt

Qt Networking

For some complicated diagrams of how Qt’s networking classes work, see Inside the Qt HTTP stack.

In short, Qt provides the QTcpSocket class, which is based on a socket SELECT that is integrated into the event loop of the GUI thread. For HTTP and FTP, it also provides the higher-level QNetworkAcccessManager, or “QNAM”, which is based on a QTcpSocket. Clients for daemons with custom protocols—redis, mongodb, MPD—should be implemented using a QTcpSocket. Web service clients should be implemented using using a QNAM.

If you need Qt-based client and it doesn’t exist, you should, if it’s feasible, write it. If your client’s scope encompasses most of the protocol or web service API, then you should open-source and release it. The fact that there isn’t already a large selection of Qt-based clients to choose from is a shortcoming of the Qt programming community.

Exceptions

At work, I had an external web service client that I needed to use in Qt applications. The client was of a blocking, synchronous design. The latency was high enough to block the event loop. It was impossible to use with a QNAM. The protocol was undocumented, and rewriting it wasn’t an option.

The most obvious solution would have been to make the calls in a separate thread, but I came up with a better solution. It was to set up an internal web service (written in Flask). The Qt application would use a QNAM to call the internal web service. The internal web service would instantiate the client, use it to call the external web service, and then return the results to the Qt application. I also added a cache (I used redis, but memcached would have worked just as well) to the “proxy” web service, to cut the latency to zero.

Alternatives

I’m aware of the QtReactor project, to use Twisted with PyQt and PySide. I’m not going to say you should never use it, but do you really want to “integrate” a completely separate event loop and a completely different Observer implementation into your Qt application? Qt has its own event loop, and its own Observer implementation (signals and slots).

The MPD Protocol

Getting Quetzalcoatl’s architecture right means writing my own Qt-based MPD client. This client will implement the MPD protocol. MPD is a server and, as with all *nix servers, you can simply telnet to it and issue it commands. The client will write commands to it, via a socket.

The protocol includes the “idle” command, to subscribe to real-time notifications. When you are subscribed, you can issue no commands other than “noidle”, which fetches the results and cancels the subscription. When you are not subscribed, you can issue it any other command. Those commands return, with any data fetched, immediately.

A client should have first-class support for real-time notifications. It should go into “idle” mode when connected. When you need to issue a command, it should “noidle”, handle any results sent back, issue the command, handle any results sent back, and then go into “idle” mode again. If notifications arrive when it’s in “idle’ mode, then it should handle them, and then “idle” again.

The MPD Client

My client, therefore, will be similar to the clients available for node.js. Here is a simplified view, with one “idle” signal and one command.

class MPDClient(QtNetwork.QTcpSocket):
    playlist = QtCore.Signal()

    def status(self):
        # Implementation goes here.

The “playlist” signal is emitted whenever there is a change in the current playlist.

On the other hand, status returns an object similar to a QNetworkReply. It has a “completed” signal, which is emitted, along with the deserialized data, when (and not before) the reply arrives on the socket. After emitting the “completed” signal, the reply object cleans itself up (the implementation calls self.deleteLater()). You use it like this:

def handler(self, status):
    pprint.pprint(status)

reply = client.status()
reply.completed.connect(handler)

Note that handler can be an inner function. Both PyQt and PySide allow you to bind signals to anything that you can call. It’s not quite as elegant as a corresponding node.js client, where the language allows anonymous functions of arbitrary length, but it’s probably as good as you can get with Python and Qt.

Qt 5, as a C++ framework, allow you to bind signals to C++ lambdas. I would expect a C++ implementation to take advantage of that.

Implementation Plans

The current standard in Python MPD clients, python-mpd2, instantiates and encapsulates its sockets. It is designed for a synchronous pattern where you send data, then block until the reply appears. It is tightly coupled to Python’s socket implementation. The parts that serialize and deserialize data are tightly coupled to the socket operations. Even though it can be used asynchronously, in a Qt event loop (the source code has examples of using “idle” in a GTK event loop), it is not what I need.

What I want from python-mpd2 are its serializers and deserializers, decoupled from any socket operations. I’ve been extracting them into a separate library, MPD Serializers. This library is LGPL licensed, because it uses code from python-mpd2, and python-mpd2 is LGPL-licensed. I will release this library, separately, first.

The MPD client itself, the QMPDSocket, will have MPD Serializers as a dependency. It will be based on a QTcpSocket. I will release it after MPDSerializers.

A new version of Quetzalcoatl, with an architecture based on these building blocks, will follow.

Quetzalcoatl also uses last.fm. For now, it uses only a tiny percentage of the last.fm API. If I use more, I will think about releasing a QNAM-based last.fm client separately. That might be something to consider for version 3.