{"id":4386,"date":"2018-08-13T14:59:14","date_gmt":"2018-08-13T04:59:14","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4386"},"modified":"2018-08-13T14:59:27","modified_gmt":"2018-08-13T04:59:27","slug":"optimizing-database-access-in-django-a-patchwork-story","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2018\/08\/13\/optimizing-database-access-in-django-a-patchwork-story\/","title":{"rendered":"Optimizing database access in Django: A patchwork story"},"content":{"rendered":"\n

tl;dr:<\/strong> I made Patchwork a lot faster by looking at what database queries were being generated and optimizing them either by making Django produce better queries or by adding better indexes.<\/p>\n\n\n\n

Introduction to Patchwork<\/h2>\n\n\n\n

One of the key bits of infrastructure a bunch of maintainers of Open Source Software use is a tool called Patchwork<\/a>. We use it for a bunch of OpenPOWER firmware development, several Linux subsystems use it as well as freedesktop.org.<\/p>\n\n\n\n

The purpose of Patchwork is to supplement the patches-to-a-mailing-list development work flow. It allows a maintainer to see all the patches that have been posted on the list, How many Acked-by\/Reviewed-by\/Tested-by replies they have, delegate responsibility for the patch to a co-maintainer, track (and change) the state of the patch (e.g. to “Under Review”, “Changes Requested”, or “Accepted”), and create bundles of patches to help in review and testing.<\/p>\n\n\n\n

Since patchwork is an open source project itself, there’s several instances of it out there in common use. One of the main instances is https:\/\/patchwork.ozlabs.org\/<\/a> which is (funnily enough) used by a bunch of people connected to OzLabs for projects that are somewhat connected to OzLabs. e.g. the linuxppc-dev project and the skiboot and petitboot projects. There’s also a kernel.org instance, which is used by some kernel subsystems.<\/p>\n\n\n\n

Recent versions of Patchwork have added some pretty cool features such as the ability to integrate with CI systems such as Snowpatch<\/a> which helps maintainers see if patches submitted are likely to break things.<\/p>\n\n\n\n

Unfortunately, there’s also been some complaints that recent version of patchwork have gotten slower than previous ones. This may well be the case, or it could just be that the volume of patches is much higher and there’s load on the database. Anyway, after asking a few questions about what the size and scope was of the patchwork database on ozlabs.org, I went “hrm… this sounds like it shouldn’t really be a problem… perhaps I should look into this”.<\/p>\n\n\n\n

Attacking the problem…<\/h2>\n\n\n\n

Every so often it is revealed that I know a little bit about databases.<\/p>\n\n\n\n

Getting a development environment up for Patchwork is amazingly easy thanks to Docker and the great work of the Patchwork maintainers. The only thing you need to load in is an example dataset. I started by importing mail from a few mailing lists I’m subscribed to, which was Good Enough(TM) for an initial look.<\/p>\n\n\n\n

Due to how Django forces us to design a database schema though, the suggested method of getting a sample data set will not<\/strong> mirror what occurs in a production system with multiple lists. It’s for this reason that I ended up using a copy of a live dataset for much of my work rather than constructing an artificial one.<\/p>\n\n\n\n

Patchwork supports both a MySQL and PostgreSQL database backend. Since the one on ozlabs.org is backed by PostgreSQL, I ended up loading a large dataset into PostgreSQL for most of my work, although I also did some testing with MySQL.<\/p>\n\n\n\n

The current patchwork.ozlabs.org instance has a database of around 13GB in side, with about a million patches. You may think this is big, my database brain goes “no, this is actually quite small and everything should be a lot faster than it is even on quite limited hardware”<\/p>\n\n\n\n

The problem with ORMs<\/h2>\n\n\n\n

It turns out that Patchwork is written in Django, an ORM (Object-Relational Mapping) framework in Python – and thus something that pretty effectively obfuscates application code from the SQL being run.<\/p>\n\n\n\n

There is one thing that Django misses that could be a pretty big general performance boost to many applications: it doesn’t support composite primary keys. For some databases (e.g. MySQL’s InnoDB engine) the PRIMARY KEY is a clustered index – that is, the physical layout of the rows on disk reflect primary key order. You can use this feature to your advantage and have much higher cache hits of your database pages.<\/p>\n\n\n\n

Unfortunately though, we cannot do that with Django, so we lose a bunch of possible performance because of it (especially for queries that are going to have to bring in data from disk). In fact, we’re forced<\/strong> to use an ID <\/strong>field that’ll scatter our rows all over the place rather than do something efficient. You can somewhat get back some of the performance by creating covering indexes, but this costs in terms of index maintenance and disk space.<\/p>\n\n\n\n

It should be noted that PostgreSQL doesn’t have a similar concept, although there is a (locking) CLUSTER statement that can (as an offline operation for the table) re-arrange existing rows to be in index order. In my testing, this can give a bit of a boost to performance of some of the Patchwork queries.<\/p>\n\n\n\n

With MySQL, you’d look at a bunch of statistics on what pages are being brought in and paged out of the InnoDB buffer pool. With PostgreSQL it’s a bit more complex as it relies heavily on the OS page cache.<\/p>\n\n\n\n

My main experience is with MySQL like environment, so I’ve had to re-learn a bunch of PostgreSQL things in this work which was kind of fun. It may be “because of my upbringing” but it seems as if there’s a lot more resources and documentation out in the wild about optimizing MySQL environments than PostgreSQL ones, especially when it comes to documentation around a bunch of things inside the database server. A lot<\/strong> of credit should go to the MySQL Documentation team – I wish the PostgreSQL documentation was up to the same standard.<\/p>\n\n\n\n

Another issue is that fetching BLOBs is generally an expensive<\/strong> operation that you want to avoid unless you’re going to use them. Thus, fetching the whole “object” at once isn’t always optimal. The Django query generation appears to be somewhat buggy when it comes to “hey, don’t fetch these columns, I don’t need them”, so you do have to watch what query is produced<\/strong> not just what query you expect<\/strong> to be produced. For example, [01\/11] Improve patch listing performance (~3x).<\/a>
<\/p>\n\n\n\n

Another issue with Django is how you go from your Python code to an actual SQL query, especially when the produced SQL query is needlessly complex or inefficient. I’ve seen Django always<\/strong> produce an ORDER BY for one table, even when not needed, I’ve also seen it always<\/strong> join tables even when you’re getting no columns from one of them and there’s no way you’re asking for it. In fact, I had to revert to raw SQL for one of my performance improvements as I just couldn’t beat it into submission: [10\/11] Be sensible computing project patch counts<\/a>.<\/p>\n\n\n\n

An ORM can be great for getting apps out quickly, or programming in a familiar way. But like many things, an understanding of what is going on underneath is key for extracting maximum performance.<\/p>\n\n\n\n

Also, if you ever hear something like “ORM $x doesn’t scale” then maybe that person just hasn’t looked at how to use the ORM better. The same goes for if they say “database $y doesn’t scale”- especially if it’s a long existing relational database such as MySQL or PostgreSQL.<\/p>\n\n\n\n

Speeding up listing current patches for a project<\/h2>\n\n\n
\n
\"17
More than 4 seconds in the database
does not make page load time great.<\/figcaption><\/figure>\n<\/div>\n\n\n

Fortunately though, the Django development environment lets you really<\/strong> easily dive into what queries are being generated and (at least roughly) where they’re being generated from. There’s a sidebar in your browser that shows how many SQL queries were needed to generate the page and how long they took. The key to making your application go faster is to run fewer queries in less time.<\/p>\n\n\n\n

I was incredibly impressed with how easy it was to see what queries were run, where they were run from, and the EXPLAIN output for them.<\/p>\n\n\n\n

By clicking on that SQL button on the right side of your browser, you get this wonderful<\/strong> chart of what queries were executed, when, and how long they took. From this, it is incredibly<\/strong> obvious which query is the most problematic: the one that took more than four seconds!<\/strong><\/p>\n\n\n\n

In the dim dark days of web development, you’d have to turn on a Slow Query Log on the database server and then grep through your source code or some other miserable activity. I am so glad<\/strong> I didn’t have to do that.<\/p>\n\n\n\n

\"\"
More than four seconds for a single database query does not make for a nice UX.<\/figcaption><\/figure>\n\n\n\n

This particular query was a real hairy one, the EXPLAIN output from PostgreSQL (and MySQL) was certainly long and involved and would most certainly not<\/strong> feature in the first half of an “Introduction to query optimization” day long workshop. If you haven’t brushed up on various bits of documentation on understanding EXPLAIN, you should! The MySQL EXPLAIN FORMAT=JSON is especially fantastic for getting deep details as to what’s going on with query execution.<\/p>\n\n\n\n

The big performance gain here was to have the database be able to execute the query in a much more efficient way by creating two covering indexes for part of the query. To work out what indexes to create, one has to look at the EXPLAIN output and work out why<\/strong> the database is choosing to do either a sequential scan of a large table, or use an index that doesn’t exclude that many rows. In this case, I tweaked the code to slightly change the query that was generated as well as adding a covering index. What we ended up with is something that is dramatically<\/strong> faster.<\/p>\n\n\n\n

\"\"
The main query is ~350x faster than before<\/figcaption><\/figure>\n\n\n\n

You’ll notice that it appears<\/strong> that the first query there takes a lot more time but it doesn’t, it just takes a lot more time relative to<\/strong> the main query.<\/p>\n\n\n\n

In fact, this particular page is one that people have mentioned at being really, really slow to load. With the main query now about 350 times faster than it was originally, it shouldn’t be a problem anymore.<\/p>\n\n\n\n

A future improvement would be to cache the COUNT() for the common case, as it’s pretty easily computed when new patches come in or states change.<\/p>\n\n\n\n

The patches that help this particular page have been submitted upstream here:<\/p>\n\n\n\n