{"id":4397,"date":"2018-08-22T18:15:41","date_gmt":"2018-08-22T08:15:41","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4397"},"modified":"2019-07-23T14:24:38","modified_gmt":"2019-07-23T04:24:38","slug":"pwnm-sync-synchronizing-patchwork-and-notmuch","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2018\/08\/22\/pwnm-sync-synchronizing-patchwork-and-notmuch\/","title":{"rendered":"pwnm-sync: Synchronizing Patchwork and Notmuch"},"content":{"rendered":"\n

One of the core bits of infrastructure I use as a maintainer is Patchwork (I wrote about making it faster recently<\/a>). Patchwork tracks patches sent to a mailing list, allowing me as a maintainer to track the state of them (New|Under Review|Changes Requested|Accepted etc), combine them into patch bundles, look at specific series, test results etc.<\/p>\n\n\n\n

One of the core bits of software I use is my email client, notmuch<\/a>. Most other mail clients are laughably slow and clunky, or just plain annoying for absorbing a torrent of mail and being able to deal with it or just plain ignore it but have it searchable locally.<\/p>\n\n\n\n

You may think your mail client is better than notmuch, but you’re wrong.
<\/p>\n\n\n\n

A key feature of notmuch is tagging email. It doesn’t do the traditional “folders” but instead does tags (if you’ve used gmail, you’d be somewhat familiar).<\/p>\n\n\n\n

One of my key work flows as a maintainer is looking at what patches are outstanding for a project, and then reviewing them. This should also be a core part of any contributor to a project too. You may think that a tag:unread and to:project-list@foo<\/em> query would be enough, but that doesn’t correspond with what’s in patchwork.
<\/p>\n\n\n\n

So, I decided to make a tool that would add tags to messages in notmuch corresponding with the state of the patch in patchwork. This way, I could easily search for “patches marked as New in patchwork” (or Under Review or whatever) and see what I should be reviewing and looking at merging or commenting on.<\/p>\n\n\n\n

Logically, this wouldn’t be that hard, just use the (new) Patchwork REST API to get the state of everything and issue the appropriate notmuch commands.<\/p>\n\n\n\n

But just going one way isn’t that<\/strong> interesting, I wanted to be able to change the tags in notmuch and have them sync back up to Patchwork. So, I made that part of the tool too.<\/p>\n\n\n\n

Introducing pwnm-sync<\/a>: a tool to sync patchwork and notmuch.<\/p>\n\n\n\n

\"\"
notmuch-hello tag counts for pwnm-sync tagged
patches in patchwork
<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
notmuch-hello tag counts for pwnm-sync tagged
patches in patchwork<\/figcaption><\/figure><\/div>\n\n\n\n

With this tool I can easily see the patchwork state of any patch that I have in my notmuch database. For projects that I’m a maintainer on (i.e. can change the state of patches), If I update the patches of that email and run pwnm-sync again, it’ll update the state in patchwork.<\/p>\n\n\n\n

I’ve been using this for a few weeks myself and it’s made my maintainer workflow significantly nicer.<\/p>\n\n\n\n

It may also be useful to people who want to find what patches need some review.<\/p>\n\n\n\n

The sync time is mostly dependent on how fast your patchwork instance is for API requests. Unfortunately, we need to make some improvements on the Patchwork side of things here, but a full sync of the above takes about 4 minutes for me. You can also add a –epoch option (with a date\/time) to say “only fetch things from patchwork since that date” which makes things a lot<\/strong> quicker for incremental syncs. For me, I typically run it with an epoch of a couple of months ago, and that takes ~20-30 seconds to run. In this case, if you’ve locally updated a old patch, it will still sync that change up to patchwork.<\/p>\n\n\n\n

Implementation Details<\/h2>\n\n\n\n

It’s a python3 script using the notmuch python bindings, the requests-futures module for asynchronous HTTP requests (so we can have the patchwork server assemble the next page of results while we process the previous one), and a local sqlite3 database to store state in so we can work out what changed locally \/ server side.<\/p>\n\n\n\n

Getting it<\/h2>\n\n\n\n

Head to https:\/\/github.com\/stewartsmith\/pwnm-sync<\/a> or just:<\/p>\n\n\n\n

git clone https:\/\/github.com\/stewartsmith\/pwnm-sync.git<\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
One of the core bits of infrastructure I use as a maintainer is Patchwork (I wrote about making it faster recently). Patchwork tracks patches sent to a mailing list, allowing me as a maintainer to track the state of them … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-4397","post","type-post","status-publish","format-standard","hentry","category-general"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-18V","jetpack-related-posts":[{"id":4293,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/11\/28\/how-i-do-email-at-home\/","url_meta":{"origin":4397,"position":0},"title":"How I do email (at home)","author":"Stewart Smith","date":"2017-11-28","format":false,"excerpt":"I thought I might write something up on how I've been doing email both at home and at work. I very much on purpose keep the two completely separate, and have slightly different use cases for both of them. For work, I do not want mail on my phone. For\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4342,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/12\/11\/how-i-do-email-at-work\/","url_meta":{"origin":4397,"position":1},"title":"How I do email (at work)","author":"Stewart Smith","date":"2017-12-11","format":false,"excerpt":"Recently, I blogged on my home email setup and\u00c2\u00a0in that post, I hinted that my work setup was rather different. I have entirely separate computing devices for work and personal, a setup I strongly recommend. This also lets me \"go home\" from work even when working from home, I use\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/Screenshot-from-2017-12-11-13-43-37.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/Screenshot-from-2017-12-11-13-43-37.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/Screenshot-from-2017-12-11-13-43-37.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":2393,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/09\/17\/without-notmuch-i-would-simply-delete-your-email\/","url_meta":{"origin":4397,"position":2},"title":"Without notmuch, I would simply delete your email","author":"Stewart Smith","date":"2011-09-17","format":false,"excerpt":"I have been using notmuch (http:\/\/notmuchmail.org\/) as my email client for quite a while now. It's fast. I don't mean that everything happens instantly (some actions do take a bit longer than ideally they would), but with the quantity of mail I (and others) throw at it? Beats everything else\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4386,"url":"https:\/\/www.flamingspork.com\/blog\/2018\/08\/13\/optimizing-database-access-in-django-a-patchwork-story\/","url_meta":{"origin":4397,"position":3},"title":"Optimizing database access in Django: A patchwork story","author":"Stewart Smith","date":"2018-08-13","format":false,"excerpt":"tl;dr: I made Patchwork a lot faster by looking at what database queries were being generated and optimizing them either by making Django produce better queries or by adding better indexes. Introduction to Patchwork One of the key bits of infrastructure a bunch of maintainers of Open Source Software use\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2018\/08\/Screenshot-from-2018-08-13-14-32-28.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2018\/08\/Screenshot-from-2018-08-13-14-32-28.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2018\/08\/Screenshot-from-2018-08-13-14-32-28.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2018\/08\/Screenshot-from-2018-08-13-14-32-28.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2018\/08\/Screenshot-from-2018-08-13-14-32-28.png?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":3051,"url":"https:\/\/www.flamingspork.com\/blog\/2012\/06\/28\/new-jenkins-bazaar-plugin-release-1-18\/","url_meta":{"origin":4397,"position":4},"title":"New Jenkins Bazaar plugin release! 1.18","author":"Stewart Smith","date":"2012-06-28","format":false,"excerpt":"From the desk of your new Bazaar plugin for Jenkins maintainer, I give you Version 1.18. This release has two good bug fixes: UI fix for checkout option (JENKINS-12261) Auto-recover from corrupt BZR branches (e.g. bzr branch\/checkout killed at inopportune moment) by cleaning the workspace and trying again (this is\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3365,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/07\/05\/switching-to-fedora-from-ubuntu\/","url_meta":{"origin":4397,"position":5},"title":"Switching to Fedora from Ubuntu","author":"Stewart Smith","date":"2013-07-05","format":false,"excerpt":"I've run Ubuntu on my desktop (well... and laptop) since roughly the first release back in 2004. I've upgraded along the way, with reinstalls on the laptop limited to changing CPU architecture and switching full disk encryption. Yesterday I wiped Ubuntu and installed Fedora. Previously to Ubuntu I ran Debian.\u2026","rel":"","context":"In "GNOME"","block_context":{"text":"GNOME","link":"https:\/\/www.flamingspork.com\/blog\/category\/gnome\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=4397"}],"version-history":[{"count":4,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4397\/revisions"}],"predecessor-version":[{"id":4462,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4397\/revisions\/4462"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=4397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=4397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=4397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}