{"id":809,"date":"2007-03-28T22:14:18","date_gmt":"2007-03-28T12:14:18","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/2007\/03\/28\/patching-your-mission-critical-email-syncing-software-on-your-life-setup-my-offlineimap-patch-for-today\/"},"modified":"2010-05-27T17:15:25","modified_gmt":"2010-05-27T07:15:25","slug":"patching-your-mission-critical-email-syncing-software-on-your-life-setup-my-offlineimap-patch-for-today","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2007\/03\/28\/patching-your-mission-critical-email-syncing-software-on-your-life-setup-my-offlineimap-patch-for-today\/","title":{"rendered":"Patching your mission-critical email syncing software on your life setup… my OfflineIMAP patch for today"},"content":{"rendered":"

I’ve used OfflineIMAP<\/a> for quite a while now. On the whole I’m fairly happy with it. Today I sent this to the list:<\/p>\n

Forgive the potentially bad python, not my native tongue :)<\/tt><\/p>\n

This patch is motivated by three things:<\/tt>
\n- offlineimap is extremely slow at syncing lots of locally deleted<\/tt>
\nmessages<\/tt>
\n- offlineimap uses lots of memory<\/tt>
\n- LocalStatus files aren't written safely (a hard crash can cause<\/tt>
\ncorruption)<\/tt>
\n\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0- I've been bitten by this in the past, causing a complete resync of<\/tt>
\nthe folder... so I get duplicate messages.<\/tt><\/p>\n

I am currently using 4.0.14 (from Debian) with this patch. I used it to<\/tt>
\nconvert the files and everything. Seems quite reliable and quick.<\/tt><\/p>\n

In my tests, execution time for a normal sync is relatively the same.<\/tt><\/p>\n

Execution time for when lots of messages have been deleted in a<\/tt>
\nreasonably sized folder (e.g. during re-organisation of mail folders) is<\/tt>
\nas much as 10x faster.<\/tt><\/p>\n

In my tests, running with 1 thread uses as much as 20% less memory with<\/tt>
\nthis patch (i.e. about 160MB instead of 200MB+ for my maildir)<\/tt><\/p>\n

Disk space used by the LocalStatus files isn't much more... for me it<\/tt>
\nlooks like it's 6.5MB now versus 4.5MB then. We get the added benefit of<\/tt>
\nindexes for all our queries... nice :)<\/tt><\/p>\n

I had disable the threading for copying messages as this means that<\/tt>
\nLocalStatus objects are shared between threads, which pysqlite doesn't<\/tt>
\nlike (it asserts).<\/tt><\/p>\n

I think the part of this patch that implements the uidexists does<\/tt>
\nactually slow things down compared with having the messagelist.... a<\/tt>
\nmore optimal implementation may be possible, but I think the other speed<\/tt>
\nimprovements (and memory savings) are worth it.<\/tt><\/p>\n

A future patch may convert other storage types to sqlite (or similar) to<\/tt>
\nfurther reduce memory consumption (and hopefully runtime).<\/tt><\/p>\n

This does add a dependency on pysqlite... which is packaged in debian<\/tt>
\n(and ubuntu) - and i'm using the stock packages for these.<\/tt><\/p>\n

Comments very much appreciated.<\/tt>
\nOf course, the patch is here<\/a>. I’m using it now… although I’ll warn you that it does update your .offlineimap to a new format (and doesn’t provide you a way to go back, without restoring the backed-up LocalStatus files and probably getting message duplicates).<\/p>\n

So, those around the MySQL circles I tend to hang around may ask “Why not libmysqld?” (the embedded MySQL server). Well… a few reasons… sqlite is file-per-db (even though I’m essentially using file-per-table here), the python bindings are everywhere (and work), it’s tiny and crash safe.<\/p>\n

You may also ask “Why?”… well, I’ve been re-organising a bunch of mail folders, which means deleting a *lot* of messages from some folders (and moving them to others).. offlineimap has been really slow at this. So I fixed it, with code (not whining).<\/p>\n

I also wrote a bit-of-a-hack perl script to remove duplicate messages from a bunch of folders (a bug in offlineimap had caused me to get several copies of each message in a bunch of my folders a while ago). So that script is here<\/a>. Commented out are bits to do comparison via md5 as well as message-id. Don’t use unless you know what you’re doing… it may also use a few hundred MB RAM on large (few hundred thousand messages) folder.<\/p>\n

Hopefully these will help improve my productivity.
\nNow, back to my regular programming….<\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve used OfflineIMAP for quite a while now. On the whole I’m fairly happy with it. Today I sent this to the list: Forgive the potentially bad python, not my native tongue :) This patch is motivated by three things: … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[2,14],"tags":[],"class_list":["post-809","post","type-post","status-publish","format-standard","hentry","category-life-the-universe-and-everything","category-mysql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-d3","jetpack-related-posts":[{"id":4293,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/11\/28\/how-i-do-email-at-home\/","url_meta":{"origin":809,"position":0},"title":"How I do email (at home)","author":"Stewart Smith","date":"2017-11-28","format":false,"excerpt":"I thought I might write something up on how I've been doing email both at home and at work. I very much on purpose keep the two completely separate, and have slightly different use cases for both of them. For work, I do not want mail on my phone. For\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":760,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/11\/13\/86gb-of-email\/","url_meta":{"origin":809,"position":1},"title":"8.6GB of email","author":"Stewart Smith","date":"2006-11-13","format":false,"excerpt":"If you tar my Maildir, it comes out at about 8.6GB currently. That's about all my mail since October 2001. Notable exceptions are most of the Spam I've received and any messages from LKML. Doing a first time sync with offlineimap takes an amount of time that is truly scary.\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":850,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/06\/27\/things-that-have-recently-stalled\/","url_meta":{"origin":809,"position":2},"title":"Things that have recently stalled….","author":"Stewart Smith","date":"2007-06-27","format":false,"excerpt":"compressed backup patch actually works rather well... and restoring from compressed backup too. need to modify the rate-limiting code though... may as well rate limit the writing of the *compressed* data stream... otherwise the option isn't nearly as useful compressed LCP patch well... the *restoring* of compressed LCPs.... can write\u2026","rel":"","context":"In "life, the universe and everything"","block_context":{"text":"life, the universe and everything","link":"https:\/\/www.flamingspork.com\/blog\/category\/life-the-universe-and-everything\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":622,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/03\/15\/let-me-express-my-thanks\/","url_meta":{"origin":809,"position":3},"title":"Let me express my thanks","author":"Stewart Smith","date":"2006-03-15","format":false,"excerpt":"Looks like due to a bug in offlineimap, i'm now getting 1 copy of my mail on my laptop, and about 3 copies of each message on my imap server. Looks like I'm going to have to write a script to find duplicate mail messages to unfuck this. Damn you\u2026","rel":"","context":"In "grumble"","block_context":{"text":"grumble","link":"https:\/\/www.flamingspork.com\/blog\/category\/grumble\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":432,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/07\/19\/faster-imap\/","url_meta":{"origin":809,"position":4},"title":"faster IMAP","author":"Stewart Smith","date":"2005-07-19","format":false,"excerpt":"wow, courier-imap in sarge is so much faster than in woody. I see noticable speed improvements now. I also don't see weird stuff happenning with offlineimap (terminating due to unexpected EOF). Much happier with mail. Also, on another note, I've started using Evolution's vfolders again. Talk about cool. Also, I\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1016,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/02\/22\/bzr-loom-a-bzr-plugin-with-quilt-like-functionality\/","url_meta":{"origin":809,"position":5},"title":"bzr-loom – a bzr plugin with quilt like functionality","author":"Stewart Smith","date":"2008-02-22","format":false,"excerpt":"A bzr plugin to assist in developing focused patches. in Launchpad I use quilt a lot for development. Currently, If I had to choose between BK and quilt - I'd choose quilt. I use bzr in other development projects like MemberDB. I use git as a frontend for SVN (it\u2026","rel":"","context":"In "MemberDB"","block_context":{"text":"MemberDB","link":"https:\/\/www.flamingspork.com\/blog\/category\/memberdb\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/809","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=809"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/809\/revisions"}],"predecessor-version":[{"id":2008,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/809\/revisions\/2008"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=809"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=809"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=809"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}