{"id":3043,"date":"2012-06-28T15:10:44","date_gmt":"2012-06-28T05:10:44","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=3043"},"modified":"2012-06-28T15:10:44","modified_gmt":"2012-06-28T05:10:44","slug":"hacking-the-jenkins-bzr-plugin","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2012\/06\/28\/hacking-the-jenkins-bzr-plugin\/","title":{"rendered":"Hacking the Jenkins BZR plugin"},"content":{"rendered":"

For Drizzle and for all of the projects we work on at Percona we use the Bazaar<\/a> revision control system (largely because it’s what we were using at MySQL and it’s what MySQL still uses). We also use Jenkins<\/a>.<\/p>\n

We have a lot<\/strong> of jobs in our Jenkins. A lot<\/strong>. We build upstream MySQL 5.1, 5.5 and 5.6, Percona Server 5.1, Percona Server 5.5, XtraBackup 1.6, 2.0 and 2.1. For each of these we also have the normal trunk builds as well as parameterised ones that allow a developer to test out a tree before they ask for it to be merged. We also have each of these products across seven operating systems and for each of those both x86 32bit and 64bit. If we weren’t already in the hundreds of jobs, we certainly are once you multiply out between release and debug and XtraBackup being across so many MySQL and Percona Server versions.<\/p>\n

I honestly would not be surprised if we had the most jobs of any user of the Bazaar plugin to Jenkins, and we’re probably amongst the top few of all Jenkins installations.<\/p>\n

So, in August last year we discovered a file descriptor leak in the Bazaar plugin. Basically, garbage collection doesn’t get kicked off when you run out of file descriptors. This prevented us from even starting back up Jenkins until I found and fixed the bug. Good times.<\/p>\n

We later hit a bug that was triggered in the parallel loading of jobs during startup. We could get stuck in an infinite loop during Jenkins starting that would just eat CPU and get nowhere. Luckily Jenkins provides a workaround: specify “-Djenkins.model.Jenkins.parallelLoad=false” as an argument and it just does it single threaded. For us, this solves that problem.<\/p>\n

We were also hitting another problem. If you kill bzr at just the wrong time, you can leave the repository in not an entirely happy state. An initial branch can be killed at a time where it’ll think it’s a repository rather than a checkout and there’s a bunch of other weirdness (including file system corruption if you happen to use bad VM software).<\/p>\n

The way we were solving this was to sometimes go and “clean workspace” on the jobs that needed it (annoying with matrix builds). We’d switched to just doing “clean tree” for a bunch of builds. The problem with doing a clean tree was that “bzr branch” to check out the source code could take a very<\/strong> long time – especially for Percona Server which is a branch of MySQL and hence has hundreds of megabytes of history.<\/p>\n

We couldn’t use bzr shared repositories as we kept hitting concurrency bugs when more than one jenkins job was trying to do a bzr operation at the same time (common when matrix builds kick off builds for release and debug for example).<\/p>\n

So.. I fixed that in the Jenkins bazaar plugin too (which should be in an upcoming release) and we’ve been running it on our Jenkins instance for the past ~2 months.<\/p>\n

Basically, if we fail to check out the Bazaar tree, we wipe it clean and try again (Jenkins has a “retry count” for source checkouts). This is a really awesome form of self healing. Even if the bazaar team fixed all the bugs, we’d still have to go and get that new version of bzr on all our build machines – including ancient systems such as CentOS 5. Not as much fun as bashing your head into a vice.<\/p>\n

After all of that, I seem to now be the maintainer of the Bazaar plugin for Jenkins as Monty pointed out I was using it a lot more than him and kept finding and fixing bugs.<\/p>\n

Soooo… say hello to the new Jenkins Bazaar plugin maintainer, me.<\/p>\n

Yes, I maintain Java code now. Be afraid. Be very<\/strong> afraid.<\/p>\n","protected":false},"excerpt":{"rendered":"

For Drizzle and for all of the projects we work on at Percona we use the Bazaar revision control system (largely because it’s what we were using at MySQL and it’s what MySQL still uses). We also use Jenkins. We … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[76,75,423],"tags":[480,100,481,438],"class_list":["post-3043","post","type-post","status-publish","format-standard","hentry","category-code","category-drizzle-work-et-al","category-percona","tag-bazaar","tag-bzr","tag-java","tag-jenkins"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-N5","jetpack-related-posts":[{"id":3655,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/01\/14\/converting-mysql-trees-to-git\/","url_meta":{"origin":3043,"position":0},"title":"Converting MySQL trees to git","author":"Stewart Smith","date":"2014-01-14","format":false,"excerpt":"I have put up a set of scripts on github: https:\/\/github.com\/stewartsmith\/bzr-to-git-conversion-scripts. Why do I need these? Well... if only bzr fast-export|git fast-import worked flawlessly for large, complex and old trees. It doesn't. Basically, when you clone this repo you can run \".\/sync-BLAH.sh\" and it'll pull BZR trees for the project,\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2380,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/05\/21\/xtrabackup-bazaar-repositories-upgraded-to-2a-format\/","url_meta":{"origin":3043,"position":1},"title":"xtrabackup bazaar repositories upgraded to 2a format","author":"Stewart Smith","date":"2011-05-21","format":false,"excerpt":"I have just upgraded the xtrabackup bazaar code repositories to the 2a format. This means that bzr 1.16 is required to access the source code repositories now. If you get an error like the one below when working with a local branch, you'll need to run \"bzr upgrade\" in it\u2026","rel":"","context":"In "percona"","block_context":{"text":"percona","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/percona\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3242,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/03\/07\/other-mysql-code-size\/","url_meta":{"origin":3043,"position":2},"title":"Other MySQL branch code sizes\u00ef\u00bb\u00bf\u00ef\u00bb\u00bf","author":"Stewart Smith","date":"2013-03-07","format":false,"excerpt":"Continuing on from my previous posts,\u00c2\u00a0MySQL code size over releases\u00c2\u00a0and\u00c2\u00a0MariaDB code size\u00c2\u00a0I've decided to also look into some other code branches. I've used the same methodology as my previous few posts: sloccount for C and C++ code only. There are also other branches around in pretty widespread use (if only\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2468,"url":"https:\/\/www.flamingspork.com\/blog\/2012\/03\/09\/sessions-at-the-percona-live-mysql-conference-that-interest-me\/","url_meta":{"origin":3043,"position":3},"title":"Sessions at the Percona Live MySQL Conference that interest me","author":"Stewart Smith","date":"2012-03-09","format":false,"excerpt":"For the past many years, there's been a conference in April, at the Santa Clara Convention Centre where the topic has been MySQL and the surrounding ecosystem. The first year I went, I gave a talk on the new features in MySQL Cluster 5.1 to a overflowing room of attendees.\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3298,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/04\/19\/detecting-if-a-mysql-server-supports-partitioning\/","url_meta":{"origin":3043,"position":4},"title":"Detecting if a MySQL server supports partitioning","author":"Stewart Smith","date":"2013-04-19","format":false,"excerpt":"This morning, this Percona XtraBackup bug came to my attention:\u00c2\u00a0https:\/\/bugs.launchpad.net\/bugs\/1170340\u00c2\u00a0- basically, it's now really quite tricky to determine if a MySQL server you're connected to supports partitioning or not. If you're connected to anything less than MySQL 5.6, you can use have_partitioning variable. But since that's gone in 5.6, you're\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3419,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/10\/02\/the-end-of-bazaar\/","url_meta":{"origin":3043,"position":5},"title":"The end of Bazaar","author":"Stewart Smith","date":"2013-10-02","format":false,"excerpt":"I've used the Bazaar (bzr) version control system since roughly 2005. The focus on usability was fantastic and the team at Canonical managed to get the entire MySQL BitKeeper history into Bazaar - facilitating the switch from BitKeeper to Bazaar. There were some things that weren't so great. Early on\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"Screenshot from 2013-10-02 10:32:19","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2013\/10\/Screenshot-from-2013-10-02-103219-300x59.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3043","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=3043"}],"version-history":[{"count":2,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3043\/revisions"}],"predecessor-version":[{"id":3050,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3043\/revisions\/3050"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=3043"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=3043"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=3043"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}