{"id":334,"date":"2005-01-06T14:10:02","date_gmt":"2005-01-06T04:10:02","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=334"},"modified":"2010-05-27T17:04:10","modified_gmt":"2010-05-27T07:04:10","slug":"effective-bk-usage","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2005\/01\/06\/effective-bk-usage\/","title":{"rendered":"effective bk usage"},"content":{"rendered":"

(inspired by jimw talking about it on Planet MySQL<\/a>)<\/p>\n

I take a bit of a different approach…<\/p>\n

I’ve got directories for 4.0, 4.1 and 5.0, and within them, i have clones of the main ndb tree (called ndb, so there’s a path like “MySQL\/5.0\/ndb”). I don’t ever edit in this tree, it’s my clean one. I use it for pulling and pushing.<\/p>\n

I then ‘bk clone -lq ndb foobar’ (where foobar is what i’m doing, or ndb-foobar, depending on mood). If i have a seperate part to foobar (e.g. stage2 foobar), i’d clone it off the foobar tree (e.g. ‘bk clone -lq foobar foobar2’). The idea being I can work on stage 2 before stage 1 is pushed (and find any problems with stage 1 and pull before i pull into stage2 and can no longer do work).<\/p>\n

i use the -q option (quiet) to bk often because printing out a few thousand lines to screen tends to slow things down.<\/p>\n

It would be interesting to investigate ways to improve hardlinking performance as a clone takes longer than it really should (read about 20000 inodes, write about 20000 inodes). Although (i haven’t checked this) – a clone probably *copies* the checked out files, not link them. So it’s really copy about 10000 files and link about 10000 files.<\/p>\n

A ‘du -sh’ on a 5.0-ndb clone says 232MB. After ‘bk -r clean’ (i.e. have no files checked out) is 136MB. i.e. you are saving 136MB for cloning with the -l option. Now, if BK (and us really), we’d check out files always as read-only, which would also be hard linked across clones. This would further save 96MB per clone. When you check out a file for writing, it then creates a copy of it – so you only use the disk space for the files your editing (and only use the disk space for the sfile when you check in).<\/p>\n

“What about directory disk usage?” I hear you ask. Well, a MySQL 5.0 clone has 1053 directories. So, for each clone, we’re using 1053 inodes. On XFS with 256 byte inodes (the default) this works out to be 263kb of disk space. Let’s consider a bad case where we need another block of disk space for each of these directories (to hold all those directory entries for those 10000 files). This would mean 4MB of disk usage.<\/p>\n

For a really clever bk (and config) you could have a clean tree of 232MB and each clone only take up less than 5MB of disk space. As it is, the checked out files aren’t hard linked, so each clone takes up an extra 96MB of disk space.<\/p>\n

In essence, we’re using 19 times more disk space per clone than we need to.<\/p>\n

Now, if ccache is really clever (i don’t know if it is) it would hardlink object files so we use even *less* disk space for a compiled tree (a ‘du -sh’ on a compiled max-debug tree is about 1GB).<\/p>\n

Why have I gone on about this so much? Well, disk is cheap – except in laptops where replacing disk or getting a big one isn’t easy or cheap.<\/p>\n

Also, backup is expensive, slow and awkward.<\/p>\n

maybe at some later time i’ll talk about the theoretical IO usage of some things….<\/p>\n","protected":false},"excerpt":{"rendered":"

(inspired by jimw talking about it on Planet MySQL) I take a bit of a different approach… I’ve got directories for 4.0, 4.1 and 5.0, and within them, i have clones of the main ndb tree (called ndb, so there’s … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[14],"tags":[],"class_list":["post-334","post","type-post","status-publish","format-standard","hentry","category-mysql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-5o","jetpack-related-posts":[{"id":1383,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/02\/02\/row-id-in-mysql-and-drizzle-and-the-engines\/","url_meta":{"origin":334,"position":0},"title":"row id in MySQL and Drizzle (and the engines)","author":"Stewart Smith","date":"2009-02-02","format":false,"excerpt":"Some database engines have a fundamental concept of a row id. The row id is everything you need to know to locate a row. Common uses include secondary indexes (key is what's indexed, value is rowid which you then use to lookup the row). One design is the InnoDB method\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":781,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/01\/23\/bitkeeper-unlock-s-magic\/","url_meta":{"origin":334,"position":1},"title":"bitkeeper unlock -s magic","author":"Stewart Smith","date":"2007-01-23","format":false,"excerpt":"If you ever get something like this: bk clone -lq ndb bug25567 clone: unable to readlock \/home\/stewart\/Documents\/MySQL\/5.1\/ndb then try this: bk unlock -s to remove stale locks. I have no idea how anybody is meant to come to the command from the error message... blindly guessing 'bk help unlock' worked\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":386,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/03\/29\/revision-control\/","url_meta":{"origin":334,"position":2},"title":"revision control","author":"Stewart Smith","date":"2005-03-29","format":false,"excerpt":"Brian Aker has blogged about BitKeeper versus CVS no doubt this has stemmed from somebody's rant on the BK license. Now, this is a valid rant, but, really - it's getting[1] old. Personally, I quite like the GNU Arch Revision control system. Unfortunately, the UI is sort of sucky and\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":914,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/11\/09\/mysql-5122-stew2\/","url_meta":{"origin":334,"position":3},"title":"mysql-5.1.22-stew2","author":"Stewart Smith","date":"2007-11-09","format":false,"excerpt":"New: Updated NDB Compressed LCP and BACKUP patches (now with O_DIRECT support) InnoDB patch for Windows that should give ~5x improvement on commits\/sec (Bug31876) Everything in current telco-6.3 tree (ndb ~6.3.5) Lots of NDB improvements and new features over regular 5.1. WL3686 Remove read before update WL2680 NDB Batched Update\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1013,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/02\/20\/a-world-of-fail\/","url_meta":{"origin":334,"position":4},"title":"A world of FAIL","author":"Stewart Smith","date":"2008-02-20","format":false,"excerpt":"=================================== ERROR ==================================== File holyfoot\/hf@mysql.com\/deer.(none)|mysql-test\/r\/bdb_notembedded.result|20061113160642|60022|276fa5181da9a588 is marked as gone in this repository and therefor cannot accept updates. The fact that you are getting updates indicates that the file is not gone in the other repository and could be restored in this repository. if you want to \"un-gone\" the file(s) using\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":904,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/10\/16\/mysql-5122ish-stew1\/","url_meta":{"origin":334,"position":5},"title":"MySQL 5.1.22(ish)-stew1","author":"Stewart Smith","date":"2007-10-16","format":false,"excerpt":"I've decided to publish my patch series. The goal of the -stew patches is to collect things I find interesting and that at some point could (should) make it into the main MySQL tree (even if others don't think so). It's not designed for use in production.. I don't really\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/334","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=334"}],"version-history":[{"count":2,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/334\/revisions"}],"predecessor-version":[{"id":2826,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/334\/revisions\/2826"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=334"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=334"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=334"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}