{"id":939,"date":"2007-11-27T16:13:50","date_gmt":"2007-11-28T00:13:50","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/2007\/11\/27\/libeatmydata\/"},"modified":"2020-07-29T19:01:16","modified_gmt":"2020-07-30T02:01:16","slug":"libeatmydata","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2007\/11\/27\/libeatmydata\/","title":{"rendered":"libeatmydata"},"content":{"rendered":"

Following my successful linux.conf.au<\/a> talk “Eat My Data: How Everybody Gets POSIX File I\/O Wrong<\/a>“, I started to feel the need to easily be able to have my data eaten.<\/p>\n

Okay, not quite. However, when you’ve written your software properly, so it uses fsync() correctly, opening files with O_SYNC or whatever – tests take longer as you’re having to wait for things to hit the rust.<\/p>\n

So….. LD_PRELOAD=libeatmydata.so to the rescue! With a POSIX compliant fsync() (that does nothing) and filtering on open(2), it can take your test run times down dramatically.<\/p>\n

The only time you shouldn’t use it for your tests is when you end up crashing the machine to test durability (i.e. when the OS doesn’t have the opportunity to cleanly write out the data to disk).<\/p>\n

See the libeatmydata project page<\/a>: http:\/\/www.flamingspork.com\/projects\/libeatmydata\/<\/a><\/p>\n

and the bazaar<\/a> repository: http:\/\/www.flamingspork.com\/src\/libeatmydata<\/a><\/p>\n

(it’s seemed to have saved somewhere between 20 and 30% of the time for innodb\/ndb tests in mysql-test-run).<\/p>\n","protected":false},"excerpt":{"rendered":"

Following my successful linux.conf.au talk “Eat My Data: How Everybody Gets POSIX File I\/O Wrong“, I started to feel the need to easily be able to have my data eaten. Okay, not quite. However, when you’ve written your software properly, … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[2,14,758],"tags":[633,628,54],"class_list":["post-939","post","type-post","status-publish","format-standard","hentry","category-life-the-universe-and-everything","category-mysql","category-talks","tag-code","tag-mysql","tag-ndb"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-f9","jetpack-related-posts":[{"id":4264,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/11\/27\/ten-years-of-libeatmydata\/","url_meta":{"origin":939,"position":0},"title":"Ten years of libeatmydata!","author":"Stewart Smith","date":"2017-11-27","format":false,"excerpt":"So, ten years ago (how is that even possible... it seems like it was just a couple of years ago), there was the first commit in the libeatmydata repository (now in git on github rather than in bzr on launchpad). The first implementation was literally just this: #include #include\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1587,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/04\/08\/using-dtrace-to-find-out-why-solaris-is-so-slow-and-if-its-actually-solaris\/","url_meta":{"origin":939,"position":1},"title":"Using Dtrace to find out if the hardware or Solaris is slow (but really just working around the problem)","author":"Stewart Smith","date":"2009-04-08","format":false,"excerpt":"A little while ago, I was the brave soul tasked with making sure Drizzle was working properly and passing all tests on Solaris and OpenSolaris. Brian recently blogged about some of the advantages of also running on Solaris and the SunStudio compilers - more warnings from the compiler is a\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1866,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/22\/libeatmydata-for-solaris\/","url_meta":{"origin":939,"position":2},"title":"libeatmydata for Solaris","author":"Stewart Smith","date":"2010-03-22","format":false,"excerpt":"Thanks to Olly Betts, libeatmydata now has Solaris support as of release-15. So for those of you living on Solaris and actually doing a real fsync() during your test runs... do not fret! Feedback much appreciated (even better in patch form).","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":940,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/11\/27\/testing-crashes-in-file-io\/","url_meta":{"origin":939,"position":3},"title":"testing crashes in file IO","author":"Stewart Smith","date":"2007-11-27","format":false,"excerpt":"So Rusty gave me this idea for simulating crashes in libeatmydata to check durability of application data with stuff hitting the rust in interesting ways. Preserving fsync barriers, but mucking with the data before the last one and crashing. Then, recovery should all work. Interesting... something to soon hack on...","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2223,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/12\/06\/replication-log-inside-innodb\/","url_meta":{"origin":939,"position":4},"title":"Replication log inside InnoDB","author":"Stewart Smith","date":"2010-12-06","format":false,"excerpt":"The MySQL replication system has always had the replication log (\"binlog\") as a separate set of files on disk. Originally, this really didn't matter as, well, MyISAM wasn't transactional or crash safe so the binlog didn't need to be either. If you crashed on a busy write workload, your replication\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1935,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/05\/27\/a-warning-to-solaris-users-fsync-possibly-doesnt\/","url_meta":{"origin":939,"position":5},"title":"A warning to Solaris users…. (fsync possibly doesn’t)","author":"Stewart Smith","date":"2010-05-27","format":false,"excerpt":"Read the following: Oracle\/Sun ZFS Data Loss - Still Vulnerable OpenSolaris Bug 6880764 Data loss running Oracle on ZFS on Solaris 10, pre 142900-09 Linux has its fair share of dumb things with data too (ext3 not defaulting to using write barriers is a good one). This is however particularly\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/939","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=939"}],"version-history":[{"count":4,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/939\/revisions"}],"predecessor-version":[{"id":4636,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/939\/revisions\/4636"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=939"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=939"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=939"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}