{"id":854,"date":"2007-06-27T11:46:37","date_gmt":"2007-06-27T01:46:37","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/2007\/06\/27\/run-backup-run\/"},"modified":"2014-10-08T09:22:10","modified_gmt":"2014-10-07T23:22:10","slug":"run-backup-run","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2007\/06\/27\/run-backup-run\/","title":{"rendered":"Run Backup, Run!"},"content":{"rendered":"

Over the past N weeks\/couple of months, we’ve been making a number of improvements to how backups are done in MySQL Cluster.<\/p>\n

Once you get to large data sets, you start to really care about how long a backup takes.<\/p>\n

Traditionally, MySQL Cluster has been in-memory only. The way to back this up is to just write from memory to disk (rate limited) and synchronised across the cluster.\u00c2\u00a0 Since memory is really fast (compared to the rate we’re writing out to disk) – never had a problem.<\/p>\n

In MySQL 5.1 (and Cluster Carrier Grade Edition- CGE), disk based attributes are supported. This means that a row has both in memory and disk based parts.\u00c2\u00a0 As we all (should) know, disk seeks take a very long time. We don’t want to seek.<\/p>\n

So, at some point recently we changed the scanning order from in-memory order (which previously made perfect sense) to on disk order. Randomly seeking through RAM is much cheaper than all the disk seeks. This greatly improved backup performance.<\/p>\n

We also did some read-ahead work, which again, greatly improved performance.<\/p>\n

Today, I see mail from Jonas about changing the way we read tuples for backup (and LCP) to make it even more efficient (READ_PACKED). This should also reduce CPU usage for LCP\/Backup… which is a casual issue. I should really take the time to look closely at this and review.<\/p>\n

I also wrote a patch to the code in NDB that writes files to disk to write a compressed gzio stream instead of an uncompressed one. This happens in a different thread, so potentially using one of those CPU cores that ndb wouldn’t otherwise use… and also dramatically reducing the amount of data written to disk…. this patch isn’t in any tree yet, and I’ve yet to try it with the READ_PACKED patch, which together should work rather well.<\/p>\n

I also need to grab Brian<\/a> at some point and find out why azio (as used by the ARCHIVE engine) doesn’t work the same way as gzio for basic stream writing…<\/p>\n","protected":false},"excerpt":{"rendered":"

Over the past N weeks\/couple of months, we’ve been making a number of improvements to how backups are done in MySQL Cluster. Once you get to large data sets, you start to really care about how long a backup takes. … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[14],"tags":[123,117,124,628,54],"class_list":["post-854","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-azio","tag-backup","tag-compression","tag-mysql","tag-ndb"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-dM","jetpack-related-posts":[{"id":814,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/04\/03\/qa-with-mysql-cluster-content-my-2c-thrown-in\/","url_meta":{"origin":854,"position":0},"title":"Q&A with MySQL Cluster content (my 2c thrown in)","author":"Stewart Smith","date":"2007-04-03","format":false,"excerpt":"Ivan mentions the Q and As from a Q&A session in which MySQL Cluster is mentioned - I thought I'd add my perspective here as well: Q from Matthew: When are we likely to see disk based indexing for ndb? Disk based indexing is planned in one of the future\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":616,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/03\/11\/correcting-some-incorrect-information-about-cluster\/","url_meta":{"origin":854,"position":1},"title":"Correcting some incorrect information about Cluster","author":"Stewart Smith","date":"2006-03-11","format":false,"excerpt":"While googling this morning, I came across: Arguments in favour of PostgreSQL - MoodleDocs. Unfortunately, there's some factual errors about both MySQL and MySQL Cluster. For a start, it seems as though the comparison is Postgresql versus MyISAM. Now, this isn't exactly a fair comparison when it comes to some\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3304,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/05\/13\/the-mysql-cluster-storage-engine\/","url_meta":{"origin":854,"position":2},"title":"The MySQL Cluster storage engine","author":"Stewart Smith","date":"2013-05-13","format":false,"excerpt":"This is one close to my heart. I've recently written on other storage engines:\u00c2\u00a0Where are they now: MySQL Storage Engines,\u00c2\u00a0The MERGE storage engine: not dead, just resting\u00e2\u20ac\u00a6. or forgotten\u00c2\u00a0and The MEMORY storage engine. Today, it's the turn of MySQL Cluster. Like InnoDB, MySQL Cluster started outside of MySQL. Those of\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1229,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/10\/14\/mysql-cluster-ndb-on-win32-progress\/","url_meta":{"origin":854,"position":3},"title":"MySQL Cluster (NDB) on Win32 progress","author":"Stewart Smith","date":"2008-10-14","format":false,"excerpt":"Many things have been happenning in the land of NDB on Win32 as of late. I've fixed about 700 compiler warnings (some of which were real bugs) leaving about 161 to go on Win32 (VS2003). We're getting a few more warnings on Win64 (some of which look merely semantic, while\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1650,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/05\/27\/pluggable-metadata-stores-or-the-revenge-of-table-discovery\/","url_meta":{"origin":854,"position":4},"title":"Pluggable Metadata stores (or… the revenge of table discovery)","author":"Stewart Smith","date":"2009-05-27","format":false,"excerpt":"Users of the ARCHIVE or NDB storage engines in MySQL may be aware of a MySQL feature known as \"table discovery\". For ARCHIVE, you can copy the archive data file around between servers and it magically works (you don't need to copy the FRM). For MySQL Cluster (NDB) it works\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3899,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/11\/11\/mysql-cluster-on-power8\/","url_meta":{"origin":854,"position":5},"title":"MySQL Cluster on POWER8","author":"Stewart Smith","date":"2014-11-11","format":false,"excerpt":"So, I've written previously on MySQL on POWER, and today is a quick bit of news about MySQL Cluster on POWER - specifically MySQL Cluster 7.3.7. I ran into three main issues in getting some flexAsync benchmark results. One of them was the fact that I wanted to do this\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=854"}],"version-history":[{"count":3,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/854\/revisions"}],"predecessor-version":[{"id":3875,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/854\/revisions\/3875"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}