{"id":549,"date":"2006-01-11T13:08:57","date_gmt":"2006-01-11T03:08:57","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/2006\/01\/11\/bug-15695-and-ndb-initial-start\/"},"modified":"2006-01-11T13:08:57","modified_gmt":"2006-01-11T03:08:57","slug":"bug-15695-and-ndb-initial-start","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2006\/01\/11\/bug-15695-and-ndb-initial-start\/","title":{"rendered":"Bug 15695 and NDB initial start"},"content":{"rendered":"

The process for starting up a cluster is pretty interesting. Where, of course, “interesting” is translated to “complex”. There’s a lot of things you have to watch out for (namely you want one<\/strong> cluster, not two or ten or anything). You also want to actually start a cluster, not just wait forever for everybody to show up.<\/p>\n

Except in some situations. For example, initial start. With an initial start, you really want to have all the nodes present (you don’t want to run the risk of starting up two separate clusters!).<\/p>\n

Bug 15695<\/a> is a bug to do with Initial Start. If you have three nodes (a management node and two data nodes) and break the network connection just between the two data nodes, and then reconnect it (at the wrong time – where the wrong time means you trigger the bug) the cluster will never start. A workaround is to restart one of the data nodes and everything comes up.<\/p>\n

Note that this is just during initial start so it’s not a HA bug or anything. Just really annoying.<\/p>\n

This seems to get hit when people have firewalls stopping the nodes talking to each other and then fix the firewall (but not shutting down the cluster).<\/p>\n

As is documented in the bug, you can replicate this with some iptables<\/a> foo.<\/p>\n

One of the main blocks involed in starting the cluster (and managing it once it’s up) is the Quorum manager – QMGR. You’ll find the code in ndb\/src\/kernel\/blocks\/qmgr\/. You’ll also find some in the older CMVMI (Cluster Manager Virtual Machine Interface).<\/p>\n

A useful thing to do is to define DEBUG_QMGR_START in your build. This gives you some debugging output printed to the ndb_X_out.log file.<\/p>\n

The first bit of code in QmgrMain.cpp is the heartbeat code. execCM_HEARTBEAT simply resets the number of outstanding heartbeats for the node that sent the heartbeat. Really simple signal there.<\/p>\n

During SR (System Restart) there is a timeout period for which we try to wait for nodes to start. This means we’ll be starting the cluster with the most number of nodes present (it’s much easier doing a SR with as many nodes as possible than doing NR – Node Recovery – on lots of nodes). NR requires copying of data over the wire. SR probably doesn’t. Jonas is working on optimised node recovery which is going to be really needed for disk data. This will only copy the changed\/updated data over the wire instead of all data that that node needs. Pretty neat stuff.<\/p>\n

We implement the timeout by sending a delayed signal to ourself. Every 3 seconds we check how the election of a president is going. If we reach our limit (30seconds) we try to start the cluster anyway – not allowing other nodes to join).<\/p>\n

Current problem is that each node in this two node not-quite-yet cluster thinks it has won the election and so switches what state it’s in to ZRUNNING (see Qmgr.h) hence stopping the search for other nodes. When the link between the two nodes is brought back up – hugs and puppies do not ensue.<\/p>\n

I should have a patch soon too.<\/p>\n

For a more complete explanation on the stages of startup, have a look at the text files in ndb\/src\/kernel\/blocks. Start.txt is a good one to read.<\/p>\n

\n","protected":false},"excerpt":{"rendered":"

The process for starting up a cluster is pretty interesting. Where, of course, “interesting” is translated to “complex”. There’s a lot of things you have to watch out for (namely you want one cluster, not two or ten or anything). … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[14],"tags":[],"class_list":["post-549","post","type-post","status-publish","format-standard","hentry","category-mysql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-8R","jetpack-related-posts":[{"id":805,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/03\/17\/ndb-online-add-node-progress-or-rather-testing-it\/","url_meta":{"origin":549,"position":0},"title":"NDB Online Add Node Progress (or rather, testing it)","author":"Stewart Smith","date":"2007-03-17","format":false,"excerpt":"So, the sitch as of today: Added ndb_mgm_set_configuration() call to the mgmapi - which is not-so-casually evil API call that sends a packed ndb_mgm_configuration object (like what you get from ndb_mgm_get_configuration) to the management server, who then resets its lists of nodes for event reporting and for ClusterMgr and starts\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1783,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/02\/03\/ndbinfo-with-sql-hits-beta\/","url_meta":{"origin":549,"position":1},"title":"NDB$INFO with SQL hits beta","author":"Stewart Smith","date":"2010-02-03","format":false,"excerpt":"\u00ef\u00bb\u00bfBernhard blogged over at http:\/\/ocklin.blogspot.com\/2010\/02\/mysql-cluster-711-is-there.html that MySQL Cluster 7.1.1 Beta has been released. The big feature (from my point of view) is the SQL interface on top of NDB$INFO. This means there is now full infrastructure from the NDB data nodes right out to SQL in the MySQL Server for\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3901,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/11\/12\/preliminary-mysql-cluster-benchmark-results-on-power8\/","url_meta":{"origin":549,"position":2},"title":"Preliminary MySQL Cluster benchmark results on POWER8","author":"Stewart Smith","date":"2014-11-12","format":false,"excerpt":"Yesterday, I got the basics going for MySQL Cluster on POWER. Today, I finished up a couple more patches to improve performance and ran some benchmarks. This is on a 3.7Ghz POWER8 machine with non-balanced memory (only 2 of the 4 NUMA nodes have memory, so we have less total\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":836,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/05\/23\/ndbrecord\/","url_meta":{"origin":549,"position":3},"title":"NdbRecord","author":"Stewart Smith","date":"2007-05-23","format":false,"excerpt":"Kristian is currently talking about the new NdbRecord API for the NDBAPI and how it relates to ha_ndbcluster (the mysql storage engine, which uses ndbapi to talk to the cluster nodes) and how it can be used by ndbapi applications. It looks like we're getting a really neat API that\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":616,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/03\/11\/correcting-some-incorrect-information-about-cluster\/","url_meta":{"origin":549,"position":4},"title":"Correcting some incorrect information about Cluster","author":"Stewart Smith","date":"2006-03-11","format":false,"excerpt":"While googling this morning, I came across: Arguments in favour of PostgreSQL - MoodleDocs. Unfortunately, there's some factual errors about both MySQL and MySQL Cluster. For a start, it seems as though the comparison is Postgresql versus MyISAM. Now, this isn't exactly a fair comparison when it comes to some\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":798,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/03\/09\/mgmapi-timeouts-and-resurrecting-the-online-add-node\/","url_meta":{"origin":549,"position":5},"title":"mgmapi timeouts and resurrecting the online add node","author":"Stewart Smith","date":"2007-03-09","format":false,"excerpt":"The other day I managed to send off what's nearly the final patches for adding proper timeout support to the MySQL Cluster management API. Jonas has had a bit of a look, found one thing I've missed, but it'll probably get in somewhere soon (probably the carrier grade edition first,\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/549","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=549"}],"version-history":[{"count":0,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/549\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}