{"id":1650,"date":"2009-05-27T01:43:49","date_gmt":"2009-05-26T15:43:49","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1650"},"modified":"2009-05-27T01:43:49","modified_gmt":"2009-05-26T15:43:49","slug":"pluggable-metadata-stores-or-the-revenge-of-table-discovery","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2009\/05\/27\/pluggable-metadata-stores-or-the-revenge-of-table-discovery\/","title":{"rendered":"Pluggable Metadata stores (or&#8230; the revenge of table discovery)"},"content":{"rendered":"<p>Users of the ARCHIVE or NDB storage engines in MySQL may be aware of a MySQL feature known as &#8220;table discovery&#8221;. For ARCHIVE, you can copy the archive data file around between servers and it magically works (you don&#8217;t need to copy the FRM). For MySQL Cluster (NDB) it works so that when you CREATE TABLE on another MySQL server,\u00c2\u00a0 other MySQL servers can get the FRM for these tables from the cluster.<\/p>\n<p>With my work to replace the FRM with a protobuf structure in Drizzle and clean up parts of the API around it, this feature didn&#8217;t really survive in any working state.<\/p>\n<p>Instead, I&#8217;m now doing things closer to the right way: pluggable metadata stores. The idea being that the whole &#8220;table proto on disk&#8221; (in MySQL it&#8217;s the FRM, but in Drizzle we&#8217;re now using a protobuf structure) code is pluggable and could be replaced by an implementation specific to an engine (e.g. the innodb or ndb data dictionaries) or a different gerenic one.<\/p>\n<p>Currently, the default plugin is the same way we&#8217;ve been doing it forever: file-per-table on disk in a directory that&#8217;s the database. The API has a nasty bit now (mmmm&#8230; table name encoding), but that&#8217;ll be fixed in the future.<\/p>\n<p>The rest of this week will be dedicated to plugging this into all the bits in the server that manipulate the files manually.<\/p>\n<p>With luck, I&#8217;ll have modified the ARCHIVE engine by then too so that there&#8217;ll just be the archive data file on disk with the table metadata stored in it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Users of the ARCHIVE or NDB storage engines in MySQL may be aware of a MySQL feature known as &#8220;table discovery&#8221;. For ARCHIVE, you can copy the archive data file around between servers and it magically works (you don&#8217;t need &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2009\/05\/27\/pluggable-metadata-stores-or-the-revenge-of-table-discovery\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[75,14],"tags":[],"class_list":["post-1650","post","type-post","status-publish","format-standard","hentry","category-drizzle-work-et-al","category-mysql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-qC","jetpack-related-posts":[{"id":1655,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/06\/09\/drizzle-pluggable-metadatastore-or-no-table-definition-file-on-disk\/","url_meta":{"origin":1650,"position":0},"title":"Drizzle pluggable MetadataStore (or: no table definition file on disk)","author":"Stewart Smith","date":"2009-06-09","format":false,"excerpt":"My code is shaping up rather nicely (see https:\/\/code.launchpad.net\/~stewart\/drizzle\/discovery) and I'm planning to submit a merge-request for it later today. I'm about to commit code that implements a MetadataStore for the ARCHIVE engine. This means that for ARCHIVE tables, you only have the .ARZ file on disk. The table definition\u2026","rel":"","context":"In &quot;General&quot;","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1745,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/12\/09\/drizzle-frm-replacement-the-table-proto\/","url_meta":{"origin":1650,"position":1},"title":"Drizzle FRM replacement: the table proto","author":"Stewart Smith","date":"2009-12-09","format":false,"excerpt":"Drizzle originally inherited the FRM file from MySQL (which inherited it from UNIREG). The FRM file stores metadata about a table; what columns it has, what type those columns are, what indexes, any default values, comments etc are all stored in the FRM. In the days of MyISAM, this worked\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1251,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/11\/06\/goodbye-frm-or-at-least-the-steps-to-it\/","url_meta":{"origin":1650,"position":2},"title":"Goodbye FRM (or at least the steps to it)","author":"Stewart Smith","date":"2008-11-06","format":false,"excerpt":"Since before MySQL was MySQL, there has been the .FRM file. Of course, what it really wanted to be was \".form\" -\u00c2\u00a0 a file that stored how to display a form on your (green) CRT. Jan blogged earlier in the year on this still being there, even in MySQL 5.1\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4180,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/09\/27\/mysql-removes-the-frm-7-years-after-drizzle-did\/","url_meta":{"origin":1650,"position":3},"title":"MySQL removes the FRM (7 years after Drizzle did)","author":"Stewart Smith","date":"2016-09-27","format":false,"excerpt":"The new MySQL 8.0.0 milestone release that was recently announced brings something that has been a looooong time coming: the removal of the FRM file. I was the one who implemented this in Drizzle way back in 2009 (July 28th 2009 according to Brian)- and I may have had a\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2041,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/06\/29\/enum-now-works-properly-in-drizzle\/","url_meta":{"origin":1650,"position":4},"title":"ENUM now works properly (in Drizzle)","author":"Stewart Smith","date":"2010-06-29","format":false,"excerpt":"Over at the Drizzle blog, the recent 2010-06-07 tarball was announced. This tarball release has my fixes for the ENUM type, so that it now works as it should. I was quite amazed that such a small block of code could have so many bugs! One of the most interesting\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2323,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/03\/17\/things-ive-done-in-drizzle\/","url_meta":{"origin":1650,"position":5},"title":"Things I&#8217;ve done in Drizzle","author":"Stewart Smith","date":"2011-03-17","format":false,"excerpt":"When writing my Dropping ACID: Eating Data in a Web 2.0 Cloud World talk for LCA2011 I came to the realisation that I had forgotten a lot of the things I had worked on in MySQL and MySQL Cluster. So, as a bit of a retrospective as part of the\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1650","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1650"}],"version-history":[{"count":2,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1650\/revisions"}],"predecessor-version":[{"id":1652,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1650\/revisions\/1652"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1650"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1650"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1650"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}