{"id":1765,"date":"2009-12-11T08:00:34","date_gmt":"2009-12-10T22:00:34","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1765"},"modified":"2014-10-08T09:15:27","modified_gmt":"2014-10-07T23:15:27","slug":"the-schema-protobuf-message-drizzles-metadata-on-a-schema","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2009\/12\/11\/the-schema-protobuf-message-drizzles-metadata-on-a-schema\/","title":{"rendered":"The Schema protobuf message: Drizzle&#8217;s metadata on a schema"},"content":{"rendered":"<p>I&#8217;ve previously talked about table metadata in <a href=\"http:\/\/www.drizzle.org\">Drizzle<\/a> and how we use the table protobuf message to describe a table (see <a href=\"http:\/\/www.flamingspork.com\/blog\/2009\/12\/09\/drizzle-frm-replacement-the-table-proto\/\">Drizzle FRM Replacement<\/a> and others). The model in <a href=\"http:\/\/www.drizzle.org\">Drizzle<\/a> is that the engine is responsible for its metadata. For schemas (you may be thinking &#8216;database&#8217; but we&#8217;re moving to the Schema terminology in Drizzle) we also have a small amount of metadata.<\/p>\n<p>The protobuf message is specified in <a href=\"https:\/\/bazaar.launchpad.net\/~drizzle-trunk\/drizzle\/7.2\/annotate\/head:\/drizzled\/message\/schema.proto\">drizzled\/message\/schema.proto<\/a> and is incredibly short. In fact, here it is in its\u00c2\u00a0entirety:<\/p>\n<pre>\r\n<table id=\"logentries\" style=\"caption-side: top; width: 965px; margin-top: 0px; margin-right: auto; margin-bottom: 10px; margin-left: auto; background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #ffffff; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; border-collapse: collapse; background-position: initial initial; background-repeat: initial initial; padding: 0px; border: initial none initial;\" border=\"0\">\r\n<tbody>\r\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #f8fbfd; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\r\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L1\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L1\">1<\/a><\/td>\r\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\r\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\r\n<pre style=\"margin: 0px;\"><span>package drizzled.message;<\/span><\/pre>\n<\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #eef6fa; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L2\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L2\">2<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\n<pre style=\"margin: 0px;\"><span>option optimize_for = SPEED;<\/span><\/pre>\n<\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #f8fbfd; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L3\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L3\">3<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\"><\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #f8fbfd; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L4\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L4\">4<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\n<pre style=\"margin: 0px;\"><span>message Schema {<\/span><\/pre>\n<\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #f8fbfd; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L5\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L5\">5<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\n<pre style=\"margin: 0px;\"><span>  required string name = 1;<\/span><\/pre>\n<\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #eef6fa; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L6\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L6\">6<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\n<pre style=\"margin: 0px;\"><span>  optional string collation = 2;<\/span><\/pre>\n<\/td>\n<\/tr>\n<tr style=\"background-image: initial; background-attachment: initial; background-origin: initial; background-clip: initial; background-color: #f8fbfd; font: normal normal normal 14px\/normal sans-serif; background-position: initial initial; background-repeat: initial initial; padding: 0px; margin: 0px;\">\n<td style=\"vertical-align: top; width: 37px; font: normal normal normal 12px\/normal monospace; text-align: right; border: 1px none #d2d2d2;\"><a id=\"L7\" style=\"color: #0f597f; text-decoration: none; font-weight: normal; outline-style: none; outline-width: initial; outline-color: initial;\" href=\"#L7\">7<\/a><\/td>\n<td style=\"vertical-align: top; width: 70px; text-align: center; border: 1px none #d2d2d2;\"><\/td>\n<td style=\"vertical-align: top; width: auto; font: normal normal normal 12px\/normal monospace; border: 1px none #d2d2d2;\">\n<pre style=\"margin: 0px;\"><span>}<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>We don&#8217;t keep an awful lot of metadata about schemas. A Schema has a name and it has a default collation.<\/p>\n<p>You can also read the db.opt file directly using the provided (and very simple) <a href=\"http:\/\/bazaar.launchpad.net\/~drizzle-developers\/drizzle\/development\/annotate\/head:\/drizzled\/message\/schema_reader.cc\">schema_reader<\/a> utility.<\/p>\n<p>In the near future, we could have CREATE DATABASE and CREATE SCHEMA replicated via this protobuf message. This would make it extremely easy to parse for utilities parsing the replication stream.<\/p>\n<p>We&#8217;ll also (rather shortly) have key,value pairs for options to CREATE SCHEMA\/CREATE DATABASE. More on that later :)<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve previously talked about table metadata in Drizzle and how we use the table protobuf message to describe a table (see Drizzle FRM Replacement and others). The model in Drizzle is that the engine is responsible for its metadata. For &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2009\/12\/11\/the-schema-protobuf-message-drizzles-metadata-on-a-schema\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[75],"tags":[70,185,204],"class_list":["post-1765","post","type-post","status-publish","format-standard","hentry","category-drizzle-work-et-al","tag-drizzle","tag-protobuf","tag-schema"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-st","jetpack-related-posts":[{"id":1813,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/12\/writing-a-storage-engine-for-drizzle-part-2-create-table\/","url_meta":{"origin":1765,"position":0},"title":"Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE","author":"Stewart Smith","date":"2010-03-12","format":false,"excerpt":"The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1763,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/12\/12\/the-table-protobuf-message-format\/","url_meta":{"origin":1765,"position":1},"title":"The Table protobuf message format","author":"Stewart Smith","date":"2009-12-12","format":false,"excerpt":"If you've ever opened up drizzled\/message\/table.proto in the Drizzle source tree you will have seen what's in the table message: the structure that describes a database table in Drizzle. Previously I've talked about the Table message more generally, giving a fair bit of history of the FRM file and how\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1650,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/05\/27\/pluggable-metadata-stores-or-the-revenge-of-table-discovery\/","url_meta":{"origin":1765,"position":2},"title":"Pluggable Metadata stores (or&#8230; the revenge of table discovery)","author":"Stewart Smith","date":"2009-05-27","format":false,"excerpt":"Users of the ARCHIVE or NDB storage engines in MySQL may be aware of a MySQL feature known as \"table discovery\". For ARCHIVE, you can copy the archive data file around between servers and it magically works (you don't need to copy the FRM). For MySQL Cluster (NDB) it works\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2121,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/08\/20\/show-create-table-using-table-protobuf-message\/","url_meta":{"origin":1765,"position":3},"title":"SHOW CREATE TABLE using table protobuf message","author":"Stewart Smith","date":"2010-08-20","format":false,"excerpt":"... and really testing the replication code path for CREATE TABLE. So, for a very long time now, Drizzle has been using a protobuf based structure to describe the structure of tables. The idea was to be able to have engines rather painlessly generate this structure themselves (which several now\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1655,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/06\/09\/drizzle-pluggable-metadatastore-or-no-table-definition-file-on-disk\/","url_meta":{"origin":1765,"position":4},"title":"Drizzle pluggable MetadataStore (or: no table definition file on disk)","author":"Stewart Smith","date":"2009-06-09","format":false,"excerpt":"My code is shaping up rather nicely (see https:\/\/code.launchpad.net\/~stewart\/drizzle\/discovery) and I'm planning to submit a merge-request for it later today. I'm about to commit code that implements a MetadataStore for the ARCHIVE engine. This means that for ARCHIVE tables, you only have the .ARZ file on disk. The table definition\u2026","rel":"","context":"In &quot;General&quot;","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2323,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/03\/17\/things-ive-done-in-drizzle\/","url_meta":{"origin":1765,"position":5},"title":"Things I&#8217;ve done in Drizzle","author":"Stewart Smith","date":"2011-03-17","format":false,"excerpt":"When writing my Dropping ACID: Eating Data in a Web 2.0 Cloud World talk for LCA2011 I came to the realisation that I had forgotten a lot of the things I had worked on in MySQL and MySQL Cluster. So, as a bit of a retrospective as part of the\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1765","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1765"}],"version-history":[{"count":3,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1765\/revisions"}],"predecessor-version":[{"id":3830,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1765\/revisions\/3830"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1765"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1765"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1765"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}