It’s coming up on a year since I started working full time on Drizzle. So, I got a bit reflective…<\/p>\n
Have we done things that I (and others) really wanted done? Back in 2007, I wrote my top 5 wishlist for the MySQL Server<\/a>.<\/p>\n I am not going to pretend I speak for the MySQL development team; I’m just trying to evaluate how Drizzle is doing against some wishlists that (to me) embodied some of the reasons we started Drizzle.<\/p>\n Please think of this as “database server wishlists” and comparing them against Drizzle….<\/p>\n My wishlist was:<\/p>\n 5. Six-monthly release cycles<\/strong><\/p>\n Done. Not only does Drizzle have milestone releases, but we’re also dropping tarballs every two weeks (currently for the bell milestone<\/a>). We’re also doing a decent job of keeping trunk free of massive breakage.<\/p>\n 4. Much more in depth automated testing<\/strong><\/p>\n Done (and in progress). We have drizzle-automation<\/a> running things all the time. Hudson (and buildbot) test across many platforms (what pushbuild did and more when inside MySQL) before<\/strong> code hits trunk. We also have regular performance benchmarks<\/a> that we compare across versions, crash-me, the random query generator<\/a> as well as checking that we don’t regress in code size (via sloccount)<\/a>.<\/p>\n 3. Sane build system<\/strong><\/p>\n (slightly distorting my original words). Well, we’re not quite at the ready for packaging in distributions stage, but a debug or non debug build is just -g and\u00c2\u00a0optimization\u00c2\u00a0level for the compiler, plugins are using autofoo to work otu if they can be built… so yeah, this is pretty sane.<\/p>\n We also are building with -Werror (and more!) which increases code quality no end.<\/p>\n So, mostly done.<\/p>\n 3.5. (yes, i have a 3.5): Kill HPUX<\/strong><\/p>\n Done.<\/p>\n 2. Increased liberal use of asserts<\/strong><\/p>\n An in-progress thing, but the better compiler warnings have won us a lot<\/strong>.<\/p>\n 1. Pluggable data dictionary<\/strong><\/p>\n Not only that, but done away with FRM totally. Really happy with this.<\/p>\n What about other peoples wishlist?<\/p>\n Kostya<\/strong> had one:<\/p>\n 1. Remove excessive fuss. <\/strong><\/p>\n i.e. “just do it”.<\/p>\n I think we’re doing really well with this for Drizzle. Plugins are pretty easy to get merged, and if your patches to the kernel are good, they’re also easy. Big changes can be harder, but in the end it has turned out well.<\/p>\n 2. Open the development process.<\/strong><\/p>\n Done. There is no internal wiki, there are no “committers” versus “non-committers”.. everything is judged on merit of the idea\/code. Sometimes the most valuable contribution is somebody telling you their real world experience.<\/p>\n 3. Get to a normal release schedule.<\/strong><\/p>\n Done.<\/p>\n 4. Establish productive relationship with the majority of users.<\/strong><\/p>\n I think drizzle-discuss mailing list is doing quite well in this regard. Is quite active with discussion.<\/p>\n 5. Find a way to do incompatible changes with minimal pain for users. <\/strong><\/p>\n We’ll see how we go :)<\/p>\n \n Ronald also had a list<\/a>:<\/p>\n 1. Real time Query Monitoring<\/strong><\/p>\n With gearman logging and my recent experimentation with using CPU performance counters<\/a> I think we’ll end up somewhere rather awesome.<\/p>\n If you’re looking for MySQL monitoring though, the MySQl Enterprise query monitoring stuff looks pretty good to me.<\/p>\n 2. Consistent Release Cycles<\/strong><\/p>\n We’re doing pretty well so far!<\/p>\n 3. INFORMATION_SCHEMA Extensions<\/strong><\/p>\n We’ve inherited the architecture from MySQL 5.1 (and 6.0) of being able to pretty easily add INFORMATION_SCHEMA tables and improved it. It’s currently pretty easy to add them. We also have ongoing work having an INFORMATION_SCHEMA storage engine which means that you won’t have to have the I_S tables be\u00c2\u00a0materialized\u00c2\u00a0every time you query them.<\/p>\n 4. Online table maintenance<\/strong><\/p>\n All progress has been due to Storage Engine authors. With the data dictionary work though, this gets easier and saner to do.<\/p>\n 5. Published benchmarks<\/strong><\/p>\n We’re encouraging others who will be more objective :) Although we also do regular performance regression tests as part of our standard development process.<\/p>\n Dormando also had a list<\/a> (complete with “there is no five”):<\/p>\n 1) Logical separation of connections from threads<\/strong><\/p>\n We have this in Drizzle through plugins. Interesting ones are pool_of_threads (fixed number of threads), multithread (thread per connection) and single_thread (one thread).<\/p>\n 2) A more modular core<\/strong><\/p>\n We’re very much doing well here. It’s a long process, but I’m quite impressed by our progress.<\/p>\n 3) Better replication (better replication management\/protocol?)<\/strong><\/p>\n The work being done on Drizzle replication is really exciting. I love the fact that modularity is encouraged and the ability to replace any bit you want easily, as well as read the replication stream in about any language you want.<\/p>\n 4) Better test suite<\/strong><\/p>\n I may never be 100% happy with a test suite, but we’re doing good…<\/p>\n PeterZ<\/a>‘s view is always interesting, and he had one too:<\/p>\n 1. Be Pluggable<\/strong><\/p>\n Check. (and also, of course, in progress)<\/p>\n 2. Be Scalable<\/strong><\/p>\n We’ve done a lot of work scaling on many CPUs and many connections. Really, 8 concurrent database connections just isn’t interesting<\/strong>. We ever run as part of our regression suite up to 2048 concurrent connections<\/a>.<\/p>\n 3. Be Distributed<\/strong><\/p>\n With the new protocol work to have built in sharding, plugins for logging and replication via Gearman, we’re getting better.<\/p>\n 4. Be Solid<\/strong><\/p>\n This will be a test for us. I think we should end up pretty good because of a number of reasons:<\/p>\n So I hope<\/strong> we’re going to be okay here.<\/p>\n 5. Don’t forget about the roots<\/strong><\/p>\n The group of us working at Sun on Drizzle have said we want to focus on being awesome for large scale Web apps while enabling others to make Drizzle good for other things. I think this is the right approach to not forget our roots (and target users) while allowing it to be awesome for any use somebody wants to have of it.<\/p>\n From a Storage Engine author PoV, Paul had some insights<\/a> while thinking about PBXT:<\/p>\n 1. A generic engine test suite<\/strong><\/p>\n We’re doing pretty well… the whole Drizzle test suite runs with InnoDB, and doesn’t require much change to get going with another transactional engine. The proof is in the Drizzle PBXT branch! But I also think we could do better and have a test suite more directed at each part of an engine (including error cases!).<\/p>\n 2. Internal APIs<\/strong><\/p>\n Paul mentions FRMs, which are gone :) In their place is a simple interface that engines can implement for ever increased functionality (i.e. they own their own metadata). We’re getting better in other places too.<\/p>\n 3. Customizable table and column attributes<\/strong><\/p>\n MariaDB has this now, and we have space in our table definition proto, but not at the parser level (yet).<\/p>\n 4. Push-down restrict and join conditions.<\/strong><\/p>\n Not yet, and not for a little while.<\/p>\n 5. Custom data types<\/strong><\/p>\n It’ll be great when we rework the type system even more so that this really is as easy as it should be – not only from a SQL level but also for adding new types as server modules.<\/p>\n Finally, Antony’s wish list<\/a>:<\/p>\n 1. Modular Architecture<\/strong><\/p>\n Covered.<\/p>\n 2. libmysys as a separate project<\/strong><\/p>\n We’ve removed it where we can, and are using gnulib where we can. It very much improves the situation when you ditch weird-ass platforms and assume some level of POSIX.<\/p>\n 3. New\/modular parser<\/strong><\/p>\n We’re getting close to a stage where you could load a different parser… not there, but relatively close. It would still be messy, but a lot better than even 6 months ago.<\/p>\n 4. Unit tests for server components<\/strong><\/p>\n With our move towards modularity, this is actually getting possible!<\/p>\n 5. Aggregate Stored functions and External Stored Procedures<\/strong><\/p>\n We don’t currently have either, we have decent thoughts though.<\/p>\n Antony also cheated and added a few more:<\/p>\n A new Recursive Descent parser<\/strong><\/p>\n The required work to be able to replace the parser is in progress.<\/p>\n Abstract Syntax Trees<\/strong><\/p>\n See above.. getting the pre-work done.<\/p>\n SCTP and\/or link aggregation<\/strong><\/p>\n We’ll see improvements around this with the new drizzle protocol.<\/p>\n Parsing within the client<\/strong><\/p>\n We’ve had some very good discussion on the drizzle-discuss list. We’ll no doubt have something to help remove more of the cycles used in executing a query.<\/p>\n Integrated Federation<\/strong><\/p>\n That’s the game plan :)<\/p>\n Elimination of FRM Files<\/strong><\/p>\n I never get tired of saying this is done :)<\/p>\n Elimination of errmsg.sys<\/strong><\/p>\n We’re now just using gettext, like every other free software project on the planet. Although I think we could take a few steps in making errors more easily parsable by code.<\/p>\n \n So how do we stack up?<\/p>\n I think we’re doing pretty well. There’s still a lot of work to get where we want to be, but it’s amazing how much progress we’ve made in the short time we’ve been around.<\/p>\n I also just\u00c2\u00a0realized\u00c2\u00a0I missed Jay’s list<\/a>… but we’re doing pretty well there too.<\/p>\n","protected":false},"excerpt":{"rendered":" It’s coming up on a year since I started working full time on Drizzle. So, I got a bit reflective… Have we done things that I (and others) really wanted done? Back in 2007, I wrote my top 5 wishlist … Continue reading \n
\n