__packed

Blog | rml

struct __packed s { … }This attribute tells GCC that a type or variable should be packed into memory, using the minimum amount of space possible, potentially disregarding alignment requirements. If specified on a struct or union, all variables therein are so packed. If specified on just a specific variable, only that type is packed. As an example, a structure with a char followed by an int would most likely find the integer aligned to a memory address not immediately following the char (say, three bytes later). The compiler does this by inserting three bytes of unused packing between the two variables. A packed structure lacks this packing, potentially consuming less memory but failing to meet architecture alignment requirements.

It should also be noted that non aligned data access on some architectures (e.g. ppc) can totally cripple performance. We’re talking orders of magnatude here. IBM has a good article on their developer site.

http://www-128.ibm.com/developerworks/power/library/pa-dalign/

Apple has some good tools for this too – and docs if i remember correctly.

rml on GCC extensions (and making them portable)

Blog | rml talks about a bunch of useful GCC extensions.

We generally don’t use this within mysql code. Due (no doubt) to portability issues. Maybe we should look closer at it these days. I wonder if we’d get any noticable improvement in NDB by adding it to our ndbrequire/ndbassert and CRASH_INSERTION tests. In some areas of code we do have a number of asserts.

The place to play is ndb/src/kernel/vm/pc.hpp

as actually triggering a ndbassert or ndbrequire is something that should never happen, unlikely() is a good thing to put there.

Interestingly enough, this is probably the place we also want to play with for dtrace. either that or with EXECUTE_DIRECT (and another place that Mikael mentioned on IRC last night.

My next task is to get qemu network interface working so i can get the source across to my VM of Solaris 10 and then start playing.

Of course, the other option is to actually install it somewhere (or shell out for vmware). It would be a lot faster then though.

PortaWiki going well (Wiki for portability issues)

PortaWiki is going pretty well. We’ve got a couple of contribututors at the moment and getting good little bits on the various oddities of various platforms. I encourage you to check it out and add things that you know.

It’d be great to have a MySQL section there too. In versions previous to 5.0 for example, you may get different results from some math operations on different platforms as we used the floating point stuff. In 5.0 we have precision math so this isn’t a problem – but it probably caused somebody to raise an eyebrow in the past. Volunteers?

how not to design a website

be a speaker at OSDC. Use their web system. I’m a smart guy and getting confused. Honestly guys, send in the slides for a presentation should not be a difficult task.

Contrast this with the recent AUUG conf where i attached files to an email and then clicked on the big Send button.

If via a web form it should be the *exact* equivilent. Not four different fields to specify the darn title of the thing.

must be time to use the OSDC conference registration/paper submission site

it’s annoying. grr.

but, on the other hand, I am speaking about MySQL 5.0 at OSDC.

This is even cooler as 5.0 has gone GA. So it’s not “upcoming features” it’s the “here and now”.

I’ll now have to release MemberDB 0.4 (the MySQL release). Converting the Linux Australia installation over at some point soon too. The 0.4 tree fixes enough bugs that it’s worth it (one of which Pia found the other day).

MySQL 5.0 is GA

We at MySQL AB have unleased MySQL 5.0 upon the world. It’s now declared GA (stable) and recommended for use everywhere you can possibly fit it (yes, this means brain implants and other things we dare not mention).

On DevZone there’s also a photo titled “MySQL 5.0 Development Team” taken at the DevConf earlier this year in Prague (you can see the pretty buildings in the background). I have a bunh of nice photos from there. I plan to put the scenic ones up somewhere at some point.

There’s even a poll for your favourite new feature. There isn’t an option for “version number divisible by 5”, but hey :)

I am going to get up and dance.

This does mean I will look stupid, but it’s dancing towards ice cream. Everybody deservers ice cream. Especially those with MySQL 5.0. In fact, if you don’t have it – no ice cream for you. You know you want ice cream…

PortaWiki – collaboration on portability issues

At AUUG2005 last week, Arjen, myself and others were discussing the idea of trying to assemble some sort of common resources that multiple projects can use to contribute and find out about portability issues they stumble across.

The idea being that we can all then learn from each other and write better, more portable software.

So, I’ve set something up.

I present, the incredibly bare (okay, not quite completely bare) PortaWiki.

Please add whatever stuff you find, you know or anything. No idea how this is going to work – I plan to let it evolve.

(Arjen tells me that Peter Gutmann should receive credit as he thinks he came up with the idea. Kudos to him).

http://www.flamingspork.com/portawiki/

Solaris 10 under QEMU

I’m currently watching a Solaris 10 install under QEMU on my laptop. It seems to be taking a while, but getting there.

(I got a Solaris 10 DVD in my AUUG shwag)

Basically, I want to play with DTrace and see how easy it is to do things with it. Solaris seems to be the requirement. I don’t want to have a partition for it nor run it as a primary OS. So, qemu it is.

I can also then use the funky disk image foo with qemu so that i don’t waste a lot of space (mmm… sparse disk images).

For a 7GB qemu-img created filesystem, used intirely as /, it seems that there’s 128MB overhead for having the file system. The installer is chugging away writing things and this seems to be constant.

So, all in all i should end up using a bit less than 3GB of real disk space for a full Solaris 10 install in a qemu image.

Serenity

Spotted in a review:
“The major difference between Lucas’ movies and Whedon’s movies is the superb dialogue.”

The next day after seeing Serenity I was with friends watching Firefly on DVD and at the end of ever episode (okay, part way through it – and generally more than once) someone would ask “why the hell was this cancelled? they must be stupid or something”.

Hey, ACPI buttons work!

rockin. The buttons at the top left of my keyboard work on ubuntu!

what looks like running away makes the screensaver come on (password protected). Useful. Replaces that shortcut i’d set.

The bluetooth button always has. plugs/unplugs the internal bluetooth adapter.

The world with a ring on it doesn’t do anything.

But the little wireless button works! Rockin. no more ifup/ifdown foo!

VGA Out and presentations

I can now give presentations from my laptop – yay.

It requires running the ATI binary drivers instead of the open source ones.

Then VGA out works without being squiggly. (that’s on my Asus V6V laptop with a Radeon X600 running Ubuntu Breezy) – there’ that should be enough google juice.

However, as if being binary only wasn’t crappy enough – suspend doesn’t work. So it’s open source drivers for all other times! I don’t use GL, so that doesn’t worry me. Of course, it may start to worry me what with all the neat cairo stuff and other accelleration coming… but not yet.

This should come in handy for the Melbourne MySQL Users Group meeting tomorrow night!

mouse gestures

I once thought mouse gestures were weird and crackful. A good way to hide away functionality and make things difficult.

Well… I’ve started using them. Currently I’m using Epiphany as my web broswer (although there’s enough nice firefox things out there that I may switch at some point… if i can get my saved passwords across). The mouse gestures here (naturally) use a different bloody button than the package I’m using on firefox (right button is more logical as laptops generally only have two. or one).

But talk about useful and fast. even just back and forward. maybe integrating this with a pop-up display of what actions you can do (for learning them) would really seal the deal.

do i really have people reading this drivel?

According to my web stats for last month there were 25778 hits on /blog/ and 6531 on /blog/wp-rss2.php. Since there were 1894 hits on /blog/wp-comments-post.php, that probably means that a bit of that is just spammers.

The weird metric of “visits” is 7860 for /blog.

Do I really have anything like this number of people reading what I write here?.

Please, somebody publically smack my ego back to where it belongs!

(err… maybe not, my ego isn’t into bondage).

a funky thing done last week…

still have to talk to people about standards for this sort of thing and all that. But as a first checkin – funkyness++!

mysql> select * from INFORMATION_SCHEMA.DATAFILES;  select * 
from INFORMATION_SCHEMA.TABLESPACES;
Empty set (0.03 sec)

Empty set (0.00 sec)

mysql> CREATE TABLESPACE ts1 ADD DATAFILE 'datafile.dat' USE 
LOGFILE GROUP lg1 INITIAL_SIZE = 12M ENGINE NDB;
Query OK, 0 rows affected (2.35 sec)

mysql> select * from INFORMATION_SCHEMA.DATAFILES;  select * 
from INFORMATION_SCHEMA.TABLESPACES;
+--------------+--------+--------------+----------+------+------------+
| NAME         | ENGINE | PATH         | SIZE     | FREE | TABLESPACE |
+--------------+--------+--------------+----------+------+------------+
| datafile.dat | NDB    | datafile.dat | 12582912 |   11 |            |
+--------------+--------+--------------+----------+------+------------+
1 row in set (0.00 sec)

+------+--------+---------+-------------+-----------------------+
| NAME | ENGINE | VERSION | EXTENT_SIZE | DEFAULT_LOGFILE_GROUP |
+------+--------+---------+-------------+-----------------------+
| ts1  | NDB    |       1 |     1048576 |                     0 |
+------+--------+---------+-------------+-----------------------+
1 row in set (0.00 sec)

mysql> CREATE TABLE t1 
(pk1 int not null primary key auto_increment,
 b int not null, 
c int not null) 
tablespace ts1 storage disk engine ndb; 
Query OK, 0 rows affected (0.62 sec)

mysql> insert into t1 (b,c) values (1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),
(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),
(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4),(1,2),(2,3),(3,4);
Query OK, 36 rows affected (0.11 sec)
Records: 36  Duplicates: 0  Warnings: 0

mysql> select * from INFORMATION_SCHEMA.DATAFILES;  select * 
from INFORMATION_SCHEMA.TABLESPACES; 
+--------------+--------+--------------+----------+------+------------+ 
| NAME         | ENGINE | PATH         | SIZE     | FREE | TABLESPACE |
+--------------+--------+--------------+----------+------+------------+
| datafile.dat | NDB    | datafile.dat | 12582912 |    9 |            |
+--------------+--------+--------------+----------+------+------------+
1 row in set (0.02 sec)

+------+--------+---------+-------------+-----------------------+
| NAME | ENGINE | VERSION | EXTENT_SIZE | DEFAULT_LOGFILE_GROUP |
+------+--------+---------+-------------+-----------------------+
| ts1  | NDB    |       1 |     1048576 |                     0 |
+------+--------+---------+-------------+-----------------------+
1 row in set (0.00 sec)

mysql> CREATE TABLESPACE ts2 ADD DATAFILE 'datafile2.dat' 
USE LOGFILE GROUP lg1 INITIAL_SIZE = 12M ENGINE NDB;
Query OK, 0 rows affected (2.18 sec)

mysql> select * from INFORMATION_SCHEMA.DATAFILES;  select * 
from INFORMATION_SCHEMA.TABLESPACES;
+---------------+--------+---------------+----------+------+------------+
| NAME          | ENGINE | PATH          | SIZE     | FREE | TABLESPACE |
+---------------+--------+---------------+----------+------+------------+
| datafile2.dat | NDB    | datafile2.dat | 12582912 |   11 |            |
| datafile.dat  | NDB    | datafile.dat  | 12582912 |    9 |            |
+---------------+--------+---------------+----------+------+------------+
2 rows in set (0.02 sec)

+------+--------+---------+-------------+-----------------------+
| NAME | ENGINE | VERSION | EXTENT_SIZE | DEFAULT_LOGFILE_GROUP |
+------+--------+---------+-------------+-----------------------+
| ts1  | NDB    |       1 |     1048576 |                     0 |
| ts2  | NDB    |       1 |     1048576 |                     0 |
+------+--------+---------+-------------+-----------------------+
2 rows in set (0.00 sec)

mysql> ALTER TABLESPACE ts1 ADD DATAFILE 'datafile3.dat' 
INITIAL_SIZE=12M ENGINE NDB;
Query OK, 0 rows affected (1.85 sec)

mysql> select * from INFORMATION_SCHEMA.DATAFILES;  select *
 from INFORMATION_SCHEMA.TABLESPACES;
+---------------+--------+---------------+----------+------+------------+
| NAME          | ENGINE | PATH          | SIZE     | FREE | TABLESPACE |
+---------------+--------+---------------+----------+------+------------+
| datafile2.dat | NDB    | datafile2.dat | 12582912 |   11 |            |
| datafile3.dat | NDB    | datafile3.dat | 12582912 |   11 |            |
| datafile.dat  | NDB    | datafile.dat  | 12582912 |    9 |            |
+---------------+--------+---------------+----------+------+------------+
3 rows in set (0.02 sec)

+------+--------+---------+-------------+-----------------------+
| NAME | ENGINE | VERSION | EXTENT_SIZE | DEFAULT_LOGFILE_GROUP |
+------+--------+---------+-------------+-----------------------+
| ts1  | NDB    |       1 |     1048576 |                     0 |
| ts2  | NDB    |       1 |     1048576 |                     0 |
+------+--------+---------+-------------+-----------------------+
2 rows in set (0.00 sec)

the ‘free’ column is really the number of free extents. Not exactly ideal… maybe… but since that’s the unit of allocation in the data files, it sort of makes sense. The other option is to list number of extents * extent size. Maybe that’s clearer for people… there is the option of denormalising the tables and have extent size in the DATAFILES table too. There is something in my brain that makes that a hard leap though.

Although…. if you’re going to be querying the tables directly and not just using a pretty gui on top of it all, you should probably know what you’re doing anyway.

Although, both a great benefit (and curse) of commoditising the database market is the fact that you get all sorts as users. This is interesting in cluster as it is naturally a bit more complex than a simple client-server RDBMS.

we also need a NODE column as well. which will probably cause confusion for non-cluster users and the like :)

(for the unintiated, this is work being done in a branch off the 5.1 tree for NDB disk data. we’ll push it to the main 5.1 tree at some point). don’t go thinking this is production ready any time soon (in other words insert a standard disclaimer).