Let me know if anything broke. Seems like a decent upgrade though. WP alread was really nice, and it’s just getting better!
Adventures with OpenSolaris
So… some colleagues have been experimenting with DTrace a bit, and I’ve been (for a while now) wanting to experiment with it.
The challenge now, instead of in the past, is that I’m setting up a Solaris based system – not getting one premade.
I chose OpenSolaris as I’d previously tried Solaris 10 and just sunk too much time trying to get updates and a development environment installed (another colleague could get the opposite to me going: he got devtools but no updates. at least mine was up to date and secure… but without a compiler).
So… OpenSolaris. It isn’t 100% open, there’s binary only drivers and such… but compared to previous Solaris, a whole lot better. Now, if only it was GPL licensed so we could have cross-pollination with Linux.
I grabbed the 2008.05 ISO as soon (in fact, slightly before) it was released and installed it in VirtualBox.
The installation was shiny – one of the best OS installs I’ve seen in a while. It set up nice things (zfs, X) and (an improvement on the previous release) even managed to get all the hardware going (not sound though).
However, on first reboot, nasty surprise. DNS isn’t enabled by default.
I found out why DNS isn’t enabled by default – and (as usual) this comes down to hysterical raisins. Back in what we laughingly call the past, during install Solaris would ask you what services you wanted to use for name resolution (which I guess made sense when people used yp/NIS more often than DNS). The default didn’t include DNS.
In the graphical installer, it just chose the default without asking… which is no DNS. So my mother would be able to install OpenSolaris, but once done, she’d have to know to type in 150.101.98.214 instead of www.google.com.au into Firefox. However, I swallowed my pride, edited /etc/nsswitch.conf and went along my business (I wonder the percentage of users who would actually go from “hrrm, internet not working” to editing /etc/nsswitch.conf without intense googling).
The UI did look nice though. Nice looking GDM, GNOME desktop looked nice. You could tell that whoever did the theme had spent too much time near MacOS X, but I’ll forgive them for that. The default shell is remotely sane and even though the bash completions aren’t as funky as on Ubuntu, I managed (unlike sitting at cmd.exe, where somebody is likely to die each time my keystrokes end up there).
I even had a look at the graphical package management tool – which looked quite nice. I even tried to do an update via it… which ended in what seemed to be a locked package manager and general amounts of fail. To see if it had just stopped or was chewing up my CPU or memory, I opened a terminal and ran ‘top’.
I then found out that top isn’t installed by default. It’s 57kb on my Ubuntu 8.04 laptop so disk space couldn’t be the reason why it’s not installed. It’s certainly not a “it’s a minimal install” argument, there’s lots of other things there by default.
Next step, let’s get updates (some time had elapsed between first install and now).
Seeing as I hadn’t met too much success with the graphical utility (it was at version 0.0000001 or something, so I don’t lay blame there). I find out that ‘pkg image-update’ is what you want to run. So I do.
It chugs for a while and says there’s 1GB of updates. That’s okay, I (where I=Sun) pay for what here on the arse end of the Internet is considered a decent link to my home office. About 20-30minutes later, having downloaded about 600MB, it goes “url timedout error” and aborts. Oh well I think, that’s easy – i’ll run it again and it’ll just resume downloading (remember the revolution when that started working, you know, in 1997).
I then discovered that pkg doesn’t resume downloads. It creates a snapshot using ZFS and puts the updates in it. If anything goes wrong, it just deletes the snapshot. This is a huge benefit over (say) dpkg, which if you press the reset button at the right time will leave your system very, very fucked (magic incantations can revive it, but it’s not fun – and the dpkg developers don’t think it’s a problem – come to my “Eat My Data” talk at OSCON to find out the full story). So OpenSolaris pkg wins on the “don’t ruin my working OS install already” front, but fails on resuming downloads.
I try again. Same story.
It’s now wasted a bit over 1GB of downloads… which equates to a couple of dollars.
I wait a few days, a week, and try again. Same story. I even try with a few hints found online that should fix things (well.. they did let another 100MB on average download before dying with the same story).
I then decided to just try and do the minimal – I wanted a development environment so I could build a MySQL Server with NDB and then play with DTrace to help nut out a performance problem or two.
So i tell pkg to install SunStudio Express. I’m even using instructions off sun.com, so it has to work.
It’s only ~500MB now (IIRC). Fails with exactly the same error as before (url timedout). Gah!
So, this brings us to today. I head into the Sun office.
I figure “this just has to work from a Sun office… ” and I was right!
It got through the (now) 1500MB download of updates!
It even applied them!
Success!
Win!
Well, no, – FAIL.
It now refused to boot with the updates. Or rather, it just rebooted soon after having started booting. No panic, no error screen, no “will reboot in 120 seconds” or anything useful. Instead, you just saw a flicker of the error message before it rebooted.
So… with some very careful pause/unpause of the VM (thanks VirtualBox… I also have a feature request now – pause before reboot :) I got this:
Aparrently the successful update, not so much.
Hrrm… perhaps select the known good one from the GRUB menu? It did actually boot! But this wasn’t just the old kernel, it was the whole older system. I guess that’s a possible upside of ZFS snapshots…. but oh my, that could be sooooo subtle and lead to data loss that it’s really quite dangerous.
I was still no closer to getting an up to date opensolaris system with enough developer tools to build a MySQL Server and use dtrace.
And this was enough. It’s now gone and I get my 10GB of disk back.
Maybe I’ll try again later… but I’m finding the google-perftools to be rather exciting and they’re really satisfying shiny thing urges at the moment.
WL4271 Encrypted Online Backup: Preview 3
“WL4271 Encrypted Online Backup: Preview 3†branch in Launchpad
Now with Windows support. Many thanks to Chuck Bell for helping get the code going on Windows.
We can however, all sit around dumbfounded as to how Windows has so little of a POSIX like layer and yet doesn’t define ENOTSUP.
As a refresher, this tree implements:
- Encryption for MySQL Online backup
- Algorithms and keysizes supported:
- 3DES
- AES (128, 192 and 256bit)
- World peace
(world peace not included)
UPDATE: If you’re wondering why the branch isn’t there, it’s still pushing to launchpad. Yes, that’s over 7 hours to push a branch. ick. Can’t be too much longer, surely. I cannot wait until lp uses shared repos.
Twitter Updates for 2008-07-08
- on my way back to airport and then flying home #
- hungry #
- @kartar: not there yet :) but unlikely #
- just had Sun induction day. would have been more useful back when i actually joined #
- @garypendergast: almost… wolud have been *much* more useful a few months ago. and totally not geared towards engineers. #
- @dormando: happy 10:11PM ! :) #
Powered by Twitter Tools.
Twitter Updates for 2008-07-08
- on my way back to airport and then flying home #
- hungry #
- @kartar: not there yet :) but unlikely #
- just had Sun induction day. would have been more useful back when i actually joined #
- @garypendergast: almost… wolud have been *much* more useful a few months ago. and totally not geared towards engineers. #
- @dormando: happy 10:11PM ! :) #
Powered by Twitter Tools.
Twitter Updates for 2008-07-07
- why does it take so long to get a cab at sydney airport #
Powered by Twitter Tools.
on microblogging/twitter2blog etc…
Andrew Pollock makes the observation that a microblogging (twitter or whatever) stream added to a regular blog doesn’t add much for the reader… I’ve been debating this myself ever since I installed the WordPress Plugin that does twitter integration.
One motivation was “I don’t like everything being tied up in some proprietary system” and wanting a copy of my data somewhere. On the other hand, what I write into twitter isn’t exactly that-scottish-play and loosing it all would probably be a net gain for humanity.
So… do I turn off the Twitter plugin? I’m erring on the side of “yes, turn it off”. Thoughts?
Twitter Updates for 2008-07-03
- I just answered this buffy trivia question correctly: “what was Buffy’s SAT score?”. I may be addicted. #
- if you send HTML only mail to me: fuck you. #
Powered by Twitter Tools.
Twitter Updates for 2008-07-01
- need coffee #
- encryption with different algorithms and keysizes! #
- and…. omg… testing #
- Hi desk, meet my head…. BANG BANG BANG,…. “Keep IT Safe” training… BANG BANG. #
- just lied in a Sun security training thing… their “correct” answer was very, very, very wrong. #
- I think sun’s intranet is 50% redirect pages #
- stock photos of smiling people randomly inserted in places make me a sad panda. #
- should sleep #
Powered by Twitter Tools.
Security question fail.
Spot the problem:
You work for company X.
- Phone rings: “Hi, my name is Alice, I work for company X”
- “Hi Alice, this is Bob, in order to verify that you do actually work for X, what is your employee number and phone extension, I’ll call you back when verified”.
- “Okay Bob, it’s Alice, employee number 1234 and I’m on 555-5555”
- You look up the employee database and sure enough, Alice is there with number 1234.
Were you talking to Alice?
Will you be talking to Alice if you dial 555-5555?
Encrypted Online Backup Preview 2 (DES and AES)
New preview includes:
- 3DES support
- AES support with 128 (default), 192 or 256 bit keysize
bzr branch lp:~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview2
(you can pull this directly into the previous preview1 branch, it’s just 2 extra patches).
Examples:
- BACKUP DATABASE test to ‘test.ba’ ENCRYPTION_ALGORITHM=3des PASSWORD=’pants’;
- RESTORE FROM ‘test.ba’ ENCRYPTION_ALGORITHM=3des PASSWORD=’pants’;
- BACKUP DATABASE test to ‘test128.ba’ ENCRYPTION_ALGORITHM=aes ENCRYPTION_KEYSIZE=128 PASSWORD=’pants’;
- RESTORE FROM ‘test128.ba’ ENCRYPTION_ALGORITHM=aes ENCRYPTION_KEYSIZE=128 PASSWORD=’pants’;
Enjoy! Feedback is very muchly appreciated.
MySQL Encrypted Online Backup Preview 1
I’ve just pushed to launchpad, a set of patches that implement AES encryption support for MySQL Online Backup. You will need to build –with-ssl to get support for encrypted online backup.
Encrypted backup files have no recognisable header – they’re just a stream of random bytes. Encryption and compression also works, and when you encrypt, you should also use compression (compress first, then encrypt).
Source tree: https://code.launchpad.net/~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview1
Bzr command: bzr branch lp:~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview1
Thread on internals@ with patchset: http://lists.mysql.com/internals/35759
Build instructions: be sure to ./configure –with-ssl
Example usage:
- BACKUP DATABASE test TO ‘test.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
- BACKUP DATABASE test TO ‘test1.ba’ WITH COMPRESSION COMPRESSION_ALGORITHM=gzip ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
- RESTORE FROM ‘test.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
- RESTORE FROM ‘test1.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
UPDATE: preview1 is no more, preview2 is now out, contains extra stuff (see next blog post)
Twitter Updates for 2008-06-30
- distrated by shiny things #
- the Sun Corporate Travel Online tool is 1000 percent of total utter fucking fail. #
- hi Amex Travel, let me introduce you to ORDER BY #
- dear Amex Travel, if you displayed the prices of flights up front, you wouldn’t need to “search for lower prices”. fail. #
- Dear Amex Travel, $a == $b is not a lower fare. #
- Dear Amex Travel, remember my Qantas frequent flyer number you fail-mongers. #
- Dear Amex Travel, I’m unlikely to change my meal preference. remember it. fail++ #
- Dear Amex Travel, you’re more expensive than Qantas.com, and have a worse UI. Why do we use you? fail++ #
- loonch time #
- pushing a mysql branch to launchpad takes a *long* time… as in longer than going out for lazy long lunch. #
- waiting immense amount of times to cerate new mysql branch on launchpad.net #
- still waiting for push to launchpad to finish…. only been going about 6.5hrs #
Powered by Twitter Tools.
MySQL 5.1 Cluster DBA Certification Study Guide available again!
Vervante Books Etc — MySQL 5.1 Cluster DBA Certification Study Guide
Only $40. Written by absolute brilliant people (and me). Needing to learn MySQL Cluster so you can go and use it? test it out? work out if it’s for you? Get this book!
Twitter Updates for 2008-06-25
- “…all you have to do is find the universal perkiness constant…” #
Powered by Twitter Tools.
Twitter Updates for 2008-06-25
- “…all you have to do is find the universal perkiness constant…” #
Powered by Twitter Tools.
Twitter Updates for 2008-06-24
- coffee is very non-fail #
Powered by Twitter Tools.
Twitter Updates for 2008-06-23
- @glynnfoster: for me, image-update gets “url timedout” and just rolls back. i dream of running out of space #
Powered by Twitter Tools.
Twitter Updates for 2008-06-20
- hating OpenSSL and yaSSL. a lot. #
- it is tempting just to implement my own crypto routines instead of using these poorly documented, half implemented pieces of trash. #
- @kartar: it’s the little things… like not having any mention on how big an output buffer should be… you know, unimportant things. #
- @kartar want to start libunfuckedcrypto ? #
- the complete lack of return codes in yassl crypto_wrapper makes me want to stab it in the face #
Powered by Twitter Tools.
NDB$INFO scanning from ndb_mgm
In code just tested:
ndb_mgm> ndbinfo MEMUSAGE
RESOURCE_NAMEÂ Â Â NODE_IDÂ Â Â PAGE_SIZE_KBÂ Â Â PAGES_USEDÂ Â Â PAGES_TOTALÂ Â Â BLOCK
IndexMemory,1,8192,16,160,DBACC
DataMemory,1,32768,20,640,DBTUP
IndexMemory,2,8192,16,160,DBACC
DataMemory,2,32768,20,640,DBTUP
Win!
This is the first time that we’ve been able to get this kind of info out of the cluster without using the magic “all dump 1000” (or “all report MemUsage”) which end up using events, which go to the log file, aren’t exactly reliable etc.
This performs a scan on the NDBINFO tables (in ndbd) from ndb_mgmd and returns the result to the management client. You can then use this in scripts from the command line. e.g. to find out how many pages of datamemory are used on each node:
$ ./storage/ndb/src/mgmclient/ndb_mgm -c localhost:9311 -e ‘ndbinfo MEMUSAGE’|tail -n +3|grep ‘DataMemory’|cut -d ‘,’ -f4
20
20
now, just to clean it up a bit, fix the one bug (yes, you guessed it: in metadata caching) and get a review….
but, a milestone!