I have released version 1.19-rc3 of Perdition. This release fixes several problems. Perhaps most significantly, Thunderbird 3.1 can now re-connect to perdition SSL/TLS.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

I have released version 2.0.2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen.

There are changes since 2.0.2-rc1.

The code is available as a tarball here and in git here. More information is available in the announcement email.

kexec-tools recently moved, so the URLs above are new.

• Home: horms.org/
• Feed: horms.org/pleb_blossom/rss.xml

since I’m stupid and I scheduled the team meeting before I could actually be in The Hague, I had to move the meeting. and, again, since I’m stupid I managed to schedule it against important talks of the day — and my own.

so: the GTK+ team meeting has been moved to the common area after the last talk of Wednesday, July 28th. sorry for the inconvenience I caused, and please don’t hurt me when you see me.

• Symbian on my Nokia N97 — for spontaneously rebooting as soon as I got off the ferry.
• Google Maps — for not caching the map tiles I'd carefully downloaded while I was on the free ferry wireless, showing my route to the hotel.
• Mobile phone networks — for the insane amount of money it will have cost me to re-download the same map tiles again, as I was driving.

I really need to get myself an N900 and start using maemo-mapper again. Every time I try to use non-free software, it hurts.

When developing a website, and in particular an intranet, for an organisation there is often a debate on what should be on the frontpage.

Different departments and different individuals all seem to have "strong opinions" on the matter.

read more

Josh Berkus gave a great talk at linux.conf.au 2010 (the CFP for linux.conf.au 2011 is open until August 7th) entitled “How to destroy your community” (lwn coverage). It was a simple, patented, 10 step program, finely homed over time to have maximum effect. Each step is simple and we can all name a dozen companies that have done at least three of them.

Simon Phipps this past week at OSCON talked about Open Source Continuity in practice – specifically mentioning some open source software projects that were at Sun but have since been abandoned by Oracle and different strategies you can put in place to ensure your software survives, and check lists for software you use to see if it will survive.

So what can you do to not destroy your community, but ensure you never get one to begin with?

Similar to destroying your community, you can just make it hard: “#1 is to make the project depend as much as possible on difficult tools.”

#1 A Contributor License Agreement and Copyright Assignment.

If you happen to be in the unfortunate situation of being employed, this means you get to talk to lawyers. While your employer may well have an excellent Open Source Contribution Policy that lets you hack on GPL software on nights and weekends without a problem – if you’re handing over all the rights to another company – there gets to be lawyer time.

Your 1hr of contribution has now just ballooned. You’re going to use up resources of your employer (hey, lawyers are not cheap), it’s going to suck up your work time talking to them, and if you can get away from this in under several hours over a few weeks, you’re doing amazingly well – especially if you work for a large company.

If you are the kind of person with strong moral convictions, this is a non-starter. It is completely valid to not want to waste your employers’ time and money for a weekend project.

People scratching their own itch, however small is how free software gets to be so awesome.

I think we got this almost right with OpenStack. If you compare the agreement to the Apache License, there’s so much common wording it ends up pretty much saying that you agree you are able to submit things to the project under the Apache license.  This (of course) makes the entire thing pretty redundant as if people are going to be dishonest about submitting things under the Apache licnese there’s no reason they’re not going to be dishonest and sign this too.

You could also never make it about people – just make it about your company.

#2 Make it all about the company, and never about the project

People are not going to show up, do free work for you to make your company big, huge and yourself rich.

People are self serving. They see software they want only a few patches away, they see software that serves their company only a few patches away. They see software that is an excellent starting point for something totally different.

I’m not sure why this is down at number three… it’s possibly the biggest one for danger signs that you’re going to destroy something that doesn’t even yet exist…

#3 Open Core

This pretty much automatically means that you’re not going to accept certain patches for reasons of increasing your own company’s short term profit. i.e. software is no longer judged on technical merits, but rather political ones.

There is enough politics in free software as it is, creating more is not a feature.

So when people ask me about how I think the OpenStack launch went, I really want people to know how amazing it can be to just not fuck it up to begin with. Initial damage is very, very hard to ever undo. The number of Open Source software projects originally coming out of a company that are long running, have a wide variety of contributors and survive the original company are much smaller than you think.

PostgreSQL has survived many companies coming and going around it, and is stronger than ever. MySQL only has a developer community around it almost in spite of the companies that have shepherded the project. With Drizzle I think we’ve been doing okay – I think we need to work on some things, but they’re more generic to teams of people working on software in general rather than anything to do with a company.

I seem to have picked up some sort of virus. I'm mostly offline while I wait for this to clear up. Please forgive any delays...

LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) is used to model atoms or as a parallel particle simulator at the atomic, meso, or continuum scale. It has potentials for soft materials (biomolecules, polymers) and solid-state materials (metals, semiconductors) and coarse-grained or mesoscopic systems. You may remember a difficult install from the past.

read more

Bending into Padahastasana, the energy courses through my legs. In sympathetic unison the crickets count the seconds away.

Listening to the drone of my teachers voice, I feel a gecko scurry over my toes to hunt for moths in the glare of the lights.

Looking up from my book, I see ochre tinged cliffs melting into the jungle below. Fork-tailed swallows flit through the skies chasing insects and cuddle on the roof, sharing chirps of intimacy.

Drifting in the pool, I strain to float effortlessly. Above me, in a haze of yellow and black butterflies, swarms of red dragonflies zigzag across the water.

Playing on the beach, we are joined by two young children. They howl in Thai as we skim a frisbee over their heads and laugh with unspoken pride when they pry the disk from the sky.

Hanging out in an empty pub, we make aimless conversation with the owner. As we laugh and talk a party slowly forms around us, John and Boi on the guitar, Kris and I singing.

Struggling to shape my thoughts into words, I'm approached by a young Indian boy who stares longingly at my computer. To his astonishment I don't have any games installed, instead we entertain ourselves by taking silly pictures while his father looks on and laughs.

Watching children play in the distance with a broken surfboard in the sea. Skimming it across the wash they shriek with laughter each time one of them gets catapulted into the air by an incoming wave.

Sitting, facing north, I close my eyes. I can hear the wind in the trees and feel the beat of my heart as I begin to count my breath.

By 8am this morning, a crowd of 13 people had gathered in the drizzle outside Wellington’s Magnummac store, eagerly awaiting the New Zealand launch of Apple’s iPad.

I decided to ask them why.

“This device is changing the rules of digital engagement”, I couldn’t wait to get my hands on one; I want to be the first person in New Zealand to get one.” The hopeful, but ill-informed, gentleman at the head of the queue told me.

“You know that people have been importing them for months?”

“This device is changing the rules of digital engagement” he replied, effectively ending the interview.

I moved along the queue, stopping in front of a lady who, I quickly noticed, had a tattoo of a heart with an apple in the middle of it on each hand and “A-P-L-E” spelled out across her knuckles.  I moved along the queue.

A normal looking gentleman told me: “We want to stand at the intersection of computers and humanism”.  I noticed that he was nervously fingering a small white book on which I could make out the title “The Quotations of Chairman Jobs”, and I fled.

As I passed the last person, he grabbed me and confided that, in his inebriated state, he’d thought it was the queue for the soup kitchen.  I explained.  We fled together along Vivian Street.

The next couple of months are going to be a crazy ride. I will be visiting at least 7 countries, speaking on 8 or more days in a 5 week period. The talks will be focused on Drupal and Aegir. My schedule is below.

Horizontally Scaling Drupal - Melbourne

On 7 August I'll be running a 1 day workshop around the theme of horizontally scaling Drupal. The content is built on the knowledge I developed building, deploying and managing around 2100 sites for a client. This event has very limited capacity and has almost sold out.

DrupalCon - Denmark

Denmark is hosting the European leg of DrupalCon this year. I will be attending the full conference. I won't be presenting, but I will be getting involved with some of the BoFs. I had a ball at DrupalCon San Francisco earlier in the year.

Efficiently Managing Many Drupal Sites - Slovakia

After spending a couple of days recovering from DrupalCon, I'll be teaming up with the crew at Sven Creative in Bratslavia, to run a 2 day intensive workshop on horizontally scaling Drupal and development workflows. For more information check out the workshop website.

Free Software Balkans - Albania

On the weekend of 11-12 September, the inaugural Free Software Balkans Conference will be held at the University of Vlore, Albania. I'll be there speaking about Drupal and Aegir. In addition to this I will be running half day build your first Drupal site workshops around the country. The dates and locations for the workshops are still being finalised.

OSI Days - India

On my way back to Australia I will be taking a side trip to Chennai, via Delhi, for OSI Days 2010, Asia's largest open source conference. I will be presenting sessions on Aegir and Drupal. This looks like it will be a huge event.

Other Events

I've launched a new site workshops.davehall.com.au to list my training and speaking engagements. As dates are locked in I'll be adding them to the site.

If you would like to meet with me while I'm on the road, add me to your tripit network, follow me on identi.ca or twitter or add me to your network on LinkedIn.

So I sometimes get asked if we funnel back bug reports or patches back to MySQL from Drizzle. Also, MariaDB adds some interest here as they are a lot closer (and indeed compatible with) to MySQL. With Drizzle, we have deviated really quite heavily from the MySQL codebase. There are still some common areas, but they’re getting rarer (especially to just directly apply a patch).

Back in June 2009, while working on Drizzle at Sun, I found a bug that I knew would affect both. The patch would even directly apply (well… close, but I made one anyway).

So the typical process of me filing a MySQL bug these days is:

• Stewart files bug
• In the next window of Sveta being awake, it’s verified.

This happened within a really short time.

Unfortunately, what happens next isn’t nearly as awesome.

Namely, nothing. For a year.

So a year later, I filed it in launchpad for MariaDB.

So, MariaDB is gearing up for a release, it’s a relatively low priority bug (but it does have a working, correct and obvious patch), within 2 months, Monty applied it and improved the error checking around it.

So MariaDB bug 588599 is Fix Committed (June 2nd 2010 – July 20th 2010), MySQL Bug 45377 is still Verified (July 20th 2009 – ….).

(and yes, this tends to be a general pattern I find)

But Mark says he gets things through… so yay for him.2

This description from Newsweek on what happens in our brains when we're creatively solving problems is wonderful. I don't think I could have come up with these words but they describe exactly what it feels like to me when I'm involved in difficult problem solving.

When you try to solve a problem, you begin by concentrating on obvious facts and familiar solutions, to see if the answer lies there. This is a mostly left-brain stage of attack. If the answer doesn’t come, the right and left hemispheres of the brain activate together. Neural networks on the right side scan remote memories that could be vaguely relevant. A wide range of distant information that is normally tuned out becomes available to the left hemisphere, which searches for unseen patterns, alternative meanings, and high-level abstractions.

Having glimpsed such a connection, the left brain must quickly lock in on it before it escapes. The attention system must radically reverse gears, going from defocused attention to extremely focused attention. In a flash, the brain pulls together these disparate shreds of thought and binds them into a new single idea that enters consciousness. This is the “aha!” moment of insight, often followed by a spark of pleasure as the brain recognizes the novelty of what it’s come up with.

Now the brain must evaluate the idea it just generated. Is it worth pursuing? Creativity requires constant shifting, blender pulses of both divergent thinking and convergent thinking, to combine new information with old and forgotten ideas. Highly creative people are very good at marshaling their brains into bilateral mode, and the more creative they are, the more they dual-activate.

I’m at OSCON this week. Come say hi and talk Drizzle, Rackspace, cloud, photography, vegan food or brewing.

Due to tyre troubles with my other two bikes I was "forced" to ride my Cannondale road bike to work this morning. The first time I've ridden it since well before Hikari was born last November. Wow, that bike is light and voom it flies. (And I put the brakes back together correctly after shipping it from Sydney!)

In semi-related news, its really warm in Tokyo at the moment.

Working on plugin bugs sucks, but sometimes it's fun to watch a video clip over and over again.

Today, I spent a couple of hours thinking back to how awesome the good ol’ days of DOS games were and that now…I hardly play games. Mainly because I find them all pretty damn boring. I thought it’d be awesome to make a 1995 super-computer, say a couple of Pentium Pros and a kick-ass 3dfx Voodoo card (or two), just to play some retro games. Alas, finding rare parts, even on eBay, is rather difficult and it’d probably be a pretty expensive ordeal. Guess I’ll just have to stick with DOSbox and VMware.

On another note, I present you with the list of awesome game I’ve played. You’ll notice how it begins with a shit-ton of DOS games, ventures briefly into Windows, then heads to the PSX, then to WoW, and finally, the world of casual gaming.

The DOS Era (1991 – 1997)
Ancients
Battle Chess 4000
Bio Menace
Blake Stone
Blood
Blues Brothers – Jukebox Adventure
Bouncing Babies
Captain Comic
Carmageddon
Catacombs 3-D
CD-Man
Command & Conquer: Red Alert
Commander Keen 1, 2, 3, 4, 5, Dreams, 6
Companions of Xanth
Cosmo’s Cosmic Adventure
Crysal Caves
Dangerous Dave
Descent, 2
Doom 1, 2
Duke Nukem 1, 2, 3D (Atomic)
Dune
Earthworm Jim
ElectroMan
Epic Pinball
Fantastic Dizzy
Flashback
Freddy Pharkas – Frontier Pharmacist
Fuzzy’s World of Miniature Space Golf
Gertrude’s Secrets
Goldfields
Gorillas (QBASIC)
Grand Theft Auto 1, 2
Heretic, Shadow of the Serpent Riders
Hexen: Beyond Heretic, Deathkings of the Dark Citadel
Hocus Pocus
Hugo’s House of Horrors, Whodunit, Jungle of Doom
IndyCar Racing
Jazz Jackrabbit, 2
Jetpack
Ken’s Labyrinth
King’s Quest 1, 2, 3, 4, 5, 6, 7
Lamborghini: American Challenge
Lemmings
Links: The Challenge of Golf
Lotus III: The Ultimate Challenge
Mario Is Missing
Monster Bash
Need for Speed
Nibbles (QBASIC)
Oddworld: Abe’s Oddysee
One Must Fall: 2097
Prince of Persia
Quake
Raptor
Reader Rabbit
Rise of the Triad
Sango Fighter
Shadow Warrior
Sherlock Holmes: Consulting Detective
Sim City 2000
Sim City Classic
Sim Farm
Stunts
SWIV 3D
Terminal Velocity
The Incredible Machine
Theme Park
Tomb Raider
Turok
Wacky Wheels
Warcraft 1, 2
Wipeout
Wolfenstein 3D (Spear of Destiny)
WordMAX
Worms
Zool
The Windows Era (~1995 – 2004)
3D Movie Maker
3D Pinball for Windows
Age of Empires 1, 2, 3
Ballistics
Battlefield 1942
Catz
Chip’s Challenge
Counter Strike
Crazy Taxi
Daytona USA
Deer Hunter
Destruction Derby
DogDay
Doom 3
Doom 3, Resurrection of Evil
Dope Wars
Elasto Mania
Enter The Matrix
FIFA 97
Flight Simulator 2004
Fury3
Grim Fandango
Half Life
Halo
Heretic II
HeXen II, Portal of Praevus
Hitman: Codename 47, 2: Silent Assassin, Contracts, Blood Money
Hong Kong Mahjong
Hover!
I.G.I.
Jurassic Park: Chaos Island
Kid Pix
Kingpin: Life of Crime
Math Blaster
Math Circus
Max Payne, Max Payne II
MechWarrior 2
Metal Gear Solid 2
Micro Machines V3
Microsoft Golf
MindMaze (that game in Microsoft Encarta!)
Monster Truck Madness
Motocross Madness 1, 2
Myst, Riven, III, IV, V, realMYST
Need for Speed 2
Need for Speed: Underground, Underground 2
Postal, 2
Putt-Putt
Pyst
Quake II, The Reckoning, Ground Zero
Quake III Arena, Team Arena, Quake Live,
Rainbow Six: Rouge Spear
Rally Championship
Red Faction
Redneck Rampage
Sega Rally
Sim Park
SimTower
SkiFree
Soldier of Fortune
Sonic CD
Starcraft, Brood War
SWAT
The Land of Um
The Sims
Theme Hospital
Titanic: Adventure Out Of Time
ToCA Race Driver 2
Typing of the Dead
Unreal, Unreal Tournament, Unreal Tournament 2003, Unreal 2, Unreal Tournament 2004,
Virtua Cop
V-Rally
Where In The World Is Carmen Sandiego?
Yoot Tower
The PSX Era (1999 – 2002)
Ace Combat 1, 2
Colin McRae Rally 1, 2.0
Crash Bandicoot, 2:Cortex Strikes Back, 3:Warped
Croc 1, 2
Die Hard
Dino Crisis
Driver
Gex
Gran Turismo 1, 2
Metal Gear Solid: Integral
Point Blank
R4: Ridge Racer Type 4
Silent Hill
Siphon Filter
Spyro the Dragon
Tenchu
Time Crisis
Wargames: Defcon 1
WoW Era (2005 – 2009)
World of Warcraft, Burning Crusade, Wrath of the Lich King
Post-WoW Era (2008 – 2010)
Colin McRae: DiRT 1, DiRT 2
Counter-Strike: Source
Crysis
Knytt, Stories
Left 4 Dead 1, Left 4 Dead 2
Portal
Team Fortress 2
Unreal Tournament 3
Within A Deep Forest
iPhone Era (2009 – 2010)
Angry Birds
BubblePRO
Doodle Jump
Flight Control
Need for Speed Undercover
Paper Toss
Peggle
Plants vs. Zombies
Rock Band
Spikey’s Bounce Around
Super Monkey Ball 1, 2
Tap Tap Revenge
Other
Donkey Kong Country (SNES)
GoldenEye (N64)
Granny’s Garden (C64)
Guitar Hero (Xbox 360)
Pokemon Red, Blue, Green, Yellow (GB)
Sonic 1, 2, 3, & Knuckles (Mega Drive)
Super Mario World (SNES)
Wayne Gretzky’s 3D Hockey (N64)
Wii Sports (Wii)

I am sure to update this list as I can think of any DOS games that come to mind. I currently have one game racking my brain that I played a ton and just can’t remember the name, sigh.

Over the last few months months I've been corresponding with Dennis Ulrich of Western Digital (WDC) about my concerns with the EULA for the My Book World Edition (MBWE) and their obligations under the GPL. To say it has been a drawn out process is an understatement.

It has taken some time to get WDC to understand the situation. There has been confusing messages about what the situation is with the EULA, the GPL and what license covers what pieces of code. The bottom line is that currently users must check the header of each file to ascertain which license applies to it, even though the downloads are marked as GPL.

Although WDC is moving slowly, they do seem to be commited to making the situation clearer in the next iteration of their MBWE product line. Based on a recent phone call with Dennis, the legal and engineering teams are working together to ensure various licenses complied with and their software engineers are aware of their obligations.

The next version of the MBWE is likely to ship with a revised EULA and properly inform the users of their rights under the GPL. This text is still being developed. At the same time it is still unclear if WDC will backport their changes to their existing NAS products. As it is a simple string change it shouldn't be too hard for them to dedicate a few resources to audit the code and update the strings.

It is unfortunate that WDC appears to have little interest in developing their MBWE product range as a hacker friendly FOSS product. It appears the licensing fix that will be implemented by WDC will more clearly delinate the FOSS and non free components of the MBWE firmware. Clarity is always an important legal consideration, but doesn't help foster a community. WDC seem to have little interest in fostering a hacker community around their products. This is an unfortunate decision by the company.

Many manufacturers of embedded devices only start releasing source for their firmware after being caught out for violating the GPL. WDC is to be commended for complying with the requirements of the GPL from the start. Altough there is no legal requirement for them to make the web gui code and other non free components available, WDC already does. It would be disappointing if they chose to take a backward step and stopped distributing parts of the firmware.

It is possible, and perfectly legal, for WDC to stop distributing the source for the proprietary components. At the same time it would not take much effort for them to release the whole platform as GPL or another FOSS friendly license. WDC is already required to do a code drop every time they release a new version of their hardware or firmware. I suspect it would be faster and easier to push all code to a public git repository than pick through and dump selected components as tarballs on a website.

WDC already have their support team dealing with customer bug reports. Maintaining a mailing list, a bug tracker, a wiki and maybe a public source code repository on somewhere like gitorious is likely to take less than 1 full time employee. The benefit for WDC would be great.

Not only is the hacker community likely to contribute bugs fixes and propose or even develop new features, they can help increase sales. I'm sure the good will generated by the switch to truly open approach to the MBWE product line would outweigh the cost of the additional resources required.

Let's hope Western Digital's fix is a FOSS friendly one. I will post more news as things progress.

I’m nearly ready for PDXPUG Day 2010.

We’ve got some fun talks lined up from leaders in the Postgres community in town for OSCON. It’s free to attend. We’ll at the Oregon Convention Center, room D131.

Stop by sometime during the day if you’re in town!

Here’s our schedule:

10am: What’s new in 9.0 – Selena Deckelmann
11am: ORMs and Their Discontents – Christophe Pettus
1:30pm: Mining Your Logs For Fun and Profit – Josh Berkus
2:30pm: PORTAL – Dan Colish
3:30pm: An Introduction to Managing and Troubleshooting PostgreSQL on Windows – Tim Bruce
4:30pm: PostgreSQL in Brazilian Army and Air Force – Luis Dosso

6pm-10pm: Party at the Gotham Tavern!

Related posts:

1. A year of PDXPUG
2. Lightning talks from PgCon 2010
3. PDXPUG Day on July 20 – Register now!

It's currently set to trigger if you hit 'Reply to All' on a message with more than 15 recipients; unless it's a mailing list message. And of course you can see that it's trivial to turn it off if you never want to see it again.

I've also taken a moment to write down and post some thoughts on the 'Reply to All' vs. 'Reply to List' debate for mailing list messages.

Head on over to http://lca2011.linux.org.au/ and check it out!

You’ve got until August 7th to put in a paper, miniconf, poster or tutorial.

Things I’d like to see come from my kinda world:

• topics on running large numbers of machines
• latest in large scale web infrastructure
• latest going on in the IO space: (SSD, filesystems, SSD as L2 cache)
• Applications of above technologies and what it means for application performance
• Scalable and massive tcp daemons (i.e. Eric should come talk on scalestack)
• exploration of pain points in current technologies and discussion on ways to fix them (from people really in the know)
• A Hydra tutorial: starting with stock Ubuntu lucid, and exiting the tutorial with some analysis running on my project.
• Something that completely takes me off guard and is awesome.

I’d love to see people from the MySQL, Drizzle and Rackspace worlds have a decent presence. For those who’ve never heard of/been to an LCA before: we reject at least another whole conference worth of papers. It’s the conference on the calendar that everything else moves around.

In 2009 I got accepted to give a talk on writing good user documentation. I'd submitted several papers before then but never got accepted; the chief reason was that I had submitted papers about stuff I was interesed in but was not actually a key contributor to. LCA is crazy hard to get to speak at, but is totally worth it because they really treat speakers well. And to me it's addictive - I loved it so much in 2009 I wanted to do awesome things just to get a place in 2010.

That didn't work out for me; mainly because I'm a neophile. I tend to be interested in a whole bunch of things but only shallowly - occasionally (such as when I decided to write the doco for LMMS) I dip in but I rarely seem to be able to sustain that involvement before the next thing comes along and lures me away. But I'm more hopeful I can get a speakership at 2011 because I'm putting forward two proposals for things that I'm actually really involved in and know about.

Ah well. Now for three months of anticipation. Better keep on working on my electric motorbike then...

Cat stacking

The heroic practice of coordination or organising ("wrangling") obstreperous or otherwise uncooperative agents into a consistent, coherent pattern or unit e.g. wrestling those time sheets into a semblance of order was an exercise in cat stacking if ever there was one.

Just wanted to make sure that I claimed this term on behalf of myself and my colleagues who (we believe) coined it at Egressive where the term is in common use.

For 2 months now, I have been quietly working along on a new Mozilla contract that I received to continue working on HTML5 media accessibility. Thanks Mozilla!

Lots has been happening – the W3C HTML5 accessibility task force published a requirements document, the Media Text Associations proposal made it into the HTML5 draft as a <track> element, and there are discussions about the advantages and disadvantages of the new WebSRT caption format that Ian Hickson created in the WHATWG HTML5 draft.

In attending the Mozilla Summit last week, I had a chance to present the current state of development of HTML5 media accessibility and some of the ongoing work. I focused on the following four current activities on the technical side of things, which are key to satisfying many of the collected media accessibility requirements:

1. Multitrack Video Support
2. External Text Tracks Markup in HTML5
3. External Text Track File Format
4. Direct Access to Media Fragments

The first three now already have first drafts in the HTML5 specification, though the details still need to be improved and an external text track file format agreed on. The last has had a major push ahead with the Media Fragments WG publishing a Last Call Working Draft. So, on the specification side of things, major progress has been made. On the implementation – even on the example implementation – side of things, we still fall down badly. This is where my focus will lie in the next few months.

Follow this link to read through my slides from the Mozilla 2010 summit.

So, for example, it would be legal for me to crack the DRM on the eBooks I buy, which is necessary just so that I can read them. Currently I have to break the law just to be able to buy and use eBooks.

UK citizens, go here and add your vote; it's very simple to register if you haven't already done so.

This has been sitting in my edit queue for too long… here goes.

So, I was invited to Foocamp this year. Back in February, I attended KiwiFoo at the invitation of Nat Torkington. I realize now (sorry, Nat!) that I never blogged about going there. I have some rough notes around that I may try to edit down.

I’ll avoid any direct comparisons for a moment, and just talk about what it was like to dive into a slice of Silicon Valley, dislocated into the countryside for a weekend.

When I arrived, I felt immediately at home, welcomed and appreciated. Sara Winge was so helpful and easy-going about the Tesla Coil Josh brought. (Thanks for the ride, Josh!) I spent a lot of evening time educating people about avoiding touching the sparks (it may be a pretty toy, but it is in fact a dangerous one!) and also playing around with a metal glove that was rigged up so you could get a bit closer in to the coil.

I spent a couple delightful hours considering claude glass with Roberta Cairney. Over and over, I absorbed the positive energy tumbling out of Sumana. Once again, I spoke with Kiwis whose humor and affectionate swearing reminded me of home.

I held one session – about forgetting and the ethics of shifting our culture to assume everything may be on your permanent record, and the ways in which people try to opt out or game the system. I was surprised and overwhelmed by the people who attended. Rita King, Scott Berkun, Biella Coleman and danah boyd were all part of the discussion and I was able to talk about ideas I’ve had tumbling around in my head for years. I believe sysadmins/devops must have conversations about the ethics of the default choices made by developers around configuration and long-term management of log data. People asked provocative questions, and we had a real debate about the ethics. It was a wonderful experience and I came up with at least one good idea that I hope Jesse Robbins and I are going to act on together.

I also ran into several people who are just starting to work on user group and community issues in their geographic areas. Seattle came up over and over – and I’m looking forward to helping Ben Huh and the open gov folks who want to do grassroots organizing in their tech communities. I also met some amazing women there, and hope that we’re able to continue our discussions about business and tech in the future.

I camped, and was in a tent under the stars, and early morning fog. I enjoyed running into a fellow PostgreSQL community member, Paul Ramsey, among the early risers.

Apart from the immediate things to collaborate on, and an incredibly long list of new ideas and connection points, I came away inspired and mentally refreshed. I relished the relative lack of device obsession. The people that I wanted to have conversations with tended to have put their devices away for a few hours, and were focusing on the people in front of them.

The Ignite talks were my favorite talks. Jake Applebaum’s meditation on wikileaks was particularly inspiring, reminding me to seek out opportunities to change the world for the better.

Related posts:

1. meme: Current state of me
2. My Other Things
3. xppdx – dipping a toe in the water

Updated!: See below.

Here are some questions that came up from trying to use the current PostgreSQL hot standby documentation:

Q: If you set hot_standby = off after having it on, what happens?

A: This change requires a database restart on the hot standby (or replica) server. The database goes into “warm standby” mode, and you can no longer issue queries against it. You can change this right back by setting the parameter to ‘on’ and restarting again.

Q: Can you use hot standby with only a single schema or database?

A: No. Hot Standby is all-or-nothing for a particular PostgreSQL database cluster. A cluster is made up of all the databases that live in a particular $PGDATA instance, and Hot Standby is currently not capable of distinguishing between changes occurring on different particular databases or schemas.

Q: Is the process for setting up hot standby any different for empty databases vs. populated databases?

A: No. The setup process is the same – you must create a base backup.

Q: How do I bring my hot standby out of standby mode?

A: If you’re using something like the following in your recovery.conf file:

restore_command = 'cp xxxx'
standby_mode = 'on'


Change: standby_mode = 'off' and restart your hot standby postgresql instance.

Q: Where did my recovery.conf file go? (after your database came out of warm/hot standby)

A: PostgreSQL automatically changes the name of the file to recovery.done when recovery completes. This helps prevent accidents.

Q: What happens if my archive_timeout = 60 (which creates a 16mb file every minute) and I flood the database with so much activity that my standby falls behind?

A: This is possible, and you may be interested in trying Streaming Replication. However, for the majority of users, a delay in restoring data is acceptable (and possibly desirable). Eventually the standby server will catch up. You can monitor how delayed the server is using functions like txid_current_snapshot().

Q: Are schema changes (like CREATE TABLE or ALTER TABLE) replicated to the standby?

A: Yes! All changes to the database cluster are copied to the standby. This includes any DDL operations, new rows, the effects of autovacuum — any change to the data store on the master is copied to the standby.

Related posts:

1. Quick start on Hot Standby
2. Simon Riggs just rocked my world.
3. Setting up a PDXAPI instance

Updated.

We could have some better end-user documentation around creating a warm or hot standby system for basic postgresql replication.

To this end, I created a Quick Start doc on the wiki, but it could use more help. Maybe we should create some setup recipes for common situations?

Also – I wrote the following script during a hot standby bugbash PDXPUG had today:


#!/bin/sh

BINPATH=/usr/local/pg90/bin
CP=/bin/cp
PGCTL=${BINPATH}/pg_ctl
PSQL=${BINPATH}/psql
INITDB=${BINPATH}/initdb

sudo mkdir -p /var/tmp/archive
sudo chown ${USER} /var/tmp/archive

${INITDB} hotstandby1

echo 'wal_level = hot_standby' >> hotstandby1/postgresql.conf
echo 'archive_mode = on' >> hotstandby1/postgresql.conf
echo "archive_command = 'cp %p /var/tmp/archive/%f'" >> hotstandby1/postgresql.conf
echo "archive_timeout = 60" >> hotstandby1/postgresql.conf
echo "port = 6543" >> hotstandby1/postgresql.conf

${PGCTL} -D hotstandby1 start -l hotstandby1.log
sleep 5

${PSQL} -p 6543 postgres -c "select pg_start_backup('backup')"
${CP} -pR hotstandby1/ hotstandby2
${PSQL} -p 6543 postgres -c "select pg_stop_backup()"
rm hotstandby2/postmaster.pid
rm hotstandby2/pg_xlog/*

echo 'hot_standby = on' >> hotstandby2/postgresql.conf
echo 'port = 6544' >> hotstandby2/postgresql.conf
echo "standby_mode = 'on'" >> hotstandby2/recovery.conf
echo "restore_command = 'cp -i /var/tmp/archive/%f %p'" >> hotstandby2/recovery.conf

${PGCTL} -D hotstandby2 start -l hotstandby2.log

* Added port specification in case you’ve already got postgres running. Added a BINPATH for custom install directories.

Related posts:

1. Q&A about Hot Standby
2. Simon Riggs just rocked my world.
3. Greg’s THREE talks at PostgreSQL Conference East

I have released version 2.0.2-rc2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen. The only change since 2.0.2-rc1 is to include a number of files that were missing from the tarball which caused build failures on x86_64, ppc, ppc64 and arm.

The code is available as a tarball here and in git here. More information is available in the announcement email.

For more information, see http://luv.asn.au/2010/08/03

OSDC will be in Geelong this year. More details coming soon.

MVAPICH and MVAPICH2 is an implementation of MPI for computing systems and servers using InfiniBand, 10GigE/iWARP and RoCE & etc networking technologies.

Installation consists of downloading the tarball into an appropriate directory, extracting it, loading the enviroment variable modules, using of some previously established configuration files, runnig make and make install, and establishing the symbolic links. Note that MVAPICH and MVAPICH2 are entirely different products.


cd /usr/local/src/MVAPICH2

read more

GUADEC is one of the rare occasions for the people hacking on gtk to meet face to face and use a high bandwidth channel to discuss development of everyone’s favorite toolkit.

I’ve reserved a room under the BOF schedule page; currently, it’s on July 27th, between 16:00 and 18:00 (to avoid conflicting with talks).

we need to gather the attendance list, to verify if we should actually have the meeting or if informal discussions are more suited. please, fill out the attendance list and we’ll see you at GUADEC!

A great video podcast is Meet the GIMP. It’s quite accessible and has some useful information. The recent(ish) episode on Dynamic Range Theory is useful if you’re wondering why images look different through your eyes, on an LCD and on paper (and what the hell the difference between RAW and JPEG is).

Tuning and Analysis Utilities (TAU) is a program and performance analysis tool for high-performance parallel and distributed computing with a suite of tools for static and dynamic analysis of programs written in C, C++, FORTRAN 77/90, Python, High Performance FORTRAN, and Java.

You'll also need the Program Database Toolkit (PDT), also available from the same location. Let's install that first. You'll need to set your environment variables. You use modules, right?


module load gcc/4.4.4
modulel load openmpi-gcc/1.3.4

read more

I have released version 1.19-rc2 of Perdition. There is a single change since 1.19-rc1 to fix a crash-bug on invocation of perdition in managesieve mode. Merge madness on my part. Thanks to Filipe Carvalho for reporting it. The fix is here.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

After two years of effort, the W3C Media Fragment WG has now created a Last Call Working Draft document. This means that the working group is fairly confident that they have addressed all the required issues for media fragment URIs and their implementation on HTTP and is asking for outside experts and groups for input. This is the time for you to get active and proof-read the specification thoroughly and feed back all the concerns that you have and all the things you do not understand!

The media fragment (MF) URI specification specifies two types of MF URIs: those created with a URI fragment (“#”), e.g. video.ogv#t=10,20 and those with a URI query (“?”), e.g. video.ogv?t=10,20. There is a fundamental difference between the two that needs to be appreciated: with a URI fragment you can specify a subpart of a resource, e.g. a subpart of a video, while with a URI query you will refer to a different resource, i.e. a “new” video. This is an important difference to understand for media fragments, because only some things that we want to achieve with media fragments can be achieved with “#”, while others can only be achieved by transforming the resource into a different new bitstream.

This all sounds very abstract, so let me give you an example. Say you want to retrieve a video without its audio track. Say you’d rather not download the audio track data, since you want to save on bandwidth. So, you are only interested to get the video data. The URI that you may want to use is video.ogv#track=video. This means that you don’t want to change the video resource, but you only want to see the video. The user agent (UA) has two options to resolve such a URI: it can either map that request to byte ranges and just retrieve those – or it can download the full resource and ignore the data it has not been requested to display.

Since we do not want the extra bytes of the audio track to be retrieved, we would hope the UA can do the byte range requests. However, most Web video formats will interleave the different tracks of a media resource in time such that a video track will results in a gazillion of smaller byte ranges. This makes it impractical to retrieve just the video through a “#” media fragment. Thus, if we really want this functionality, we have to make the server more intelligent and allow creation of a new resource from the existing one which doesn’t contain the audio. Then, the server, upon receiving a request such as video.ogv#track=video can redirect that to video.ogv?track=video and actually serve a new resource that satisfies the needs.

This is in fact exactly what was implemented in a recently published Firefox Plugin written by Jakub Sendor – also described in his presentation “Media Fragment Firefox plugin”.

Media Fragment URIs are defined for four dimensions:

• temporal fragments
• spatial fragments
• track fragments
• named fragments

The temporal dimension, while not accompanied with another dimension, can be easily mapped to byte ranges, since all Web media formats interleave their tracks in time and thus create the simple relationship between time and bytes.

The spatial dimension is a very complicated beast. If you address a rectangular image region out of a video, you might want just the bytes related to that image region. That’s almost impossible since pixels are encoded both aggregated across the frame and across time. Also, actually removing the context, i.e. the image data outside the region of interest may not be what you want – you may only want to focus in on the region of interest. Thus, the proposal for what to do in the spatial dimension is to simply retrieve all the data and have the UA deal with the display of the focused region, e.g. putting a dark overlay over the regions outside the region of interest.

The track dimension is similarly complicated and here it was decided that a redirect to a URI query would be in order in the demo Firefox plugin. Since this requires an intelligent server – which is available through the Ninsuna demo server that was implemented by Davy Van Deursen, another member of the MF WG – the Firefox plugin makes use of that. If the UA doesn’t have such an intelligent server available, it may again be most useful to only blend out the non-requested data on the UA similar to the spatial dimension.

The named dimension is still a largely undefined beast. It is clear that addressing a named dimension cannot be done together with the other dimensions, since a named dimension can represent any of the other dimensions above, and even a combination of them. Thus, resolving a named dimension requires an understanding of either the UA or the server what the name maps to. If, for example, a track has a name in a media resource and that name is stored in the media header and the UA already has a copy of all the media headers, it can resolve the name to the track that is being requested and take adequate action.

But enough explaining – I have made a screencast of the Firefox plugin in action for all these dimensions, which explains things a lot more concisely than word will ever be able to – enjoy:

Click here to view the embedded video.

And do not forget to proofread the specification and send feedback to public-media-fragment@w3.org.

R is a language and environment for statistical computing and graphics. To install R download the tarball to an appropriate place and extract e.g.,

read more

It has been about 6 months since the current Linux Australia Council was voted in, and about a month since I became President, following James Turnbull’s resignation. In that time, the Council has been working on implementing the platform that we ran on. We have successfully managed to hold a Council meeting every fortnight (with a very few exceptions), to allow us to get together to organise events and implement the goals of Linux Australia.

read more

Today I ventured into making Vuze (the popular BitTorrent client) x64 compatible.

At first, it seems like a pretty simple task, but it turns out it’s slightly more complicated than it seems.

Here’s how to do it yourself. If you’re lazy, there’s a link at the bottom of this post with all the compiled files.

Step 1: Get the 64-bit JRE

First, download and install the 64-bit Java Runtime Environment (or JDK, whatever you want). Here’s a link to the JRE (make sure to download the Windows x64 version): https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_Developer-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=jre-6u21-oth-JPR@CDS-CDS_Developer

Step 2: Use the x64 Standard Widget Toolkit

Secondly, we must replace the main GUI toolkit that Vuze uses (SWT) with the x64 equivalent. Download the “SWT Binary and Source” for Windows (x86_64) from: http://www.eclipse.org/swt/ (make sure to click More). At the time of writing the stable version is at 3.5.2. Here’s a link to that version: http://download.eclipse.org/eclipse/downloads/drops/R-3.5.2-201002111343/download.php?dropFile=swt-3.5.2-win32-win32-x86_64.zip

Extract swt.jar from the zip file and place it in the Vuze folder (%PROGRAMFILES(X86)%\Vuze), replacing the file. Note that once you do this, you can’t go back to the x86 version unless you download the x86 SWT or reinstall Vuze. At this point, we can actually use Vuze in 64-bit with the command line using the 64-bit JRE: java -jar Azureus2.jar ….but we aren’t done yet.

Step 3: Make the iTunes plugin work

New versions of Vuze come with an iTunes plugin that uses native code that is x86 only. This particular plugin uses some COM calls using a bridge called JACOB. Open up your Vuze plugins folder (%PROGRAMFILES(X86)%\Vuze\plugins), open the azitunes folder and look for a file that looks like “jacob-x.xx.x-x86.dll”. We need to find the x64 version of that DLL, and match the version. So if x.xx.x is 1.14.3, we need JACOB 1.14.3. You can find all the JACOB versions at SourceForge: http://sourceforge.net/projects/jacob-project/files/ Open the folder with the relevant version and grab the zip file that DOES NOT end in “_src”. Open up the zip, and extract jacob-x.xx.x-x64.dll to %PROGRAMFILES(X86)%\Vuze\plugins\azitunes.

Step 4: Enabling Windows specific Vuze settings

Vuze also uses a small portion of native code to perform some Windows specific tasks. These will be invisible to you until you compile aereg.dll into 64-bit. You will need to checkout the source code from the Vuze SVN respository and compile it with Visual Studio, making sure you target x64 rather than x86. To make it compile you’ll need the 64-bit JDK and to add some include paths to the project, specifically: jdkx.x.x_xx\include and jdkx.x.x_xx\include\win32. Don’t worry about the warnings, if you have any, they’re normal. Here’s the Vuze repository for aereg.dll: http://svn.vuze.com/public/client/trunk/azureus2/lib/libWIN32Access/ After you compile it to x64, make sure to RENAME IT to aereg64.dll. This took me quite some time to figure out, I had to do some source code digging to figure out why Vuze refused to load the 64-bit library.

Step 5: Making Azureus.exe / AzureusUpdater.exe use the 64-bit JRE

This is difficult since Vuze use commercial software, exe4j, to create the EXE launcher for the JAR. I went ahead and well…made it work. I have included the exe4j project files if you wish to compile the EXEs using exe4j by yourself. The final binaries are included at the end of the post. Vuze do not provide these projects, so it took me a while to mimic the exact project options that Vuze use to compile their EXEs. Here’s the projects: http://digitaldj.net/vuze/Vuze_exe4j.zip

Step 6: Edit the manifest

This is a minor step, that doesn’t really need to be completed. Open up Azureus.exe.manifest and make sure to replace all references of “x86″ with “amd64″.

Step 7: Move Vuze to Program Files

If you want, you can go ahead and move the Vuze folder to Program Files, instead of Program Files (x86)….but it isn’t necessary. If you do this, you’ll have to update your shortcuts and make sure your file associations (.torrent) point to the new folder location.

Hooray! We’re done. Now double click Azureus.exe and bam, you’re now using the 64-bit JRE. You can check task manager to ensure you’re not using the 32-bit version (it’ll have a *32 next to the process name).

And for the lazy or non-technical. Here’s the files you need to make your Vuze 64-bit: http://digitaldj.net/vuze/Vuze_x64_1.20.zip Simply extract the files (and overwrite all) to C:\Program Files (x86)\Vuze (these files are current to Azureus 4.4.0.6, and may work for future versions, but maybe not)

PS. Similar steps are taken for Linux and OSX. Minus the obvious. As far as I know, Linux has no native code, but OSX has a similar library to aereg.dll. You’d have to compile the source code here with XCode: http://svn.vuze.com/public/client/trunk/azureus2/lib/libOSXAccess/

I’ve been writing a bit about my adventures with Black & White film and developing myself. I haven’t (yet) developed my own colour negative (C41 process) film. I do hope to do so at some point in the future – even though I can get the local lab to do it for $4 a roll, it’s nice to be able to do this yourself.

When I was young, I also took photos. I still use that camera sometimes too. Recently I’ve been scanning in the first ever slide film I shot – a roll of Kodachrome when I was 8 years old. I do like the look of Kodachrome, and am sad that it’s going away.

Last year, when I was in the US for Burning Man, I got introduced to Kodak Ektar 100. With the promise of colours that remind you of Kodachrome, I grabbed a bunch and headed to San Francisco and then Burning Man.

I liked the look of a bunch of stuff I shot. For example:

Recently, on my trip to Hong Kong, I shot some too. The above was all shot with an old Ricoh SLR, when I was in Hong Kong I used my Nikon F80 and the 50mm f1.8 lens.

One of my favourites was of this little statue:

In Hong Kong a lot of buildings are interconnected so you can walk between them without having to go outside (where it’s hot and humid). There are bits of sculpture in the buildings around the Rackspace office. This is one near the hotel I was staying with. During the morning and afternoons, these walkways are filled with people, exactly like streets…. but a floor above and indoors.

I’m adding more shots from Hong Kong to my Flickr Photstream as the days go on.

I really like this film. I even don’t mind it for people… the first was the test shot (have I loaded correctly, things winding, wonder if this shot will work) in the hotel lobby in San Francisco. Leah:

I should learn to scan better (I have since, this was probably the first image I scanned using my scanner, certainly the first Ektar frame). Another two people images I like on Ektar are:

Dare I say that I always seem to find the Ektar colours to be relaxed? I like it. The blues really shine through. Reds are also really nice (heck, I even love the yellow), and I plan to go and investigate how I can combine these colours in interesting ways on film.

Some of you may have noticed that blob streaming has been merged into the main Drizzle tree recently. There are a few hooks inside the Drizzle kernel that PBMS uses, and everything else is just in the plug in.

For those not familiar with PBMS it does two things: provide a place (not in the table) for BLOBs to be stored (locally on disk or even out to S3) and provide a HTTP interface to get and store BLOBs.

This means you can do really neat things such as have your BLOBs replicated, consistent and all those nice databasey things as well as easily access them in a scalable way (everybody knows how to cache HTTP).

This is a great addition to the AlsoSQL arsenal of Drizzle. I’m looking forward to it advancing and being adopted (now much easier that it’s in the main repository)

Side note: when a respected information source covers something where you have on-the-ground experience, the result is often to make you wonder how much fecal matter you’ve swallowed in areas outside your own expertise.

 — Rusty Russell in Superfreakonomics; Superplug for Intellectual Ventures

• Some of the support issues are Australia, and even Melbourne specific


• If I don't list a brand I've not tested it (vendors are welcome to offer loaners)


• I only care about managed switches, VLAN's are essential


• For L3 switches I want IPv4 & IPv6 with OSPF & OSPF3, doing MPLS as a P router is a bonus


• This is my opinion with my biases

Top Tier Vendors

Cisco

Juniper

Brocade (Foundry)

Extreme

Force10

Second Tier Vendors

HP / 3Com

HP ProCurve

3com

Dlink

When I put my hand up to help with linux.conf.au 2011 I realised how much work it was going to be. I’d (just barely) helped with the previous lca in Brisbane, and shared a workplace with the chief stress bunny (Raymond Smith) so got to see a lot of the really tedious, difficult stuff that just needs to happen.

I’m not very good at that sort of stuff, I’m much better with short sharp tasks that need to happen yesterday. I’ve put my hand up to run the Open Day as my main task, but I also let it be known I was happy to jump into the breach when needed. So far that’s happened a few times, and I’m happy I didn’t take on too much stuff and have some floating time available to me. I think it’s really important for an all volunteer group to have a few people with spare cycles.

A new website with content is soon to go live, the call for papers is almost ready to go, and we’re starting to sniff around for volunteers.

I have released version 1.19-rc1 of Perdition, a mail retrieval proxy. This release includes numerous bug fixes and support for the manage sieve protocol.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

I have released version 1.19-rc1 of Perdition, a mail retrieval proxy. This release includes numerous bug fixes and support for the manage sieve protocol.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

• Somewhere between 100-200 lines of code I start feeling the need for tests
• When I start feeling unclear in my head, I reach to tests to make them clear
• If someone wants to work with me, tests instantly become mandatory
• When I come back to some code after a while
• When I notice my first bug

I have released version 2.0.2-rc1 of kexec-tools, the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen. The code is available as a tarball here and in git here. More information is available in the announcement email.

“The Saratogian” used a mixture of Scribus, Google Docs to put out their newspaper on Independence Day as a gesture of support for the Free Software movement. The paper went out and looked good, leading one to wonder what they could do if they adopted this approach permanently and took advantages of the freedoms of Open Source to customise it. Quoting from their site:

“So why did we do it? Crazy? Maybe. Tired? Definitely. Proud? You bet.

The free software experiment is part of the Ben Franklin Project of the Journal Register Company, which owns The Saratogian, 17 other dailies and a slew of weeklies and has been expanding its online presence with a content-driven, digital-first mentality that reflects where the world of communication is today and where it’s headed.

• Figure out what the process is and write it down
• Simplify the process itself, reducing the number of steps
• Automate as many of the steps as possible, thus combining them
• Delegate the execution of the process

Software Freedom Day is an event held all over the world to celebrate and raise awareness of free software and open source software.

read more

One of my new (financial) year resolutions was to increase the readership of my blog. As part of this plan I have been trying to get my blog syndicated on relevant planets. The latest on my list has been Planet Ubuntu Users and the associated Ubuntu Universe. About 24 hours ago my blog was added to both aggregators. Thanks Tiago!

I run Ubuntu on my Dell D830 laptop (my primary machine). I run various flavours and versions of Ubuntu on desktops, servers and VMs both for my business and clients.

In 2009 I converted the local community run internet cafe to Ubuntu. When they were running XP there were problems almost weekly, since the switch, the only real issue they have had was a failed automatic update for firefox last week.

As for off the clock Linux time, my kids associate tux with computers, while my partner is more than happy running LTS releases on her laptop. The only desktop OS my mother in law has used is Ubuntu Linux.

I have been active in the Australian Ubuntu LoCo for quite a few years.

My Ubuntu related posts are generally technical howtos. Most of the stuff is system administration related with a few Ubuntu home brew packaging and webapp development tips.

The non tagged feed of this blog contains a good mix of geeky stuff about whatever we are working on at DHC, rants about annoying things and the occasional thing from real life. I hope there is something there that you find useful. Happy reading.

Not everybody smokes in Indonesia, but it seems like everybody does. Pinched between gnarled fingers or hanging carelessly from lips — stall owners, bikers and touts all peer out at the world through clouds of smoke. Within hours of landing the cloying smell of clove cigarettes has jolted me back back ten years, memories jostling for attention as they flicker before my eyes. Simultaneously my dormant addiction begins wriggling into consciousness, reaching out into my thoughts, looking for leverage to make "yes" the only answer.

My introduction to clove cigarettes was through Kaleigh and her civilised habit of smoking a single Djarum in the evenings. My ignorance was fascinated and I quickly discovered that lounging in the evening sun, a quiet buzz in hand, was the perfect antidote to whatever stresses the day had delivered. I had not yet learned about addiction, how it waits and yearns, how it lingers in the corners never quite allowing absolution. It is images from this honeymoon period, as my addiction and I become acquainted, which now fill my mind.

Driving with Kaleigh from Seattle to Anchorage over the Alcan Highway. We barely knew each other and were trapped in her jeep, desperately trying to keep the conversation alive through two and a half thousand miles of British Columbia's icy wasteland.

At the end of our relationship, standing on my porch and solemnly swearing off women for the next five years, only to meet Teresa less then a month later. Huddling with her on the back steps of Internet Alaska as she bummed cloves from me, the subzero temperatures providing the perfect excuse to sit close enough to kindle an office romance. I still didn't think of myself as a smoker and I remember my confusion when several weeks later she showed up on the steps with a pack of Camels. I discovered that she was an ex-smoker, and she discovered that cloves are three quarters tobacco.

Watching our friend Mark, as he watched Teresa and I, transition from smoking cloves to smoking Camels. No words were spoken but I understood what he had recognised, and why he no longer joined us for the occasional smoke.

Moving to Portland and discovering a city I could love, a city I still miss. Those endless hours with other volunteers from Personal Telco, sitting around and dreaming in bars and cafes about how we were going to build a free citywide wireless cloud with nothing but geek fervour. The collapse of the dotcom era, my friends and I falling in and out of employment as the economy heaved and twisted. Being trapped in a job I didn't yet realise I detested, and failing to adjust to the grim realities of an enterprise support role at a doomed startup. It still astounds me that James had the grace to allow me to work four ten hour days, so I could spend a day each week at home working to help Personal Telco's dreams become reality.

In the stinking hot week of Portland's summer, relaxing barefoot in the shade on our front steps. With sudden conviction I knew that I wanted to have kids, that I wanted to get married and that I'd really like for us to own our own home. Then the bitter realisation that despite a lifetime of privilege and opportunity, I had nothing but memories to show for it. In those moments I lost the sense that I had all the time in the world, I robbed myself of the freedom to let every experience unfurl in its own time. Years later I realise that there are many things right with having nothing but memories to show for your life.

The honeymoon ended the first time I tried to quit, and failed, but the memories don't stop.

My first extended travels — three months with Teresa chain smoking through England, France, Holland, Germany, New Zealand and Australia. My failed proposal attempt at the Eiffel Tower and a successful one in the Coromandel. Successfully quitting with Teresa at the end of our travels. Becoming a smoker again through the final months of our marriage and plunging into the abyss of divorce.

Modern etiquette won't let me say that that smoking was a good companion, but it has been a companion. As I write this I'm shocked at how much of my life has been spent as a smoker. How many memories I can conjure of me with a fag in one hand and a coffee or beer in the other.

I remember reading in Wired that a big tobacco executive thought that he was going to get rich creating genetically modified tobacco that didn't have any nicotine. Cigarettes that are just as dangerous but not addictive. I howled with laughter, reading it out loud through snorts of astonishment, at this gross misunderstanding. They had it completely backwards. What they need to do is create a cigarette which is still addictive, but that doesn't kill you. The addiction is the point.

Smoking provided a reason to get up from my desk and spend ten minutes away from the computer. A reason to meet Teresa on the front porch in the evenings where we would talk and dream about our lives. Rituals full of comfort in cars and airports, coffee shops and bars. A way to be occupied while being alone. And never underestimate the camaraderie of smokers in a world in which they are beset. Each shared cigarette is the beginning of a new friendship, and all smokers share.

For me it's not the quitting which is hard, it's the staying quit. As the challenge of abstinence fades, my vigilance wanes. It didn't take long, maybe a week? A couple beers in Nusa Lembongan and I was bumming smokes from Tania. The surfers in Medewi were more then happy to share their cloves. A week later I succumbed to the pleasures of my own pack, purchased at the Bromo Permai hotel.

Today I'm in Singapore, sitting at a table full of Nick's dive buddies, laughing my ass off as they joke and tease, teaching me swear words in Bahasa, Hokkien and Thai. Everybody at the table is smoking, drinking and laughing but only I know that I'm smoking my last pack. Tomorrow we get on the train for Malaysia, and in these final hours each one tastes sweeter.

I finally got around to installing iOS4, so I decided I’d go ahead and investigate what it takes to update the Fresh app (streaming radio) to support multitasking. Currently, my conclusion is that implementing ANY of the multitasking APIs is 110% hell unless you’re starting from scratch.

First of all, your application has to play nice while it’s sitting in the background. This means that you’re basically required to implement a whole lot of “if backgrounded” checks, unless you want to soak up the battery. This involves stopping unnecessary timers, updating the UI when the application comes back to the foreground (rather than per a specific event), and saving non-critical alerts for when the user pops back into the application. This basically requires a crapton of refactoring…and we all love how that introduces bugs. If you have any sort of complicated app, you’ll be having to manage state saving so that fast app switching works nicely. Oh, and not to mention that iOS never guarantees that your application stays backgrounded. It can be killed depending on resource usage, available memory or even just idle time. Every task (other than a few specific exceptions) is time limited, so you are constantly having to check and request more time for your task.

Now, this particular application is meant to take advantage of the background audio multitasking API. Now, as an aside, Apple does not provide any “simple” API to stream audio. What they provide is a primitive and generic audio buffer/queue that suits both local files and network streams. Basically, this means that if you want to stream audio over the network, you have to implement the protocol, manually send the requests to get data (via HTTP for example), then feed that data into a buffer (where you have to handle the size of incoming data vs. the size of the buffer), THEN…play the audio from the buffer or handle any errors that occurred along the way. Now, in any half decent modern API, surely, you’d have a perfected version of the algorithm where you could simply plug in a URL and instruct a magical black box to play it. But alas no, each developer is to implement the over complicated, primitive audio queue… just for a streaming radio application. Now combine that with the following multitasking API, and you’re in developer hell.

One of the official pages on backgrounding is nearly impossible to find on the Apple site, so here it is: http://developer.apple.com/iphone/library/documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/BackgroundExecution/BackgroundExecution.html Now, if you look closely, you’ll see that most sections are detailed nicely….except background audio, fantastic. So, taking a quick look at that section reveals that if you simply “flag” your application as one that streams audio, you’re good to go. Hooray right? Well, sort of, not really, no.

Hurdle 1: Hooking into iPod controls

Surprisingly, this is the EASIEST part of implementing background audio. It seems to “just work” as long as you have fairly simple start/stop/pause methods for your radio streamer. One gotcha is that state changes can now take place from both your play/stop button and the iPod controls… so, basically you should be using some sort of event system for state changes and to reupdate the UI whenever your app hits the foregound again.

Hurdle 2: Interruptions

Interruptions have existed since OS2. They cause your music to stop, usually when someone rings. However, now your application has to act more like the iPod application…except you can’t! Unlike the iPod application, which automatically restarts after you finish your call, the only way your application can automatically restart it’s stream is if the user brings it back to the foreground. Not only that, I’ve found that interruption events are very inconsistent. Sometimes they fire simultaneously with the interruption, sometimes they won’t fire until you bring the application to the foreground. On the times that they don’t fire with the interruption, I find that some sort of crazy auto deallocation of the streamer occurs which I can only fix with a terrible workaround. I am yet to find the actual cause of this.

Hurdle 3: iPhone Simulator

It sucks. Testing background audio with it is impossible! Even in the GM build, there is a bug that mutes all sound from your application when it is backgrounded. This makes it rather difficult to tell whether or not your application IS ACTUALLY SITTING IN THE BACKGROUND.

Hurdle 4: You’re free to do whatever you want…except when you need do something

So, as explained before, if you flag your application for streaming audio, your application can go ahead and play audio and do what it needs to do without any particular time limit or suspended state. Cool! Sort of like a free pass to do whatever the hell I want, right? Wrong. There’s a HUGE exception here: “if your application stops playing audio while in the background, your application is suspended.” Wow, that’s fantastic. So let me get this right…if my STREAMING AUDIO APPLICATION happens to need to BUFFER (i.e. STOP PLAYING AUDIO) while on a dodgy cellular connection, my application is essentially…fucked. Yes, that’s right. If you have a little bit of a network hiccup, and need to buffer, the user will need to jump back into the application to restart the audio stream! Not even the iPod controls (which are meant for external use) allow you restart the stream. I haven’t yet found a fix for this yet, but I’m assuming I’m going to have to go back to that mindfuck of a primitive audio queue to fill it with some sort of dummy audio while it buffers, so that iOS doesn’t fucking suspend my application in the middle of playing music. Doesn’t this found like fun?

Hurdle 5:Backwards Compatibility

Finally, as always, you need to aim for backwards compatibility. Apple claims to have completely culled OS2.x, but let’s face it, there are still 1st Generation iPod Touch users that DIDN’T pay the iOS3/4 upgrade tax, that want to use apps. You have to, at a minimum, ensure your application works on OS3 devices, which means making sure that some method calls, that are iOS4 specific, actually exist. Then using the simulator to test………. /facepalm.

FML.

int x, y; /* shared variables */
int r1, r2, r3; /* private variables */

void foo_0(void)
{
	ACCESS_ONCE(x) = 1;
}

void foo_1(void)
{
	r1 = x;
	smp_rmb();  /* The only change. */
	r2 = y;
}

void foo_2(void)
{
	y = 1;
	smp_mb();
	r3 = x;
}

assert(!(r1 == 1 && r2 == 0 && r3 == 0));

Hi, I'm Dave and it has been 10 days since I ran up a new slice.

I have been using slicehost for over 2 years to host various VMs. I run a Free / Open Source focused IT Consulting and programming business based in Victoria, Australia.

This blog contains a good mix of geeky stuff about whatever we are working on at DHC, rants about annoying things / companies along with the occasional thing from "real life". A lot of the stuff that gets blogged about is running on slicehost VMs or at least has been tested on them.

This blog is now being syndicated on the Slicehost Planet. The future of the Slicehost planet is unclear, support have suggested that it suffer the same fate as USA-193, while others insist it is staying. For now at least they are adding new blogs, so long as they hosted on their VMs. If you have a blog running on a slice, email support at slicehost and ask them to add your feed to the planet.

Howdy,  

The Linux Australia council is now requesting formal bid proposals for hosting LCA2012. As you can see from our timeline below, we've slipped a bit on announcements and the close for bid proposals is only two weeks away!!

read more

“As the light level decreases out of the reciprocity range, the increase in duration, and hence of total exposure, required to produce an equivalent response becomes higher than the formula states” (see Wikipedia entry).

Those of us coming from having shot a lot of digital, especially when you’re experience of low light photography is entirely with digital are going to get a bit of a shock at some point. Why didn’t this image work exactly as I wanted it to? Why isn’t there as much.. well.. image!

You’ll probably read things like “you don’t need to worry about it until you’re into really long exposures” or maybe you’ll start reading the manufacturers documents on the properties of the film and just go “whatever”.

Ilford Delta 3200 Professional is one of the films where you have to start caring about it pretty quickly. Basically, you need to overexpose once you start getting exposures greater than ~1second.

In decent light, handheld with a pretty quick exposure, things look great:

But whack things on a Tripod and have a bit of a longer exposure you’re going to start failing a bit. Even though I like this shot, I find that it’s just not quite got everything I would have liked to capture. Just exposing a bit more I think would have done it. I had to do too much in scanning and the GIMP…

So I learnt something with this roll, which is always good.

This was my first real play with darktable. It’s a fairly new “virtual lighttable and darkroom for photographers” but if you are into photography and into freedom, you need to RUN (not walk) to the install page now.

My first real use of it was for a simple image that I took from my hotel room when I was in Hong Kong last week. I whacked the fisheye on the D200, walked up to the window (and then into it, because that’s what you do when looking through a fisheye) and snapped the street scene below as the sun was going away.

I’d welcome feedback… but I kinda like the results, especially for a shot that wasn’t thought about much at all (it was intended as a just recording my surroundings shot).

The second shot I had a decent go at was one I snapped while out grabbing some beers with some of the Rackspace guys (Hi Tim and Eddie!) in Hong Kong. Darktable let me develop the RAW image from my D200 and get exactly the image I was looking for…. well, at least to my ability so far. Very, very impressed.

Being a photographer and using Ubuntu/GNOME has never been so exciting. Any inclination I had of setting up a different OS for that “real” photo stuff is completely gone.

(Incidently, I will be talking about darktable at LUV in July)

I’ve developed some more film! Here’s some shots from last time I was in Hobart. All shot on Ilford HP5+, which I quite like. I’m still getting used to this developing thing and next time should be much better!

The HP5+ was shot at the box speed of 400 with my Nikon F80 and the wonderful 50mm f1.8 lens. I developed in R09 OneShot (Rodinal) for the standard 6 minutes that the Ilford box tells me to. I used my Epson V350 Photo scanner to scan the negatives with iscan. I am wishing for better scanning software. *seriously* wishing.

These first four are probably going to be recognisable to anybody who knows Salamanca.

For those who love the Lark Distillery or English Bitter, I snapped a shot of (one of) the pint I had:

So I’d count this as fairly successful! Of course, need some animal shots:

… and there was a stop at a Sustainability Expo that had a surprising lack of bountiful vegan food when we got there…

I have to say, pulling that film out and seeing an image is incredibly rewarding.

If you want to know more about how I do it all on Linux, come to my talk at LUV this upcoming July 6th.

Monty’s talk at Velocity 2010 seemed to go down really well (at least from reading the agile admin entry on Drizzle). There are a few great bits from this article that just made me laugh:

“Oracle’s “run Java within the database” is an example of totally retarded functionality whose main job is to ruin your life”

Love it that we’re managing to get the message out.

I have kind of a weird idea for a database person.

Forgetting should be built into our applications by default. I just spent the weekend at FooCamp, and I held a session to discuss this idea, and some of the possible consequences if it were implemented.

To explain why I think this, I’m going to take an extreme stance for a moment and argue a position that I’d like to see rebutted. So, please have at it!

For too long we have allowed decisions made by developers – default application settings – to determine what ultimately become surveillance levels.

There are notable counter examples: 4chan intentionally expires postings every few days. Riseup keeps no logs. The EFF documents what we do and do not legally need to keep. These, however, are the efforts of a tiny minority when considered against the rest of the web.

Over time, our conception of what is reasonable has changed around logging and accounting for vast periods of our activities. Never before would a silly recording taken by a 15-year old be stored indefinitely, and then be documented as a watershed event because of how many times it was viewed in a vast global network, rather than for the content of the cultural artifact itself. The log of views themselves were the cultural artifact, and it is celebrated.

Fading away isn’t evil. But we act like it is when we pipe what once was ephemeral into archive.org indefinite storage.

Why have we decided to participate in this social experiment? It really wasn’t a collective decision. Some software developers and investors decided that archival on a massive scale was important or profitable. We started calling these things “part of history” and just storing them without thinking about it. Saving became default.

I’m not saying that archiving the internet, search robots or “opting in” are bad things. But those who least understand archiving’s effect on personal privacy may be the ones most likely to suffer in the future.

The ripple effects of the decision to move from “default expire” to “default save” are vast. Consider for a moment if we were to call the ability to intentionally forget on the internet a human right.

Instead, what we’ve done is to say to millions of people – you do not have the right to forget. Companies will take your locations and status updates, and never delete them. And privacy is rapidly becoming a privilege of those who can afford to buy it.

For the sake of argument, consider the difference between narrative historical documentation and collections of “facts.” The narrative is an aggregation, full of embellishments and forgetting and kernels of truth. Facts are collected, supposedly objectively. Both approaches to capturing historical thought suffer from the fallacy that historical “fact” is fixed and doesn’t evolve based on the viewer and reteller over time. How much worse is this effect when our collections of facts are now ballooning to include every blog post, photo, tweet and web access log you’ve ever made?

The point is not that individuals wish to change history or even obscure events which may reflect poorly on them. (Even though we all do!)

We need to give people a real choice – not a set of ACLs and rules. Choice about what is archived about them, control over that process and a clear delineation between personal artifact and public property.

Kathy Sierra deleted her twitter stream and was accused of removing a piece of history, and possibly the worse internet offense – taking away conversations. Taken at face value, isn’t that the point of conversation? That it is ephemeral?

Conversations leave echos in changed thoughts and light or deep impressions in the minds of the participants. Just because Twitter has by default chosen to retain these conversations indefinitely doesn’t change the nature of conversation itself. No one would argue that just because we share our thoughts that we are obligated to share every thought.

In the same way, we are not obligated to maintain a record of our sharing. And if we do maintain and share a record of our own end of a conversation, we still have the right to ultimately destroy it.

Once shared, of course, an artifact of a conversation can’t be taken away from those that have copies. But authors and owners of the original work must always retain the right to destroy.

So, that brings me to what is ethical in our applications. When we say: “we’re keeping your data forever” and “delete means your account will still be here when you come back”, application developers and companies are making an ethical choice. They are saying, “your shared thoughts aren’t your own – to remember or forget. We are going to remember all of this for you, and you no longer have the right to remove them.”

Connectedness is not the same as openness. Storing vast logs of data related to individuals which connect thousands of facts over the course of their lives should be presented as the ethical choice it is, rather than a technical choice about “defaults”. Picking what we decide to log and store is an ethical and political decision. And it should also be possible for it to be a personal decision.

No related posts.

int x, y; /* shared variables */
int r1, r2, r3; /* semi-private variables */

void foo_0(void)
{
        ACCESS_ONCE(x) = 1;
}

void foo_1(void)
{
        r1 = ACCESS_ONCE(x);
        smp_mb();
        r2 = ACCESS_ONCE(y);
}

void foo_2(void)
{
        ACCESS_ONCE(y) = 1;
        smp_mb();
        r3 = ACCESS_ONCE(x);
}

assert(!(r1 == 1 && r2 == 0 && r3 == 0));

Over at the Drizzle blog, the recent 2010-06-07 tarball was announced. This tarball release has my fixes for the ENUM type, so that it now works as it should. I was quite amazed that such a small block of code could have so many bugs! One of the most interesting was the documented limit we inherited from MySQL (see the MySQL Docs on ENUM) of a maximum of 65,535 elements for an ENUM column.

This all started out from a quite innocent comment of Jay‘s in a code review for adding support for the ENUM data type to the embedded_innodb engine. It was all pretty innocent… saying that I should use a constant instead of the magic 0×10000 number as a limit on an assert for sanity of values getting passed to the engine. Seeing as there wasn’t a constant already in the code for that (surprise number 1), I said I’d fix it properly in a separate patch (creating a bug for it so it wouldn’t get lost) and the code went in.

So, now, a few weeks after that, I got around to dealing with that bug (because hey, this was going to be an easy fix that’ll give me a nice sense of accomplishment). A quick look in the Field_enum code raised my suspicions of bugs… I initially wondered if we’d get any error message if a StorageEngine returned a table definition that had too many ENUM elements (for example, 70,000). So, I added a table to the tableprototester plugin (a simple dummy engine that is loaded for testing the parsing of specially constructed table messages) that had 70,000 elements for a single ENUM column. It didn’t throw an error. Darn. It did, however, have an incredibly large result for SHOW CREATE TABLE.

Often with bugs like this I may try to see if the problem is something inherited from MySQL. I’ll often file a bug with MySQL as well if that’s the case. If I can, I’ll sometimes attach the associated patch from Drizzle that fixes the bug, sometimes with a patch directly for and tested on MySQL (if it’s not going to take me too long). If these patches are ever applied is a whole other thing – and sometimes you get things like “each engine is meant to have auto_increment behave differently!” – which doesn’t inspire confidence.

But anyway, the MySQL limit is somewhere between 10850 and 10900. This is not at all what’s documented. I’ve filed the appropriate bug (Bug #54194) with reproducible test case and the bit of problematic code. It turns out that this is (yet another) limit of the FRM file. The limit is “about 64k FRM”. The bit of code in MySQL that was doing the checking for the ENUM limit was this:

/* Hack to avoid bugs with small static rows in MySQL */
  reclength=max(file->min_record_length(table_options),reclength);
  if (info_length+(ulong) create_fields.elements*FCOMP+288+
      n_length+int_length+com_length > 65535L || int_count > 255)
  {
    my_message(ER_TOO_MANY_FIELDS, ER(ER_TOO_MANY_FIELDS), MYF(0));
    DBUG_RETURN(1);
  }

So it’s no surprise to anyone how this specific limit (the number of elements in an ENUM) got missed when I converted Drizzle from using an FRM over to a protobuf based structure.

So a bunch of other cleanup later, a whole lot of extra testing and I can pretty confidently state that the ENUM type in Drizzle does work exactly how you think it would.

Either way, if you’re getting anywhere near 10,000 choices for an ENUM column you have no doubt already lost.

If you have tried Quick start guides: MySQL cluster in 10 minutes, you may have realized that it is really quick and easy. However, it leaves some typing to be done. Users of MySQL Sandbox have a horror of repetitive typing, and this got me thinking. "Could I integrate MySQL Sandbox and Cluster?" The answer was: "Sure." But then I started thinking of all the minor and major changes that I wanted to do to the Sandbox and have delayed for too long. What I need, is a radical refactoring. And then I remembered that it has been almost two years since I learned a new programming language and that perhaps I could expand my horizons and the Sandbox architecture at once.

$ make_cluster.py ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ tar -xzf ~/Downloads/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64.tar.gz

++ low_level_make_sandbox --basedir=$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64 --sandbox_directory=mcluster --install_version=5.1 --sandbox_port=5144 --no_ver_after_name --no_run --force --my_clause=log-error=msandbox.err --my_clause=ndbcluster 
    The MySQL Sandbox,  version 3.0.12
    (C) 2006-2010 Giuseppe Maxia
installing with the following parameters:
upper_directory                = $HOME/sandboxes
sandbox_directory              = mcluster
sandbox_port                   = 5144
check_port                     = 
no_check_port                  = 
datadir_from                   = script
install_version                = 5.1
basedir                        = $HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64
tmpdir                         = 
my_file                        = 
operating_system_user          = gmax
db_user                        = msandbox
db_password                    = msandbox
my_clause                      = log-error=msandbox.err ; ndbcluster
prompt_prefix                  = mysql
prompt_body                    =  [\h] {\u} (\d) > 
force                          = 1
no_ver_after_name              = 1
verbose                        = 
load_grants                    = 1
no_load_grants                 = 
no_run                         = 1
no_show                        = 
loading grants
.. sandbox server started
stopping server
Your sandbox server was installed in $HOME/sandboxes/mcluster

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgmd -f $HOME/sandboxes/mcluster/my_cluster/conf/config.ini --initial --configdir=$HOME/sandboxes/mcluster/my_cluster/conf/
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.44 ndb-7.1.4b
2010-06-28 21:29:57 [MgmtSrvr] INFO     -- Reading cluster configuration from '$HOME/sandboxes/mcluster/my_cluster/conf/config.ini'

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndbd -c localhost:1186
2010-06-28 21:29:57 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1

++ mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 (not connected, accepting connect from localhost)
id=4 (not connected, accepting connect from localhost)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


++ $HOME/sandboxes/mcluster/clear

++ $HOME/sandboxes/mcluster/start
... sandbox server started
Please wait. Giving the cluster time to catch up
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 0 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)


It may take up to 2 minutes to initialize ... ( 5 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @localhost  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 (not connected, accepting connect from any host)

[...]

It may take up to 2 minutes to initialize ... ( 100 )
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=3 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0, Master)
id=4 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)

[mysqld(API)] 1 node(s)
id=50 @127.0.0.1  (mysql-5.1.44 ndb-7.1.4)


++ $HOME/sandboxes/mcluster/use -vvv -e "create table test.t1(i int not null primary key)engine=ndb"
--------------
create table test.t1(i int not null primary key)engine=ndb
--------------
Query OK, 0 rows affected (0.45 sec)

++ $HOME/sandboxes/mcluster/use -vvv -e "show create table test.t1\G"
--------------
show create table test.t1
--------------

*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `i` int(11) NOT NULL,
  PRIMARY KEY (`i`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

to shut down the cluster, type:
$HOME/python/mysql-cluster-gpl-7.1.4b-osx10.6-x86_64/bin/ndb_mgm -e shutdown

iOS4 has GM’d, but Apple is yet to finalize their Game Center app. Currently, there are two different versions of iOS 4.0. Firstly, the publicly available version and an developer’s version. The developers version is identical to the official update, but includes the Game Center app for developers. Unfortunately, tools like redsn0w, sn0wbreeze and PwnageTool don’t support the developers IPSW. Fortunately, PwnageTool is incredibly customizable for patching firmware, so a few customizations makes it accept the developers IPSW.

1) Download and extract PwnageTool 4.01

2) View the package contents of PwnageTool.app and browse to PwnageTool.app/Contents/Resources/FirmwareBundles/<Your iPhone/iPod firmware version>.bundle

3) Within this bundle is an Info.plist, open it and make the following modifications to the following keys:

3a) RootFilesystem: needs to match the root FS dmg filename in the developer IPSW. Find this by opening the IPSW with a ZIP application and find the filename of the DMG that is approximately 300-400MB in size (it should be something like xxx-xxxx-xxx.dmg)

3b) RootFilesystemKey: each root FS is encrypted. Finding the key is difficult. Instead, search for the DMG filename in Google and you should find a few places that host the key for that particular DMG.

3c) RootFilesystemMountVolume: usually this is something like ApexXXXXX.iPhoneOS. Change this to ApexXXXXX.iPhoneDeveloperOS.

3d) SHA1: this needs to match the SHA1 of the developer IPSW. You can find this by using the command “openssl sha1 <IPSW filename>”

4) Now you can save the Info.plist, open up PwnageTool and select the developer IPSW. Hooray!

Apache Solr is an excellent full text index search engine based on Lucene. Solr is increasingly being used in the Drupal community for search. I use it for search for a lot of my projects. Recently Steve Edwards at Drupal Connect blogged about setting up a mutli core Solr server on Ubuntu 9.10 (aka Karmic). Ubuntu 10.04LTS was released a couple of months ago and it makes the process a bit easier, as Apache Solr 1.4 has been packaged. An additional advantage of using 10.04LTS is that it is supported until April 2015, whereas suppport for 9.10 ends in 10 months - April 2011.

As an added bonus in this howto you will be able to auto provision solr cores just by calling the right URL.

In this tutorial I will be using Jetty rather than tomcat which some tutorials recommend, as Jetty performs well and generally uses less resources.

Install Solr and Jetty

Installing jetty and Solr just requires a simple command

$ sudo apt-get install solr-jetty openjdk-6-jdk

This will pull down Solr and all of the dependencies, which can be alot if you have a very stripped down base server.

Configuring Jetty

Configuring Jetty is very straight forward. First we backup the existing /etc/default/jetty file like so:

sudo mv /etc/default/jetty /etc/default/jetty.bak

Then simply change your /etc/default/jetty to be like this (the changes are highlighted):

# Defaults for jetty see /etc/init.d/jetty for more

# change to 0 to allow Jetty to start
NO_START=0
#NO_START=1

# change to 'no' or uncomment to use the default setting in /etc/default/rcS 
VERBOSE=yes

# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
#JETTY_USER=jetty

# Listen to connections from this network host (leave empty to accept all connections)
#Uncomment to restrict access to localhost
#JETTY_HOST=$(uname -n)
JETTY_HOST=solr.example.com

# The network port used by Jetty
#JETTY_PORT=8080

# Timeout in seconds for the shutdown of all webapps
#JETTY_SHUTDOWN=30

# Additional arguments to pass to Jetty    
#JETTY_ARGS=

# Extra options to pass to the JVM         
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true"

# Home of Java installation.
#JAVA_HOME=

# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in /etc/default/jetty). Should contain a list of space separated directories.
#JDK_DIRS="/usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun"

# Java compiler to use for translating JavaServer Pages (JSPs). You can use all
# compilers that are accepted by Ant's build.compiler property.
#JSP_COMPILER=jikes

# Jetty uses a directory to store temporary files like unpacked webapps
#JETTY_TMP=/var/cache/jetty

# Jetty uses a config file to setup its boot classpath
#JETTY_START_CONFIG=/etc/jetty/start.config

# Default for number of days to keep old log files in /var/log/jetty/
#LOGFILE_DAYS=14

If you don't include the JETTY_HOST entry Jetty will only bind to the local loopback interface, which is all you need if your drupal webserver is running on the same machine. If you set the JETTY_HOST make sure you configure your firewall to restrict access to the Solr server.

Configuring Solr

I am assuming you have already installed the Apache Solr module for Drupal somewhere. If you haven't, do that now, as you will need some config files which ship with it.

First we enable the multicore support in Solr by creating a file called /usr/share/solr/solr.xml with the following contents:

<solr persistent="true" sharedLib="lib">
 <cores adminPath="/admin/cores" shareSchema="true" adminHandler="au.com.davehall.solr.plugins.DHCSolrCoreAdminHandler">
 </cores>
</solr>

You need to make sure the file is owned by the jetty user if you want it to be dymanically updated, otherwise change persistent="true" to persistent="false", don't include the adminHandler attribute and don't run the commands below. Also if you want to auto provision cores you will need to download the jar file attached to this post and drop it into /usr/share/solr/lib.

sudo chown jetty:jetty /usr/share/solr
sudo chown jetty:jetty /usr/share/solr/solr.xml
sudo chmod 640 /usr/share/solr/solr.xml
sudo mkdir /usr/share/solr/cores
sudo chown jetty:jetty /usr/share/solr/cores

To keep your configuration centralised, symlink the file from /usr/share/solr to /etc/solr. Don't do it the other way, Solr will ignore the symlink.

sudo ln -s /usr/share/solr/solr.xml /etc/solr/

Solr needs to be configured for Drupal. First we backup the existing config file, just in case, like so:

sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.orig.xml
sudo mv /etc/solr/conf/solrconfig.xml /etc/solr/conf/solrconfig.orig.xml

Now we copy the Drupal Solr config files from where you installed the module

sudo cp /path/to/drupal-install/sites/all/modules/contrib/apachesolr/{schema,solrconfig}.xml /etc/solr/conf/

Solr needs the path to exist for each core's data files, so we create them with the following commands:

sudo mkdir -p /var/lib/solr/cores/{,subdomain_}example_com/{data,conf}
sudo chown -R jetty:jetty /var/lib/solr/cores/{,subdomain_}example_com

Each of the cores need their own configuration files. We could implement some hacks to use a common set of configuration files, but that will make life more difficult if we ever have to migrate some of cores. Just copy the common configuration for all the cores:

sudo bash -c 'for core in /var/lib/solr/cores/*; do cp -a /etc/solr/conf/ $core/; done'

If everything is configured correctly, we should just be able to start Jetty like so:

sudo /etc/init.d/jetty start

If you visit http://solr.example.com:8080/solr/admin/cores?action=STATUS you should get some xml that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<response>
	<lst name="responseHeader">
		<int name="status">0</int>
		<int name="QTime">0</int>
	</lst>
	<lst name="status"/>
</response>

If you get the above output everything is working properly

If you enabled auto provisioning of Solr cores, you should now be able to create your first core. Point your browser at http://solr.example.com:8080/solr/admin/cores?action=CREATE&name=test1&i... If it works you should get output similar to the following:

<?xml version="1.0" encoding="UTF-8"?>
<response>
	<lst name="responseHeader">
		<int name="status">0</int>
		<int name="QTime">1561</int>
	</lst>
	<str name="core">test1</str>
	<str name="saved">/usr/share/solr/solr.xml</str>
</response>

I would recommend using identifiable names for your cores, so for davehall.com.au I would call the core, "davehall_com_au" so I can easily find it later on.

Security Note: As anyone who can access your server can now provision solr cores, make sure you restrict access to port 8080 to only allow access from trusted IP addresses.

For more information on the commands available, refer to the Solr Core Admin API documenation on the Solr wik.

Next in this series will be how to use this auto provisioning setup to allow aegir to provision solr cores as sites are created.

I wish that I was writing about a new code for style but instead I'm writing about poor coding style. I'll keep it brief.

1. If you have a very large function that is doing lots of things, its a sign that some refactoring might be in order. If not for yourself, to aid others who have to work on the code.
2. If the things are interlaced for no apparently good reason, you've created a maintenance monster.

It's time again for the LUV Beginner's Workshop. Come along on July 17. 

For more information, see http://luv.asn.au/2010/07/17

• Stewart Smith - Photographers workflow: from the lense to the GIMP
• Andrew Chalmers - State of the L4Ka Pistachio microkernel.

For more information, visit http://luv.asn.au/2010/07/06

I am writing this blog post with Vim, my favorite editor, instead of using the online editor offered by blogger. And I am uploading this post to my Blogger account using Google CL a tool that lets you use Google services from the command line. I am a command line geek, and as soon as I saw the announcement, I installed it in my laptop. The mere fact that you are reading this blog post shows that it works.

$  google blogger --blog="The Data Charmer" --title=partitioning list "title,url"

$  google blogger --blog="The Data Charmer" --title=".*partitioning" list "title"
Holiday gift - A deep look at MySQL 5.5 partitioning enhancements
The partition helper - Improving usability with MySQL 5.1 partitioning
A quick usability hack with partitioning
MySQL 5.1  Improving ARCHIVE performance with partitioning

$  google blogger --blog="The Data Charmer" --title="(?i).*partitioning.*" list "title"
Holiday gift - A deep look at MySQL 5.5 partitioning enhancements
Partitioning with non integer values using triggers
Tutorial on Partitioning at the MySQL Users Conference 2009
The partition helper - Improving usability with MySQL 5.1 partitioning
A quick usability hack with partitioning
MySQL 5.1  Improving ARCHIVE performance with partitioning

$google picasa post -n "Blogger Pictures" -t googlecl ~/Desktop/google_cl.png 

$ google picasa list -n "Blogger Pictures" --query googlecl title,url_direct 
google_cl.png,http://lh6.ggpht.com/_gVfZHGgf5LA/TBzjaKiJJvI/AAAAAAAAA74/dthDDhybsmc/google_cl.jpg

google blogger --blog="The Data Charmer" --draft --title "Welcome googleCL" --tags="google,mysql,partitioning,command line,blogging" post ~/blog/welcome_googlecl.html

Our site hasn't changed very much over the last 4 years, but the business has changed a lot. The biggest change was the (uneventful and long overdue) upgrade to Drupal 6 a few months ago.

During the last week or so the site has been updated and refocused. The major changes include:

• The home page focuses on our main services
• The services section has been updated to reflect the services we currently offer
• Added a projects section to showcase the contributions we have made to various projects over the last decade
• The footer has been rethought and now provides links to key parts of the site and relevant off site links
• Implemented the custom breadcrumbs modules to improve navigation within the site
• Updated service links module to provide a more diverse selection of social networking links, including identi.ca with a patch
• Offloaded some RSS feeds onto feedburner, through the drupal contrib module

This also signals our return to regular blogging. There are a few posts in the pipeline. There should be a good mix of drupal and sys admin posts in the coming weeks.

As always, feedback is welcome.

I'm pleased to announce that the eXtensible Host Controller Interface (xHCI) 1.0 specification is now publicly available on intel.com. This is the specification for the PC hardware that talks to all your USB 3.0, USB 2.0, and USB 1.1 devices. (Yes, there are no more companion controllers, xHCI is the one host controller to rule them all).

Open, public documentation is always important to the open source community. Now that the spec is open, anyone can fully understand my Linux xHCI driver (although it's currently only compliant to the 0.96 xHCI spec; anyone want to help fix that?). This also means the BSD developers can implement their own xHCI driver.

Curious what a TRB or a Set TR Deq Ptr command is? Want to know how device contexts or endpoint rings work? Go read the spec!

struct foo {
    int a;
    int b;
};
struct foo static_foo = { 42, 17 };
struct foo *foo_p = &static_foo;

p = foo_p;
if (p != NULL)
    do_something_with(p->a, p->b);

Perhaps this is a familiar refrain for some readers… but I’ll say it again: Linode rock! On their 7th birthday, they are giving current and new users a huuuge RAM upgrade —  around 42% on average. From humble beginnings as a Linode 64 five years ago, my Linode 512 is now a Linode 768. Yikes!

“Good news. Linode’s getting older.”
“… and Leon is getting laaaaarger!”

Already blessed with plenty of memory available for cache (which is very handy on a VM with relatively slow I/O), I now have even more headroom:

Video of the LFX workshop talks is available now, including my talk. Enjoy!

I recalled an event I participated in back in 1985 which was the Mosgiel 100 year centenary parade. As a pupil of one of the local primary schools (Reid Park Primary) we were primed with dress of the era and a song that was nowhere near as bad as some radio jingles. We got to ride on a float down the main street, oh what joy!

“I’m a hundred year old kid – don’t you see,
I’m proud to see the end of our first century.
From 1885 through ’til 1985,
We’ll still be century children at our next centenary.”

Author unknown, can you help?

Today, I start at Emma.

I have some clues about what I’ll be up to – working on some big PostgreSQL databases, tearing into the infrastructure and discovering what makes their small company tick. Emma’s work with small businesses, and a focus on humane communication and consensus building completely drew me in.

And I’m looking forward to riding my bike everyday over on the office on Burnside!

Related posts:

1. Change in the air
2. PostgreSQL Conference Day 1
3. Women Who Code – where are they?

Our initial WebM support has landed on mozilla-central. The main holdup was ensuring that Google's VP8 source license was GPL-compatible; that has now been resolved to everyone's satisfaction. Good stuff!

Of course, this is only the beginning. Currently we're working on support for the 'buffered' attribute, and there are many other bug fixes and improvements to make.

You know already that InnoDB in MySQL 5.5 has great improvements in performance and scalability. You will have to wait a few months for that, though, because MySQL 5.5 is not GA yet. But if you need some extra performance in MySQL 5.1, you may want to use the Innodb Plugin instead of the built-in one. As of version 5.1.47, the Innodb plugin is of GA quality, and it comes with a good out-of-the-box improvement compared to the built-in engine.

innodb_buffer_pool_size=5G

sysbench \
    --test=oltp \
    --oltp-table-size=1000000 \
    --mysql-db=test \
    --mysql-user=$USER \
    --mysql-password=$PASSWD \
    --mysql-host=$HOST \
    --mysql-port=$PORT \
    --max-time=60 \
    --oltp-read-only=$ON_OFF \
    --max-requests=0 \
    --num-threads=8 run

struct rcu_protected {
  int a;
};
struct rcu_protected elements[3];
struct rcu_protected *current = &elements[0];
int consistent;  /* must be 2 * current->a */

I have been a Palm fan for a very long time. My first PDA was a PalmPilot Professional, which eventually led to the m100 (AA batteries!), my first Palm OS phone, Pia’s hand-me-down Tungsten W, the Treo 650 and finally, my last (real) phone: the Treo 680. I’m even vaguely sad that I missed the final outing of Palm OS on the Centro.

I was disappointed when Palm split into two companies, but perhaps it was ultimately a good thing… PalmSource became victim to an almighty corporate Sarlacc, to be digested slowly over a thousand years, while Palm became nimble and determined to win out of necessity.

The ill-fated Foleo was a public relations disaster, but in retrospect, a concept way ahead of its time — consider the explosion of the netbook market soon after, and where the iPad is today (albeit generations of mobile technology beyond the capabilities of the Foleo).

Which brings us to webOS, released a year ago today on the Palm Pre. Having waited impatiently all that time (often running to the emulator to get my fix), I finally acquired one last week.

It is a truly delicious user experience. I won’t go into too much detail on this front — as with most things, you must use it — suffice to say that it carries the soul of Palm OS in a 21st century vessel.

(Okay, one thing on this: When it comes to the basic functions of a smart phone — calls, and contacts — I’m convinced this thing has both the iPhone and HTC’s Sense UI for the Desire beat. Synergy is seamless and awesome.)

Despite it’s youth, webOS is an incredibly promising and fast-moving platform… and in stark contrast to other “mobile Linux” competitors, it’s not just a bunch of goofy shit piled on top of a heavily molested Linux kernel: What runs on your phone is an utterly recognisable Open Source stack and an utterly recognisable web stack. Sure, there’s a layer of proprietary Java gumpf shoved between the good bits, but even that is getting thinner.

To some extent, webOS is the GNOME Mobile platform with a user interface and services layer built for the web generation. It’s D-Bus, GStreamer and PulseAudio under WebKit. Mojo applications, written in JavaScript, talk to services via JSON APIs, and native apps integrate into the web-based user interface via — get this! — NPAPI plugins.

Crucially, webOS will grow and improve along with the web. Everything you’re seeing in the web world — faster JavaScript, hardware accelerated CSS animation, massive growth of the JavaScript ecosystem (consider all the frameworks, CommonJS modules, nodejs event-based server, etc.) and all the incredible new APIs popping up — will find a place in webOS. Check out the Palm Developer Day keynote and podcast from Dion Almaer and Ben Galbraith for more on the near future of webOS.

On the awesomeness of Dion and Ben at Palm? I’ll just quote James Governor:

Palm tried to use Apple’s trick of secrecy first rather than investing heavily in developer good will and playing the open card. It didn’t work. Palm realised its error last year and did something incredibly smart – it hired Dion Almaer and Ben Galbraith to develop a new, web-savvy, strategy around its platform.

Using open technology as the bricks and mortar of a platform is not particularly amazing of course. Everyone’s doing it. But Palm are making friends and influencing developers by having an impressively open attitude to devices, too. Your store-bought phone — with a bit of Konami code action and the freely downloadable webOS SDK — is already “rooted” for you. Just log in. ps afx? cat /proc/cpuinfo? top? Your “first command” habit is most likely catered for.

That openness has encouraged an incredible amount of community activity. The most Open Source savvy group dedicated to the platform is WebOS Internals. Initially, they published all sorts of juicy information about the innards of webOS and the Pre… but have now rallied around distribution of “home brew” Open Source patches and apps, using their own package management interface, Preware. They’ve even published an updated kernel which supports overclocking, temperature sensors and more advanced power management than the original! It’s wonderful stuff.

Then HP bought Palm.

Despite some messaging hiccoughs (now resolved), this is an incredibly exciting move for Palm and webOS fans. I’m hoping it gives Palm the reach, resources and relationships to go global, accelerate improvement of the platform, and ship some terrific new hardware to make their software shine… that said, it better not be an almighty corporate Sarlacc!

Damn it feels good to be a Palmster.

There's a lazy insolence in the way the locals move slowly about their tasks. It's difficult for me to tell if it's a form of disinterest with their activity or a measured conservation of energy. Perhaps as the young men in front of me leisurely drag air tanks through the waves they are simply wishing that they too were out playing in the ocean like the near by children. Certainly their bodies don't speak of a easy life.

Nusa Lembongan is the first place I've felt at all at ease. It's still full of tourists but at least they are my age and friendly, unlike the crowds of dour faced Australians in Kuta and the smiling but reticent retirees in Sanur.

Down the beach from where I'm drinking coffee and watching the morning unravel, three kids are playing in the waves on a broken surfboard. Taking turns skimming headlong into an oncoming wave to catapult head over heels into the water. Each wave brings shrieks of laughter from their friends and I regret leaving my camera back in my room a little more. More then anything I think it's the happiness of the children that puts me at ease, I'm glad the we're staying here a few days.

Across from me, two men have been sitting in a boat named Chilli for the last hour. They are waiting for something, but it's not clear what. Perhaps a diving tour? A couple of boats down a young European woman is being given a scuba lesson by her boyfriend. Periodically they disappear only to resurface a few minutes later behind a different boat.

I'm fascinated by the little differences. We haven't been slumming it and each place has had the full range of modern conveniences, but with caveats. At one the shower points at my navel and sprays water everywhere from the tap. Other one we think doesn't have hot water until we realise it simply takes the duration of a shower to warm up. Many are just a combined "wet room" which shower, toilet and sink all cohabitating in a tangle of pipes and levers. Some provide soap, some towels, none have bathmats and most are slowly disintegrating. I like the differences, each reflecting the biases of the owner and towns we're staying in.

This is the morning of my eighth day of my trip and while I'm settling into the pace and realities, already I'm wondering why I'm doing this. I feel just as out of place amongst the hard bodied locals and touring surfers as I always have. What do I want from this trip, am I chasing a childhood dream? I can clearly imagine friends and family rolling their eyes as I over think this as well, but its in my nature to try and understand.

I didn't expect the trash. The constant mosaic of excrement, plastic bags and bottles, cigarette butts, food scraps, old clothes and candy wrappers. I'm not sure why I didn't expect it, but I didn't. In my own country this neglect makes me angry, here I have no right to anger and instead it leaves me numb.

I'm getting used to being measured at every encounter, my new friend quickly determining the fastest route to my callow wallet. Everyone knows somewhere I can stay, can drive me somewhere or has a pretty girl who will blow my mind. It's easy to get overwhelmed, so I've learned to step back, smile and keep walking. This seems better understood then any plea of "no", which is taken as the opening bid in the coming negotiation. I admire their collective organisation, once convinced my wallet isn't making an appearance, they are generous with advice about where to go and what to see. They have the network effect down pat, knowing that everything comes back around eventually.

The smell of rubbish is the backdrop for the constant offers of girls and cries of "transport". A Balinese taxi driver tells me about his wife and children before regaling me with tales of the western women he's fucked. A bemo driver boasts of his conquests, disdainfully describing the whores as Javanese, while in the same breath, offering to find me a nice Java girl. In the hotel pool, a traveling Malay salesman asks to hold our hands and describes in detail the perfect nipples of the girl sleeping in his room. He offers to introduce us to her and while two Canadians laugh and joke, psyching themselves up to the occasion, I slip away to my room.

It's a family affair, mothers and boyfriends watch from the corners of alleyways and bars as their girls whore themselves. Their eyes display a nonchalance which leaves me distinctly uncomfortable and confused. Husbands are texted requesting a pick up before running off into the darkness with their latest catch. The offers are so pervasive, and the environment so accepting, that it's impossible not to be curious. To wonder about these girls with their slender hips and perfect nipples, to imagine what an hour spent with them might be like. It's easy to see how men who would never be tempted at home, end up paying for the girl in their bed.

Today the Bali part of the trip ends. We travel to Gillimanuk to catch the ferry to Banyuwangi, the eastern most point of Java. The last two weeks have been spent making friends, swimming, riding scooters and surfing; but most importantly getting used to the routines of a traveler in a new country.

It's time to move.

This is the final installment of a 3-part series on building a visual status screen, or dashboard, for your company. If you missed the previous parts you can find it at:

 Building a Status Screen, part 1: Hardware

 Building a Status Screen, part 2: Collecting Data

In the first installment I showed how to physically mount LCDs on a wall, and in the second I showed how I go about collecting data from various systems (both internal and external) in a central location.

The final step is taking that data and displaying it on the status screens, and then updating the display whenever the data changes. There are a huge number of ways this could be done depending on what sort of system you are accustomed to working with, but since I normally do web application development I decided to take the simple way out and make the status screens using dynamic web pages. Yes, when I'm swinging this PHP hammer every problem looks like it could be solved with a web page!

I'll start by showing you the simplest, most brute-force approach that should be easy to get going. It's a bit crude but it'll get the job done, and later I'll mention some enhancements that can make it more elegant.

In the last installment we ended up with all the relevant data being stored in simple text files on a central machine. With all the data in one place it's easy to create a dynamic web page that is served by that machine and displays the data neatly formatted to suit your LCDs, and by displaying it in a browser like Google Chrome that can run in full-screen mode it can look really pretty.

You'll need some kind of scripting system running on your server to generate dynamic web pages, and there are many alternatives so I won't go into detail here. I'm using PHP but you could do the same thing with Perl, Python, ASP.net, or many others. It's really up to you.

Let's start with just about the simplest case possible, which is a page that loads a single value (the current temperature, in this case) from a text file on disk, displays it, and reloads itself periodically.

<html><head><title></title>

<meta http-equiv="refresh" content="10" />

</head>

<?php

  $temperature = file_get_contents( "/home/statusscreen/data/data-weather-temp" );

  echo "The temperature is " . $temperature . "&deg;";

?>

</body></html>

That's it! Not so hard, right? If you put that code into a file called something like "screen.php" inside the web tree on your server and load it up in a browser, you'll see a mostly blank page with the current temperature displayed on it - assuming your data collector script is updating the "data-weather-temp" file as described in Part 2.

The "meta" tag at the top is a special tag that tells the browser to reload the page periodically, and in this case it'll reload every 10 seconds. If you leave the page open in your browser and alter the value in the "data-weather-temp" file you'll see the change reflected in the web page after a few seconds.

Of course this is a very minimal example, and if you've done any programming it'll be obvious that there are many ways to improve the code above: for example, before attempting to retrieve the contents of the file there should be a check to make sure the file actually exists. Then once the value has been extracted it should probably be cast to a numeric value just in case something nasty ends up in the file. The page itself will also be totally unstyled, simply displaying default black text in the top left corner of the page on a default white background. You'll also notice that the entire page is reloading every single time, when really it's only the temperature value that should need to be fetched from the server.

It's a start though, and with some further development it can end up as quite a decent system. To test it you can open the Google Chrome browser on the computer connected to your wall-mounted status screen, press F11 to go to full-screen mode, and leave it open. The page will continue to reload every 10 seconds so by editing the script on the server you'll be able to see the effect of your changes almost immediately. You can also preview the page directly on your laptop or desktop, of course, but having it open on the machine and screen combination that you'll ultimately leave running can be handy because it may have a different resolution to your own computer. Normally when developing web pages it's a good principle to structure it in such a way that it looks good in different browsers at different resolutions, but in this case you're creating a web page that will only ever be loaded up by one computer running that specific browser at a known resolution, so you can tune it to look exactly the way you want without caring about how well it works on other computers.

The next step is to expand the script to include other bits of data that you want to display on the same screen. In my case I wanted a simple 4x3 grid layout for one of the screens so it made sense to put the different parameters into a table. The result could be something like this:

<?php

$datapath = "/home/statusscreen/data";

$temperature = file_get_contents( $datapath . "/data-weather-temp" );

$brillianz = file_get_contents( $datapath . "/data-ops-br" );

$sb4 = file_get_contents( $datapath . "/data-ops-sb4" );

$projnum = file_get_contents( $datapath . "/data-projects-projnum" );

 //(etc)

?>

<html><head><title></title>

<meta http-equiv="refresh" content="10" />

</head>

<table>

<tr>

  <td><?=$temperature?></td>

  <td><?=$brillianz?></td>

  <td><?=$sb4?></td>

  <td><?=$projnum?></td>

</tr>

</table>

</body></html>

That's functional, but still ugly. We're still dealing with default text on a white page in an unstyled table.

But now you can apply some styling to make it look pretty. By putting class names and IDs on the various elements and loaded an external CSS file you can now make your plain page look as classy as you like. In my version I applied styles to just about everything using CSS: the font size, font colour, text shadow, page background, borders and margins, and cell backgrounds you see in photos of the screens are all just styled with CSS.

At this point you have a system that is not only perfectly functional, but is actually starting to look pretty sweet as well. You could leave the system running exactly like this and be happy.

If you want to take things a little further though there are many things you can do. For example, rather than load the entire page every time you could have the page only load once and then use AJAX to periodically fetch and update data for the individual cells. In my version I have a bunch of Javascript that fetches values and then applies a "fade" effect to update from one value to the next, making the visual transition much smoother than a jarring reload of the whole page.

Because I have three screens on the wall I actually have three pages defined. One uses the simple grid layout described here, while one loads a web-based energy monitoring system created at IVT to display real-time power consumption in different parts of the building, and another displays bar graphs using data collected from various internal systems such as our bug tracker and package autobuilders. What you want to display is totally up to you.

The energy monitoring system is actually based on slightly different technology to what I've described here. It's still a web page, but it uses Adobe Flex to display real-time data that updates dynamically without having to reload the whole page.

I've deliberately become more vague towards the end because this is where you really have to decide what data you want to display and how you want it to appear. I built a system that suits my requirements, but it may not suit yours.

If you've built a status screen for your company I'd love to hear about it, so please drop me an email!

Today I spent a little time setting up a PDXAPI instance of the CivicApps data. There are a few different tools out there for grabbing the data and loading it up, and so I’m documenting the basic steps here for setting up a spatial SQLite using @lokkju’s python projects.

hg clone https://pyspatialite.googlecode.com/hg/ pyspatialite

cd pyspatialite
mv setup.cfg.OSX setup.cfg
python setup.py build
sudo python setup.py install

cd ..
cd pyod

hg clone https://pyod.googlecode.com/hg/ pyod

# unsatisfied dependency!
sudo easy_install pyyaml

pypython fetcher.py

This creates a 1 GB sqlite database called ‘test.sqlite’.

Next, I’ll be testing out loading this into a CouchDB instance and maybe playing with Max Ogden’s initial PostGIS export.

Related posts:

1. Q&A about Hot Standby

I’ve been struggling to write anything of consequence at all recently… but apparently the best way to get back on the horse is to write what you know, so I have embarked on what might be described as an act of niche journalism.

Ubuntu’s Bleeding Edge is a new blog about the action, intrigue and occasional romance of the Ubuntu development branch, for everyone who thinks Release Day parties are only meant to celebrate the start of a new release cycle. It’s an easy way to keep up-to-date about the goings-on of Maverick Meerkat… particularly if the idea of running an ever-changing, alpha-quality operating system scares the bejesus out of you.

The story so far…

• Welcome to Ubuntu’s Bleeding Edge — an introduction and kickoff
• Previously on Maverick… — the first upload summary, documenting the first month of Maverick
• It’s June already… and Alpha 1 Beckons — second upload summary
• A couple of goofy asides about the boredom of freeze week and relief via the first alpha release (both include video for hilarity’s sake)
• Before the floodgates opened — summary of uploads during Alpha 1’s freeze

It’s a nice way to tackle a problem in bite-sized chunks, have a ready excuse to write regularly, and recall how much I used to enjoy those six month roller-coaster release cycles. It sure helps to have an enjoyable subject that I know pretty well, too!

Check it out, and subscribe to the feed if you’re keen to keep up with the ups and downs of Maverick.

Update: My latest devel branch update covers the removal of aptitude from the default desktop install and the switch to building for i686, both of which have become minor controversies.

4 min 20 sec

So next time somebody complains about NDB taking a long time in CREATE TABLE, you’re welcome to point them to this :)

• A single CREATE TABLE statement
• It had ONE column
• It was an ENUM column.
• With 70,000 possible values.
• It was 605kb of SQL.
• It ran on Drizzle

This was to test if you could create an ENUM column with greater than 2¹⁶ possible values (you’re not supposed to be able to) – bug 589031 has been filed.

How does it compare to MySQL? Well… there are other problems (Bug 54194 – ENUM limit of 65535 elements isn’t true filed). Since we don’t have any limitations in Drizzle due to the FRM file format, we actually get to execute the CREATE TABLE statement.

Still, why did this take four and a half minutes? I luckily managed to run poor man’s profiler during query execution. I very easily found out that I had this thread constantly running check_duplicates_in_interval(), which does a stupid linear search for duplicates. It turns out, that for 70,000 items, this takes approximately four minutes and 19.5 seconds. Bug 589055 CREATE TABLE with ENUM fields with large elements takes forever (where forever is defined as a bit over four minutes) filed.

So I replaced check_duplicates_in_interval() with a implementation using a hash table (boost::unordered_set actually) as I wasn’t quite immediately in the mood for ripping out all of TYPELIB from the server. I can now run the CREATE TABLE statement in less than half a second.

So now, I can run my test case in much less time and indeed check for correct behaviour rather quickly.

I do have an urge to find out how big I can get a valid table definition file to though…. should be over 32MB…