SORTING DATA FRAMES IN R

This is a grandfathered post copied across from my old blog when I was using MovableType (who remembers MovableType?!)

I frequently find myself having to re-order rows of a data.frame based on the levels of an ordered factor in R.

For example, I want to take this data.frame:

       product store sales
        1       a    s1    12
        2       b    s1    24
        3       a    s2    32
        4       c    s2    12
        5       a    s3     9
        6       b    s3     2
        7       c    s3    29

And sort it so that the sales data from the stores with the most sales occur first:

   product store sales
   3       a    s2    32
   4       c    s2    12
   5       a    s3     9
   6       b    s3     2
   7       c    s3    29
   1       a    s1    12
   2       b    s1    24

I keep forgetting the exact semantics of how its done and Google never offers any assistance on the topic, so here is a quick post to get it down once and for all, both for my own benefit and the greater good. First we need some data:

   productSalesByStore = data.frame(
         product = c('a', 'b', 'a', 'c', 'a', 'b', 'c'),
         store = c('s1', 's1', 's2', 's2', 's3', 's3', 's3'),
         sales = c(12, 24, 32, 12, 9, 2, 29)
      )

Now construct a sorted summary of sales by store:

   storeSalesSummary =
         aggregate(
                  productSalesByStore$sales,
                  list(store = productSalesByStore$store),
         sum)
   storeSalesSummary =
      storeSalesSummary[ 
         order(storeSalesSummary$x, decreasing=TRUE), 
         ]

storeSalesSummary should look like this:

   store  x
    2    s2 44
    3    s3 40
    1    s1 36

Use that summary data to construct an ordered factor of store names:

   storesBySales =
      ordered(
         storeSalesSummary$store,
         levels=storeSalesSummary$store
         )

storesBySales is now an ordered factor that looks like this:

      [1] s2 s3 s1
  Levels: s2 < s3 < s1

Re-construct productSalesByStore$store so that it is an ordered factor with the same levels as storesBySales

   productSalesByStore$store =
      ordered(productSalesByStore$store, levels=storesBySales)

Note that neither the contents nor the order of productSalesByStore has changed (yet). Just the datatype of the store column. Finally, we use the implicit ordering of store to generate an explicit permutation of productSalesByStore so that we can sort the rows in a stable manner:

   productSalesByStore = 
      productSalesByStore[ order(productSalesByStore$store), ]

And we are done!

PARALLEL READ/WRITE LOCKING AND MY ORACLE ADVENTURE

Back when Oracle 8i was a thing, Fibre Channel was all the rage and on the Oracle roadmap was Oracle Parallel Database, I was called into the Oracle HQ in Redwood City, Silicon Valley.

They'd pushed back the release of "Parallel" quarter after quarter and a couple senior engineers caught wind that I was the guy for doing custom built memory memory managers and I was doing big highly parallelised data structures on large SGI platforms. I had one data structure running on 16 racks of Silicon Graphics kit at a large data centre off highway 101 for an investment bank (aka hedge fund), and I'd earned a reputation for being able to do parallel distributted read/write locking of vast data structures with all CPU threads running at full speed and without any mutex locking or blocking. So, I was summoned to Silicon Valley "for a chat".

Oracle had this grand idea for Oracle 8i to share fibre channel LUNs between hosts, and your federated database would sit on one LUN with multiple Oracle 8i instances on separate machines all accessing the same database in parallel (hence the name 'Parallel'). Oracle at the time was actively influencing the specs of fibe channel (FCAL), but they just couldn't get it to work -- so I was called in so they could pick my brains. The visit was fun, but I was no dummy, and I certainly wasn't going to give up the secrets to how to build the worlds fastest computing systems.

I found the meeting quite entertaining and it descended into an argument over Oracle's outrageous pricing. On a multi-cpu system craylinked together with other multi-cpu systems, why should I pay Oracle a licensing fee for every damn CPU when we had called in Mark Gurry (the guy that wrote the book on Oracle Performance Tuning), tuned the crap out of Oracle so that it barely used a single CPU, maybe two under heavy load. I won the argument and secured special pricing for Oracle moving forward (possibly not what they had intended for our meeting - oh well, that's AP for you!)

A much more youthful looking me standing next to the Oracle lake after our meeting

GEEKZ ON DEMAND

Geekz on Demand (G.O.D) was a HR consultancy I started back in 1997 with Richard Taylor. At the time it was tech boom 1.0, and there was a dearth of talent that properly knew what the Internet was and how to get things onto it.

Entre The Geekbase, and it took off like a rocket, landing us in the news quite consistently:

Rowan and I on the cover of The Age, 27 Jan 1998.

THE START OF THE TECH BOOM

I spent tech boom 1.0 putting as much as I could online.

PRE-TECH BOOM

Not unlike Steve Jobs, I started my tech-career with phone phreaking.

Fresh out of the AFP banging on the front door of my Brighton home in 1993 for phone phreaking and hacking (of course, being underage, no conviction was recorded). We were quite prolific and formed a loose nameless collective of the absolute best of the day. We were awash with so many zero day c0dez and exploits that we had to invent "minus days". Feared by the rest of the hacking community, we had the ability to access any computer system or telephone network we desired. Even Julian Assange (aka 'prof') ph3ared us. Our threat-hunting was extrordinary, we learned how to come up with 5 new minus-days exploits in 24 hours, we'd scan 1000 telephone numbers on a bad day, and 10,000 on a good one (basically an entire suburb). We even had a team of computer illiterate guys that had a van and spent their evenings going through corporate rubbish bins looking for manuals and other goodies.

Our innovations were countless, ranging from fax bombing telephone numbers (A DOS attack directed towards a voice line/mobile), digital phreaking using diverters and pads, hacking network switches to create conference bridges (or 'party lines' as we called them), drag-net hacking, and we even developed a code of ethics which kept us firmly in the realm of 'explorers' and away from the less desirable labels of 'fraudsters' or 'terrorists'.

The Black Hat Code of Ethics

• Don't take
• Don't break
• Don't change
• Don't steal (eg, credit card fraud)

The general rule was that if you broke one of those rules, then you had to put it right before you were done with the system you had hacked into.

We weren't without oversight either. For example, a senior cryptographer from the DSD would dial into our party line and give me and my mates a fortnightly lecture on number theory and cryptography (which at 13 years old is something special that lives with you the rest of your life). This went on for at least 9 months. We also had techs from Telstra watching over our activites as well, because we were basically pen-testing (penetration testing) digital telephony. Occasionally the techs would get in touch and give us a dressing down for some of our activites, so we had boundaries. The DSD guy also found out what was the latest and greatest thing in the black-hat world, so he could keep his finger on-the-pulse. A really chilled, very Australian and very hillarious symbiotic 'free flow of information' type of relationship was shared by all us insiders. Basically, we were a harmless bunch of Aussie bogans doing lots of hysterical phone pranks on a global scale for the most part. For the outsiders though, we were the pinacle of the 'Trust no-one' ethos and were scarry. Very scarry. In the black-hat world, hax0rs would have wars and we won every single one.

We became so internationally renowned, that the second ever DEFCON was held right here in Melbourne, and Captain Crunch (aka John Draper) flew out to meet us.

THE GOLDEN ERA

After that little event, I decided I was done with hax0ring and phreakx0ring and moved to the next big thing: emersing myself in what would become known as 'The Tech Boom', being famous and doing haute-tech. It was so much more fun, and considerably less anti-social.

Me on the cover of the Computer Age, circa early 1995.

The Netcafe

We were everywhere, putting anything and everything online, helping anyone and everyone understand what this new Internet thing was. I quickly became the poster boy for the Internet here in Melbourne, and by 1995 we had stood up Australia's first Netcafe on Acland Street. It was a hive of activity, and attracted everyone. I quickly caught the attention of the media, including Wired Magazine, but they didn't have any reporters in Australia. After a couple calls, they did the next best thing: pro sponsorship. The Wired folk didn't usually do sponsorships, but someone at the office called Absolut Vodka and did the next best thing, sending us a crate of 6 Absolut Vodka bottles (including whichever one was on the back cover for that month), along with a fresh, air-freighted current month edition of Wired Magazine, direct to The Netcafe. Back then, Wired was the bible, but it also arrived in Australia 3 months late thanks to it being sea-freighted.

Me sitting at a PC in the upstairs room at The Netcafe, featuring Win95 and surrounded by Marcsta artworks
Business Review Weekly (BRW), June 1995


Chris Beaumont briefly discussing the Netcafe at an exclusive Melbourne tech-futurist group
discussing The Netcafe and mentions moi!

An old pic circa 1996 of my first home lab. Note the Beaumont oil painting on the wall and
the Absolut Vodka tshirt from being sponsored by Wired Magazine

The Netcafe was a special place. With support from Michael Bethune from OzOnline, Adam from Standard Computers and of course Micrcosoft, we created a bunch of rooms above the Deluxe cafe on Acland street that people could experience both The Internet and Windows 95. 95 was a good operating system, it felt like a Silicon Graphics or Sun workstation and it was fast. Everyone that came in we showed it to, taking Melbourne from zero-knowledge about The 'Net to being as educated as anyone in Silicon Valley. It was an amazing time.

Even Jeff Kennet humself gave me a little tiny gold badge of Victoria as an acknowledgement of my contribution to the state.

Beat Magazine

During this period I crossed paths with Rob Furst, founder of the inky street-press music scene publication Beat Magazine. He quickly employed me as the e-editor, responsible for getting street press mag onto the Internet. I was still a kid at the time, and still studying at Brighton Grammar. How I fitted everything in I don't know, but I did. It was a great time and fun was had by all:

List of contributors to Beat Magazine, circa 1995. Note the e-editor :)

Ted Nelson

In 1996 a second luminary from Silicon Valley flew out to meet me -- Theodore ('Ted') Holm Nelson, inventor of hypertext. He heard that something amazing was happening in Australia and he came out here to take a look for himself. We chatted, solved a few problems then he went on his way. I still have his business card today.