I was recently asked:
what’s the worst project you ever worked on?
After a quick scan through my mental graveyard of half-finished, half-assed ideas too embarrassing for me to dignify with a link here, I settled upon my first job out of university.
I was hired as a Software Engineer by IBM, to work out of their main UK development laboratory, Hursley Park, home of the mighty CICS, WebSphere MQ and MessageBroker products.
These are the gold standard in messaging and transaction processing backends that do the heavy-lifting behind the scenes of lots of Big Enterprise Systems.
I was part of the WebSphere MQ build team. We were responsible for taking the big MQ codebase and compiling it on all the different supported platforms. Linux on several architectures, Windows, AIX, HP-UX, iSeries, zSeries, …
The main challenge of the role was reliably managing jobs in wildly heterogeneous environments and getting useful information back if something went wrong.
The team had originally used a bunch of cron-style jobs on each build machine, which – a couple of times a day – would pull the code, compile it, and make the results available on a network share.
By the time I arrived, however, the cron jobs had been replaced by JEM: the Job Execution Manager.
JEM was a central server, communicating with agents on each build machine. It meant that builds could be started on arbitrary machines, at arbitrary times, with arbitrary changesets; it also gated the jobs so that build machines didn’t get swamped with several at once.
JEM was kind of A Big Deal and – if the scuttlebutt was to be believed – had launched the guy behind it onto a promotion hyper-fast track.
Despite JEM only being two or three years old at that point, it had already become something of a black box to everyone on the team, as the creators had moved on to other roles:
Every developer is familiar with Not Invented Here, and has probably been responsible for it at some point. I know I have!
But let’s look at the requirements for JEM:
Does that remind you of anything?
How about WebSphere MQ?
A team of four people was required to administer and manage JEM, when the very thing it was building had been the perfect solution all along. It should have been self-hosting!
To roll your own solution when a suitable alternative already exists is a poor show, but understandable and very common.
Still rolling your own solution when the alternative is well-known to you and you can use it for free – and with effectively unlimited support from the world’s top experts – is inexcusable.
Still rolling your own solution when the tangible output of your daily work is itself the perfect solution (which you can use for free, with unlimited support)… that’s the worst project I ever worked on.