By Jorrit BoekelOn time, on budget, with quality. Choose one, young bioinformatician.
In the canonical phrasing you could only pick two out of three, but obviously you can’t pick “on budget” if you’re not in the position to hire people, unless there is a throw-money-at-the-problem solution like hardware or outsourcing. So you’re not a boss, and this saying may need modification even further. Often when doing scientific development you are pressed to deliver faster, maybe simply by fear of being scooped. Not to mention the possibility to see your project become obsolete when it comes out. Science moves fast and thus your work moves to the garbage bin fast too.
My first moves in software were heavily influenced by reading too much Hacker News. Having a lot of fun but neither CS degree nor lots of experience in programming meant I had to learn on the spot. After the first phase of very dirty code, came other phases, which included the obvious too-many-abstractions phase, unit testing volatile components phase, and the phase were I used frameworks too heavy for the job. At least I managed to dodge the rewrite everything in $LANGUAGE phase. Having had those, I feel quite well about my process as it evolved. But there is always quicker and dirtier on the one hand, and cleaner, more scalable, and more secure on the other. What to do?
The technology stack above was of course an obvious example, but there are frameworks that have value and slow you down at the same time. I feel I am well versed in tool development for and a fan of the Galaxy project for instance, but I can’t help but thinking now and then that I would have analysis pipelines and tools available much faster when deploying them in R, bash or python (especially since I dislike editing XML). There is a price to pay for flexible, shareable tools that install problem free. But maybe a solution where you can have the cake and eat it pops up one day.
Until I find that, nothing is for free. To use a terrible metaphor, technical debt can be cheap as project-inflation eats it up. But anything long-lived, collaborative or critical enough should probably marinate in your development machine longer. Choose wisely.