We recently organised the SpaceHack project as part of the de.NBI BioHackathon, where we aimed to investigate different computation pipelines for cell segmentation to increase comparability between single cell and spatial transcriptomics. The project was very popular, with over 60 users registered for participation - such a large number of participants makes organisation extremely difficult. While indeed managing the scientific content of the hackathon was challenging, the compute was another story...
We used the de.NBI cloud for our core compute and storage infrastructure. The cloud team were extremely helpful from the early stages of conceptualisation, all the way to support after the event. Since this was my first use of the de.NBI cloud for a hackathon, I heavily relied on advice from the de.NBI cloud team in configuring the compute and storage resources, and paradigms for accessing those resources. Given that the participants of the hackathon would be more familiar with exploratory data analysis in tools such as Python and R, it was important to simplify the access procedure as much as possible while still allow flexibility for choice of resources depending on resources requirements of bioinformatics and machine learning tools. The cloud team set up a JupyterHub instance for accessing compute instances with varying levels of resources (normal, medium, high memory and GPU). They also provided examples of how custom software environments could be configured. Furthermore, before the start of the hackathon we had wrangled and processed a number of datasets that needed to be shared between all users, and the cloud team set up both user specific and shared volumes for accessing those datasets to be shared. User authentication was via linking the cloud instance to a GitHub Organisation, which simplified the user authentication steps.
How did it work? It was fantastic! I have never organised an event with shared compute that worked so well with regards to helpful consultation, accessibility, scalability and flexibility. We are already organising the follow up of our hackathon project on the de.NBI cloud, and strongly encourage other hackathon and workshop organisers to do the same.
Naveed Ishaque, on behalf of the SpaceHack organisation team.
