scaling-automating-r-workflows-kubernetes.md

talk_id	talk_slug	talk_type	talk_tags	session_slug	talk_title	talk_title_short	talk_materials_url	speakers
22199	scaling-automating-r-workflows-kubernetes	regular	interop production	rapid-response	Scaling and automating R workflows with Kubernetes and Airflow	Scaling and automating R workflows with Kubernetes and Airflow		name affiliation url username photo bio Isaac Florence UK Health Security Agency webpage twitter github linkedin affiliation https://isaacflorence.co.uk/ https://twitter.com/isaacatflorence https://github.com/isaac-florence https://uk.linkedin.com/in/isaacflorence https://www.gov.uk/government/organisations/uk-health-security-agency isaac_florence /assets/img/2022Conf/_talks/22199_isaac-florence.jpg I am an epidemiologist and public health policy professional working for the UK Health Security Agency (UKHSA). I have been using R and RStudio products since university and both have formed a major part of my career so far. At UKHSA I write in - and teach - R, SQL, Python, and other programming and computing skills. Although currently much of my work is geared towards data engineering and infrastructure (for R, computing & data platforms, and team building), my favourite part of work is epidemiology analysis (in R!). I also work on R infrastructure for "Applied Epi" (the team behind the "Epidemiologists' R Handbook"), as well as on some soon-to-be-announced open source projects on scaling and containerising R workloads.

During the pandemic, epidemiologists have been forced to adapt to the unprecedented scale of the data and high cadence of reporting.

At the UK Health Security Agency, we have created a platform for teams to easily deploy R and/or Python tasks onto our High-Performance Computing resources, scheduling their execution, and allowing previously unthinkable workloads to be executed with ease. Thanks to Kubernetes, git, Docker, and Airflow, our epidemiologists can stop worrying about their laptop's memory and bandwidth, and focus on answering the crucial questions of the pandemic. We'd like to tell you how we did it.