12-06, 14:30–15:00 (UTC), General Track
Cloud UX kinda sucks. It was written for cloud engineers who like very explicit systems, and always read the docs. This makes it a bad fit for data people (data scientists, data engineers, machine learning researchers) who rapidly learn and use several tools on a day-to-day basis. This mismatch in UX expectations results in poor utilization and wasted resources.
This talk goes through the challenges we faced when building a cloud UX for data people, and the kinds of solutions we ended up adopting when supporting Dask (parallel python) in a cloud environment.
Cloud UX is pretty bad today.
Cloud computing delivers a compelling promise: large scale computing close to your data on any hardware with by-the-minute pricing. Game-changing.
However, cloud utilization remains low in most organizations largely because usability remains poor. This is especially for data professionals unused to cloud APIs and UIs.
In making a cloud platform for Dask (an open source library for parallel computing) we ran into issues like the following:
- Deploying and robustly cleaning up cloud resources
- Package management
- Network security
- User and cost management
- Access to rare resources, like spot or GPUs
- Cost optimization and hardware selection
In talking to other organizations it’s clear that these problems are common among cloud practitioners. In this talk we go through how we solved the problems above in our platform. Our hope is that this educates organizations starting their cloud journey so that they know what to expect, and gives tips to organizations struggling with these problems today.
No previous knowledge expected
I'm a software engineer and open source community member. Most of my coding activities center around the Python data science stack. In particular, I'm a core maintainer of Dask and I work at Coiled where I focus on scaling Python. Previously I studied at the University of Wisconsin-Madison where I received my PhD in Physics.