Interactive, Cloud-Native Workflows on HPC Using KNoC

Cloud and HPC platforms differentiate by many aspects, but both can run applications in identical contexts using containers. In this paper we present KNoC, an open-source virtual node (kubelet) for Kubernetes that transparently manages the container lifecycle on a remote HPC cluster using Slurm and Singularity. Our goal is on one hand to allow HPC users to leverage existing cloud-native tools, such as the popular Argo Workflows language to express complex data-processing pipelines, while on the other hand enabling Cloud setups to exploit computing resources available in HPC centers. KNoC bridges Cloud and HPC, transforming Argo to a cross-environment, portable solution, which allows the combination of Cloud-based tools and HPC steps into the same workflow, controlled and monitored through an interactive frontend. Deploying KNoC requires only a secure shell connection to the cluster's login node. We describe the design and implementation of KNoC, and evaluate the integration using several proof-of-concept workflows.