CNCF-Hosted Co-Located Events Europe 2023

Multi tenant batch processing is one of the difficult resource-intensive data engineering use cases in Kubernetes. Today, a simple rule that governs both scheduling order and the preemption behavior in the default scheduler is that a pod with a higher priority is more important than a pod with a lower priority, and this crosses namespaces. We need more. When looking at preemption from a batch workload, we need to take into account the possibility of an opt-out from preemption by considering the context of the type of workload, including user quotas, hierarchical queues, Spark driver pod, etc. How do we get that accuracy and flexibility? What are the “The Laws of Preemption” for batch processing? Most of the rules given around tasks and queues are there to prevent a preemption storm or loop. How to define guardrails around preempting a workload from one tenant and giving the resources to another? In a cluster that runs workloads for multiple tenants, preemption should be tenant aware. A tenant could map to a queue hierarchy, thus crossing tenant boundaries. Preemption for YuniKorn is based on the hierarchical queue model and guaranteed resources assigned to a queue. In this talk, we will dive deep into the batch preemption capability of YuniKorn as a K8s scheduler plugin.