A KubeCon + CloudNativeCon Europe 2024 talk by Michał Woźniak of Google and Yuki Iwai of CyberAgent, Inc. introduces Kueue, a Kubernetes job-level scheduler for AI and ML workloads. The deck begins with a title slide, then explains what Kueue is, its kube-native design principles, job lifecycle, and core concepts such as ResourceFlavor, ClusterQueue, LocalQueue, Cohort, and AdmissionCheck. It walks through a batch reference architecture using a series of build-up diagrams that show quota borrowing, preemption, and requeueing across reservation, on-demand, and spot capacity. A middle section presents CyberAgent's on-premise ML platform, including its multi-tenant cluster, workload types, the gap between ResourceQuota and Kueue, and the two approaches it evaluated. Newer features follow: LendingLimit, partial admission, Provisioning Request integration, MultiKueue, fair sharing, and hierarchical cohorts. The deck closes with pointers to try Kueue, join the WG-Batch community, and contact the speakers.
Slides worth studying




Questions about this deck
How many slides does the Google Kueue presentation deck have?
The deck has 49 slides, and slide 1 is the title slide: Advanced Resource Management for Running AI/ML Workloads with Kueue, by Michał Woźniak of Google and Yuki Iwai of CyberAgent, Inc.
What does the Google Kueue deck say about Kueue's design principles?
Slide 3 says Kueue is "kube-native": there is no external database, it is compatible with cluster-autoscaler, kube-scheduler, and the Job controller, and no kube components are forked or replaced. It adds that required enhancements are requested upstream.
What does the Google Kueue deck say about the batch reference architecture?
Slide 10 says the batch reference architecture is intended to help platform administrators, cloud architects, and operations professionals deploy a batch processing platform. It lists best practices for running batch workloads and says it is customizable for user preferences.
What does the Google Kueue deck say about fair sharing?
Slide 45 says fair sharing gives fair access to unused resources to different teams, and that without fair sharing a FIFO scheme is used. It also says quota imbalances within the structure are resolved by preemption.
Details
- Company: Google
- Year: 2024
- Type: Conference talk
- Industry: Cloud infrastructure
- Pages: 49
More from Google
All Google decks in the library
Review score: 6 out of 10, Needs substantial work
Presentation quality, judged with the SlideDeck Co review heuristic. Ask, evidence and idea merit are not rated here.
The deck has a consistent template and a sensible route from what Kueue is, to concepts, to a worked example, to a case study, to new features. The concept build on slides 6 to 9 and the stepwise batch diagram on slides 11 to 20 show real teaching craft for a technical conference audience. But many pages are weak: slide 10 is a near-empty goals page with a tiny link, slides 21, 24, 36 and 47 are decorative or unexplained images with no message, and slides 28 to 31 are bullet-heavy, with a callout box covering the text on slides 30 and 31. Slides 11 to 20 carry small labels and unlabelled numbers that depend heavily on the speaker, and several late feature slides (37, 39, 40, 45) are thin or loosely tied to the argument, so the core is usable but needs substantial revision.
Worth keeping: Keep the slide 6 to 9 concept build-up and the slide 11 to 20 stepwise cohort diagram, with colour coding and short callouts like 'Preemption' and 'Requeued'.
Most useful repair: Fix the overlapping callout on slides 30 and 31, give the empty slides 10, 21, 24, 36 and 47 a real message, and enlarge the small diagram text on slides 11 to 20 and 41 to 44.
What would change the score: I would replace the stock photos with meaningful visuals, cut the repeated slide-build duplicates such as 28 to 29 and 32 to 33 into clearer states, and write titles that state the point instead of repeating section labels like 'Kueue - main concepts'.
Confidence medium. 49 of 49 pages inspected on contact sheets, 4 at readable size. Reviewed 2026-11-08. Model-assisted review (claude-sonnet-5-5).
Full text of the slides (49 slides)
Every slide's words and a description of its charts and images, so the deck can be searched and rebuilt. Also available as Markdown.
Slide 1
Advanced Resource Management for
Running AI/ML Workloads with Kueue
Michał Woźniak (@mimowo), Google
Yuki Iwai (@tenzen-y), CyberAgent, Inc.
Visuals: Decorative background graphic only: a large diagonal boundary runs from the top edge near x 69% down to the bottom edge near x 12%, dividing a saturated sky-blue upper-left triangle from a light warm-grey lower-right region. In the lower right a large sky-blue circle or arc segment (edge rising from about x 56%, y 100% to the right edge near y 39%) contains a lighter cyan rectangle block (about x 75% to 100%, y 54% to 87%). No photographs, charts, logos or icons.
Slide 2
What is Kueue? / Job-level scheduler / ● Determines when to start a job / ○ delayed pod creation off-loads api-server and kube-scheduler / ● Support all-or-nothing semantics (without it jobs could “deadlock”) / ● Resource quota management / ○ Multi-tenant quota management / ○ Quota management for various types of hardware / ○ Control over resource preferences (reservations vs. on-demand vs spot) / Jobs “soup” / Team A / Queued workloads / Pods in cluster / Team B
Visuals: Diagram, left to right. (1) Two person icons: Team A has a bright green head and shoulders icon, Team B has a cyan head and royal blue shoulders icon; each has a grey-outlined hollow arrow pointing right. (2) Jobs “soup”: about 14 loose circles of varying size scattered in a cluster, outlined in red, blue, dark purple, teal, green, grey and yellow on light grey fills, with the largest circle at lower right outlined in bright green. (3) A solid blue octagon with three white chevron arrows, the Kueue logo, centred between the soup and the queued workloads. (4) Queued workloads: a title above two stacked horizontal rounded rectangles with dark purple-grey outlines. The upper rounded rectangle holds four boxes: a 2x2 grid of green-outlined squares, a 2x2 grid of magenta-outlined squares, one large green-outlined square, and a grey-filled box with two small red squares. The lower rounded rectangle holds three boxes: an empty first slot, a 2x2 grid of green-outlined squares, a 2x2 grid of blue-outlined squares, and one large red-outlined square. (5) A faded blue octagon with white chevrons, a blurred variant of the Kueue logo, centred between queued workloads and pods. (6) Pods in cluster: a thin dark purple-outlined rectangle containing two rounded panels. The upper panel is pink (#D5A6BD) and holds a 2x2 grid of yellow squares, a 2x2 grid of green squares and a network interface card icon at right. The lower panel is lavender (#B4A7D6) and holds a solid red square, a 2x2 grid of yellow squares and a green computer chip icon at right. Lighting is flat, with no shadows or gradients.
Slide 3
Kueue - Design principles
KubeCon
CloudNativeCon
Europe 2024
It is “kube-native”:
● There is no external database
● Compatible with kube components:
○ cluster-autoscaler, kube-scheduler, Job controller
○ No kube components are forked or replaced
● Required enhancements are requested upstream
source
Unsplash+ (repeated watermark across the photo)
Visuals: One photograph, boxed as a rectangle at x 27.4%, y 48.9%, w 39.5%, h 44.7%. It shows two stylised 3D cartoon hands reaching toward each other from the left and right. The left hand holds a mustard-yellow jigsaw puzzle piece and the right hand holds a teal jigsaw puzzle piece, and the two pieces interlock at the centre. The left sleeve is teal and the right sleeve is mustard-brown; both sleeves run off the left and right edges of the box, so the image is cropped at those edges. The lighting is soft and diffuse, with no strong shadows. The dominant colours are light grey background, teal, mustard-yellow and skin tones. The background is light grey with a faint repeated grey 'Unsplash+' watermark pattern covering the whole image. The image is not full-bleed and has no overlay or tint. Logos: the KubeCon logo (a dark purple helm-wheel icon with the word KubeCon) and the CloudNativeCon logo (a dark purple square icon with the words CloudNativeCon and Europe 2024), separated by a thin vertical divider, with thin horizontal rules on either side of Europe 2024. A small dark dot sits at about x 11.8%, y 50.1%, a stray mark left by a bullet.
Slide 4
Kueue - Design principles
KubeCon
CloudNativeCon
Europe 2024
Pods creation is gated by job admission
Introduce “suspend” semantics across Job CRDs
Supported Job-based integrations:
k8s: Job, JobSet
Kubeflow Jobs
Ray: RayJob, RayCluster
In-house Jobs via Job framework
BTW: Plain pod groups are also supported
JobSet
Kubeflow
RAY
PyTorch
TensorFlow
Visuals: Logos (not photographs) arranged in the right half of the slide: (1) Kubernetes Job icon, a blue hexagon with a white grid of squares and the word 'job' beneath, at about x 50.3 y 51.3 w 7.8 h 13.5 region; (2) Kubernetes JobSet icon, a blue hexagon with white ship's wheel (helm) symbol, with the label 'JobSet' in black sans-serif below it; (3) Kubeflow logo, a blue geometric mark with the wordmark 'Kubeflow' in blue and dark navy; (4) TensorFlow logo, an orange geometric mark with the wordmark 'TensorFlow' in grey-navy; (5) RAY logo, blue circular node-network mark with the wordmark 'RAY' in dark grey; (6) PyTorch logo, red-orange flame-like circle mark with the wordmark 'PyTorch' in dark grey. No charts, photographs or diagrams. A small grey dot sits left of the third sub-bullet.
Slide 5
Kueue - Job lifecycle
KubeCon
CloudNativeCon
Europe 2024
Job Creation
Quota reservation
Admission
Optional
Admission checks
Cluster Autoscaler
Job controller
kube-scheduler
Preemption
Visuals: Diagram. Shapes and connections: (1) A green person icon (bright green #00FF55 head, deeper green #00AA00 shoulders) at about x 10.0 y 49.7, w 3.0 h 6.7. A thick grey block arrow labelled 'Job Creation' runs from the person to the dotted cycle box. (2) A dotted-outline rounded rectangle (dark grey dots, no fill, corner radius about 3% of slide width) at about x 23.6 y 35.9 w 53.0 h 37.0 encloses the lifecycle loop. (3) Inside the loop are three light-blue (#4FC3F7) outlined circles, each about 5 to 6% of slide width: left circle at x 24.3 y 49.1 containing a red STOP octagon (#E8401C); top circle at x 35.0 y 39.2 containing a red STOP octagon; right circle at x 46.5 y 48.5 containing blue gears and a small green checkmark box (#8BC34A) at its upper right. (4) At the centre of the loop is a blue hexagon (#2962FF) with three white chevrons, the Kueue logo, at x 35.7 y 51.0 w 3.7 h 6.7. (5) Thick light-grey (#EAE9E8) block arrows with dark purple-grey (#3D3762) outlines connect the stages clockwise: 'Quota reservation' arrow from the left circle upward then right to the top circle; 'Admission' arrow from the top circle right and down to the right circle; a 'Preemption' arrow from the right circle down, left and up back to the left circle. (6) A rounded-rectangle callout (white fill, dark purple #4B3F72 outline about 0.3% of height, corner radius about 1.5% of height) at x 36.6 y 25.3 w 14.6 h 14.1 with text 'Optional / Admission checks' reads 'Optional' and 'Admission checks' centred. Its small square tail at x 36.7 y 40.9 points to the top STOP circle. (7) A grey block arrow labelled 'Job controller' runs right from the dotted box to a grid icon of four green-outlined squares (#6AA84F outlines) at x 63.7 y 51.2 w 2.7 h 4.8. (8) A grey curved block arrow from the grid icon up, right and down to a resource box at x 72.6 y 48.5 w 6.2 h 9.3, labelled 'Cluster Autoscaler' above it. The box has a green (#6CC39A) border around a green-filled microchip (#9FCC9A fill, dark slate #46596B inner rectangle, gold pins). (9) A grey vertical block arrow from that box down to a label 'kube-scheduler' and then to a second resource box at x 66.7 y 75.6 w 10.1 h 9.3 with a green (#6CC39A) border containing a four-square pod grid (green squares with dark purple-grey #3D3762 outlines) and a microchip with gold pins. Grey arrow fill #EAE9E8 with outline #3D3762 throughout. No photographs.
Slide 6
Kueue - main concepts
KubeCon
CloudNativeCon
Europe 2024
User
Kueue
Admin
Namespace
Job
Workload
LocalQueue
ClusterQueue
Resource quota definition
Visuals: Diagram. Three dotted rounded-rectangle frames with no fill and a dark slate-indigo dotted stroke: the User frame (top-left, containing a green person icon and the label User), the Kueue frame (containing a blue hexagonal Kueue icon with white chevrons and the label Kueue), and the large Admin frame (right, containing a yellow-and-orange person icon and the label Admin). Inside the left part, three overlapping rounded rectangles offset diagonally down-right, each filled light blue #9FC5E8 with a dark indigo #434374 stroke, labelled Namespace on the front card. On the front card: a green circle (#B6D7A8, black outline) labelled Job; a light grey square (#EAE9E8, indigo outline) labelled Workload; a yellow rounded rectangle (#FFE599, indigo outline) labelled LocalQueue. A dashed thin arrow runs from Job to Workload, and another dashed thin arrow runs from Workload to LocalQueue. A solid thin indigo arrow runs from LocalQueue to a yellow rounded rectangle (#FFE599, near-black outline) labelled ClusterQueue at right. Beneath ClusterQueue is a speech-bubble callout with a white fill and gold #BF9000 outline, its pointer rising to the bottom edge of ClusterQueue, containing the text Resource quota definition. Icons: green user icon, Kueue hexagon icon, and yellow-orange admin icon as described. Logos: KubeCon wordmark with ship-wheel emblem and CloudNativeCon wordmark with square emblem, both in dark indigo, separated by a thin vertical divider, with Europe 2024 beneath in a thin-ruled line.
Slide 7
Kueue - main concepts
User
Kueue
Admin
Namespace
Job
Workload
LocalQueue
ClusterQueue
ResourceFlavor
Represents machines of
similar characteristics, like
GPU model or reservations
vs. spot.
Contains labels and
tolerations added to
admitted jobs
KubeCon
CloudNativeCon
Europe 2024
Visuals: Diagram with the following shapes and connections. Three dotted-outline rounded rectangles (dark slate #3F3F6B dotted stroke): one labelled User containing a green person icon (bright green head #00FF66-ish over dark green #00A000 shoulders); one labelled Kueue containing a blue faceted gem icon (#4A7FD4 to white highlights); one large dotted rounded rectangle labelled Admin containing a yellow person icon (#FFFF00 head over orange #FFA500 shoulders), spanning from about x 40 to 96 and y 20 to 80. Three stacked rounded rectangles in light blue #9FC5E8 with dark slate #3F3F6B 2px outlines, offset diagonally, the front one labelled Namespace at top centre. Inside the front Namespace: a green circle #B6D7A8 with black 2px outline labelled Job; a light grey square #EAE9E8 with slate outline labelled Workload; a yellow rounded rectangle #FFE599 with dark outline labelled LocalQueue. A thin dashed slate arrow runs Job to Workload and Workload to LocalQueue. A solid thin slate arrow runs from LocalQueue to ClusterQueue, a yellow rounded rectangle #FFE599 with dark outline at about x 63 to 75 and y 45 to 57. A solid thin slate arrow runs from ResourceFlavor (yellow rounded rectangle #FFE599 with dark outline, at about x 84 to 96, y 45 to 57) leftward to ClusterQueue. A gold #C09A1A outlined speech-bubble callout with white fill points upward from its tail at ResourceFlavor's bottom edge; its tail is at about x 90 to 94, y 54 to 66. The dotted Admin boundary runs down and around to meet the callout at its lower-left. Icons are small raster images, none are logos except the conference marks.
Slide 8
Kueue - main concepts
User
Kueue
Admin
Namespace
Scope for borrowing and
preemption
Cohort
Job
Workload
LocalQueue
ClusterQueue
ResourceFlavor
Visuals: Diagram. Shapes and labels: (1) Dotted rounded rectangle labelled 'User' at top left containing a green person icon (bright green head with lighter green torso) and a 'User' label above it. (2) Dotted rounded rectangle labelled 'Kueue' containing a dark blue heptagon-like Kubernetes-style hexagon icon. (3) Large dotted rounded rectangle labelled 'Admin' containing an olive/gold person icon with a yellow visor band, and holding the Cohort, ClusterQueue and ResourceFlavor boxes. (4) Three overlapping light blue (#9FC5E8) rounded rectangles with dark indigo outlines stacked diagonally, the front one labelled 'Namespace'; inside the front card are a green circle labelled 'Job' (#B6D7A8 fill, black outline), a light grey rectangle labelled 'Workload' (#EAE9E8 fill, indigo outline), and a yellow rounded box labelled 'LocalQueue'. (5) Yellow (#FFE599) rounded rectangles: 'Cohort' (indigo outline), 'ClusterQueue' (dark outline), 'ResourceFlavor' (dark outline). (6) Gold-outlined white speech-bubble callout pointing at Cohort, text 'Scope for borrowing and preemption'. Connectors: a dotted thin indigo arrow from Job to Workload and from Workload to LocalQueue; a solid thin indigo arrow from LocalQueue right to ClusterQueue left; a solid thin indigo arrow from ClusterQueue top up to Cohort bottom; a solid thin arrow from ResourceFlavor left to ClusterQueue right. Logos: KubeCon logo (dark indigo Kubernetes wheel with 'KubeCon' wordmark) and CloudNativeCon Europe 2024 logo (indigo stylised square with 'CloudNativeCon' wordmark and 'Europe 2024' beneath), separated by a thin vertical divider. Lighting is flat; colours are flat fills.
Slide 9
Kueue - main concepts
KubeCon
CloudNativeCon
Europe 2024
User
Kueue
Admin
Namespace
Job
Workload
LocalQueue
Cohort
ClusterQueue
ResourceFlavor
AdmissionCheck
Additional check before admission
by internal or external controller
Visuals: Diagram. Shapes and labels: a dotted rounded box labelled User containing a green person icon; a dotted rounded box labelled Kueue containing a blue Kubernetes-style heptagon icon; a large dotted rounded box labelled Admin containing an olive person icon. Three stacked light blue (#9FC5E8) rounded rectangles, each with a dark indigo outline, form the Namespace stack, the front one labelled Namespace at its top centre. Inside the front namespace: a green (#B6D7A8) circle with black outline labelled Job; a light grey (#EAE9E8) rectangle with indigo outline labelled Workload; a yellow (#FFE599) rounded box labelled LocalQueue. Right of the namespace: a yellow rounded box labelled Cohort (top), a yellow rounded box labelled ClusterQueue (centre), a yellow rounded box labelled ResourceFlavor (right of ClusterQueue), and a yellow rounded box labelled AdmissionCheck (below ClusterQueue). Connectors: a thin dashed indigo arrow from Job to Workload; a thin dashed indigo arrow from Workload to LocalQueue; a solid thin indigo arrow from LocalQueue to ClusterQueue; a solid thin arrow from ResourceFlavor pointing left into ClusterQueue; a solid thin arrow from ClusterQueue pointing up to Cohort; a solid thin arrow from AdmissionCheck pointing up into ClusterQueue. A gold-outlined white speech-bubble callout below AdmissionCheck, its pointer tip touching AdmissionCheck's bottom edge, contains the text Additional check before admission by internal or external controller. Icons: green person (User), blue heptagon (Kueue), olive person (Admin). Logos: KubeCon wordmark with a kubernetes-style wheel mark, separated by a thin vertical rule from the CloudNativeCon wordmark with a dark indigo square mark, and the line Europe 2024 beneath both with thin horizontal rules. Nothing is photographed or cropped; no overlays or tints.
Slide 10
Batch reference architecture
Goals:
Intended to help platform administrators, cloud architects, and operations professionals deploy a batch processing platform
Best practices for running batch workloads
Customizable for user preferences
https://github.com/GoogleCloudPlatform/ai-on-gke/tree/main/gke-batch-refarch
Acknowledgements: Ali Zaidi
Visuals: One small grey dot near x 11.9%, y 50% (about 0.3% of slide width), likely a stray mark, with no other imagery. Logos at top right: KubeCon logo (dark navy Kubernetes helm wheel icon above the word KubeCon) and CloudNativeCon logo (dark navy square icon with a diagonal stripe above the word CloudNativeCon), with a thin vertical divider between them and the text 'Europe 2024' centred beneath between two thin horizontal navy rules. The logos are navy #1F2A5A on the light blue band.
Slide 11
Batch reference architecture / KubeCon / CloudNativeCon / Europe 2024 / cohort / Can preempt lower / priority workloads / LocalQueues / ClusterQueues / Nodes / RESERVATION / NominalQuota: 4 / BorrowingLimit: 0 / ON-DEMAND / NominalQuota: 4 / BorrowingLimit: 4 / Cannot preempt / RESERVATION / NominalQuota: 0 / BorrowingLimit: 4 / SPOT / NominalQuota: 12 / BorrowingLimit: 4 / RESERVATION / ON-DEMAND / SPOT
Visuals: Diagram, no chart or photograph. Two user icons (flat person silhouettes): top user at about x 4.2% y 45.8% w 3.0% h 6.7%, head light cyan #22CCEE, body blue #1E6FF0; bottom user at about x 4.2% y 74.1% w 3.0% h 6.7%, head bright green #22EE66, body green #00B52E. Each user has a right-pointing block arrow (light grey #E8E8E8 fill, thin dark slate #44436B outline) running to a white rectangle labelled LocalQueues, which has a second block arrow to the cohort. Top LocalQueue rectangle at about x 17.1% y 45.3% w 18.2% h 7.9%, white fill, thin #44436B outline, empty; bottom LocalQueue rectangle at about x 17.1% y 73.5% w 18.2% h 7.9%, same style, empty. Cohort: a large rounded rectangle with dotted grey outline (about x 41.3% y 20.7% w 25.0% h 74.7%) labelled cohort above it. Inside it, two ClusterQueue containers (thin black outline, white fill): upper at about x 42.6% y 36.4% w 21.3% h 24.7%, containing RESERVATION (blue #0563C1, NominalQuota 4, BorrowingLimit 0) and ON-DEMAND (dark red #990000, NominalQuota 4, BorrowingLimit 4); lower at about x 42.6% y 64.6% w 21.3% h 24.7%, containing RESERVATION (blue, NominalQuota 0, BorrowingLimit 4) and SPOT (green #38761D, NominalQuota 12, BorrowingLimit 4). Each quota entry is a thin black-outlined inner box on white. Callout 1: rounded speech bubble, white fill, red #CC0000 outline about 2px, at about x 25.6% y 26.6% w 16.3% h 10.8%, text 'Can preempt lower priority workloads', with its pointer tail running down-right to the upper RESERVATION box at about (42.8%, 53.8%). Callout 2: rounded speech bubble, white fill, green #228B22 outline, at about x 25.0% y 58.4% w 16.9% h 9.8%, text 'Cannot preempt', with its tail running down-right to the lower RESERVATION/SPOT boundary at about (42.7%, 77.6%). Nodes column: three white rectangles with thin #44436B outline. Top at about x 72.6% y 36.4% w 15.4% h 12.3%, labelled RESERVATION in blue, containing a 2x2 grid of small outlined squares in dark purple #3F3A6B at about x 73.5% y 40.6% w 3.6% h 5.9%. Middle at about x 72.6% y 51.3% w 15.4% h 12.4%, labelled ON-DEMAND in dark red. Bottom at about x 72.7% y 66.2% w 15.4% h 12.4%, labelled SPOT in green. Logos (top-right header): KubeCon mark (dark navy #2B2D6E Kubernetes wheel-style hexagon icon) with wordmark KubeCon, a thin vertical divider, and the CloudNativeCon mark (navy square icon) with wordmark CloudNativeCon, with a horizontal rule and Europe 2024 beneath, all in dark navy #2B2D6E, at about x 83% y 2% w 14% h 10%.
Slide 12
Batch reference architecture
KubeCon
CloudNativeCon
Europe 2024
cohort
LocalQueues
ClusterQueues
Nodes
RESERVATION
NominalQuota: 4
BorrowingLimit: 0
ON-DEMAND
NominalQuota: 4
BorrowingLimit: 4
Low priority
RESERVATION
NominalQuota: 0
BorrowingLimit: 4
SPOT
NominalQuota: 12
BorrowingLimit: 4
RESERVATION
ON-DEMAND
SPOT
Visuals: Diagram. Two user icons on the left (blue person at top-left of the LocalQueues row, green person at bottom-left) each feed a grey block arrow pointing right into a LocalQueue box. The top LocalQueue box is empty. The bottom LocalQueue box contains three grey-backed tiles, each holding a 2x2 grid of green-outlined squares, and a rounded speech-bubble callout reading 'Low priority' above it with a black pointer line to the tiles. Grey block arrows connect the top LocalQueue to the top ClusterQueue and the bottom LocalQueue to the bottom ClusterQueue. Inside a dotted rounded rectangle labelled 'cohort' sit two outlined ClusterQueue containers. The upper ClusterQueue holds a RESERVATION quota (NominalQuota 4, BorrowingLimit 0) and an ON-DEMAND quota (NominalQuota 4, BorrowingLimit 4). The lower ClusterQueue holds a RESERVATION quota (NominalQuota 0, BorrowingLimit 4) and a SPOT quota (NominalQuota 12, BorrowingLimit 4). On the right, three outlined Node boxes stacked vertically: RESERVATION (containing a 2x2 grid of purple-outlined squares), ON-DEMAND (empty) and SPOT (empty). No photographs, logos apart from the header logos, or charts.
Slide 13
Batch reference architecture / KubeCon / CloudNativeCon / Europe 2024 / cohort / LocalQueues / ClusterQueues / Nodes / RESERVATION / NominalQuota: 4 / BorrowingLimit: 0 / ON-DEMAND / NominalQuota: 4 / BorrowingLimit: 4 / 4 / RESERVATION / NominalQuota: 0 / BorrowingLimit: 4 / SPOT / NominalQuota: 12 / BorrowingLimit: 4 / RESERVATION / ON-DEMAND / SPOT
Visuals: Diagram. Two user avatar icons at left, one teal-to-blue (top row, at about x 4.2 y 45.8) and one green (bottom row, at about x 4.2 y 74.1). Each avatar has a grey-outlined block arrow pointing right to a LocalQueues box. The top LocalQueues box is an empty rectangle with a thin dark indigo outline (about x 17.1 y 45.3, w 18.2 h 7.9). The bottom LocalQueues box (about x 17.1 y 73.5, w 18.2 h 7.9) contains two small green 2x2 grid icons at its right end, each in a light grey-outlined square. A grey block arrow then points from each LocalQueues box to the cohort. The cohort is a large rounded rectangle with a dotted grey outline and label 'cohort' above it. Inside it are two solid black-outlined ClusterQueues boxes stacked vertically. The top ClusterQueues box contains two stacked sub-boxes: RESERVATION (blue, NominalQuota 4, BorrowingLimit 0) and ON-DEMAND (dark red, NominalQuota 4, BorrowingLimit 4). The bottom ClusterQueues box contains RESERVATION (blue, NominalQuota 0, BorrowingLimit 4) with a large indigo '4' at its right, and SPOT (green, NominalQuota 12, BorrowingLimit 4). At right, the Nodes column has three thin indigo-outlined white boxes: RESERVATION (blue label, containing four solid dark-green squares in a 2x2 grid), ON-DEMAND (dark red label, empty) and SPOT (green label, empty). Lines are thin dark indigo, arrows are light grey fill with indigo outline.
Slide 14
Batch reference architecture / KubeCon / CloudNativeCon / Europe 2024 / cohort / LocalQueues / ClusterQueues / Nodes / RESERVATION / NominalQuota: 4 / BorrowingLimit: 0 / ON-DEMAND / NominalQuota: 4 / BorrowingLimit: 4 / RESERVATION / NominalQuota: 0 / BorrowingLimit: 4 / 4 / SPOT / NominalQuota: 12 / BorrowingLimit: 4 / 8 / RESERVATION / ON-DEMAND / SPOT
Visuals: Diagram: two user icons (top one teal-to-blue gradient person silhouette at about x 4.2 y 45.8 w 3.0 h 6.7; bottom one dark green person silhouette at about x 4.2 y 74.1 w 3.0 h 6.7) each followed by a light grey (#E8E8E8) right-pointing block arrow with dark slate outline (#4B3B71) into an empty white rectangle with thin dark slate outline (#4B3B71), labelled none. From each empty rectangle a smaller light grey block arrow points right into the cohort. The cohort is a large rounded rectangle with dotted grey outline, titled 'cohort' above it. Inside it, the top ClusterQueue group is a black-outlined box containing two boxes: RESERVATION (NominalQuota: 4, BorrowingLimit: 0) and ON-DEMAND (NominalQuota: 4, BorrowingLimit: 4). The bottom ClusterQueue group is a black-outlined box containing RESERVATION (NominalQuota: 0, BorrowingLimit: 4) with a large purple number 4 at right, and SPOT (NominalQuota: 12, BorrowingLimit: 4) with a large purple number 8 at right. Nodes column: a RESERVATION box containing four dark green squares arranged 2 by 2 with purple outlines; an ON-DEMAND box with no squares; a SPOT box containing eight dark green squares arranged in two rows of four with purple outlines. Node boxes have thin dark slate outlines (#4B3B71) and white fill.
Slide 15
Batch reference architecture
KubeCon / CloudNativeCon
Europe 2024
LocalQueues
High priority
cohort
ClusterQueues
RESERVATION
NominalQuota: 4
BorrowingLimit: 0
ON-DEMAND
NominalQuota: 4
BorrowingLimit: 4
RESERVATION
NominalQuota: 0
BorrowingLimit: 4
4
SPOT
NominalQuota: 12
BorrowingLimit: 4
8
Nodes
RESERVATION
ON-DEMAND
SPOT
Visuals: Diagram. Two user icons (top: teal-to-blue silhouette; bottom: green silhouette) each with a grey outlined right arrow pointing to a rectangular LocalQueue box. The top LocalQueue box (about x 17.1-35.3, y 45.3-53.2) contains two small grid icons with dark red outlined squares (2x2 each) on light grey tiles, at about x 27.3-31.2 and 31.2-34.7; a speech-bubble callout labelled 'High priority' with a black outline and white fill points down to these icons. The bottom LocalQueue box (about x 17.1-35.3, y 73.5-81.4) is empty. Grey outlined arrows connect each LocalQueue box to the cohort: top arrow to ClusterQueue group 1, bottom arrow to ClusterQueue group 2. The cohort is a large rounded rectangle with dotted grey outline and white fill (about x 50.3-65.0, y 16.6-96.0 for its label and border). Inside it, group 1 is a black-outlined box (about x 52.0-72.3, y 36.4-60.8) holding two sub-boxes: RESERVATION (NominalQuota 4, BorrowingLimit 0) and ON-DEMAND (NominalQuota 4, BorrowingLimit 4). Group 2 is a black-outlined box (about x 52.0-72.3, y 64.6-89.6) holding RESERVATION (NominalQuota 0, BorrowingLimit 4, value 4 at right in purple #4B3B71) and SPOT (NominalQuota 12, BorrowingLimit 4, value 8 at right in purple #4B3B71). Nodes column: three thin purple-outlined boxes: RESERVATION (about x 72.6-88.1, y 36.4-48.7) containing four green squares in a 2x2 grid with purple outlines; ON-DEMAND (about x 72.6-88.0, y 51.4-63.7) empty; SPOT (about x 72.7-88.1, y 66.3-78.6) containing eight green squares in two rows of four with purple outlines.
Slide 16
Batch reference architecture
KubeCon
CloudNativeCon
Europe 2024
cohort
LocalQueues
ClusterQueues
Nodes
RESERVATION
NominalQuota: 4
BorrowingLimit: 0
ON-DEMAND
NominalQuota: 4
BorrowingLimit: 4
4
RESERVATION
NominalQuota: 0
BorrowingLimit: 4
SPOT
NominalQuota: 12
BorrowingLimit: 4
8
RESERVATION
ON-DEMAND
SPOT
4
4
8
Visuals: Diagram: two user person icons (top one teal-to-blue, bottom one green) each followed by a grey block arrow pointing right into a LocalQueue box. The top LocalQueue box is empty except for a small red 2x2 grid icon at its right end (four red-outlined squares, no fill). The bottom LocalQueue box is empty. Each LocalQueue box has a thin dark slate-blue outline (#3B3A6B) and sits on a white fill. Grey block arrows (fill #E8E8E8, outline #3B3A6B) carry the flow from each LocalQueue into a ClusterQueue outer box inside a large dotted rounded rectangle labelled cohort (dotted dark grey outline, corner radius about 8% of slide width, no fill). Upper ClusterQueue outer box (thin black outline, white fill) contains two stacked inner boxes: RESERVATION (blue text, NominalQuota 4, BorrowingLimit 0, no number at right) and ON-DEMAND (dark red text, NominalQuota 4, BorrowingLimit 4, large slate-purple number 4 at right). Lower ClusterQueue outer box contains RESERVATION (NominalQuota 0, BorrowingLimit 4, large number 4 at right) and SPOT (green text, NominalQuota 12, BorrowingLimit 4, large number 8 at right). Nodes column: three boxes with thin slate-blue outlines. RESERVATION node box (blue title) holds a 2x2 grid of green squares. ON-DEMAND node box (dark red title) holds a 2x2 grid of dark red squares. SPOT node box (green title) holds eight green squares in two rows of four, with a small gap between each pair. Squares have slate-blue outlines.
Slide 17
Batch reference architecture / cohort / LocalQueues / ClusterQueues / Nodes / Preemption / RESERVATION / NominalQuota: 4 / BorrowingLimit: 0 / ON-DEMAND / NominalQuota: 4 / BorrowingLimit: 4 / 4 / RESERVATION / NominalQuota: 0 / BorrowingLimit: 4 / SPOT / NominalQuota: 12 / BorrowingLimit: 4 / 8 / RESERVATION / ON-DEMAND / SPOT / KubeCon / CloudNativeCon / Europe 2024
Visuals: Diagram. Two user icons (a blue person at the upper left and a green person at the lower left) each point by a grey block arrow into a LocalQueue box. The upper LocalQueue box holds a small red 2x2 grid icon at its right end and its output arrow points into the first ClusterQueues container. The lower LocalQueue box is empty and its output arrow points into the second ClusterQueues container. A speech-bubble callout labelled 'Preemption' sits between the two LocalQueues, with its tail pointing at a light grey square icon containing four green outlined squares. A thin dark purple arrow runs from that icon down to the lower LocalQueue box, with its arrowhead touching the box's upper edge. A second thin arrow runs from the lower ClusterQueues box 'RESERVATION' up to the same grey icon, with its arrowhead touching the icon. The dotted rounded rectangle labelled 'cohort' encloses two ClusterQueues containers. The upper container holds RESERVATION (NominalQuota 4, BorrowingLimit 0, no large number) and ON-DEMAND (NominalQuota 4, BorrowingLimit 4, large number 4). The lower container holds RESERVATION (NominalQuota 0, BorrowingLimit 4, large number 4) and SPOT (NominalQuota 12, BorrowingLimit 4, large number 8). The Nodes column holds three boxes: RESERVATION with four green squares in a 2x2 grid, ON-DEMAND with four red squares in a 2x2 grid, and SPOT with eight green squares in two rows of four. The header logos show the KubeCon wheel mark with the word KubeCon, and the CloudNativeCon mark with the words CloudNativeCon and Europe 2024, all in dark navy. Two tiny embedded images of 2 by 3 pixels sit at the left edge of each LocalQueue area and are effectively invisible.
Slide 18
Batch reference architecture
KubeCon
CloudNativeCon
Europe 2024
cohort
LocalQueues
ClusterQueues
Nodes
RESERVATION
NominalQuota: 4
BorrowingLimit: 0
ON-DEMAND
NominalQuota: 4
BorrowingLimit: 4
4
Requeued
RESERVATION
NominalQuota: 0
BorrowingLimit: 4
SPOT
NominalQuota: 12
BorrowingLimit: 4
8
RESERVATION
ON-DEMAND
SPOT
Visuals: Diagram. Left column: two user icons (top one teal-to-blue, bottom one green). Each has a right-pointing outlined arrow (grey fill #E8E8E8, outline #4B3B71) into a LocalQueue rectangle (white, thin outline #4B3B71). The top LocalQueue contains a 2x2 grid of red-outlined squares (#990000) and has an arrow out to the upper ClusterQueue group. The bottom LocalQueue contains a 2x2 grid of green-outlined squares (#1E7B2A) and a speech bubble labelled 'Requeued' (white, outline #4B3B71, tail pointing down into the icon) sits above it, with an arrow out to the lower ClusterQueue group. Centre: a dotted grey rounded rectangle labelled 'cohort' containing two ClusterQueue frames (black 1px outline, white fill). Upper frame: a RESERVATION box (blue #0563C1 title, NominalQuota: 4, BorrowingLimit: 0) and an ON-DEMAND box (red #990000 title, NominalQuota: 4, BorrowingLimit: 4) with a large purple number 4 (#4B3B71) at right. Lower frame: a RESERVATION box (NominalQuota: 0, BorrowingLimit: 4) and a SPOT box (green #38761D title, NominalQuota: 12, BorrowingLimit: 4) with a large purple number 8 at right. Right column: three boxes with purple outline (#4B3B71): RESERVATION (blue title, four outlined empty squares), ON-DEMAND (red title, four filled red squares #990000 with purple outline) and SPOT (green title, eight filled green squares #1E7B2A with purple outline, two rows of four). Logos in the top right: a navy KubeCon wheel mark above the 'KubeCon' wordmark, a thin vertical divider, a navy CloudNativeCon mark beside the 'CloudNativeCon' wordmark, and a horizontal rule either side of 'Europe 2024'. Lighting and photography: none. Nothing is cropped or boxed beyond the diagram frames.
Slide 19
Batch reference architecture / cohort / LocalQueues / ClusterQueues / Nodes / RESERVATION / NominalQuota: 4 / BorrowingLimit: 0 / 4 / ON-DEMAND / NominalQuota: 4 / BorrowingLimit: 4 / 4 / RESERVATION / NominalQuota: 0 / BorrowingLimit: 4 / SPOT / NominalQuota: 12 / BorrowingLimit: 4 / 8 / RESERVATION / ON-DEMAND / SPOT
Visuals: Diagram. Left column: two person icons (top one blue, bottom one green) each followed by a grey block arrow pointing right into a rectangular LocalQueue box; the top LocalQueue box is empty, the bottom LocalQueue box holds a small 2x2 grid icon of green-outlined squares at its right end. Each LocalQueue box has a grey block arrow pointing right into the cohort. Centre: a dotted-outline rounded rectangle labelled 'cohort' above it, containing two black-outlined ClusterQueue frames. Upper ClusterQueue frame holds two stacked boxes: RESERVATION (blue label) with NominalQuota: 4 and BorrowingLimit: 0 and a large purple count 4 at right; ON-DEMAND (dark red label) with NominalQuota: 4 and BorrowingLimit: 4 and a large purple count 4 at right. Lower ClusterQueue frame holds two stacked boxes: RESERVATION (blue label) with NominalQuota: 0 and BorrowingLimit: 4; SPOT (green label) with NominalQuota: 12 and BorrowingLimit: 4 and a large purple count 8 at right. Right column: three boxes labelled Nodes. RESERVATION node shows four dark red squares in a 2x2 grid. ON-DEMAND node shows four dark red squares in a 2x2 grid. SPOT node shows eight green squares in two rows of four. No chart, table or photograph; no legend or highlighted points.
Slide 20
Batch reference architecture
cohort
LocalQueues
ClusterQueues
Nodes
RESERVATION
NominalQuota: 4
BorrowingLimit: 0
4
ON-DEMAND
NominalQuota: 4
BorrowingLimit: 4
4
RESERVATION
NominalQuota: 0
BorrowingLimit: 4
SPOT
NominalQuota: 12
BorrowingLimit: 4
12
RESERVATION
ON-DEMAND
SPOT
Visuals: Diagram. Two rows of flow from users to nodes. Row 1 (top): a blue person icon, a grey block arrow pointing right, an empty LocalQueue box, a grey block arrow pointing right into the cohort's first ClusterQueue group. Row 2 (bottom): a green person icon, a grey block arrow, an empty LocalQueue box, a grey block arrow into the cohort's second ClusterQueue group. Inside the dotted cohort (labelled 'cohort' above it), the top ClusterQueue group contains two quota cards: RESERVATION (NominalQuota 4, BorrowingLimit 0, large number 4 at right) and ON-DEMAND (NominalQuota 4, BorrowingLimit 4, large number 4 at right). The bottom ClusterQueue group contains RESERVATION (NominalQuota 0, BorrowingLimit 4, no large number) and SPOT (NominalQuota 12, BorrowingLimit 4, large number 12 at right). Right column 'Nodes' has three boxes: RESERVATION with 4 dark red squares in a 2x2 grid; ON-DEMAND with 4 dark red squares in a 2x2 grid; SPOT with 12 green squares in two rows of six. Squares are about 1.5% of slide width with a slate-purple outline. Icons are flat, with no shading beyond a soft vertical gradient on the person heads and bodies. No photograph; no tint or overlay. Logos top-right: KubeCon wheel-and-hexagon mark in dark navy-purple with the word 'KubeCon' beneath; a thin vertical divider; CloudNativeCon square-N mark in dark navy-purple with 'CloudNativeCon' beneath and 'Europe 2024' centred below both with thin horizontal rules on either side.
Slide 21
Kueue in CyberAgent
source
Visuals: One photograph of the Fushimi Inari shrine gate (Japan): a vermilion two-storey torii-style gatehouse with a dark brown tiled roof and green-tinted eaves, flanked by two bronze fox statues with red bibs on stone pedestals and red lanterns on pillars with green roofs at left and right. Several people walk up a wide grey stone staircase toward the gate, two people sit on the left steps, and pink azalea shrubs line the path; a tall green pine stands at the right. Lighting is soft overcast daylight with a pale blue-grey sky. Dominant colours are vermilion red, white, grey stone, and pale blue-grey sky with pink flowers and green foliage. The photo is a rectangular, unrounded, unboxed image with no overlay or tint, full frame with no visible crop of the main subject; it sits on the white background. Logo lockup at top right: the KubeCon wordmark with a purple ship-wheel emblem on the left, a thin vertical divider, and the CloudNativeCon wordmark with a purple square-knot emblem, with the line 'Europe 2024' centred beneath between two thin horizontal rules; all in dark purple/navy #2E2A6B on the cyan band.
Slide 22
Kueue in CyberAgent / CyberAgent Internal on-premise ML Platform / Infrastructure / Bare metal machines for GPU Nodes / Heterogeneous computing resources with 7 types GPUs / NVIDIA H100 / NVIDIA A100 40GB / NVIDIA A100 80GB / NVIDIA L4 / etc … / Kubernetes Cluster / A Single Vanilla Multi-Tenant Cluster / The number of tenants is over 300 / Operation Period is over 4 years in the same cluster / KubeCon / CloudNativeCon / Europe 2024 / ML PLATFORM
Visuals: Two outlined polygon frames with chamfered corners, thin dark slate-purple stroke about 1 pt, white fill, no shadow. The left frame spans about x 1.8 to 48.8 percent and y 26.8 to 92.8 percent, with its top-right and bottom-left corners cut diagonally. The right frame spans about x 50.8 to 97.8 percent and y 26.8 to 74.3 percent, with its top-right and bottom-left corners cut diagonally. Top-right logo: KubeCon Kubernetes wheel mark in dark purple next to CloudNativeCon hexagon-style mark in dark purple, separated by a thin vertical divider, with the words KubeCon and CloudNativeCon and a line reading Europe 2024 beneath, all in dark purple and navy text. Bottom-right logo: ML PLATFORM, a stylised black bird silhouette (raven-like, facing left with a small eye) with bold black uppercase sans-serif text ML over PLATFORM to its right; the bird is black on white.
Slide 23
Kueue in CyberAgent
Workloads and Frameworks
● Training Machine Learning Models
○ batch/v1 Job
○ Kubeflow TFJob / PyTorchJob / MPIJob
● Jupyter Notebook
○ Managed by in-house system
■ Managed by in-house kueue-job manager
● Serving Machine Learning Model
○ Kserve (formerly KFServing)
○ Managed by ResourceQuota
KubeCon
CloudNativeCon
Europe 2024
job
Kubeflow
jupyter
KServe
Visuals: Four logos on white, none boxed or cropped: (1) a blue heptagon (dark-blue-to-royal-blue #2F5BD6 approx.) with a white 3x3-style grid of squares and the word 'job' in white below, at x 64.8% y 25.1% w 8.3% h 14.2%; (2) the Kubeflow logo, a blue faceted emblem in light blue #7FB2F5 and royal blue #1F5FD1 above the blue wordmark 'Kubeflow' (the word in blue #2F6FDB with 'flow' in dark navy #0B2E8A approx.), at x 77.5% y 35.0% w 10.1% h 17.8%; (3) the Jupyter logo, an orange (#F37626 approx.) ring-shaped mark with three grey (#8C8C8C approx.) dots and the grey lowercase wordmark 'jupyter' across the centre, at x 62.6% y 53.7% w 9.7% h 19.9%; (4) the KServe logo, a blue (#2F6FDB approx.) heptagon containing a white cloud with an upward trend arrow and the white word 'KServe' below, at x 86.3% y 57.2% w 10.1% h 13.6%. Header logos (within full-slide template image): KubeCon emblem in dark navy #2B2B5E approx. with wordmark, a vertical divider, and CloudNativeCon emblem in dark navy with wordmark, with the line 'Europe 2024' flanked by thin rules beneath, all in navy #1E1E4A approx. on the blue band.
Slide 24
Kueue in CyberAgent
Training Machine Learning Models: What do we work on the platform?
cyberagent
/calm2-7b-chat
huggingface.co
KubeCon
CloudNativeCon
Europe 2024
Visuals: One photograph/illustration: a blurred colour-gradient graphic, boxed (not full-bleed, not cropped) within the white slide. It shows the text 'cyberagent' with a small CyberAgent logo at top left, the monospace path '/calm2-7b-chat' in bold white in the middle, and the Hugging Face smiling-face emoji logo with 'huggingface.co' at the lower left. The gradient runs from dark teal at the top, through soft blue in the upper middle, to an orange-brown glow at the lower left, a violet-blue centre and a near-black navy corner at the lower right. A faint stray dot sits at about x 11.9%, y 50%, just outside the photo's left edge. Two logos appear in the header: the KubeCon mark (a purple ship-wheel hexagon emblem with the word 'KubeCon' below) and the CloudNativeCon mark (a purple square with a diagonal-striped N-like shape with the word 'CloudNativeCon' below), separated by a thin vertical divider and followed by 'Europe 2024' with thin horizontal rules on either side.
Slide 25
Slide 26
Slide 27
Kueue in CyberAgent / Bridging the Gap between Ideal and Real worlds / Ideal world / All resources and workloads are completely managed by Kueue / Real world / Conflicting between existing quota management system and Kueue / Some workloads and frameworks are not yet supported by Kueue / Kueue's functions are insufficient / (e.g. Model Serving Server and Auto Scaling Semantics etc.) / KubeCon / CloudNativeCon / Europe 2024
Visuals: Logo lockup at top right: a KubeCon logo (dark purple/navy Kubernetes wheel mark above the wordmark 'KubeCon') separated by a thin vertical divider from a CloudNativeCon logo (dark purple/navy square mark with a diagonal stripe above the wordmark 'CloudNativeCon'), with 'Europe 2024' centred beneath between two thin horizontal rules. The lockup is dark purple/navy (about #2E2A5A) on the cyan band. No charts, photographs or other illustrations.
Slide 28
Kueue in CyberAgent
Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue
● Ideal world: Quota management and Queueing system by Kueue
○ Kueue admits jobs based on quota defined in ClusterQueues
● Real world: Existing quota management system
○ It depends on ResourceQuota
○ It distributes ResourceQuota to namespaces (tenants) to reserve quota
KubeCon | CloudNativeCon
Europe 2024
Visuals: Two logos at top right: the KubeCon logo (purple Kubernetes helm wheel above the word KubeCon) separated by a thin vertical divider from the CloudNativeCon logo (purple square with diagonal stripes above the word CloudNativeCon), with the text Europe 2024 centred beneath between two thin horizontal rules. Both logos are dark purple (about #3B2A5A) on the blue band. No photographs, charts or diagrams.
Slide 29
Kueue in CyberAgent
Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue
● Ideal world: Quota management and Queueing system by Kueue
○ Kueue admits jobs based on quota defined in ClusterQueues
● Real world: Existing quota management system
○ It depends on ResourceQuota
○ It distributes ResourceQuota to namespaces (tenants) to reserve quota
● Gap
○ Over-assignment could occur
■ Over-assigned Job continues to be rejected by ResourceQuota until resources are free
○ Kueue can not admit jobs based on ResourceQuota
Visuals: Two conference logos in the top-right corner of the header band. Left logo: KubeCon, a dark purple Kubernetes ship-wheel icon above the wordmark KubeCon in bold dark purple. A thin vertical divider separates it from the right logo. Right logo: CloudNativeCon, a dark purple square with a stylised white-on-purple diagonal mark above the wordmark CloudNativeCon, then Europe 2024 in small bold dark purple between two thin horizontal rules. No photographs, charts, or diagrams.
Slide 30
Kueue in CyberAgent
Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue
● Ideal world: Quota management and Queueing system by Kueue
○ Kueue admits jobs based on quota defined in ClusterQueues
Approaches to bridge the gap:
1. WaitForPodsReady with Configurable RequeuingStrategy
● Real world: Existing quota management system
2. AdmissionCheck for ResourceQuota
○ It depends on ResourceQuota
○ It distributes ResourceQuota to namespaces (tenants) to reserve quota
● Gap
○ Over-assignment could occur
■ Over-assigned Job continues to be rejected by ResourceQuota
until resources are free
○ Kueue can not admit jobs based on ResourceQuota
KubeCon
CloudNativeCon
Europe 2024
Visuals: Two logos at top right: the KubeCon logo (a dark purple-navy Kubernetes-style wheel-in-hexagon mark above the word KubeCon) and the CloudNativeCon logo (a dark purple-navy square mark with a stylised N-like cut-out above the words CloudNativeCon and a thin rule with Europe 2024 beneath). A callout box at x10.1 y23.3 w79.7 h36.3 with white fill and a dark grey #595959 stroke about 2 px wide, square corners, no shadow, containing the text Approaches to bridge the gap: and two numbered items. A thin dark grey #595959 wedge outline (triangle with white interior, about 2 px stroke) with its apex at about x13.5 y59.6 and base points on the lower edge of the callout box at about x23.4 and x43.4, pointing to the Gap bullet. No photographs, charts or diagrams.
Slide 31
Kueue in CyberAgent
Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue
Approaches to bridge the gap:
1. WaitForPodsReady with Configurable RequeuingStrategy
2. AdmissionCheck for ResourceQuota (※1)
● Ideal world: Quota management and Queueing system by Kueue
○ Kueue admits jobs based on quota defined in ClusterQueues
● Real world: Existing quota management system
○ It depends on ResourceQuota
○ It distributes ResourceQuota to namespaces (tenants) to reserve quota
● Gap
○ Over-assignment could occur
■ Over-assigned Job continues to be rejected by ResourceQuota until resources are free
○ Kueue can not admit jobs based on ResourceQuota
※1 Currently, it is evaluating in NON-production environment
KubeCon
CloudNativeCon
Europe 2024
Visuals: Two conference logos at top right: the KubeCon logo (a dark purple-navy Kubernetes-style wheel in a hexagon with the word KubeCon beneath it) and the CloudNativeCon Europe 2024 logo (a dark purple-navy square emblem with a stylised N-like mark, the words CloudNativeCon beneath, and Europe 2024 below in a thin rule). A vertical thin grey divider separates the two logos. Over the bullet list is a white rectangle with a thin dark grey (#595959) 1 pt outline, containing the text Approaches to bridge the gap: and a numbered list; a dark grey thin triangular callout pointer (two straight lines meeting at a point near the Gap bullet) extends from the bottom edge of the box down to the Gap line. No charts, photographs or icons.
Slide 32
Kueue in CyberAgent / Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue / ● WaitForPodsReady with Configurable RequeuingStrategy / ○ Pros: / ■ Ready to use by a Kueue's configuration setting / ○ Cons: / ■ Increased API-server load due to repeated try to create Pods / and re-queueing jobs
Visuals: Two event logos at top right of the header band. Left logo: KubeCon wordmark with a dark purple/navy Kubernetes-style helm wheel emblem above the word 'KubeCon' in bold dark navy. A thin vertical divider separates it from the right logo. Right logo: a dark purple/navy square with a stylised diagonal cut mark above the words 'CloudNativeCon' in bold dark navy, with 'Europe 2024' centred beneath in small bold text flanked by thin horizontal rules. No photographs, charts, tables or diagrams. A tiny faint grey speck sits near x 11.8%, y 50% just above the 'Increased API-server' bullet; it is an artifact with no content.
Slide 33
Kueue in CyberAgent
Migrate K8s’s ResourceQuota to Kueue’s ClusterQueue
● WaitForPodsReady with Configurable RequeuingStrategy
○ Pros:
■ Ready to use by a Kueue's configuration setting
○ Cons:
■ Increased API-server load due to repeated try to create Pods
and re-queueing jobs
● AdmissionCheck for ResourceQuota
○ Pros:
■ It could avoid higher kube-api server load
○ Cons:
■ It needs to implement small custom controller
KubeCon | CloudNativeCon
Europe 2024
Visuals: Visual elements are the cyan header band, which is a flat filled rectangle spanning the full slide width, and two conference logos at the top right of the band. The left logo is the Kubernetes wheel icon in dark navy with the caption KubeCon beneath it, separated by a thin vertical divider line from the right logo, which is a navy square icon with a diagonal slash mark and the caption CloudNativeCon beneath it. Beneath both captions is the text Europe 2024 between two thin horizontal rules. No charts, photographs, tables or diagrams are present.
Slide 34
Kueue in CyberAgent / KubeCon / CloudNativeCon / Europe 2024 / Overlapping Quota Model between User and Admin / ● Conflicting Demands against GPUs / ○ All GPUs always SHOULD be allocated to user’s workloads / ○ Cluster Admins WANT to verify platform features using GPUs / ● Admins can submit Jobs only when there are some free quota in Shared-CQ / Cohort Shared-Admin / Shared-CQ NominalQuota / Admin-CQ doesn’t have / any NominalQuota / Admin-CQ BorrowingLimit
Visuals: One diagram: a black-outlined rectangular box labelled 'Cohort Shared-Admin' at its top-left, containing two stacked horizontal bands. The upper band is a light blue (#9FC5E8) rectangle with a solid blue (#1155CC) outline, labelled 'Shared-CQ NominalQuota' at its left. Over its lower part and the lower band sits a semi-transparent red (#FF0000 at 35% opacity) rectangle, which tints the blue band to mauve (about #BF8097) in the middle section and tints white to light pink (about #FFA6A6) in the bottom band. The mauve section has a dashed purple (about #674EA7) lower edge and a dash-dot mauve border, and the pink band has a dash-dot mauve (about #A64D79) border; the bottom band is labelled 'Admin-CQ BorrowingLimit' right-aligned. A speech-bubble callout (white fill, thin grey #595959 outline, about 1.5 pt) on the left reads 'Admin-CQ doesn’t have / any NominalQuota' centred, and its pointer tail extends down and right to the lower-left corner of the pink band. No photographs, charts or tables.
Slide 35
Kueue in CyberAgent
Elastically Reserved Quota Model
between the same priority important projects
● Conflicting Demands against GPUs
○ Important projects WANT to reserve GPUs
so that they can use GPUs when they want to use it
○ Any GPUs SHOULD NOT be left over for efficient usage
● Any Jobs borrowed by other ClusterQueues always be preempted
Cohort Important-Projects
Project-A
NominalQuota
Project-B
NominalQuota
Project-B
BorrowingLimit
Project-A
BorrowingLimit
preemption:
reclaimWithinCohort: Any
Visuals: Diagram: a black-outlined rectangular frame labelled 'Cohort Important-Projects' containing a 2x2 grid of rectangles. Top-left cell is light blue (#9FC5E8) with a solid blue (#1155CC) border and the label 'Project-A / NominalQuota'. Top-right cell is light red (#EA9999) with a solid mauve (#A64D79) border and the label 'Project-B / NominalQuota'. Middle row: a blue-tinted band at 35% opacity spanning the right half with a dash-dot blue border, and a red-tinted band at 30% opacity spanning the left half with a dash-dot mauve border; the two overlap the top cells' lower edges. Bottom-left cell is pale pink (about #F9DEDE) with a dash-dot mauve border labelled 'Project-B / BorrowingLimit'. Bottom-right cell is pale blue (about #DDEAF8) with a dash-dot blue border labelled 'Project-A / BorrowingLimit'. Cells are flat with no shadow. Also a code snippet image (illustration) at top right showing monospace text 'preemption:' and indented 'reclaimWithinCohort: Any' with keys in tan (about #C8956A) and values in dark grey, on a very light grey (#F8F8F8) rounded box with a thin grey (about #B8BEC6) border. Logo lockup at top right in dark navy (about #2E2A6B): Kubernetes wheel icon, KubeCon wordmark, vertical divider, CloudNativeCon icon, CloudNativeCon wordmark, and 'Europe 2024' line.
Slide 36
New features / source
Visuals: One photograph: a sparkler held in a hand against a near-black background. The sparkler is lit at its tip, throwing bright white sparks and short streaks in all directions, with some warm orange glints on the stick. A hand, seen from the side, holds the stick from the lower-left-centre; fingers are curled around the handle. A soft blue bokeh light sits to the right of the sparkler's centre. Composition is landscape, centred, with the sparkler near the middle and the hand at the bottom. The photo is not cropped at its edges (rectangular, boxed, not full-bleed) and has no overlay or tint. Lighting is low-key and dramatic, lit only by the sparkler. Dominant colours are near-black #0E1512, white #FFFFFF highlights and warm skin tones around #B5785A. The header logo is a non-text element: the KubeCon and CloudNativeCon Europe 2024 logo, with a Kubernetes wheel icon over the word KubeCon and a stylised N-shaped square icon over the word CloudNativeCon, separated by a vertical divider line and with the line 'Europe 2024' below, all in dark indigo #2E2A6B.
Slide 37
LendingLimit
● LendingLimit indicates how much resources can this ClusterQueue lend to other ClusterQueues within the same Cohort.
● Use cases
○ Control the size of the guaranteed quota for a ClusterQueue in a Cohort
○ Reserve quota for latency sensitive workloads such as ML Model Serving
■ In the future, it is possible to integrate with Kserve (formerly KFServing)
Visuals: Two logos at top right of the header band: the KubeCon logo (a dark navy-purple Kubernetes helm-wheel hexagon icon above the bold wordmark 'KubeCon') separated by a thin vertical divider from the CloudNativeCon logo (a dark navy-purple square icon with a diagonal stripe above the bold wordmark 'CloudNativeCon'), with 'Europe 2024' in small bold navy text centred beneath both, flanked by thin horizontal rules. No photographs, charts, or diagrams.
Slides 38 to 49 continue in the full text file.
Download this deck
The hosted PDF of every library deck is free for readers of the SlideDeck Co newsletter. Join with your email and the viewer and download appear here.
Building a deck of your own? The SlideDeck Co Standard Deck is a slide library for business presentations: reviews, updates and plans.






