Lesson learned from moving one java product to kubernetes

src: [VDBUH2023] - Carlos Sanchez - We Moved one Java Product to Kubernetes and This Is What We Learned - Invidious

Abstract

Using namespaces to provide a scope:

• network isolation
• permission
• quotas

You can puts lots of sidecars:

• service warmup
• storage initialization
• httpd fronting the Java app
• exporting metrics
• fluent-bit to send logs
• Java threaddump collection
• envoy proxying
• autoupdater

For JVM memory configuration, do not trust JVM ergonomics and configure it with:

• -XX:InitialRAMPercentage
• -XX:MaxRAMPercentage
• -XX:MinRAMPercentage only allows setting the max heap size for a JVM running with less than 200MB Typically can use up to 75% of container memory.

JVM memory use is hidden from Kubernetes, which sees all of it as used ⇒ set request and limits to the same value.

Configure GC with:

• -XX:UseSerialGC
• -XX:UseParallelGC
• -XX:UseG1GC
• -XX:UseZGC
• -XX:UseShenandoahGC

Configure CPUs with -SS:activeProcessorCount. CPU requests in k8s:

• 1 CPU means it can consume one CPU cycle per CPU period
• 2 containers with 0.1 CPU requests each can use 50% of the CPU time of the node

CPU Limits in k8s:

• 500ms in k8s ⇒ 50ms of CPU usage in each 100ms period
• 1000m in k8s ⇒ 100m of CPU usage in each 100ms period
• for 1000m in k8s and 4 threads, you can consume all the CPU time in 25ms and be throttled for 75ms.
• ⇒ do NOT set CPU limits for your production workloads, you would be wasting unused CPU cycles
• limits are interesting for dev/sage to guarantee you are not using more CPU than you should.