Configuring BuildKit for High Performance
Optimizing BuildKit performance is essential for a smooth development experience with Okteto. A high-performance BuildKit setup accelerates image builds, test executions, and deploy commands when using Remote Execution.
1. Dedicated Node Pool for BuildKit
BuildKit is resource-intensive. We recommend deploying BuildKit in a dedicated node pool for optimal performance and to minimize interference from other workloads.
To deploy BuildKit on a dedicated build pool, you can add taints and tolerations to the BuildKit node pool in Kubernetes, and then add the following settings to your Okteto Helm configuration:
buildkit:
nodeSelectors:
okteto-node-pool: build
tolerations:
- effect: NoSchedule
key: okteto-node-pool
operator: Equal
value: build
2. Scale Vertically: Increase CPUs for Faster Builds
BuildKit performance depends heavily on allocated CPU and memory resources. Start with 2 nodes for higher availability, with the following recommended instance types (4 CPUs and 16GB RAM):
- Amazon Web Services:
m6a.xlarge
- Google Cloud Platform:
t2d-standard-4
- Microsoft Azure Cloud Platform:
Standard_D4as_v5
Set the replicaCount to match the number of nodes in the BuildKit node pool:
buildkit:
replicaCount: 2
Monitor performance and adjust node resources as needed. For instance, upgrading your BuildKit nodes to 8 CPUs and 32 GB of memory each can provide approximately a 2x performance improvement 😎.
3. Configure SSD Storage
BuildKit is I/O intensive, especially for pulling, extracting, and pushing container images. Using a SSD storage class can significantly improve your BuildKit performance.
buildkit:
persistence:
enabled: true
class: <<your-ssd-storage-class>>
You may also increase the size of the BuildKit cache to increase the number of BuildKit cache hits (default: 100 GB):
buildkit:
persistence:
enabled: true
size: 200Gi
4. Dedicated Load Balancer for BuildKit
By default, BuildKit is exposed via gRPC behind the Okteto Nginx Ingress Controller. This setup simplifies the initial installation, but we recommend exposing BuildKit on a dedicated load balancer for improved performance and control:
Pros
- More control over how build requests are distributed across BuildKit pods.
- Reduces load on the Nginx Ingress Controller, improving performance for other services.
Cons
- Extra costs for provisioning and maintaining an additional load balancer.
To expose BuildKit via a dedicated load balancer, add the following to your Okteto Helm configuration:
buildkit:
service:
type: LoadBalancer
sessionAffinity: ClientIP
ingress:
enabled: false
Setting sessionAffinity
to ClientIP
directs all client requests to the same BuildKit pod, leveraging caching for improved performance.
Additional BuildKit service annotations might be needed to support ClientIP
affinity, refer to your Cloud Provider documentation for additional assistance.
For example, for AWS Load Balancer Controller you would need the following annotations:
buildkit:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "stickiness.enabled=true,stickiness.type=source_ip"
After upgrading Okteto, use kubectl
to retrieve the external IP address assigned to the BuildKit service:
kubectl get service -l=app.kubernetes.io/part-of=okteto,app.kubernetes.io/component=buildkit --namespace=okteto
The output will look something like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
okteto-buildkit LoadBalancer 10.245.142.73 a519c8b3b27f95...elb.amazonaws.com 443:31597/TCP 5m
Take the EXTERNAL-IP
address and create an A
record for it with the name buildkit.<<your-okteto-subdomain>>
.
5. Horizontal Pod Autoscaling (optional)
Horizontal Pod Autoscaling (HPA) enables the BuildKit pods to scale based on CPU and memory utilization, automatically adjusting pod numbers to meet demand.
Requirements
- The Metrics API must be available in your cluster.
- Configure your BuildKit node pool in Kubernetes with autoscalability.
The default Metrics API only works with buildkit.rootless.enabled: true
. If you prefer not to enable BuildKit rootless mode, you will need to configure your custom metrics in order for the HPA to behave correctly
You can configure BuildKit HPA with the buildkit.hpa configuration settings:
buildkit:
rootless:
enabled: true
hpa:
enabled: true
min: 2
max: 6
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
6. Save Costs with Spot Instances (optional)
Use spot instances to reduce costs while maintaining sufficient resources for BuildKit.
Ensure high availability by configuring at least 2 nodes across different zones.
The Okteto CLI automatically retries any command if interrupted by BuildKit instance termination or failure to minimize the effect of using spot instances. If using remote execution, it is very important to ensure that your deploy/test/destroy commands are idempotent, as commands may run again if a node terminates during the execution of the deploy or destroy operations.
7. Follow Best Practices for Dockerfiles and Remote Execution
Even the best BuildKit setup cannot compensate for un-optimized Dockerfiles or inefficient Remote Execution configurations. To make the most of your BuildKit configuration, we have the following recommendations:
- Adhere to Dockerfile best practices.
- Reduce file transfer times to BuildKit by using .dockerignore and .oktetoignore files.
- Use BuildKit cache mounts to persist cache folders between image builds. This can significantly speed up build times by caching dependencies and build artifacts.
- Use Test Container caches to persist cache folders between test executions. This ensures that test dependencies are cached, reducing the time taken for subsequent test runs.