Skip to main content
Version: 1.28

Configuring BuildKit for High Performance

Optimizing BuildKit performance is essential for a smooth development experience with Okteto. A high-performance BuildKit setup accelerates image builds, test executions, and deploy commands when using Remote Execution.

1. Dedicated Node Pool for BuildKit​

BuildKit is resource-intensive. We recommend deploying BuildKit in a dedicated node pool for optimal performance and to minimize interference from other workloads.

To deploy BuildKit on a dedicated build pool, you can add taints and tolerations to the BuildKit node pool in Kubernetes, and then add the following settings to your Okteto Helm configuration:

buildkit:
  nodeSelectors:
    okteto-node-pool: build
  tolerations:
    - effect: NoSchedule
      key: okteto-node-pool
      operator: Equal
      value: build

2. Scale Vertically: Increase CPUs for Faster Builds​

BuildKit performance depends heavily on allocated CPU and memory resources. Start with 2 nodes for higher availability, with the following recommended instance types (4 CPUs and 16GB RAM):

  • Amazon Web Services: m6a.xlarge
  • Google Cloud Platform: t2d-standard-4
  • Microsoft Azure Cloud Platform: Standard_D4as_v5

Set the replicaCount to match the number of nodes in the BuildKit node pool:

buildkit:
  replicaCount: 2

Monitor performance and adjust node resources as needed. For instance, upgrading your BuildKit nodes to 8 CPUs and 32 GB of memory each can provide approximately a 2x performance improvement 😎.

3. Configure SSD Storage​

BuildKit is I/O intensive, especially for pulling, extracting, and pushing container images. Using a SSD storage class can significantly improve your BuildKit performance.

buildkit:
  persistence:
    enabled: true
    class: <<your-ssd-storage-class>>

You may also increase the size of the BuildKit cache to increase the number of BuildKit cache hits (default: 100 GB):

buildkit:
  persistence:
    enabled: true
    size: 200Gi

4. Dedicated Load Balancer for BuildKit​

By default, BuildKit is exposed via gRPC behind the Okteto Nginx Ingress Controller. This setup simplifies the initial installation, but we recommend exposing BuildKit on a dedicated load balancer for improved performance and control:

Pros

  • More control over how build requests are distributed across BuildKit pods.
  • Reduces load on the Nginx Ingress Controller, improving performance for other services.

Cons

  • Extra costs for provisioning and maintaining an additional load balancer.

To expose BuildKit via a dedicated load balancer, add the following to your Okteto Helm configuration:

buildkit:
  service:
    type: LoadBalancer
    sessionAffinity: ClientIP
  ingress:
    enabled: false
info

Setting sessionAffinity to ClientIP directs all client requests to the same BuildKit pod, leveraging caching for improved performance. Additional BuildKit service annotations might be needed to support ClientIP affinity, refer to your Cloud Provider documentation for additional assistance. For example, for AWS Load Balancer Controller you would need the following annotations:

buildkit:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: "stickiness.enabled=true,stickiness.type=source_ip"

After upgrading Okteto, use kubectl to retrieve the external IP address assigned to the BuildKit service:

kubectl get service -l=app.kubernetes.io/part-of=okteto,app.kubernetes.io/component=buildkit --namespace=okteto

The output will look something like this:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP                           PORT(S)          AGE
okteto-buildkit     LoadBalancer   10.245.142.73   a519c8b3b27f95...elb.amazonaws.com    443:31597/TCP    5m

Take the EXTERNAL-IP address and create an A record for it with the name buildkit.<<your-okteto-subdomain>>.

5. Horizontal Pod Autoscaling (optional)​

Horizontal Pod Autoscaling (HPA) enables the BuildKit pods to scale based on CPU and memory utilization, automatically adjusting pod numbers to meet demand.

Requirements​

  • The Metrics API must be available in your cluster.
  • Configure your BuildKit node pool in Kubernetes with autoscalability.
warning

The default Metrics API only works with buildkit.rootless.enabled: true. If you prefer not to enable BuildKit rootless mode, you will need to configure your custom metrics in order for the HPA to behave correctly

You can configure BuildKit HPA with the buildkit.hpa configuration settings:

buildkit:
  rootless:
    enabled: true
  hpa:
    enabled: true
    min: 2
    max: 6
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

6. Save Costs with Spot Instances (optional)​

Use spot instances to reduce costs while maintaining sufficient resources for BuildKit.

tip

Ensure high availability by configuring at least 2 nodes across different zones.

The Okteto CLI automatically retries any command if interrupted by BuildKit instance termination or failure to minimize the effect of using spot instances. If using remote execution, it is very important to ensure that your deploy/test/destroy commands are idempotent, as commands may run again if a node terminates during the execution of the deploy or destroy operations.

7. Follow Best Practices for Dockerfiles and Remote Execution​

Even the best BuildKit setup cannot compensate for un-optimized Dockerfiles or inefficient Remote Execution configurations. To make the most of your BuildKit configuration, we have the following recommendations: