Support for multiple buildkit pods on the same node

[shabbirsaifee92]

Describe the problem/challenge you have

We use vclusters to provide ephemeral k8s environment to our devs and recently we have trying to get the images built within the cluster while using tilt.

Since the permission for each user is restricted their ephemeral cluster we need to deploy buildkit within each ephemeral environment. Since buildkit daemons creates a lock on the /var/lib/buildkit/buildkit.lock of the host, currently it is not possible to deploy multiple buildkit pods to the same node.

I was able to find a work around by changing the hostpath of buildkit volume from /var/lib/buildkit to unique file path for each buildkit deployment.

It seems to work fine and I can see multiple pods on the same node and tilt is able to build the images.
But since its more of a hack we are unsure to roll this out. (any guidance if this is an ok hack for now?)

Description of the solution you'd like
Be able to deploy multiple buildkit pods on the same host.

Design/Architecture Details
[If you're have thoughts on how this should be implemented, please share those details]

Environment Details:

• kubectl buildkit version 0.1.6 (use kubectl buildkit version)
• Kubernetes version 1.24.9 (use kubectl version)
• Where are you running kubernetes - AKS
• Container Runtime and version - containerd

Vote on this request

This is an invitation to the community to vote on issues. Use the "smiley face" up to the right of this comment to vote.

• 👍 "This project will be more useful if this feature were added"
• 👎 "This feature will not enhance the project in a meaningful way"