Building docker images in kuberentes cluster

1. Is it a good practice is build docker images in kubernetes cluster where other containers will be run/executed?Are there any obvious challenges with kaniko?

Yes, it is possible to build images inside Kubernetes containers, but it could be a bit of a challenge.

Some users use it to build a workflow for CI/CD with Jenkins. In fact, it is better to use tools to simplify the process.

Kubernetes also have rules to prepare containers development kit, they are described here

Another way is to use Kaniko, this tool builds container images from a Dockerfile inside a container or Kubernetes cluster.

I found this article interesting to read on this topic.

On the other hand, there was a successful attempt to build images without Docker daemon running. You may be interested in Bazel project and story how to use it.

2. Should separate dedicated VMs be created to manage the build process?

Regarding your second question: It is not necessary to set up dedicated VM to run Docker images creation workflow.

Finally, it may be interesting to have a private registry in Kubernetes cluster and use it for building purposes.

It's possible to build images on kubernetes nodes. But i wouldn't recommend it. The reason being, a application build process is memory and compute intensive, frequent image builds could cause disruption to services being scheduled by that kubernetes node.

Use a dedicated Jenkins server(s) instead, create pipelines according to your requirements and delivery.

You can get started here!

Hope that helps!