Migrating a Virtual machine running customized Guest OS to containers

I strongly suggest to avoid to try something like this, concepts are quite different between process virtualization and machine virtualization.

Please check this answer How is Docker different from a virtual machine?

This is a very broad question with lots of potential detail, but effectively the containers do not share the OS, but rather they share the kernel within the OS.

To quote Docker's own wording on this subject:

Each container shares the kernel within the host OS, allowing you to run multiple Docker containers on the same host. Unlike VMs, containers do not have an OS within it.

Source of the quote

To broadly answer your question, yes its possible to move the 'VM' running a custom linux OS to a container, but you're not really moving the 'VM' but rather just moving the application of interest to a container (if that's what you want to do).

You'll need to choose or custom build a docker image that is based on the same architecture and linux distribution that your VM OS is currently using to make things easier for yourself. This is generally a process of a designing a Dockerfile and then building your image from that.

You'll build/bake your application files into that image and provide configuration values for the application to use inside the docker container that runs from the image most likely as custom configuration passed in with environment variables.