Rootless docker-compose cannot build timescale image
So I managed to make it work. In my Dockerfile I also set the uid of a user because I share some volumes and want the UIDs of users be consistent between the containers. So on top of my Dockerfile I had the following:
RUN useradd --uid 80000 postgresreplacing the uid with the lower value solved the issue
RUN useradd --uid 18000 postgres