How to safegaurd prometheus data?

Short term answer use a PV, but probably not NFS, you don’t need multiple writers. A simple network block device (EBS, GCPDisk, etc) is fine. Long term, HA Prometheus is a complex topic, check out the Thanos project for some ideas and tech. Also the Grafana Labs folks have been experimenting with some new HA layouts for it. Expect full HA Prometheus to be a very substantial project requiring you to dive deep into the internals.

There is also another option in addition to storing the prometheus data as Persistence Volume(PV). You can use exporters by prometheus as mentioned here. These exporters get the scraped data and stores them in some external db like elastic search or mysql and can be used another prometheus instance in case the previous prometheus instance crashed.

Prometheus can replicate data to remote storage via remote_write API. This means that the data isn't lost on Prometheus pod restart in k8s clustet, since it is already replicated to remote storage. See how to set up remote storage in Prometheus. This example uses VictoriaMetrics - cost efficient open source remote storage for Prometheus with the following features:

• PromQL support.
• Low resource usage - RAM, CPU, disk, network
• Scales vertically and horizontally

I wouldn't recommend Thanos, since it doesn't prevent from data loss for the recent 2 hours on Prometheus pod restarts. See this article for details.