Monitor k8s apps from "operator" cluster

I have seen that you though about using Thanos, its not bad and we had it running in production, for a while. But it didn't fit well for our requirements, yours looks familiar to ours so I suggest you to take a look at VictoriaMetricsyou have a nice article just here : https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683

Also a big up is their support on Slack!Good luck implementing it!

Ideally you would instrument your code and expose Prometheus compatible metrics for whatever needs monitored. But, there is something to be said for blackbox and/or 3rd party monitoring/smoke testing.

The http module in Blackbox Exporter is probably what you want (I've used it similarly before). If that isn't flexible enough for the testing you need to do, I like to run custom testing scripts in Lambda that record the results in Cloudwatch (if running in AWS, otherwise use the equivalent in your environment). If you haven't done that before, there is a bit of a learning curve, but it is well worth the effort.

If the APIs are externally accessible, services like Pingdom and Site24x7 offer flexible testing options (for a price), and it is generally recommended to utilize a 3rd party for at least basic up-time testing for the cases where your entire environment goes down--along with all of your monitoring!

But, it does sound like you just want to do some basic blackbox style monitoring which the Blackbox Exporter would be well suited to. It will need a host to run on, and then you'll need to add a job for it to Prometheus' scrape config. Best practice is to use each host for a single purpose, so I'd provision a specific host for the purpose of running blackbox exporter (even if it is just another container in the cluster).