Prometheus with multi-container pod on kubernetes
Here's an example job for Prometheus. Put it in your own config.
Next, add:
annotations: prometheus.io/scrape: 'true'to your pod metadata.
And on every container, which provides /metrics to prom, create an appropriate port, named metrics.
That's it. Prometheus would scrape only those ports, and there would be no situation, like when your redis instance would get http requests on its 6379 port.
The annotations you propose should work. Create one scrape_config per port annotation, keeping only targets matching the corresponding annotation port name.