Query on custom metrics exposed via prometheus node exporter textfile collector fails

I had a similar situation, but it was not a configuration problem.

Instead, my data included timestamps:

# HELP network_connectivity_rtt Round Trip Time to each node# TYPE network_connectivity_rtt gaugenetwork_connectivity_rtt{host="home"} 53.87 1541426242network_connectivity_rtt{host="hop_1"} 58.8 1541426242network_connectivity_rtt{host="hop_2"} 21.93 1541426242network_connectivity_rtt{host="hop_3"} 71.69 1541426242

PNE was picking them up without any problem once I reloaded it. As prometheus is running under systemd, I had to check the logs like this:

journalctl --system -u prometheus.service --follow

There I read this line:

msg="Error on ingesting samples that are too old or are too far into the future"

Once I removed the timestamps, values started appearing. This lead me to read more in detail about the timestamps, and I found out they have to be in miliseconds. So this format now is ok:

# HELP network_connectivity_rtt Round Trip Time to each node# TYPE network_connectivity_rtt gaugenetwork_connectivity_rtt{host="home"} 50.47 1541429581376network_connectivity_rtt{host="hop_1"} 3.38 1541429581376network_connectivity_rtt{host="hop_2"} 11.2 1541429581376network_connectivity_rtt{host="hop_3"} 20.72 1541429581376

I hope it helps someone else.

Its my bad. I did not included scrape instructions for node-exporter in prometheus.yaml file. It worked after including them.

This issue is happening because of stale metrics.Lets say you have written you metric in file at 13.00by default after 5min prometheus will consider you metric stale and it might disappear from there at the time you are making query.