How to measure exactly what "data out" in Azure web app?
After some very good help from a Ukraine developer I found on Upwork, we've finally solved the issue.
The challenge was in our robots.txt.
It turned out, that we had SO many requests on our pages - and we have 3.6 mill address pages - that it simply was a HUGE amount of requests. That's why the data out was so big.
We have now solved it by:
- Adding a robots.txt which disallow all bots but Google and Bing
- Adjusted the Google crawl speed in the Webmaster Tools
- Adjusted our sitemap from monthly to yearly changefreq for our address pages to avoid re-crawling
I'm happy!
Follow guidance given in Understand your bill for Microsoft Azure.Review billing from subscription level perspective.
Find out whether egress is sended/requested into/from azure services in other regions or largely requested from website visitors. Verify backup panel of web app as well or any other backup running regularly.
Review performance monitoring or performance test. Any tests from other regions responsible for larger egress?
Find out if egress follows site load during business times. If not dig deeper.Find out if SEO visitors trigger any downloads, if yes adjust links accordingly.