IE Web Driver runs 3-5 times slower running on TeamCity Windows Server 2012
One reason i can think of is system resource utilization when the tests are running. That is how many processes are running and how much of CPU/RAM are they consuming. I implemented a Jenkins CI setup and saw that the resource utilization on the test machine was very high and the tests were running very slow. May not be a similar situation for you, but wanted to share my experience.
I had this problem but with chromedriver. The tests ran up to 10 times slower via TeamCity compared to locally. I narrowed it down to setting DNS refresh timeout to 0.
ServicePointManager.DnsRefreshTimeout = 0;Once I increased this to a few seconds the tests took the same time on TeamCity vs locally.