Why use Page Factory?
Here's Simon Stewart, Selenium project lead and creator of the Page Factory, at the 2017 SeleniumConf in Austin. During his keynote address he says not to use Page Factory. This section of the talk starts here:
https://youtu.be/gyfUpOysIF8?t=1517
Actual statement is at 27:25.
A few answers have said that the PageFactory "loads" all the WebElements when it is instantiated - this is not actually correct.
The elements don't load until you access them. It's done through the base classes and a RealProxy. It does use the standard FindElement(s)By methods under the hood, so there is no real performance benefit to having WebElements vs storing the By's and loading them when you need them.
One reason I choose to not use the PageFactory model, is I may have a search for an element that I don't want to exist, and by using the auto-wired approach, it searches to see if it exists before I can say "doesn't exist" in the test.
Another issue is there are subtle differences between how the PageFactory instantiates the WebElement and how Driver.FindBy instantiates them. One that bugs me is that PageFactory's version doesn't implement IWrapsDriver, meaning you can't get the driver used to find the element from the element. This may not seem like much, but it means when you want to write extensions to WebElement methods that in turn need a driver, you have to work out a (much more complicated) way of getting the driver, especially since I believe the PageObjectModel should not have a direct reference to the driver...
But that said, for a lot of cases, the out of the box PageFactory approach is very good. I think that the key to using- or not using- the PageFactory is just to have a consistent approach to how your test code and page object models work and interract - as that is key to maintainability.
Why do I want to instantiate all the elements instead of doing it on the fly?
If I remember correctly, PageFactory scans for any WebElement properties/fields and their attributes and wraps them with a proxy. At that point you're not touching Selenium server yet (you can check this in server console output). Once you try to access the property the WebElement gets instantiated. So if you only accessed one PO property/field only for that one WebElement is created.
What are the advantages of Page Factory that I'm missing?
The use of attributes make the code much more readable and also easy to generate. It is common to create a tool that generates PageObjects for you.
PageFactory was created to support PageObject pattern, nothing more. You don't have to necessarily use it in order to go PO way.
Finally, if you're curious about how it works in detail, I'd suggest you check the source code. It's what I did when I was just starting with Selenium.