What's difference between protractor (Selenium webdriver) VS ghostdriver (phantomjs webdriver)?

While I'm not entirely sure I understand your question, I'll take a stab at answering it. With WebDriver, driving a browser is done via a standardized JSON-over-HTTP wire protocol. This means that you need a "server" component that understands the wire protocol to drive any particular browser. For each of the major desktop browsers (Internet Explorer, Chrome, and Firefox), there is a server component that your WebDriver code talks to (IEDriverServer.exe, chromedriver.exe, or a Firefox browser extension, respectively). PhantomJS also implements a server component that understands the WebDriver wire protocol, so the same high-level WebDriver code can be used with PhantomJS that is used with other browsers. Note that the Selenium server is not required to drive any of the browsers on the local machine.

Now, since the protocol used is simply transmitted over HTTP, that gives WebDriver the opportunity to run the WebDriver code on one machine, while driving a browser located on an entirely different machine. That's where the Selenium server comes in. The Selenium server starts an HTTP server that understands the WebDriver JSON wire protocol. When that server receives a WebDriver command, it can forward that command to another "server" component, either running on that machine (as a standalone remote server), or on yet another machine running another instance of the Selenium server (in a "grid" configuration).

So to answer your question, yes, WebDriver code can be executed against PhantomJS without using the Selenium server. It can likewise be executed against Internet Explorer, Firefox, Chrome, Safari, and some versions of Opera, all without using the Selenium server. Notice that all of this is true without mentioning Protractor at all. Since Protractor is based on WebDriverJS, as long as there's a "server" component running, whether that's a Selenium server, chromedriver.exe, IEDriverServer.exe, or PhantomJS, the driver should be able to communicate with and drive that browser. Looking at the code, it appears that WebDriverJS (and, by extension, Protractor), should be able to execute against Chrome and PhantomJS without requiring the Selenium server, but I don't know enough about Protractor's wrapping of WebDriverJS to speak with authority.