What exactly is a Private API, and why will Apple reject an iOS App if one is used?

Q. What is the main difference between a "Private API" and a "Non-private API?"

• Private is one that isn't publicly defined (there isn't much to it)

Q. Are the "Non-private" APIs only the APIs provided and verified by Apple?

• Private and Public APIs are both 'provided' by Apple but only public APIs are provided for you to use.
• The private APIs are for use by Apple only

Q. Isn't an API just a way of interacting with a Framework, and a Framework is just a set of encapsulated classes/headers that people can use for trivial purposes?

• Yes

Q. Wouldn't this mean that I cannot reuse anyone's code (other than Apple's) at all in my app?

• No

Q. If this is true, whenever a programmer makes his or her own custom classes and methods, isn't he technically designing a mini Framework just for his specific purpose? So wouldn't this mean that he is using his own private API?

• No (a framework is different to an app)

Q. What would even be the difference between using someone else's classes/methods alongside my classes/methods, how could they even tell the difference, and why would this be a problem?

• It usually wouldn't

The point you need to get is that when referring to private APIs you should only be thinking about private Apple supplied APIs.
Other APIs from 3rd parties are different (and Apple doesn't care so long as that 3rd party code doesn't use Apple private APIs).