How to create paid plugins / themes and sell them securely

I'm not familiar with any books on the subject, but I'll tell you what I've seen as one of a founders of a component / plug-in marketplace that has many such plug-ins.

There are a few approaches -

1. Some plugins do not require an API key at all. Either the plug-in is only available after purchase, or has some limitations on the free downloadable version that encourages people to pay for the commercial version. This approach relies more on people's integrity and low motivation to try and hack the free version into the commercial one, especially if they are not technical users (as many CMS users are).
2. Set up a check against your server that happens periodically. You do not need a full blown API for this, just set up an endpoint on your server that the plug-in can send the API key and according to the response allows the use of the plug-in. You need to plan it so that this check doesn't happen every time the plug-in is run, especially if it a plug-in that runs on the public site and not only in the administration panel - it will seriously degrade the performance of the site using it and create unnecessary load on your server. Use some kind of time based checked - either absolutely or from the time of the last check.
3. In addition to or instead of doing an API check, some people will obfuscate their code to make it harder to modify and bypass the check. This often requires that the server has a module installed that can parse the obfuscated files - this requirement often makes it less viable for most people. You can see some examples of obfuscators in another question.

Personally, I lean more toward the first option, as someone determined enough will break whatever protection you put (people break much more complicated solutions in no time). This is one of the problems of delivering source-code instead of binaries (and those are broken just as easily by more experienced hackers). Let those who are willing pay, and the others just let them do what they want as you won't be able to create something truly secure anyway.