Algorithms for named entity recognition Algorithms for named entity recognition python python

Algorithms for named entity recognition


To start with check out http://www.nltk.org/ if you plan working with python although as far as I know the code isn't "industrial strength" but it will get you started.

Check out section 7.5 from http://nltk.googlecode.com/svn/trunk/doc/book/ch07.html but to understand the algorithms you probably will have to read through a lot of the book.

Also check this out http://nlp.stanford.edu/software/CRF-NER.shtml. It's done with java,

NER isn't an easy subject and probably nobody will tell you "this is the best algorithm", most of them have their pro/cons.

My 0.05 of a dollar.

Cheers,


It depends on whether you want:

To learn about NER: An excellent place to start is with NLTK, and the associated book.

To implement the best solution: Here you're going to need to look for the state of the art. Have a look at publications in TREC. A more specialised meeting is Biocreative (a good example of NER applied to a narrow field).

To implement the easiest solution: In this case you basically just want to do simple tagging, and pull out the words tagged as nouns. You could use a tagger from nltk, or even just look up each word in PyWordnet and tag it with the most common wordsense.


Most algorithms required some sort of training, and perform best when they're trained on content that represents what you're going to be asking it to tag.


There's a few tools and API's out there.

There's a tool built on top of DBPedia called DBPedia Spotlight (https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki). You can use their REST interface or download and install your own server. The great thing is it maps entities to their DBPedia presence, which means you can extract interesting linked data.

AlchemyAPI (www.alchemyapi.com) have an API that will do this via REST as well, and they use a freemium model.

I think most techniques rely on a bit of NLP to find entities, then use an underlying database like Wikipedia, DBPedia, Freebase, etc to do disambiguation and relevance (so for instance, trying to decide whether an article that mentions Apple is about the fruit or the company... we would choose the company if the article includes other entities that are linked to Apple the company).