Nutch + Solr on top level page only Nutch + Solr on top level page only hadoop hadoop

Nutch + Solr on top level page only


Try a nutch inject command to insert the "no-incomming-link" URL into the nutch DB.

I guess that if you don't see anything in your solr indexes, it is because no data for those URLs is stored in the nutch DB (since nutch will take care to sync its DB with the indexes). Not having data in the DB may be explained by the fact that the URLs are isolated, hence you can try the inject command to include those sites.

I would try to actually see the internal DB to verify the nutch behavior, since before inserting values in the indexes, nutch stores data inside its DBs.

Assigning a higher score has no effect, since lucene will give you a result as long as the data is in the index.


Solr now reads HTML files using Tika by default, so that's not a problem.

http://wiki.apache.org/solr/TikaEntityProcessor

If all you want is listed pages, is there a specific reason to use the Nutch crawler? Or could you just feed URLs to Solr and go from there?