TypeError when putting scraped data from scrapy into elasticsearch TypeError when putting scraped data from scrapy into elasticsearch elasticsearch elasticsearch

TypeError when putting scraped data from scrapy into elasticsearch


As you can see in the error message: Error processing {'link': [u'http://www.meetup.com/Search-Meetup-Karlsruhe/events/221907250/'], 'title': [u'Alles rund um Elasticsearch']} your item's link and title fields are lists (the square brackets around the values indicate this).

This is because of your extraction in Scrapy. You did not post it with your question but you should use response.xpath().extract()[0] to get the first result of the list. Naturally in this case you should prepare to encounter empty result sets to avoid index-errors.

Update

For the situation where you do not extract anything you could prepare with the following:

linkSelection = response.xpath().extract()item['link'] = linkSelection[0] if linkSelection else ""

Or something alike depending on your data and fields. Perhaps None could be valid too if the list is empty.

The basic idea is to split up XPath extraction and list-item selection. And you should select an item from the list if it contains the required elements.