My Scrapy/Elasticsearch script returns "Unable to serialize" error

Okay, after consuming 9 cups of coffee and banging my head on the wall for 20 hours, I was able to fix the issue. It's so simple I'm almost ashamed to post it here, but here goes nothing;

When I first got the error yesterday, I tried decoding the referrer like this

    referring_url = response.request.headers.get('Referer')    item["referring_url"] = referring_url.decode('utf-8')

It didn't work... Until I change it to this;

    referring_url = response.request.headers.get('Referer').decode('utf-8')    item["referring_url"] = referring_url

I don't know why or how, but it works.

Huge thanks to @alecxe and @furas for pushing me in the right direction.

Make sure your installed Elasticsearch module is compatible with the Elasticsearch server.

Scrapy-elasticsearch uses v7~ of Elasticsearch module but your server might not be updated. I had the same problem, downgrading the module fixed it.

Docs:

For Elasticsearch 7.0 and later, use the major version 7 (7.x.y) of the library.

For Elasticsearch 6.0 and later, use the major version 6 (6.x.y) of the library.

For Elasticsearch 5.0 and later, use the major version 5 (5.x.y) of the library.

For Elasticsearch 2.0 and later, use the major version 2 (2.x.y) of the library, and so on.