Passing a argument to a callback function

This is what you'd use the meta Keyword for.

def parse(self, response):    for sel in response.xpath('//tbody/tr'):        item = HeroItem()        # Item assignment here        url = 'https://' + item['server'] + '.battle.net/' + sel.xpath('td[@class="cell-BattleTag"]//a/@href').extract()[0].strip()        yield Request(url, callback=self.parse_profile, meta={'hero_item': item})def parse_profile(self, response):    item = response.meta.get('hero_item')    item['weapon'] = response.xpath('//li[@class="slot-mainHand"]/a[@class="slot-link"]/@href').extract()[0].split('/')[4]    yield item

Also note, doing sel = Selector(response) is a waste of resources and differs from what you did earlier, so I changed it. It's automatically mapped in the response as response.selector, which also has the convenience shortcut of response.xpath.

Here's a better way to pass args to callback function:

def parse(self, response):    request = scrapy.Request('http://www.example.com/index.html',                             callback=self.parse_page2,                             cb_kwargs=dict(main_url=response.url))    request.cb_kwargs['foo'] = 'bar'  # add more arguments for the callback    yield requestdef parse_page2(self, response, main_url, foo):    yield dict(        main_url=main_url,        other_url=response.url,        foo=foo,    )

source: https://docs.scrapy.org/en/latest/topics/request-response.html#topics-request-response-ref-request-callback-arguments

@peduDev

Tried your approach but something failed due to an unexpected keyword.

scrapy_req = scrapy.Request(url=url, callback=self.parseDetailPage,cb_kwargs=dict(participant_id=nParticipantId))def parseDetailPage(self, response, participant_id ):    .. Some code here..    yield MyParseResult (        .. some code here ..        participant_id = participant_id    )Error reported, cb_kwargs=dict(participant_id=nParticipantId)TypeError: _init_() got an unexpected keyword argument 'cb_kwargs'

Any idea what caused the unexpected keyword argument other than perhaps an to old scrapy version?

Yep. I verified my own suggestion and after an upgrade it all worked as suspected.

sudo pip install --upgrade scrapy