Debugging Scrapy Project in Visual Studio Code

In order to execute the typical scrapy runspider <PYTHON_FILE> command you must to set the following config into your launch.json:

{    "version": "0.1.0",    "configurations": [        {            "name": "Python: Launch Scrapy Spider",            "type": "python",            "request": "launch",            "module": "scrapy",            "args": [                "runspider",                "${file}"            ],            "console": "integratedTerminal"        }    ]}

Set the breakpoints wherever you want and then debug.

1. Inside your scrapy project folder create a runner.py module with the following:

   import osfrom scrapy.cmdline import executeos.chdir(os.path.dirname(os.path.realpath(__file__)))try:    execute(        [            'scrapy',            'crawl',            'SPIDER NAME',            '-o',            'out.json',        ]    )except SystemExit:    pass

2. Place a breakpoint in the line you wish to debug

3. Run runner.py with vscode debugger

Configure your json file like that:

"version": "0.2.0","configurations": [    {        "name": "Crawl with scrapy",        "type": "python",        "request": "launch",        "module": "scrapy",        "cwd": "${fileDirname}",        "args": [            "crawl",            "<SPIDER NAME>"        ],        "console": "internalConsole"    }]

Click on the tab in VSCode corresponding to your spider, then launch a debug session corresponding to the json file.