curl truncating parts of query strings?
I looked at the documentation and curl in the case of the -d option indeed interprets & as a delimiter and therefore you can't use it there and need to excape it properly. In python you could do this using urllib.parse.quote() and the result would look like the following:
https%3A//items.initrode.com/TGWebHost/searchitems.aspx%3Fpartnerid%3D25233%26siteid%3D5296Then you would run curl as follows:
curl 'http://localhost:6800/schedule.json' \ -d project=MYPROJECT \ -d spider=SOMESPIDER \ -d spider_specific_option=1 \ -d "url=https%3A//items.initrode.com/TGWebHost/searchitems.aspx%3Fpartnerid%3D25233%26siteid%3D5296"So the problem is not in the shell quoting as I originally suggested, and it's not strictly in the way curl interprets the argument, but more in the way the query string is interpretted on the server side where & needs to be interpreted as the delimiter.