python requests randomly breaks with JSONDecodeError

Looking at the documentation for this API it seems the only responses are in JSON format, so receiving HTML is strange. To increase the likelihood of receiving a JSON response, you can set the 'Accept' header to 'application/json'.

I tried querying this API many times with parameters and did not encounter a JSONDecodeError. This error is likely the result of another error on the server side. To handle it, except a json.decoder.JSONDecodeError in addition to the ConnectionError error you currently except and handle this error in the same way as the ConnectionError.

Here is an example with all that in mind:

import requests, json, time, randomdef get_submission_records(client, since, try_number=1):    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since    headers = {'Accept': 'application/json'}    try:        response = requests.get(url, headers=headers).json()    except (requests.exceptions.ConnectionError, json.decoder.JSONDecodeError):        time.sleep(2**try_number + random.random()*0.01) #exponential backoff        return get_submission_records(client, since, try_number=try_number+1)    else:        return response['submission']['records']

I've also wrapped this logic in a recursive function, rather than using while loop because I think it is semantically clearer. This function also waits before trying again using exponential backoff (waiting twice as long after each failure).

Edit: For Python 2.7, the error from trying to parse bad json is a ValueError, not a JSONDecodeError

import requests, time, randomdef get_submission_records(client, since, try_number=1):    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since    headers = {'Accept': 'application/json'}    try:        response = requests.get(url, headers=headers).json()    except (requests.exceptions.ConnectionError, ValueError):        time.sleep(2**try_number + random.random()*0.01) #exponential backoff        return get_submission_records(client, since, try_number=try_number+1)    else:        return response['submission']['records']

so just change that except line to include a ValueError instead of json.decoder.JSONDecodeError.

Try this. it might work

while True:        try:            submissions = requests.get('http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since).json()['submission']['records']            sub = json.loads(submissions.text)            print(sub)            break        except requests.exceptions.ConnectionError:            time.sleep(100)