cURL post size limitations cURL post size limitations curl curl

cURL post size limitations


According to Infile Statement for Microsoft Windows, the limit to LRECL should be 1GB (well over what you have). However, I have in the past had issues where 1MB (1024^3) seemed to be the limit. On the other hand, that's smaller than what you see. As such I'm not sure I can say I know what the issue is, directly.

One possibility is that you are reading in UTF8 characters (or some other unicode variant). In that case, the LRECL limit is probably bytes, not characters. So I could see a reduction - depending on the unicode variant, up to 4x or so - from the maximum. You could try MAX in the lrecl= field, and see what it did; hopefully not 32767 (as the LRECL system option still claims to). Still doesn't explain the discrepancy, but a possibility to consider. If raising from 10MB to MAX or 30MB or 100MB or any higher number raises the total length, then that could explain it. I would also raise the curl maximum filesize to the same amount (or at least try raising it to something bigger) to see if some issue exists with how SAS deals with the pipe.

One thing to try might be to read it in with RECFM=F. This would be in the infile statement when you read from the filename, I think; you could parse it in bunches of 1MB (or smaller) and deal with it that way. You could also use a termstr or some other way to parse it with a delimiter, if you have such ability in your data (a common one is to use termstr='>' for html, for json I think ; is common). You don't post what your json parsing is doing, so I can't comment, but I will say that from my html parsing experience (and some json as well), it's easier to parse it when it's not gigantic lines at first.


Per Joe's suggestion I used an x command to run cURL and then saved the output directly to a text file. This circumvents any restrictions in SAS. I can then parse the json data using proc groovy. Final cURL code is above in the first post of the thread.


Just so you know, SAS has a max linesize of 32767, so unless the response contains multiple lines you will need to find a way to deal with that anyway. Check the length of the truncated value. Should tell you if this is the issue.

EDIT : Only true for SAS 9.2 and prior apparently. Thanks to Joe for the clarification.