Why is the subprocess.Popen argument length limit smaller than what the OS reports?

The maximum size for a single string argument is limited to 131072. It has nothing to do with python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/nullbash: /bin/echo: Argument list too long

It is actually the MAX_ARG_STRLEN that decides the max size for a single string:

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'". (pointed out by Xan Lopez and Ralf Wildenhues)

See this discussion of ARG_MAX, under "Number of arguments and maximum length of one argument", and this question on unix.stackexchange.

You can see it in binfmts.h:

/* * These are the maximum length and maximum number of strings passed to the * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to * prevent the kernel from being unduly impacted by misaddressed pointers. * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer. */#define MAX_ARG_STRLEN (PAGE_SIZE * 32)#define MAX_ARG_STRINGS 0x7FFFFFFF~$ echo $(( $(getconf PAGE_SIZE)*32 )) 131072

You can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))

But a single string arg cannot be longer than 131071 bytes.

This other question is similar to yours, but for Windows. As in that scenario, you can bypass any shell limitations by avoiding the shell=True option.

Otherwise, you can provide a list of files to subprocess.Popen() as done in that scenario, and as suggested by @Aaron Digulla.