What is the most efficient way of writing a JSON file with Bash? What is the most efficient way of writing a JSON file with Bash? json json

What is the most efficient way of writing a JSON file with Bash?


Efficiently generating output

echo is a built-in, not an external command, so it's not nearly as inefficient as you think. What is inefficient is putting >> filename on the end of each echo.

This is bad:

# EVIL!echo "something" >fileecho "first line" >>fileecho "second line" >>file

This is good:

# NOT EVIL!{  echo "something" >&3  printf '%s\n' "first line" "$second line" >&3  # ... etc ...} 3>file

...opens the output file only once, eliminating the major inefficiency.

To be clear: Calling echo, say, 20 times is considerably more efficient than calling cat once, since cat is an external process, not part of the shell. What's highly inefficient about running echo "foo" >>file 20 times is opening and closing the output file 20 times; it's not echo itself.


Correctly generating JSON

Don't use cat, echo, printf, or anything else of the sort. Instead, use a tool that understands JSON -- any other approach will lead to potentially incorrect (perhaps even exploitable via injection attacks) results.

For instance:

jq \  --arg something "$some_value_here" \  --arg another "$another_value" \  '.["something"]=$something | .["another_value"]=$another' \  <template.json >output.json

...will generate a JSON file, based on template.json, with something set to the value in the shell variable "$some_value_here" and another_value set to, well, a second value. Unlike naive approaches, this will correctly handle variable values which contain literal quotes or other characters which need to be escaped to be correctly represented.


An aside on echo

All the above having been said -- echo should be avoided in favor of printf (with an appropriate, static format string). Per the POSIX sh standard:

APPLICATION USAGE

It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.

The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility as follows (assuming that IFS has its standard value or is unset):

[...]

New applications are encouraged to use printf instead of echo.

RATIONALE

The echo utility has not been made obsolescent because of its extremely widespread use in historical applications. Conforming applications that wish to do prompting without s or that could possibly be expecting to echo a -n, should use the printf utility derived from the Ninth Edition system.

As specified, echo writes its arguments in the simplest of ways. The two different historical versions of echo vary in fatally incompatible ways.

The BSD echo checks the first argument for the string -n which causes it to suppress the that would otherwise follow the final argument in the output.

The System V echo does not support any options, but allows escape sequences within its operands, as described for XSI implementations in the OPERANDS section.

The echo utility does not support Utility Syntax Guideline 10 because historical applications depend on echo to echo all of its arguments, except for the -n option in the BSD version.


You can use cat and here-document format:

cat <<'EOF' > output.json{    "key": "value",    "num": 5,    "tags": ["good", "bad"],    "money": "$0"}EOF

Note the single ticks around the here-document anchor. This prevents interpolation of the document's contents. Without it, the $0 can be substituted.

If you define efficiency as raw speed as opposed to readability, you should consider using Charles Duffy's answer instead as it's almost an order of magnitude faster for small number of lines (echo 0.01s vs cat 0.1s).
If you need to create files larger than a few hundred lines, you should consider a method other than cat/echo.


Construct the data in an environment variable, and echo it once.

var=somethingvar="$var something else"var="$var and another thing"echo "$var" > file