What is the most efficient way of writing a JSON file with Bash?
Efficiently generating output
echo
is a built-in, not an external command, so it's not nearly as inefficient as you think. What is inefficient is putting >> filename
on the end of each echo
.
This is bad:
# EVIL!echo "something" >fileecho "first line" >>fileecho "second line" >>file
This is good:
# NOT EVIL!{ echo "something" >&3 printf '%s\n' "first line" "$second line" >&3 # ... etc ...} 3>file
...opens the output file only once, eliminating the major inefficiency.
To be clear: Calling echo
, say, 20 times is considerably more efficient than calling cat
once, since cat
is an external process, not part of the shell. What's highly inefficient about running echo "foo" >>file
20 times is opening and closing the output file 20 times; it's not echo
itself.
Correctly generating JSON
Don't use cat
, echo
, printf
, or anything else of the sort. Instead, use a tool that understands JSON -- any other approach will lead to potentially incorrect (perhaps even exploitable via injection attacks) results.
For instance:
jq \ --arg something "$some_value_here" \ --arg another "$another_value" \ '.["something"]=$something | .["another_value"]=$another' \ <template.json >output.json
...will generate a JSON file, based on template.json
, with something
set to the value in the shell variable "$some_value_here"
and another_value
set to, well, a second value. Unlike naive approaches, this will correctly handle variable values which contain literal quotes or other characters which need to be escaped to be correctly represented.
An aside on echo
All the above having been said -- echo
should be avoided in favor of printf
(with an appropriate, static format string). Per the POSIX sh standard:
APPLICATION USAGE
It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.
The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility as follows (assuming that IFS has its standard value or is unset):
[...]
New applications are encouraged to use printf instead of echo.
RATIONALE
The echo utility has not been made obsolescent because of its extremely widespread use in historical applications. Conforming applications that wish to do prompting without s or that could possibly be expecting to echo a -n, should use the printf utility derived from the Ninth Edition system.
As specified, echo writes its arguments in the simplest of ways. The two different historical versions of echo vary in fatally incompatible ways.
The BSD echo checks the first argument for the string -n which causes it to suppress the that would otherwise follow the final argument in the output.
The System V echo does not support any options, but allows escape sequences within its operands, as described for XSI implementations in the OPERANDS section.
The echo utility does not support Utility Syntax Guideline 10 because historical applications depend on echo to echo all of its arguments, except for the -n option in the BSD version.
You can use cat
and here-document format:
cat <<'EOF' > output.json{ "key": "value", "num": 5, "tags": ["good", "bad"], "money": "$0"}EOF
Note the single ticks around the here-document anchor. This prevents interpolation of the document's contents. Without it, the $0
can be substituted.
If you define efficiency as raw speed as opposed to readability, you should consider using Charles Duffy's answer instead as it's almost an order of magnitude faster for small number of lines (echo
0.01s vs cat
0.1s).
If you need to create files larger than a few hundred lines, you should consider a method other than cat
/echo
.