How can I use Mac OS X (and UNIX) command line tools like grep with UTF-16 files?
Mac OS X comes with an old version of BSD grep out of the box, which is limited and very slow. Even so, both BSD and GNU grep do not handle UTF-16 files. Other grep tools like ag, rg, and ugrep are designed to support Unicode and UTF files. Of these three, ugrep is closer to GNU grep, so there is no learning curve to use it as a compatible replacement for grep:
ugrep "PATTERN" FILE ...If your files contain UTF byte order marks then there is no need to convert them to search with ugrep, ag, or rg.
To search files without byte order marks requires a flag, e.g. --encoding with ugrep:
ugrep --encoding=UTF-16 "PATTERN" FILE ...