Remove non-English and accented characters from a flat file using Unix shell script

Using GNU sed, you can do the following:

sed 's/[^\o51-\o57\o64-\o89\o96-\o105\o112-\o121\o128-\o137\o144-\o145\o147\o150\o291-\o293]/ /g' inputfile

Note that those are letter "O" rather than digit zero after the backslashes.

This isn't a terribly specific answer, but it should give you a few keywords to search for.

First, the easy bit. It's straightforward to have sed match regexp characters. For example:

% echo 'one tw? f*ur' | sed 's/\*/ /' one tw? f ur% echo 'one tw? f*ur' | sed 's/[*?]/ /'one tw  f*ur%

Handling the non-ASCII characters is messier.

Some seds can handle non-ASCII characters, usually unicode files. Some seds can't. Unfortunately, it may not be obvious from your sed's manpage which it is. Life is hard.

One thing you'll have to find out is what encoding the input file is in. A unicode file will be encoded in one or other of UTF-8 or UTF-16 (or possibly one of a couple of less common ones). This isn't the place for an expansion of unicode and encodings, but those are the keywords to scan the manpages for....

Even if you can't find a sed which can handle unicode, then you might be able to use perl, python, or some other scripting language to do the processing -- these generally have regexp engines which can do unicode. The perl -n option creates an implicit loop which might make the transformation you want a one-liner.

If your input document is in a different (non-unicode) encoding, such as one of the ISO-8859 ones, then I would guess that the best thing to do would be to convert it to UTF-8 using something like iconv, and proceed from there.

If your accented characters are single-byte you can use tr with character sets to accomplish this. If you can identify a range of characters to match, that's probably easiest:

tr '\192-\255' ' ' < infile > outfile

If you're dealing with larger-than-8-bit characters, awk and sed can probably handle it, but you need to make sure your inputs are properly quoted. Try using the decimal or hexadecimal representations instead of the characters themselves.