Read a file in R with mixed character encodings
There do seem to be R library functions for guessing character encodings, like stringi::stri_enc_detect
, but when possible, it's probably better to use the simpler determinstic method of trying a fixed set of encodings in order. It looks like the best way to do this is to take advantage of the fact that when iconv
fails to convert a string, it returns NA
.
linewise.decode = function(path) sapply(readLines(path), USE.NAMES = F, function(line) { if (validUTF8(line)) return(line) l2 = iconv(line, "Windows-1252", "UTF-8") if (!is.na(l2)) return(l2) l2 = iconv(line, "Shift-JIS", "UTF-8") if (!is.na(l2)) return(l2) stop("Encoding not detected") })
If you create a test file with
$ python3 -c 'with open("inptest", "wb") as o: o.write(b"This line is ASCII\n" + "This line is UTF-8: I like π\n".encode("UTF-8") + "This line is Windows-1252: Müller\n".encode("Windows-1252") + "This line is Shift-JIS: ハローワールド\n".encode("Shift-JIS"))'
then linewise.decode("inptest")
indeed returns
[1] "This line is ASCII" [2] "This line is UTF-8: I like π" [3] "This line is Windows-1252: Müller" [4] "This line is Shift-JIS: ハローワールド"
To use linewise.decode
with XML::readHTMLTable
, just say something like XML::readHTMLTable(linewise.decode("http://example.com"))
.