Validate (X)HTML in Python

PyTidyLib is a nice python binding for HTML Tidy. Their example:

from tidylib import tidy_documentdocument, errors = tidy_document('''<p>fõo <img src="bar.jpg">''',    options={'numeric-entities':1})print documentprint errors

Moreover it's compatible with both legacy HTML Tidy and the new tidy-html5.

I think the most elegant way it to invoke the W3C Validation Service at

http://validator.w3.org/

programmatically. Few people know that you do not have to screen-scrape the results in order to get the results, because the service returns non-standard HTTP header paramaters

X-W3C-Validator-Recursion: 1X-W3C-Validator-Status: Invalid (or Valid)X-W3C-Validator-Errors: 6X-W3C-Validator-Warnings: 0

for indicating the validity and the number of errors and warnings.

For instance, the command line

curl -I "http://validator.w3.org/check?uri=http%3A%2F%2Fwww.stalsoft.com"

returns

HTTP/1.1 200 OKDate: Wed, 09 May 2012 15:23:58 GMTServer: Apache/2.2.9 (Debian) mod_python/3.3.1 Python/2.5.2Content-Language: enX-W3C-Validator-Recursion: 1X-W3C-Validator-Status: InvalidX-W3C-Validator-Errors: 6X-W3C-Validator-Warnings: 0Content-Type: text/html; charset=UTF-8Vary: Accept-EncodingConnection: close

Thus, you can elegantly invoke the W3C Validation Service and extract the results from the HTTP header:

# Programmatic XHTML Validations in Python# Martin Hepp and Alex Stolz# mhepp@computer.org / alex.stolz@ebusiness-unibw.orgimport urllibimport urllib2URL = "http://validator.w3.org/check?uri=%s"SITE_URL = "http://www.heppnetz.de"# pattern for HEAD request taken from # http://stackoverflow.com/questions/4421170/python-head-request-with-urllib2request = urllib2.Request(URL % urllib.quote(SITE_URL))request.get_method = lambda : 'HEAD'response = urllib2.urlopen(request)valid = response.info().getheader('X-W3C-Validator-Status')if valid == "Valid":    valid = Trueelse:    valid = Falseerrors = int(response.info().getheader('X-W3C-Validator-Errors'))warnings = int(response.info().getheader('X-W3C-Validator-Warnings'))print "Valid markup: %s (Errors: %i, Warnings: %i) " % (valid, errors, warnings)

XHTML is easy, use lxml.

from lxml import etreefrom StringIO import StringIOetree.parse(StringIO(html), etree.HTMLParser(recover=False))

HTML is harder, since there's traditionally not been as much interest in validation among the HTML crowd (run StackOverflow itself through a validator, yikes). The easiest solution would be to execute external applications such as nsgmls or OpenJade, and then parse their output.