How can I use Jinja with Twisted? How can I use Jinja with Twisted? flask flask

How can I use Jinja with Twisted?


You can render web pages using Jinja the same way you would use any other Python library in Twisted. You just call into it. This will work fine with Twisted, although you may run into performance issues if Jinja does something blocking. Note that it is possible to use blocking libraries with Twisted just fine, either via deferToThread or just blocking the mainloop if it's not a performance problem. So, I am inferring that your question is really about how to use Jinja without blocking.

Jinja is a templating library, which means that it reads a template, invokes some view logic on the template, and writes some HTML output. So there are 3 things that can block:

  1. reading the template,
  2. writing the result.
  3. running the view logic (your application code),

I don't know Jinja, so I don't know exactly how the APIs for each of these things is structured and I can't tell you what to do, but my guess is that that part's easy; so, I'm going to give you a general answer about 3rd-party templating libraries and Twisted.

So I'll address each of these concerns, although not quite in order:

1. Reading the Template

Really the most reasonable thing to do here is to not care about it. Reading the template is probably really fast. These are frequently-accessed, tiny files, which your operating system is almost certainly keeping in its filesystem cache. It's unlikely that you're ever going to block on reading them unless you're doing something nuts like putting them on NFS. If you profile your application and find that this is a problem – because, let's say, you have extremely slow disks or a remote filesystem – just read the template into a cStringIO or something similar at startup time and feed it to jinja after that.

3. Writing the Response

Web pages are not all that big, and Twisted doesn't provide a blocking API to write to sockets. Instead, it offers an API which just buffers the whole result in memory until it can be written out. My suggestion is to do basically the same thing here as with reading the template: unless you have seriously huge output, it's probably fine to burn up a little bit of RAM while the response is being fed to the client.

2. Running your View Logic

This is the area where you are most likely to run into problems. Jinja probably doesn't handle the results of Deferreds. But actually, it's not actually Jinja which is going to give you problems directly: it's Storm. Storm expects to be able to block, to make database queries, when you access certain attributes. Talking to the database blocks, and this is the most signifiant source of blocking I/O in most web applications. So you need to decide how you're going to deal with that. You have a few options:

  1. Just do it in the main thread and don't worry about it. Maybe your application is for a workgroup of 10 people and your database is local. Sure, your I/O is going to block, but if it still meets its performance requirements, who cares? Not every application has to scale to the moon and back.
  2. Pre-fetch everything from storm in a deferToThread call (or similar) and make sure Jinja is only accessing objects in memory. This way you can run your renderers in your main thread, in a callback on a Deferred that was doing database I/O. If you're only accessing objects in memory, your renderer still might take a little while, but that's fine. This question prompted me to post an article to my blog about the distinction between "blocking" and "running" which had been hanging out as a draft for quite a while; you may want to go read it.
  3. Do your whole render in a thread or subprocess, and treat it as a blocking component of your program. This loses some of the benefits of using Twisted, but it's still a perfectly viable strategy to integrate a blocking Jinja/Storm component and a non-blocking pure-Twisted component, the actual chat-message relaying part.

If none of these options work for you, Twisted has included a templating library that does support Deferreds since version 11.0. You may consider using twisted.web.template as an alternative to Jinja.


Here is sample example on how to implement solution 3, with basic deferred returning function support:

from jinja2 import Templatefrom twisted.internet import threads, reactor, deferdef inThread(f):    def new_f(*args, **kwargs):        return threads.deferToThread(f, *args, **kwargs)    return new_fdef fromThread(f):    def new_f(*args, **kwargs):        return threads.blockingCallFromThread(reactor, lambda: defer.maybeDeferred(f, *args, **kwargs))    return new_fclass DeferredTemplate(Template):    def render(self, **kw):        hooked_kw = {}        for k, v in kw.iteritems():            # decorate the callable so that they are run in the main thread            if callable(v):                v = fromThread(v)            hooked_kw[k] = v        return inThread(Template.render)(self, **hooked_kw)from twisted.trial import unittestclass TestJinjaDeferred(unittest.TestCase):    @defer.inlineCallbacks    def test_basic(self):        def getHello():            d = defer.Deferred()            reactor.callLater(0.0, lambda: d.callback("Hello"))            return d        def getWorldSync():            return "world"        template = DeferredTemplate("{{ getHello() }} {{ getWorldSync() }}")        res = yield template.render(getHello=getHello, getWorldSync=getWorldSync)        self.assertEqual(u"Hello world", res)


i think Tornado template system (it's like the Jinja2 template since it's a Django-like...) can be used without the tornado itself:

We attempted to clean up the code base to reduce interdependencies between modules, so you should (theoretically) be able to use any of the modules independently in your project without using the whole package.

Tornado Templates