Using Django minus the web server
Read up on standalone Django scripts and you'll be on your path to victory. Basically all you're really doing is referencing the Django settings.py (which Django expects) and then using models without web views or urls.
If all you're really interested in is using Django's ORM to manage your models and database interaction, you might want to consider using SQLAlchemy instead.
You'll still have to run the Django app as a web server, but you can restrict it to serve to only localhost or something. And sure, you can use QtWebKit as the client.