Heroku web dyno running exactly to memory limit?
Slightly ridiculous but traced the issue to django-avatar the app is using for user profile avatars. Almost 50% of response time for any page in the app was spent in {% block header %} of the template, which didn't make sense, and turned out to be the { avatar } tag.
AVATAR_STORAGE_DIR wasn't properly configured for S3 in settings.py