Memory consumed by a thread

To my knowledge, there is no reliable way to do this at runtime. And as pointed out in the source question, the heap is a shared resource and thus the heap size of a single thread does not make sense as it will overlap with objects references from other threads.

That said, when I do want to know the 'retained' size of a single thread, and yes retained size is a different but similar metric to the one that you asked for, then I do it by taking a heap dump and then using MAT (http://www.eclipse.org/mat/).

I have known people to use Java Agents to instrument the allocation of objects and then to use a weak reference to monitor when it gets GC'd. However the performance impact of doing this is high. Very high.

You may be best off using a heuristic at runtime and unit testing to ensure that memory stays within bounds. For example, you could use JMX to monitor the heap sizes and when you see the old gen growing then you can raise an alert. Using getThreadAllocatedBytes to calculate rate of allocation could also be useful.

Good run time monitoring tools: appdynamics, newrelic, visualvm and yourkit

For offline memory analysis, mat and jclarity are very good.

A very useful tool to help one spot whether there is a leak, or at least is running different to expectations is to print a count of how many instances of each class are currently on the heap: jcmd <pid> GC.class_histogram.

Java VisualVM can be used "to monitor a local application and view real-time, high-level data on the memory heap, thread activity, and the classes loaded in the Java Virtual Machine (JVM). Monitoring an application imposes low overhead and can be used for extended periods."

See also How to monitor Java memory usage?for other possibilities.