Do Dalvik VM Processes Release System RAM?

Yes. The basic idea is that, if there's a 4K page with nothing on it, the page will be returned to the system.

The function that does this in the VM is called trimHeaps(), in dalvik/vm/alloc/HeapSource.cpp. You can see it using mspace_trim(), which uses OS calls to unmap chunks that are no longer needed (see malloc_trim() comments around line 1203 in malloc.c). It then traverses the heap with mspace_inspect_all(), which calls into releasePagesInRange() for every region. The callback tests to see if it was passed a region with no allocations in it, and if so, truncates the boundaries to 4K alignment. If the result isn't empty, we know the region spans one or more physical 4K pages, which can be returned to the system with madvise(MADV_DONTNEED).

trimHeaps() is called from a few places, most notably gcDaemonThread(), which will initiate the trim five seconds after a concurrent GC. The timer gets reset if the concurrent GC happens before five seconds elapse, the idea being that if we're GCing then the VM is busily allocating and this sort of idle-time trimming will be counter-productive.

Because the Dalvik GC doesn't do compaction, this isn't as effective as it could be. Fragmentation tends to build up over time, so the situation may get worse the longer a process lives. The app framework can "recycle" long-lived services to alleviate this.