Should be able to modify the memory settings in %JDK_HOME%\lib\visualvm\etc\visualvm.conf
%JDK_HOME%\lib\visualvm\etc\visualvm.conf
Xms and Xmx are in the default_options line.
Xms
Xmx
default_options
or I think this works too:
jvisualvm.exe -J-Xmx512m(or whatever amount you need)
jvisualvm.exe -J-Xmx512m
-J on the .exe command line for onetime settings, or the .conf file noted in the other answer for changing defaults
I started with
jvisualvm -J-Xms1024m -J-Xmx2048m
and it worked.