How to find the list of processes using a particular kernel module?
This was asked before, but there was no great answer, probably because there is no good way to accomplish to find what is using what.
The best suggestion is to dmesg and look for any indication of what loaded the module.
You might also try using the --force parameter to rmmod in case your kernel was built with support for it.