how to limit the amount of data being sent by the client through websocket?
That ability does not (currently) exist in that library.
Poking around their source code, it appears that the place to start would be processPacket() method in https://github.com/websockets/ws/blob/master/lib/Receiver.js .
Once you have the packet header available, you can see the size of the message being sent. If it's above a certain threshold, there should be a way to close the connection before all of the bytes are even hitting your network.
Of course, the nice thing to do would be to fork their repository, issue a feature request, add in a configuration option that defaults to not taking any action if it's not set (don't break backwards compatibility), and submit a pull request.
If they like it, they'll merge. If not, you'll still be able to merge their future versions into your own repo and stay up to date without having to re-do your work each time they submit a new release.