How to kill a scan operation in hbase shell without Ctrl-C

You could use scan with LIMIT to limit the scan output on shell :

scan 'table', LIMIT => 5

This will show you only 5 rows. You could also press ctlr+s to hold the scan, if you have done a full scan and want to hold on at a specific row.