How do I test to see that a custom uploader works?

You could try some variation of

1. Make a screenshot.
2. Click your button
3. Make another screenshot
4. Compare them

but I'm not sure if selenium webdriver screenshots are OS level and show the picker or browser level and just show the page.

Otherwise something like sikuli that enables you to script with screenshots, could be an options. Sikuli uses Python, though I saw a gem called sikuli, but I don't know in what state that is in.

Are you sure you want to test the browser <-> OS interactions like this. Or do you just want to make sure that firing the click event on a hidden input-element of the file-type will spawn a filepicker. It is in no way defined by any HTML spec. Some browser OS combination may not even launch a picker.

If you come to the conclusion that testing the browser OS interaction is beyond the scope of your app and falls in the "testing the system" —similar to testing String—, then testing that the button.click fires a click on the input with Jasmine would be sufficient.

When the dialog opens, is there a new div or something that shows up? you could just assert that that css is there via a page.should have_css('div.fake-popup')