How can I determine if HTML5 player is running in browser?

Shell-Browser interaction

You can combine MozRepl with netcat. MozRepl is a read-evaluate-print loop for XUL-based applications like Firefox. This allows you to send Javascript expressions to Firefox and receive the evaluations (carried out by Firefox in the context of a given browser tab) from stdin. So everything what's left for you to do is to figure out which objects hang around at the time an html5 video is being played back. (Hint: Checking the paused attribute of the $('video') elements is a good start.)

Dom Traversal on the shell

Depending on your specific goal, dom traversal might be necessary. If you pre-feed the repl with the Jquery source, you can traverse the dom tree very comfortably using jQuery expressions.

Steps to get on track using MozRepl

1. Download the add-on
2. Start Mozrepl (Tools -> MozRepl -> Start)
3. Connect to MozRepl from command line: telnet localhost 4242
4. Play around with MozRepl directly. Ther's a nice video on YouTube which demonstrates some things you can do.

5. Use MozRepl with netcat. As an example

   echo 'repl.look()'|nc localhost 4242|grep -i tab

   shows you objects associated to the browser window whoose name contains the string "tab"