How can I automate both E2E and unit tests with Yeoman & AngularJS?

As I mentioned in a comment to your question - PhantomJS saves a lot of hassle. That aside, I believe you can handle everything from within your Gruntfile and just continue to run grunt test to start the whole thing.

grunt-karma allows full customization of your karma options with some handy add-ons.

From the docs:

....

You can override any of the config file's settings directly:

karma: {     unit: {    configFile: 'karma.conf.js',    runnerPort: 9999,    singleRun: true,    browsers: ['PhantomJS']     }}

Sharing Configs

If you have multiple targets, it may be helpful to share common configuration settings between them. Grunt-karma supports this by using the options property:

karma: {  options: {    configFile: 'karma.conf.js',    runnerPort: 9999,    browsers: ['Chrome', 'Firefox']  },  continuous: {    singleRun: true    browsers: ['PhantomJS']  },  dev: {    reporters: 'dots'  }}

Additionally you may want to snoop around in Yeoman's generator-angular Gruntfile code to see what else may be available or at least mockable.

You can try this to run only e2e tests

grunt karma:e2e

Within the karma.conf.js file (approx line:38) find autoWatch = false; and change it to true.

Now if you run grunt karma:unit you will find that it leaves the test server running and any changes to project files immediately run the tests again.