Message queues in ASP.Net Web API

I've used message based service bus frameworks for this in the past.

You write an application (running as a windows service), that listens for messages broadcast across a event bus.

Your frontend can publish these messages into the bus.

The most popular framework for this in .NET is NServiceBus, however it recently became commercial. You can also look into MassTransit, though this one has very poor documentation.

The workflow you would do:

• MVC App accepts upload and places it into some directory accessible by the windows service
• MVC App publishes "UploadReady" message.
• Service receives message, processes file, and updates some state for the polling controller.
• Polling controller watches for this state to change. Usually a DB record etc.

The nice bit about using a framework like this is that if your service goes down, or you redeploy it, any processing can queue and resume, so you won't have any downtime.

For long running operations you need separate Windows Service application (or Worker Role, if it is Windows Azure). IIS may kill ASP.NET processes on pool recycling and your operation will not finish.

Message queue is mostly for communication. You can use it between your web and worker parts. But it is not required there unless your data is not super critical. You can establish communication using database, cache, file system or 100 other different ways :)

You can use SignalR to notify your client about finished processing.