Windows Azure Worker Role not getting past first line of code

Given the OnStart() trace is called, but not the Initialize(), my guess is that one of the assemblies referenced by the code in Initialize() is not being copied to the deployment. Remember that .Net JIT-compiles one method at a time, and because of that behavior it would make sense that the OnStart trace message shows up (as there's little other than the Windows Azure and standard .Net framework assemblies referenced up to that point). However, when the CLR goes to JIT the Initialize method, it then tries to load several third-party assemblies (AutoMapper and Windsor) that may not be packaged correctly, but could be GACced or otherwise available locally when the emulator runs.

A couple of things to try:

1. Manually "Package" your deployment from Visual Studio and take a careful look at the build output. Many times, VS will catch your missing assemblies and tell you (unfortunately as a warning, not an error) that you're missing something.
2. If you don't see anything in the output that looks obvious, take a look at the cspkg file itself (remember it's just a ZIP file with more ZIP files in it) and make sure any referenced assemblies your application/role needs are in there. Alternately, connect to the VM and check the approot for those assemblies.
3. You may be able to find an entry in the VM's event log that shows that your application failed to load an assembly.

More often than not, the root cause of problems like this is missing dependencies. There are already good suggestions on this front in previous answers.

The trace logs are transferred to Azure storage once per minute as per your configuration. If your worker process crashes, you might lose some of the last trace messages. To work around this, try adding a Thread.Sleep(TimeSpan.FromMinutes(2)) in your exception handlers to ensure that the exception log gets flushed to storage.

Finally, if everything else fails, I suggest you try debugging your role with WinDbg. Enable Remote Desktop for your role. Log in to the role and switch off IE safe browsing so that you can install stuff. Then download and install Debugging Tools for Windows from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8279 . This package contains the entire Windows SDK, but you can select to install the Debugging Tools for Windows only.

Then run WinDbg and attach to WaWorkerHost.exe. In WinDbg, do

.loadby sos clr   // load the SOS extension that allows you to do managed debuggingsxe clr           // break on CLR exceptionsg                 // continue

WinDbg should now break execution when there is an CLR exception. When it breaks, execute

!PrintException

to see exception details. You may want to add another Thread.Sleep call in your role startup to give you time to attach the debugger before the process exits.

Thanks to an answer on the MSDN forum, I have been able to troubleshoot and resolve my problem.

The reason why my jobs were not being executed was due to the line below:

_container.Install(FromAssembly.InDirectory(                    new AssemblyFilter(Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot))));

The role root on staging is E:. Path.Combine() has an obscure implementation which you can read more about in this SO answer. What this meant was that Castle was searching for assemblies in E:approot rather than E:\approot as I expected. I am now constructing the approot path with the method below:

    private string GetAppRoot()    {        string root = Environment.GetEnvironmentVariable(Constants.RoleRoot);        if (root.EndsWith(Path.VolumeSeparatorChar.ToString()))            root += Path.DirectorySeparatorChar;        return Path.Combine(root, Constants.AppRoot);    }

That has resolved my main issue and I am now seeing the jobs execute as expected.

I was only able to troubleshoot this issue by running the worker role in an elevated execution context so that my trace data could be written to a file. I still don't know why, and would like to hear of any ideas why, the trace statements were not being transferred to storage correctly.