How to run gdb against a daemon in the background?

Assuming you have appropriate permissions, you can have gdb attach to any process. You can do it on the command line with:

gdb /path/to/binary _pid_

or from within gdb with the attach command:

attach _pid_

So, once your daemon has started, you can use either of these techniques to attach to the final PID your daemon is running as. Attaching gdb stops the process which you are tracing so you will need to issue a "continue" to restart it.

I don't know a direct way to get gdb to run arbitrary commands when the program crashes. Here is one workaround I can think of:

1. Create and register a signal handlers for SIGSEGV.
2. Tell gdb not to stop on that signal (handle SIGSEGV nostop)
3. Set a breakpoint at the first line of your signal handler.
4. Assign commands to the breakpoint from step 3

Why not just run the process interactively in a persistent screen session? Why must it be a daemon when debugging? Or just run gdb in the screen session and attach it to the running process (e.g. gdb /path/to/binary -p PID_of_binary) after it forks.

First, I'd setup your shell / environment to give you a core dump. In bash:

ulimit -c unlimited

Once you have the core dump, you can use gdb to examine the stack trace:

gdb /path/to/app /path/to/core/file