Performance profiling on Linux
I recommend taking stackshots, for which pstack is useful. Here's some more information:
If you want to spend money, Zoom looks like a pretty good tool.
For performance, you can try Callgrind, a Valgrind tool. Here is a nice article showing it in action.
The canonical example of a full system profiling tool (for Solaris, OS X, FreeBSD) is DTrace. But it is not yet fully available on Linux (you can try here but the site is down for me at the moment, and I haven't tried it myself). There are many tools, in various states of usefulness, for doing full system profiling and kernel profiling on Linux.
You might consider investigating:
- oprofile
- SystemTap
- bootchart
- strace (e.g. this SO answer