How to identify not responding process programmatically

I'm assuming you mean a spinning wheel application hang? There are many ways to freeze. The particular cause is important. If it's a Cocoa app, you could try sending your main thread / Window an event... or scripting up Spin Control.

A random answer … I'm not a programmer but I stumbled across something of possible interest whilst working through stuff in Ask Different …

sched_prim.c (Scheduling primitives) in relatively old xnu-124.7 includes:

#define MAX_STUCK_THREADS   128/* *  do_thread_scan: scan for stuck threads.  A thread is stuck if *  it is runnable but its priority is so low that it has not *  run for several seconds.  Its priority should be higher, but *  won't be until it runs and calls update_priority.  The scanner *  finds these threads and does the updates. * *  Scanner runs in two passes.  Pass one squirrels likely *  thread ids away in an array  (takes out references for them). *  Pass two does the priority updates.  This is necessary because *  the run queue lock is required for the candidate scan, but *  cannot be held during updates [set_pri will deadlock]. * *  Array length should be enough so that restart isn't necessary, *  but restart logic is included.  Does not scan processor runqs. * */thread_t        stuck_threads[MAX_STUCK_THREADS];int             stuck_count = 0;/* *  do_runq_scan is the guts of pass 1.  It scans a runq for *  stuck threads.  A boolean is returned indicating whether *  a retry is needed. */

– is that, about stuck threads, food for thought?

Or too way off from the question about processes?

At a glance, no comparable block of code in sched_prim.c in xnu-1699.26.8 source for Mac OS X 10.7.4.