Find which assembly instruction caused an Illegal Instruction error without debugging

Recently I experienced a crash due to a 132 exit status code (128 + 4: program interrupted by a signal + illegal instruction signal). Here's how I figured out what instruction was causing the crash.

First, I enabled core dumps:

$ ulimit -c unlimited

Interestingly, the folder from where I was running the binary contained a folder named core. I had to tell Linux to add the PID to the core dump:

$ sudo sysctl -w kernel.core_uses_pid=1

Then I run my program and got a core named core.23650. I loaded the binary and the core with gdb.

$ gdb program core.23650

Once I got into gdb, it showed up the following information:

Program terminated with signal SIGILL, Illegal instruction.#0  0x00007f58e9efd019 in ?? ()

That means my program crashed due to an illegal instruction at 0x00007f58e9efd019 address memory. Then I switched to asm layout to check the last instruction executed:

(gdb) layout asm>|0x7f58e9efd019  vpmaskmovd (%r8),%ymm15,%ymm0 |0x7f58e9efd01e  vpmaskmovd %ymm0,%ymm15,(%rdi) |0x7f58e9efd023  add    $0x4,%rdi |0x7f58e9efd027  add    $0x0,%rdi

It was instruction vpmaskmovd that caused the error. Apparently, I was trying to run a program aimed for AVX2 architecture on a system which lacks support for AVX2 instruction set.

$ cat /proc/cpuinfo | grep avx2

Lastly, I confirmed vpmaskmovd is an AVX2 only instruction.

Actually often you get an illegal instruction error not because your program contain an illegal opcode but because there is a bug in your program (e.g., a buffer overflow) that makes your program jumps in a random address with plain data or in code but not in the start of the opcode.

If you can enable core dumps on that system, just run the program, let it crash, then pull the core dump off the target machine onto your development machine and load it into a GDB built to debug the target architecture - that should tell you exactly where the crash occurred. Just use GDB's core command to load the core file into the debugger.

• To enable core dumps on the target:

  ulimit -c unlimited

• pseudo-files that control how the core file will be named (cat these to see the current configuration, write to them to change the configuration):

  /proc/sys/kernel/core_pattern/proc/sys/kernel/core_uses_pid

On my system, once core dumps are enabled, a crashing program will write a file simply named "core" in the working directory. That's probably good enough for your purposes, but changing how the core dump file is named lets you keep a history of core dumps if that's necessary (maybe for a more intermittent problem).