Executable prevents itself from getting debugged

There is an executable c file on my linux workstation for which I would like to know what exactly it does, since I don't trust it. I tried debugging the executable with the GNU Project Debugger (gdb) but when I use the "run" command in it, the execution of the program stops with the following output from gdb: [Inferior 1 (process 4111) exited with code 01] I think the program employs some kind of anti-debugging mechanism but I am not sure which one and how I could circumvent it. I would like to know how I can check what is done to prevent me from debugging the program and how to disable it.
Submitted by Alexander Götz on Sat, 12/03/2016 - 12:24
1 answer

Prevent ptrace anti-debugging

The first step to find out what the executables does to prevent debugging is to use the ltrace command on it, which will list the dynamic library calls which are called by the executed process and the signals which are received by that process.

If "ptrace" is called, then it means that the program likely uses it to check if any other process (such as a debugger) is attached to it and will terminate if it detects one. There's several ways to cirumvent this behavior, one of them being to overwrite the ptrace call with 0x90 which is the NOP instruction in assembly.

To do that, first use the command readelf on the executable to find out its entry point address. Open the program in gdb and set a breakpoint at the entry point address, then run it and use gdb's single step functionality to step through the program's assembly code until you find the ptrace call.

Note the address of the ptrace call and the next instruction. The difference between the two is the amount of bytes we need to replace with 0x90. To check which bytes exactly need to be replaced examine the ptrace access with gdb's x command, for example x/10xb would let you look at the next 10 bytes in hexadecimal format.

Note down the bytes that need to be replaced, then open the executable with a hex editor and find them. Replace the bytes with 0x90. It should now be possible to reverse engineer the program with gdb.

Submitted by Alexander Götz on Sat, 12/03/2016 - 12:45