Crash with alloca function but ok with malloc

Sounds like a stack overflow, alloca will allocate from the stack and so if you attempt to allocate too much that will result in a stack overflow while malloc will allocate from the heap which is much larger. There is no way to know when you overflow the stack with alloca.

Also, note that since alloca allocates on the stack returning that memory from the function won't work since that memory will no longer be available once you exit the function. So if you need to use the memory outside the function you will need to use malloc.

here you allocate memory in function,

static int overwrite_selector(struct srm_target *srm, MARFileRemoverProgressBlock progressBlock){    srm->buffer = (unsigned char *)alloca(srm->buffer_size);    if(overwrite_byte(srm, 1, 0xF6, progressBlock) < 0) return -1;    return 0;}

and since you use alloca, the memory allocated will be freed when the function returns. there for when you use it from other function it results in program to crash.
Here you called other function within the function allocating memory. so the reason seems to be stackoverflow. and there is no way to know this condition from alloca. because of this undefined behaviour of alloca on stackoverflow, it is told that its not good to use alloca in such situations.

Since alloca attempts to allocate bytes on the stack, it's important to know what the stack size is. You can check the limits on the stack size with code like this

#include <sys/resource.h>void checkStackSize( void ){    struct rlimit limits;    if ( getrlimit( RLIMIT_STACK, &limits ) == 0 )        printf( "soft_limit=%lld hard_limit=%lld\n", limits.rlim_cur, limits.rlim_max );}

When I tested on osx, the soft limit was 8MB and the hard limit was 64MB. On an iOS device, both limits were 1MB.

Note: the following comes from the man page for alloca

alloca() is machine and compiler dependent; its use is discouraged.

alloca() is slightly unsafe because it cannot ensure that the pointer returned points to a valid and usable block of memory. The allocation made may exceed the bounds of the stack, or even go further into other objects in memory, and alloca() cannot determine such an error. Avoid alloca() with large unbounded allocations.