how to implement overlap-checking memcpy in C

The only portable way to determine if two memory ranges overlap is:

int overlap_p(void *a, void *b, size_t n){    char *x = a, *y =  b;    for (i=0; i<n; i++) if (x+i==y || y+i==x) return 1;    return 0;}

This is because comparison of pointers with the relational operators is undefined unless they point into the same array. In reality, the comparison does work on most real-world implementations, so you could do something like:

int overlap_p(void *a, void *b, size_t n){    char *x = a, *y =  b;    return (x<=y && x+n>y) || (y<=x && y+n>x);}

I hope I got that logic right; you should check it. You can simplify it even more if you want to assume you can take differences of arbitrary pointers.

What you want to check is the position in memory of the source relatively to the destination:

If the source is ahead of the destination (ie. source < destination), then you should start from the end. If the source is after, you start from the beginning. If they are equal, you don't have to do anything (trivial case).

Here are some crude ASCII drawings to visualize the problem.

|_;_;_;_;_;_|          (source)      |_;_;_;_;_;_|    (destination)            >-----^    start from the end to shift the values to the right      |_;_;_;_;_;_|    (source)|_;_;_;_;_;_|          (destination)^-----<                 start from the beginning to shift the values to the left

Following a very accurate comment below, I should add that you can use the difference of the pointers (destination - source), but to be on the safe side cast those pointers to char * beforehand.

In your current setting, I don't think that you can check if the operation will fail. Your memcpy prototype prevents you from doing any form of checking for that, and with the rule given above for deciding how to copy, the operation will succeed (outside of any other considerations, like prior memory corruption or invalid pointers).

I don't believe that "attempting to augment memcpy by notifying the user if the copy operation will pass or fail before it begins." is a well-formed notion.

First, memcpy() doesn't succeed or fail in the normal sense. It just copies the data, which might cause a fault/exception if it reads outside the source array or writes outside the destination array, and it might also read or write outside one of those arrays without causing any fault/exception and just silently corrupting data. When I say "memcpy does this" I'm not talking just about the implementation of the C stdlib memcpy, but about any function with the same signature -- it doesn't have enough information to do otherwise.

Second, if your definition of "succeed" is "assuming the buffers are big enough but may be overlapping, copy the data from source to dst without tripping over yourself while copying" -- that is indeed what memmove() does, and it's always possible. Again, there's no "return failure" case. If the buffers don't overlap it's easy, if the source is overlapping the end of the destination then you just copy byte by byte from the beginning; if the source is overlapping the beginning of the destination then you just copy byte by byte from the end. Which is what memmove() does.

Third, when writing this kind of code, you have to be very careful about overflow cases for your pointer arithmetic (including addition, subtraction, and array indexing). In val = abs(ptr1 - ptr2), ptr1 - ptr2 could be a very large number, and it will be unsigned, so abs() won't do anything to it, and int is the wrong type to store that in. Just so you know.