How does Instagram customize the UIKeyboard returnKey to an '@' and a '#"?
That seems to be a text field whose keyboardType is set to UIKeyboardTypeTwitter, first introduced in iOS 5. It's exactly the same keyboard layout that is used by Tweet Sheets in iOS 5, with the return key being replaced by the @ and # keys respectively.
Since Instagram happens to use @ for referring to users and # for tagging posts, just like Twitter, I'm guessing that they're borrowing UIKeyboardTypeTwitter for convenient use with their own app.