xcode 5 archive build fail, but normal build successfully
When you build to run, Xcode usually builds for the active architecture only. When you archive it builds all the architectures specified by the intersection of the Build Settings "Architectures and Valid Architectures".
So it's likely that Xcode is trying to build for some architecture that isn't supported by your protobuf library, e.g. Arm64.
You can see reasons for failure in Report Navigator present in Navigator window.
Menu -> View -> Navigator -> Hide/Show NavigatorExtra icon will appear/disappear next to Breakpoint icon after Hide/Show Navigator
Open Report Navigator by pressing last button present on list of buttons in Navigator window.
Here you can view reasons either By Group or By Time.