You would probably have easier time using a makefile-based project with custom compilers and linkers.
http://owensd.io/2015/01/14/compiling-swift-without-xcode.html - on compiling swift without xcode.