Adding Swift to project increase size substantiously. How and when can it be avoided

Swift is still changing. So at the moment the runtime has to be included with every app. Maybe Apple will include the Swift runtime in iOS once the development of Swift slows down.

Swift source code had to be adjusted after almost every release of Xcode since the 6.0 betas. The runtime has changed at the same time the compiler has. iOS can't use a standard swift library, but has to use the one the app was compiled and linked with.

See this explanation by Apple.

you can trust that your app will work well into the future. In fact, you can target back to OS X Mavericks or iOS 7 with that same app. This is possible because Xcode embeds a small Swift runtime library within your app’s bundle. Because the library is embedded, your app uses a consistent version of Swift that runs on past, present, and future OS releases

While your app’s runtime compatibility is ensured, the Swift language itself will continue to evolve, and the binary interface will also change.

As Swift changes, those frameworks will be incompatible with the rest of your app. When the binary interface stabilizes in a year or two, the Swift runtime will become part of the host OS and this limitation will no longer exist

Not using Swift is the only way to keep your app size down.

Since Swift 3.0 won't deliver a stable ABI, this will remain the same for the time being. So in a year or two probably translates to Swift 5.0 in 2018.