Your code won't be as fast if your Swift types are copy-on-write wrappers around NSObject subclasses, instead of pure-Swift types.
I agree that you'll get the most coverage of potential developers by supporting both languages, but you're already doing the world a service by releasing open-source software. I don't think putting additional demands on maintainers (you must support CocoaPods, you must support Objective-C) is a reasonable request. If a consumer of the framework wants those features, they can submit a pull request (or just maintain the Podspec independently, as is done with ReactiveCocoa).
Performance is the least compelling reason IMO because if it's slow, you should optimize the slow bits. Otherwise it won't even register as a percentage of total time; looking at it this way is just arguing for premature optimization.
I agree that you'll get the most coverage of potential developers by supporting both languages, but you're already doing the world a service by releasing open-source software. I don't think putting additional demands on maintainers (you must support CocoaPods, you must support Objective-C) is a reasonable request. If a consumer of the framework wants those features, they can submit a pull request (or just maintain the Podspec independently, as is done with ReactiveCocoa).