Is there any open source library that does that? I know in theory many things are possible but it would be great to actually test it/see it in action. I never thought of Swift as a language with zero cost abstractions and deterministic performance. If it works I think it really gives it a leg up.
Or peruse the standard library documentation for UnsafePointer, UnsafeRawPointer, UnsafeMutablePointer, and UnsafeRawMutablePointer. These are built-in types that the compiler knows about, and code that uses them should compile down to essentially the same stuff as C pointer code would.