In this case, the direct memory addresses are literally needed.
The addresses aren't being generated internal to the program, so there are no "handles". They are referencing external data by design.
That's like saying "you shouldn't use a hard-coded volatile pointer to reference a hardware device". No, you literally need to do that sometimes; especially in embedded software.
Yes you still need to validate the keys to avoid logic errors, but you can avoid the memory errors.