It's notable that the controller and memory card slot are on the same bus-- in fact, the system isn't talking to the memory card directly! It has to send requests and data through the controller, back to the system, and then to the card port.
This is because the original specification would have had a memory card slot in the controller but it was too expensive. They also had considered using floppies for storage, but they were too unreliable (and my experiences with 1.44MB 3.5s definitely agrees that consumer-grade cheap disks sucked hard)
Bumping the timer spreed is not the creative part, creative part was zeroing on the issue. For software guys used to reliable hardware and no prior knowledge how to debug hardware bugs this is hell.