ext4 is irrelevant to what happens when a file is backed by swap; even with swapfiles, the mm subsystem more or less goes behind the back of the filesystem to access the disk corresponding to the swapfile.
The overhead of making (size-of-read / 4kb) requests (potentially stalling the reading process for every page) is relevant even on an ssd; there are costs to random access beyond moving a disk head and waiting for a platter to spin into position, and those costs are still relevant with solid-state storage.
You wrote your comment like it was a rebuttal of the person above you, but the text supports what they said: A filesystem is faster than swap for this.
