So, something that easily fits into RAM, One might even keep a copy of the whole dataset in GPU memory to avoid copying to and from even if matrix operations are done on smaller batches.

E.g. ImageNet is 50 gb of compressed data, and there are many much larger datasets in practical use.