With scrolling couldn't you just shift the existing framebuffer up/down and only draw the newly scrolled-to parts?

The GPU buffering model doesn't usually let you read the previous frame like that these days, so you would need a scratch surface. That increases your bandwidth and memory requirements, so shifting doesn't end up being much faster if it's faster at all.

I suspect there are enough edge cases (semi-transparent things jumps to mind) that it's unlikely to be lossless, anyway.