I have an even wimpier computer (2GB ram, 1.4GHz Core 2 Duo). The start up time is large, and it will happily use a couple of hundred megs of RAM, but I never felt that it was too slow. Maybe I'm using an old version (5.1), but I doubt it's gotten much worse lately.
I had racket 5 running fine on an atom based eee pc. The most important thing is to turn off unused 'tools'. you can also restrict memory use by DrRacket. These days Minimal Racket does all the for you.
