Programmatic access is a big thing, and is what makes EC2, GCE, Rackspace and friends, different from dedicated hosting. (In the case of AWS / GCE, global availability is something else).
Using the Cloud isn't only about instant scaling and per hour billing. Those are useful features, but mostly for specific use cases.
--
The Cloud it's about the ability to have software that controls the hardware.
It's about being able to orchestrate automated failovers that provision new servers.
It's about being able to have services that grow and shrink depending on usage.
It's about being able to replace your app servers with data crunching servers at night and relaunch new app servers at day - all transparently.
Of course, those are just a few examples, but the general idea is that programmatic access (APIs) is the big thing. The rest is secondary.
---
Is is, however, true that this is not useful to everyone.
On the topic of succeeding using the Cloud, it's indeed difficult. That's why companies are building cloud management tools to help users do this.
You can orchestrate automated failovers that provisions new servers even if you have dedicated servers to handle your base load. And the price difference with EC2 is so huge that you need really massive traffic spikes for it not to be cheaper to have a bunch of extra dedicated servers on standby.
Nothing stops your from spinning up data crunching EC2 instances at night either, if you want to, and if it really is more cost-effective for you than having VM on your dedicated hardware.
There's plenty of API's available if you want to run your own "private cloud" on those dedicated servers too. I never deploy outside VM's any more, even though I also mainly use dedicated servers.
EC2 is cost effective if you truly have really short term (< 4-6 hours per day) batch processing needs. It continues to shock me how many people take the pain and cost of dealing with EC2 for more typical web app usage.
Switching load or transferring data between a remote DC (your dedicated servers) and AWS is not that easy when you start having large-scale infrastructure though.
This is a great idea on paper - but running your baseline infrastructure at your dedicated provider and the rest on EC2 is definitely a challenging task.
Not that it's impossible, but that's probably going to be extra work at the app level.
---
Running OpenStack / CloudStack, or similar software on dedicated servers is indeed a relevant solution too, but this is an extra maintenance cost to bear in mind.
Using the Cloud isn't only about instant scaling and per hour billing. Those are useful features, but mostly for specific use cases.
--
The Cloud it's about the ability to have software that controls the hardware.
It's about being able to orchestrate automated failovers that provision new servers.
It's about being able to have services that grow and shrink depending on usage.
It's about being able to replace your app servers with data crunching servers at night and relaunch new app servers at day - all transparently.
Of course, those are just a few examples, but the general idea is that programmatic access (APIs) is the big thing. The rest is secondary.
---
Is is, however, true that this is not useful to everyone.
On the topic of succeeding using the Cloud, it's indeed difficult. That's why companies are building cloud management tools to help users do this.
Disclaimer: I work for one of these companies.