Startup has a very specific definition. It refers to a company that aims to solve a problem and/or create something new in a particular industry. This means that it is a subset of the small business category; the corner deli that just opened up last week is a small business, but not a startup. On the other hand, a new tech firm that is trying to figure out an innovative alternative to email is both a small business and a startup.
I'd say the biggest thing that distinguishes startups from other small businesses is the risk factor. When your goal is to solve challenging problems in your field, then by definition your chances of success are low. After all, the problems are categorized as "challenging for a reason: nobody has been able to find a simple and elegant solution to them yet. Whereas a typical small business operates on relatively safe territory - they are most likely providing identical products and services as other players in their industry. So their risk is a lot lower.
I've generally worked in "startup" environments, and I actually do mentally translate that word into "small business." I consider it to have more realistic connotations. YMMV of course.
I'd say the biggest thing that distinguishes startups from other small businesses is the risk factor. When your goal is to solve challenging problems in your field, then by definition your chances of success are low. After all, the problems are categorized as "challenging for a reason: nobody has been able to find a simple and elegant solution to them yet. Whereas a typical small business operates on relatively safe territory - they are most likely providing identical products and services as other players in their industry. So their risk is a lot lower.