How much caffeine do you consume regularly? Sounds to me like your body potentially has too much already, and is reacting by making you drowsy. Note that caffeine is also in non-coffee/tea related beverages like sodas (e.g. Coke/Pepsi/...), or sometimes certain pain killers (e.g. migraine pain killers frequently contain something like 50mg of caffeine).