Last note: another thing you can do when you have a prior is factor it into your estimator and take the maximum likelihood of the posterior as a point estimate. This is called Maximum a Posteriori (MAP) and called by some people "Bayesian", but I don't think Bayesians agree with that.
The general perspective is that, of course, you'd like to get the fully marginalized, exact posterior probability distribution. However, that's not computationally viable for most problems, so you have to resort to approximations, like MCMC, Variational methods and MAP. I would say that they're all definitely "Bayesian" methods, as long as you're aware of what you're doing, and that you check the quality of your approximation.
Thanks for your clarification! I'm not a statistician, I just happen to be surrounded by them :)
My impression is that among the Bayesians I know there is a general negative bias towards MAP, and Variational methods are vastly preferred. However I agree with you that, all being approximations, none of them is intrinsically better than the others.
In particular I don't understand all the hype around Variational Bayes, to me it seems like a "fat MAP", a MAP estimate with a Gaussian around it.
Right - MAP may (debatably) be a lousy Bayesian approximation, but it's still Bayesian :)
David MacKay's wonderful book made the observation that MAP is a variational method that uses a delta function. "From this perspective, any approximating distribution Q(x; θ) [like the Gaussian], no matter how crummy it is, has to be an improvement on the spike produced by the standard method! [MAP]"
I've only recently come across the technique myself. I think the hype is because it is new (well new in old new thing kind of way). What I find interesting is the duality like relationship between MCMC and variational methods. Variational methods are an optimization alogrithm. I don't understand variational methods enough to say anything insightful but given the work showing the duality between optimization and probability* I find this new development of hype to be highly interesting.
Drawing on the concept of duality, I think Variational methods will come to be seen as holding no more power than probabilistic techniques. But a Duality is still great cause you can plumb old techniques to get new results.
The general perspective is that, of course, you'd like to get the fully marginalized, exact posterior probability distribution. However, that's not computationally viable for most problems, so you have to resort to approximations, like MCMC, Variational methods and MAP. I would say that they're all definitely "Bayesian" methods, as long as you're aware of what you're doing, and that you check the quality of your approximation.