It's not about being a prima Donna. It's about business value. Too many meetings over the years should either be better planned, not taken place at all or could have been an email/chat message.
Meetings with an agenda are generally better, but that doesn't mean meetings without one can't have any business value. If you skip it, you make sure you at least don't contribute to anything decided in it.
Business value first