Did you update all your Java installations – client and server – to at least Java 6 update 26 in June 2011?
There were a dozen "unauthorized Operating System takeover including arbitrary code execution" bugs fixed at that time, some exploitable via untrusted applets, others via tricking server installs to submit certain data to standard APIs:
I've had the Java plugin disabled in firefox for a long time now. On the very rare occasions I need it, you can re-enable it without restarting the browser (unlike extensions.)
Lately, yes. Over the last couple of years (Read: after Microsoft mostly cleaned up its act), Java has been been one of the primary sources of client exploits, along with Adobe products.
I don't know that there's anything special wrong with it other than that anything deployed widely enough makes a good target.
Of course, the JVM is the JVM wherever it runs, but when one is under the impression of a blanket statement like "Java is secure", they're likely to be thinking of server-side processes which rarely get compromised for reasons you've stated - despite having the same "level of security" wrt vulnerabilities.
I have always considered it a relatively secure platform... am I so wrong?