Not ferns specifically, but I've read a simplistic explanation that plants lack behavioral defenses, so they rely on chemical defenses. And more chemical defenses requires more genes.

Because it somehow survived millions of years despite that massive inefficiency holding it back. Quite remarkable luck not getting out-competed to extinction.

> despite that massive inefficiency holding it back

I assume you are referring to the size of the genome. Has anyone been able to prove that it is causing an inefficiency? Maybe it isn't. In classical computer programming languages sometimes more code is more efficient, such as unrolled loops. That analogy may not apply here. I am far from knowledgeable in this realm.

If it's really just inefficiency, wouldn't a mutation that removes some of the surplus genome bring an evolutionary advantage? Those mutations are probably rare and the advantage miniscule, but anything adds up over a long enough timeframe