Though shouldn't be that hard to know if you are being spoofed. You probably have a decent idea of how much the IRS drifts and any large deviations from that or unexpected jumps in GPS should be noted and possibly, maybe manually, rolled-back so that the IRS only considers data before that point.

I understand that current civilian aircraft wasn't designed with that in mind though.

Avionics were designed with the opposite assumption. When a high accuracy location source provides a location (GNSS or DME/DME) the position is updated and it's assumed that could mean a significant jump for the INS.

Practically that was what happened all the time before GPS. You would fly for a few hours over the ocean with no ground based reference, having a reasonable but not perfect INS location. Then get close to the coast where a DME/DME fix was done which updated the INS position as a big jump of up to a few miles.

Filtering GPS updates that are too far "off" the INS state would be an almost opposite design to the original assumptions that DME and GNSS are highly accurate.

some amount of IRS drift should be ~predictable, an anticipated deviation from GPS could be used as a measure of trust for GPS

It seems like there’d be some use for something that correlates what it sees on the ground with known satellite imagery as a check? Especially for anything low-flying.