One scenario I can think of is a non-IE user telling an IE user to go to a particular website and click the button that's at the top of the right sidebar. For the IE user, it might be a stacked layout where the sidebar actually appears under the main content. I would call this a nuisance more so than a major issue, but nevertheless it is something to consider.

you'd have the same issue between a desktop and mobile user. i don't see it as an issue at all.

I think the average computer user knows that websites often look different on computers vs. phones, so they might be confused when a website looks different than normal on a computer. Like I said, I would only call this a nuisance. Certainly not a big deal.

I think it depends of if you need trafic from those browser or not. Because if you do, you'll have to maintain several version, or increase your time on working on it to do gracefull degradation (which is of course a good way to do, but more expensive)

but that's kind of the beauty of css grid, the gracefull fallback is basically mobile view which you already put effort into.