So in my experience, that was not the hard part of learning French, or Spanish for that matter. For someone coming from a non-gendered language, the gender of nouns is taxing to memorize. Often knowing what the correct word is and how it is spelled will have nothing to do with what gender it is.

The other hard part of learning French (for me) was the conjugation of verbs in various tenses, and specifically any words that are exceptions to the normal rules. If you take a look at a french grammar book for (e.g.) English speakers, it's likely that it will mostly concern conjugation.