Plus the book was actually laid down two decades later, by recounting memories to a jail mate who took note of it. Plus there were actually several retellings and rewritings of the book, so much that we're not even sure of the language it was first written in (French? Franco-Italian? Some Italian dialect?) So exaggeration, misremembering and lies are not only probable, but almost certain.
https://books.google.com/books?id=iN9Tdfdap5MC&lpg=PP1&pg=PA...
It would seem there is no consensus. Clearly some scholars think he went there.