When I worked as a freelancer, in the dotcom era, I've participated in a number of projects led by some rich people. That included having meetings in their personal homes (some of them conduct a lot of business from their home office, apparently). For most of them, I had very vague idea who they are outside of the project I was in (some of them were rich enough so I tangentially heard about them before, some weren't) and if they conducted some nefarious crap in secret, I would never know. There weren't private islands involved, but I don't think it's too far out necessarily if you upgrade from a nobody IT freelancer to the leading MIT researcher.

So, if that lunch happened _before_ Epstein conviction, I'd give some benefit of the doubt. After, "not with a ten mile pole" is the appropriate policy and any other conduct is on him.