I don't speak French, but in Italian you can't have intransitive passive verbs. `manque` is intransitive, `me` in not a direct object.

I'd expect French and Italian to be similar here.