Yes, but the servers only transfer encrypted payloads for which the servers do not have the decryption keys, and you can verify that just by looking at the clients (which are open source in this scenario). That is the entire point of end-to-end encryption.
Are you saying that MITM is not possible? For example your client will receive a key prepared by rogue server and it will decrypt and encrypt conversations on the fly. You wouldn't be able to tell unless you find a way to verify the person on the other side tried to exchange different keys.
Resisting MITM is the entire point of end-to-end encryption.
Verification can be made with the security code that WhatsApp uses, and the safety number that Signal uses (same thing, different name). Other systems have other, similar methods.
You can verify that they match in order to verify that you're not communicating with a man-in-the-middle, and if the key changes then both apps show a prominent warning.
Granted, a lot of people may not actually bother to verify.
