A few years back I ran `qpsmtpd` which is a flexible mailserver written in Perl. It was extremely easy to hack upon, so you could write plugins for SMTP-time spam-detection, rejection, etc.

These days the project is stalled, I think most of the development moved over to a nodejs based project.

Anyway the project was pretty simple to work with, and scaled wonderfully:

http://smtpd.github.io/qpsmtpd/