I don’t know how many folks really know how much money is spent by big (and medium and small) companies on sending email but I can tell you it is considerable. My last job was a small retailer and we spent maybe 2m a year just on platform costs and at my current role it’s closer to 200m (USD). There are some places where it probably makes sense to use one of the vendors. But I think a carefully architected custom solution like this could get most organizations where they need to be. I’m just not sure how much the talent costs. Do AWS experts charge a premium? Or would it just be cheaper to buy Acoustic and let the account team handle all the work.
SES is good at sending stuff, but if you're mailing a lot you'll want to handle both hard and soft bounces correctly. And not just bounces you get live from the end SMTP daemon, but bounce emails as well.
If you keep attempting to email an address that bounces, over time, you may get marked as a spammy sender.
As far as I can tell, this setup doesn't do much with bounces other than quitting on a clear hard bounce from the end SMTP daemon.
This is one of the reasons email services aren't cheap.
You're right! Bounces and complaints needs to be handled properly and the design does not reflect that.
SES uses account-level suppression lists, to avoid sending emails to addresses that has complaints, hard bounced or soft bounced a number of times. It does not automatically handle bounces sent over email, and I've not seen a provider out there doing that but I could be missing it.
We do manually monitor those however, by having the FROM address directed at an inbox that we monitor. Out of the 4M emails we send per month, we only get one bounce email back. And that's not really a bounce but a "please confirm you are real" bounce.
This could probably be automated with SES too, by parsing incoming emails and adding addresses to bounce list. For that 1 email per month we decided not to add that complexity.
Note: This setup is used for transactional emails, where users opt in via account creation and the first email sent is a verification email, to check that the address is real and that the user owns it.
However, my experience was that you also had to do something with soft bounces. For example, if someone's mailbox is perpetually full, re-sending to it over and over still can still eventually hurt your sending reputation.
Yeah, SES puts those soft bounces on suppression list too after a number of failed attempts. That is actually an issue we've struggled with. A large number of users using a mail service with just 10MB of storage. Those mailboxes were always full and we got soft bounced over and over until SES put them on suppression list. User contacted support and got told to clean out the inbox to make room but emails could still not be delivered until we had manually remove the address from the suppression list.
It should be noted that CAN-SPAM has very high fees for spanming and major email services can aggressively block spammers. Legitimate businesses have to carefully monitor and correct their issues. AWS can and will suspend your usage until you fix the problems.
From what i saw/experienced deliverability is usually only an issue when spamming people against their will. Sure, Microsoft/Google will block you at first but after a few weeks/months of going through administrative hoops you'll be allowlisted and your mails will deliver just fine.
Sounds like paying an office worker a few weeks to keep in touch with Microsoft/Google support plus hardware + employing sysadmins would be considerably cheaper than 200m/year.
That is, of course, unless you're spamming your users with undesired messages.
I'm not so sure. Running transactional-message-only (i.e. double opt-in, password resets) for a small site I own with DKIM and SPF on its own IP for years ... GMail will still block a large percentage of it.
Using SES with the exact same emails, Gmail will block significantly less. The large providers are whitelisted and unless you're as large, you're not going to get that privilege.
Granted, for $200m you should be large enough to bribe a few Google people to get into the club, but for anything where it reasonably might be profitable to run it yourself and employ a sysadmin, you won't be able to afford that.
> GMail will still block a large percentage of it.
Yeah Gmail is actively filtering smaller providers. But from what i heard/read it appears after personally contacting them several times they will end up allowlisting you (like Microsoft).
Unfortunately these evil tech businesses have very little official avenues to reach them. Usually it's either through public outcry, or through the official form you see in error logs, which requires creating a google account if i remember well.
Arguably, if you're operating a business and have a client using Gmail, you could have you and your client contact Google support to ask allowlisting, then sue Google for anti-competitive (monopolist) behavior if they don't comply. I don't know of a case like this, but it should be an easy win and would set an interesting precedent.
I'm personally not into legal stuff, but if some startup people around here with a few lawyers would try this approach... :)
Not sure about the rest of you, but i'm exactly in the opposite team. email is as close to "universal messaging" as we'll ever get. It's the best example of interop we've seen so far and i'm afraid no new protocol could ever catch up with the almost 4 decades of email legacy.
Personally i don't care that Google won't let me talk to people. I treat google's email server like i treat a bourgeois café in a fancy district: there may be friendly people to have a chat with in there, but they definitely don't want me around and i have better places to be and better people to meet. Only a revolution can change this balance, and in the meantime i won't loose my energy trying to get into their club, but rather develop our decentralized clubs.
It's silly to claim that almost everyone has problems with Gmail deliverability. The fact is that Gmail has problems with receiveability. Gmail is the email provider for people who don't care if they get all their email.
So true. If only we had more techcoops providing free-price (or 10€/y) services for the masses with a UX experience close to (or better than) google services...
We send multiple billions of emails a year, and text messages, and push messages, and outbound ivr messages, and two way chat messages, and print messages, and in some cases fax messages lol. Our outbound volume is extreme and luckily only a few of our communication streams have trouble with deliverability. But it does take a few different teams to manage it all.
Neat setup there, using SQS to ease up to not hit SES limits.
If one wants to go further than just one-off emails, listmonk by Zerodha, a fintech company that builds most of its tech in-house, is pretty decent for an email-list manager. It one-click deploys to Heroku and the only dependency is Postgres. It could use SES as its SMTP server.
Listmonk is fantastic -- I use it for the newsletter of my own blog and am amazed by how simple it was to set up and how easy it is to operate and use every time I send out a newsletter.
Prior to Listmonk I used Mailtrain[0] and like Listmonk so much I want to run it as a service (with 30% donated back to the Listmonk project of course) -- deliverability worries and initial setup aside it's just the kind of minimal tool I've come to love these days.
> like Listmonk so much I want to run it as a service (with 30% donated back to the Listmonk project of course)
Hey hardwaresofton, this is exactly what we're planning to do on https://srv.io/ :)
The platform is mostly ready, and we're just finishing some infrastructure changes to allow the platform to run third-party applications safely. We are planning to include Listmonk in a second batch of the applications to release.
I personally enjoy working with Listmonk, it has replaced Sendy[0] for me.
Just the other day I was wondering why there aren't "software maintainence/repair shops" as there are for hardware (PCs, gadgets, appliances). A tech shop like srv.io is exactly what I was wondering existed. Surely there is a market for it. I wish you (and your competitors) all the very best!
Wow that’s an interesting concept! I’ve only ever thought of these services one by one, I hadn’t thought of bundling them under the same service (except when they’re thematically linked like prometheus & jaeger for example).
Glad someone else is thinking along the same lines! F/OSS could be so much better with the help of platforms — there’s space for a win win as long as the platform side just pares down the greed a tiny bit.
Title is misleading, delivery is done by SES which is not built by the author.
Might be ignorant here, but to me the article reads like an excuse to use serverless. What I really miss is the testing story though, in my admittedly limited experience with serverless (Lambda and Workers) this was always a pain point I never really managed to overcome.
I think an interesting approach would be to send secure email via programmatically generating the public string sequence within a domains TXT DNS record (short time to live expiry) you could even establish a very slow Diffie Hellman key exchange with two paired domains where their TXT records alterations reflect the key exchange. There are multiple dimensions of communication possible here too since you can have a near infinite number of TXT records per domain. You can even in theory transmit TCP packets (albeit slowly) over back and forth DNS modifications. If the encoded info were spread across multiple unrelated domains, it would require any interceptor to know which domains are the carriers of the information. This is pretty similar to blockchain transactions except as long as you have icanns authority to use that domain you control your own private part of the blockchain that is able to interact with any other as long as it operates on the same protocols.
Agree, not intentionally misleading though. Just a lack of a better word I guess. Original title was "Building a serverless emailing service".
About testing: It's not that different from any other runtime. I admit, I put a lot of code into the same file in the examples as it was easier displaying the code that way. I tend to keep the handler of the lambda very lightweight and then test the functions that the handler calls. That way it's just unit testing and mocking away dependencies. The functions are pretty small so naive dependency injection works fine.
AWS docs covers testing of Lambdas a bit here: https://docs.aws.amazon.com/whitepapers/latest/serverless-ar...
Nice write up. We set up a very similar system using, Serverless instead of SAM, and our costs are about the same. I need to take a good look at MJML. Our templates our stored in S3 and editing them is somewhat primitive.
We have been using something similar called “Sendy” for a few years now to leverage the cost benefits of AWS and SES. The licence fees are very reasonable and setup instructions are good.
