I've used both. I was introduced to Rabbit at one job and at another, was "fed" Kafka during a selection process. At that time, I was definitely not opposed to Kafka because, hey new resume item. I ended up yearning for Rabbit for three reasons.
1) Much easier to implement and maintain for small to medium architectures. However, war stories I've heard is that it starts to become a hassle for large clustering architectures.
2) Because it's a traditional message broker, the input and output ends, which I was responsible for, were much simpler to write because I didn't have to worry about replays when it came back online. Rabbit knows which client it has already routed to and where messages went. Kafka is not that sophisticated in that regard. Kafka has been described as "dumb broker/smart clients" while Rabbit is "smart broker, dumb clients."
3) The scaling. Rabbit is very scalable. Once you get to the Uber/Paypal level (like, a couple of million writes per second), then Kafka becomes the obvious choice. Rabbit handles thousands or writes per second just fine. However, at that second company and like many others, they thought they'd have to suck up all the data, so of course, Kafka was the more scalable tool long term. Spoiler: We were never, ever close to PayPal-level transactions. If the size of the sun represents paypal/Uber transactions, we were basically Manhattan.
Kafka is one of those things where if you're new to it, especially if you're coming from Rabbit or similar, you might tend to assume the happy path - exactly once delivery. This is a bad mistake (whether that's possible and to what definition is not a debate I'd like to dive into now). What you should expect from Kafka is at least once delivery.
There will be times when you lose offsets or when you actually want to replay every message, so take an hour and figure out what that means to your app. It's usually only a few lines of code in your consumer that compares source timestamps, but it's by far the most beneficial thing you can do when working with Kafka in my experience.
It's also relatively easy to hit "tens of thousands" messages/second, especially in replay or bootstrapping scenarios, and that's when Kafka becomes useful to the non-FAANG companies.
1) Much easier to implement and maintain for small to medium architectures. However, war stories I've heard is that it starts to become a hassle for large clustering architectures.
2) Because it's a traditional message broker, the input and output ends, which I was responsible for, were much simpler to write because I didn't have to worry about replays when it came back online. Rabbit knows which client it has already routed to and where messages went. Kafka is not that sophisticated in that regard. Kafka has been described as "dumb broker/smart clients" while Rabbit is "smart broker, dumb clients."
3) The scaling. Rabbit is very scalable. Once you get to the Uber/Paypal level (like, a couple of million writes per second), then Kafka becomes the obvious choice. Rabbit handles thousands or writes per second just fine. However, at that second company and like many others, they thought they'd have to suck up all the data, so of course, Kafka was the more scalable tool long term. Spoiler: We were never, ever close to PayPal-level transactions. If the size of the sun represents paypal/Uber transactions, we were basically Manhattan.