I agree with my sibling comment, and reiterate my cousin comment that you've replied to (commenting here to complete this sub-tree).
Queuing technologies will come and go, IMO it's better to focus on the interface, and allow people to swap in whatever implementation they prefer and are accustomed to. It also benefits you in the long-term too, because an application that is less-coupled to a particular external dependency will be easier to test.
Some examples of queuing tech that's deployed successfully at scale: Redis Streams, RabbitMQ, Amazon's SQS. Since this is written in Go, you could even offer an in-memory, channel-oriented stream implementation, with no external dependencies.
Not one of these is universally better than Kafka: each offers a set of trade-offs, but a very similar interface from SigNoz's point of view.
For SigNoz's hosted/tenant-based solution, it might absolutely make more sense to use Kafka. But self-hosted users bring different trade-offs to the table, and might prefer to use another solution.
Strategically, can write/maintain the plugin for Kafka (very similar to how you operate right now, except it leaves the door open to more plugins existing in the future), and encourage community contributions for other tech. Or, when you're big enough, you might want to employ people to maintain those plugins too, since they're good for adoption.
Queuing technologies will come and go, IMO it's better to focus on the interface, and allow people to swap in whatever implementation they prefer and are accustomed to. It also benefits you in the long-term too, because an application that is less-coupled to a particular external dependency will be easier to test.
Some examples of queuing tech that's deployed successfully at scale: Redis Streams, RabbitMQ, Amazon's SQS. Since this is written in Go, you could even offer an in-memory, channel-oriented stream implementation, with no external dependencies.
Not one of these is universally better than Kafka: each offers a set of trade-offs, but a very similar interface from SigNoz's point of view.
For SigNoz's hosted/tenant-based solution, it might absolutely make more sense to use Kafka. But self-hosted users bring different trade-offs to the table, and might prefer to use another solution.
Strategically, can write/maintain the plugin for Kafka (very similar to how you operate right now, except it leaves the door open to more plugins existing in the future), and encourage community contributions for other tech. Or, when you're big enough, you might want to employ people to maintain those plugins too, since they're good for adoption.