Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows managing one business Kafka. What do you think?
Grafana, for long, has been used to monitor time-series data and recently has been moving towards observability (including traces and logs). We are different in quite a few fronts.
1. There are specific observabilty specific UI widgets like serviceMap, SLOs and error budgets, I don't know whether Grafana provides it now. Also, last I used Grafana, linking and moving from one dashboard to another is still a pain. You can get a better idea of how different observability UI can get from Grafana by looking into LightStep demo.
2. We can run aggregated on filtered traces. Eg, I can get 99th percentile response time of a tag say payment_channel. am afraid this can be extracted from traces by Grafana.
3. SigNoz is easily extendible by adding your stream processing application to slice n dice data in your own way
I was pretty much surprised to see the results too. A single node Kafka with 2GB as xmx value, was ingesting at 4500 events/sec (around 1MB/s) on a single partition.
I think the concerns raised in this thread are less regarding raw throughput, and more about (1) the complexity of the typical production Kafka deployment (2) the arguably unnecessary, highly complex ecosystem around Kafka that you have to pay people or companies to use effectively, (3) the history of problems regarding data loss with ZK/Kafka, caused by leadership election bugs.
Exactly right. In my personal experience, Kafka's reputation for data loss and other mishaps is well-earned. Some of them are well explained by Jepsen tests.
hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer which can handle equivalent production scale?
I agree with my sibling comment, and reiterate my cousin comment that you've replied to (commenting here to complete this sub-tree).
Queuing technologies will come and go, IMO it's better to focus on the interface, and allow people to swap in whatever implementation they prefer and are accustomed to. It also benefits you in the long-term too, because an application that is less-coupled to a particular external dependency will be easier to test.
Some examples of queuing tech that's deployed successfully at scale: Redis Streams, RabbitMQ, Amazon's SQS. Since this is written in Go, you could even offer an in-memory, channel-oriented stream implementation, with no external dependencies.
Not one of these is universally better than Kafka: each offers a set of trade-offs, but a very similar interface from SigNoz's point of view.
For SigNoz's hosted/tenant-based solution, it might absolutely make more sense to use Kafka. But self-hosted users bring different trade-offs to the table, and might prefer to use another solution.
Strategically, can write/maintain the plugin for Kafka (very similar to how you operate right now, except it leaves the door open to more plugins existing in the future), and encourage community contributions for other tech. Or, when you're big enough, you might want to employ people to maintain those plugins too, since they're good for adoption.
Have an interface for a queuing system and support other things, not just Kafka. Ideally, you want a default/dev instance to ship with something super simple, zero setup and maybe in-memory - but allowing you to swap-in kafka or something more capable as needed.
That's an interesting point. Curious, would you use a project which supports a simple/in-memory datastore, but not anything which would be useful in production environment? Do you think that easy to get running and setup in dev environment valuable for adoption - even if it won't work in prod?
I am trying to understand - what would be a good way to prioritise.
> Do you think that easy to get running and setup in dev environment valuable for adoption
Yes, it'll make a big difference to adoption. If step one of your setup instructions are "provision a Kafka cluster", then you are going to lose 90% of people right there.
Ideally, your dev install is super simple and has a built-in in-memory queue thing. The key here, is to make it as simple as possible to get started. Once people have tried it out, and become invested in you, then you can say "for production scale, use Kafka instead of FastQ/SimpleQue/Whatever".
They key to that second step, is to have your product abstract the queue functionality it needs, into an interface that it uses to talk to the queue - allowing people to swap out queue backends with a simple configuration change.
So, make it simple to get started - and simple to scale up when you decide to.
hmm..got your point. We shall definitely look into other queuing system to be integrated as interface. Trying to understand better, what's a super simple dev setup like (to get the adoption)?
Right now, we can run SigNoz with all components including Kafka and Druid in 4GB memory supporting around 200 events/sec. Though, will need to check whether this micro setup passes a run of a few days.
What you have now isn't too bad, but figure out if you can get it down to one single command. Have a look at what netdata does (https://learn.netdata.cloud/docs/agent/packaging/installer#a...) - this is a single command to install, works really well and is super quick to get started on a single node.
Redis is like this. It runs in-memory by default, but can be trivially configured to write to disk / be persistent between server sessions. Many credit this feature as a catalyst to its adoption.
Azure has Azure Service Bus (a "full" messaging system, with subscriptions, topics, routing, AMQP 1.0 etc), and also Azure Storage Queues (a very simple queueing system where a client polls a single queue for messages).
Nice thoughts, A few other users also pointed this out.
We observed enterprise and other Observability SaaS vendors have some scripts and controllers to keep running these components. We plan to open-source that too. As you rightly pointed out running OSS needs man hours and we will try to remove those frictions.
Also when working with Prometheus and Jaeger, we observed people anyhow have to use Kafka to handle scale and mostly OSS are good at start but become pretty complicated at handling scale. Eg, Prometheus long term storage solution is Cortex which itself is difficult to manage. In that case, Kafka should be better beast to handle than multiple moving components inside Cortex. We built SigNoz as a scalable alternative inspired from stream processing architecture.
We will also be proving sampling strategies including tail-based sampling to retain important data and not unnecessarily clogging disks.
Hey, I am one of the maintainers of SigNoz.
ELK is tightly coupled to Elastic which may not be the ideal database to handle opentelemetry data. We wanted to be more of a platform where we can provide different DBs as plugins. Users can also build their own usecases by building more stream processing applications.
On the other hand, Druid powers analytical queries on data and is efficient in handling high-dimensional data. Many companies use Druid at scale (https://druid.apache.org/druid-powered).
Also Jaeger, a distributed tracing tool, provides plugin for cassandra, elastic, badger, etc. Some users found limitation in running fast aggregation of filtered traces. With Druid we can now search by annotations(without need of service name) and get aggregates on filtered traces, like p99 of version=xyz filters.
Hosting your own solution on Heroku might be costly and not worth maintaining. Go for commercial products like Graphite or Collectd + Grafana if all you need is to send your own custom structured metrics. If you need APM capabilities, checkout DataDog
If I have 8 services sending a combined total of 100GB of traces per day and 1 extra service that alone sends 100GB of traces per day, how is your pricing justified either to LightStep or as a customer?
