Hi Prabhat,
I am one of the maintainers at SigNoz. Nice to see OpenObserve's belief that the future of Observability should be OpenSource. We chose clickhouse rather than building a database as it takes multi-year effort to move a db to maturity and clickhouse has been battle-tested at Yandex, Uber and Cloudflare. Clickhouse also provides native integration with s3 and other blob storages. Our users have been using disk as hot storage for a week and moving the data to s3 after that. Tiered storage is really cool in terms of query performance.
We have also seen logs data of our users at a compression ratio of 30x/40x. We have published a logs benchmark (https://github.com/SigNoz/logs-benchmark) where the data is very high-cardinal (causing a compression factor of only 2.5x). Would love to see how does OpenObserve perform in that dataset someday.
Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast
What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to high dimensionality(can need millions of columns). Do you store the labels in an array?
I would love to see some benchmarks when making such a heavy claim. I would be interested in knowing performance of ingestion rate, query timings and resource usage.
We have recently been talking to a few companies in HealthCare and Fintech, and it seems they have strong needs for data to remain on-premise. One of such verticals in the company is around monitoring, logs and traces. They may contain personal information and these companies want to keep it under their own control. Hence, they can't send data to SaaS vendors due to regulations around data sovereignty.
They cannot use DataDog, NewRelic and onPrem Dynatrace & AppDynamics are closed source. A recent good alternatives are Prometheus for Metrics, Jaeger/Zipkin for distributed tracing and Loki for logs. First of all setting up all these take months and running a scaled version of these OSS is pretty complex.
We are building SigNoz (https://github.com/SigNoz/signoz) to address these above problems. SigNoz is an open source alternative to DataDog. SigNoz has Kafka and Apache Druid, both of them are industry tested highly scalable components. Druid powers analytical capabilities on tracing data which is missed in both Jaeger and Zipkin.
Below are the motivations for building the project:
- Out-of-box features that other OSS miss. Get started in minutes and not months
- Seamless integration of Metrics and Traces in a single dashboard
- SaaS vendors charge an insane amount to provide Application Monitoring. They often surprise you with huge month end bills without any transparency of data sent to them
- Data privacy and compliance demands data to not leave the network boundary
- Highly scalable architecture
- No more magic happening in agents installed in your infra. You take control of sampling, uptime, configuration
- Build modules over SigNoz to extend business specific capabilities
- Full support for OpenTelemetry, which is a vendor neutral instrumentation framework
