The tl;dr is it's just a lockless hashmap attached to a TCP server with a log. Simple Get/Set operations are highly optimized, so with high batching they are able to efficiently fetch a lot of data efficiently. The architectures scales very well when you add threads and data access that is uniform.
It struggles a bit on certain types of workloads like hot keys, think heavy hitting a single sorted set. It's a cool architecture.
The difference is that Redis is single-threaded, the value proposition of Redis was/is a fast (in-memory) simple (single-threaded) server to serve things so much faster than a traditional DB. It was a perfect fit and became popular with the Ruby/Rails community,etc since those environments combined with traditional SQL servers are slogs compared to what a fast server like Redis could do.
As good as Redis is, modern computers with multiple cores potentially leaves a lot of performance on the table. Garnet seems to be a well designed multithread KV-store (though sadly the benchmark page doesn't list benchmark for more complex objects even if the simple cases looks good).
I feel like most of the obsession over Redis came from old school cgi-esque server side apps that couldn't easily maintain state themselves...
Nowadays your webapp is often a persistent, multi-threaded web server where you can cache whatever temporary state you want far more efficiently than reaching out to a KV store.
Using postgres and caching common results becomes a non-issue even at scale. The main area where a super fast KV store shines is when you need to share a very large number of keys, where many servers only need to access a random subset and isn't interested in maintaining the full set and getting updates on a bus, where the values often change and needs to be revalidated so caching is ineffective, and where persistence is not needed so a database doesn't matter and a lighter KV store is acceptable.
Do you have data to back up your claims? I see a lot of customer claims for Valkey here, https://aws.amazon.com/elasticache/customers/. Neither of the AWS or GCP offerings are in preview.
The Valkey implementation of multi-threading is fundamentally different than what existed in Redis. The history dates back to work done in ElastiCache that was released as "Enhanced IO", https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-el.... The version released in Redis could only do about 350k RPS because of poor memory locality of operations, the inability to do command processing while handling I/O, and the inability to offload much of the TCP read path. The new version in Valkey can achieve 1.2M RPS.
"They made certain improvements later", should be "we threw away the old implementation and built a better one."
1. The fundamental idea was mine. Ideas can be improved, of course, and that's good. That was my point. However, allow me to say that Redis is fundamentally a "software of ideas" than anything else. Technologically it is far from impressive (Redis, ValKey, all the forks).
2. Redis 8 improves the same idea, too, released today.
3. If you claim [in a different comment here] you provided a lot of code to Redis, why you didn't send a pull request for that? So, you are practically saying you were using, at Amazon, all the BSD code we provided, but could not provide an important part of the code to us? You see how broken such model was? At least stop defending it.
4. We can now copy the implementation: the parts are reversed (the irony!), and your code is BSD as our was for 15 years. When we avoid doing things like that, is because we have issues with how certain things were made.
5. I don't understand the motivations of you and other AWS people commenting here today. You work for a company that is creating issues to the OSS ecosystem: this is hard to deny. You cloned (and, yes, the license allowed for it) the code of Redis, and work on it so that hyperscalers can continued to do what they used to do. We bring Redis back to AGPL, and you are here to do the interests of Amazon in the comments. Did you see me commenting your stuff, when you release your things, with comments like "ah! But this is unfair"?
There is to make choices. I understand that it was cool to continue to work at a Redis fork, and part of the incredible thing open source is, is that forks survive in the hands of different teams (but design ideas can be misunderstood and projects may turn into other projects). So if you are happy to hack on ValKey, I hope you'll have the best experience out of it. But there is to make choices on how/when to interact.
This exchange makes me sad. I know we can do better.
I don't understand why so many people think that it's impossible to have open source in your heart while working for a big company in your day job. I don't understand why people who have dedicated a lot of their time and emotional energy to keep open source ways alive and help build a community effort are attacked because they work for a company that needs to be made the villain in the narrative.
Of course Redis is free to copy BSD licensed code that Valkey contributors add to the project [1]. I only wish that the blog post about this advancement in Redis would give some credit, rather than claiming "We also improved the performance of CRC64 calculations" [2].
We can all do better, and engage with one another with mutual respect and admiration for what has been freely given.
> I don't understand why so many people think that it's impossible to have open source in your heart while working for a big company in your day job.
Because a big company like Amazon has produced almost no open source work(yes some random collection of repos and 2 PRs here and there is not the same thing) compared to how much it has benefited from open source. (I know I know OSS allows for all that so not claiming anything wrong). But it does show what the company policy must be towards open source (Consume all you can, contribute only when absolutely essential for the company barring exceptional circumstances).
> I don't understand why people who have dedicated a lot of their time and emotional energy to keep open source ways alive and help build a community effort are attacked because they work for a company that needs to be made the villain in the narrative.
Antirez is not attacking anyone above. English is not his first language and he is just putting out some of his thoughts (which people are free to disagree with but those are what he thinks), like how you casually slipped that you didn't get credit for one commit copied from valkey (where the copier is giving due credit in the PR by linking the source PR and the authors inline). So they copied a PR from valkey, and the redis blog post should give lots and lots of credits? If that was the standard, so many of AWS services will spend all their reinvent time giving credits to their source OSS projects.
> Because a big company like Amazon has produced almost no open source work
This may have been true a decade ago, but things are quite different now.
> compared to how much it has benefited from open source.
This is the nature of digital public goods. We are all going to disproportionately benefit from digital public goods relative to what can be produced as new digital public goods. No one will ever, EVER be able to "contribute proportionately" given the endless bounty of software made freely available for all to use.
> If that was the standard, so many of AWS services will spend all their reinvent time giving credits to their source OSS projects.
The observant should notice a change in this over the years. For example, the announcement for Amazon Q Code Transformation [1] acknowledged that OpenRewrite was used under the covers, even though it was an implementation detail that didn't have to be disclosed...
Of course these disclosures and good-faith intentions to engage on open-source community terms under long-established community norms don't always work out the way we hope. [2]
Just make it closed source (or source-available) and give out no-cost licenses how you see fit. You are the author, you decide what to do with your code. This is a well-supported model too, plenty of products are like that.
There are plenty of licenses around, lack of alternatives isn't why people use MIT or Apache.
My friend, Amazon being legally allowed to behave like a schmuck doesn't imply the community can't point that out and complain about it. AWS (legally) exploits open source projects, and that's a fact.
There are many actions and behaviours in life that are not illegal but actively worsen society at large if you do them.
That companies that are the main contributors to OSS are forced to take drastic measures is just consequence of AWS not being a team player, you should have at least the decency of not commenting here.
PS. I don't have a horse in the race, I'm not a Redis user, I'm just appalled by your behavior.
> So, you are practically saying you were using, at Amazon, all the BSD code we provided, provide an important part of the code to us? You see how broken such model was? At least stop defending it.
I'm not defending it, I'm trying to fix it. I want Amazon to contribute back. That's what I spend most of my time doing, but I can't just sit in a meeting and tell people we should give away code. It takes time to convince people that we should collaborate on the core and just compete on what we want to differentiate on. It takes time to convince people that building open-source in a vendor neutral space makes software that is better for everyone.
You have all the rights. Usually I do my best to avoid confrontations of that kind, and to respect the work of others. But I see certain communication patterns that are, for me, too much, and I needed to tell it. I think go on HN, and comment in an aggressive way the work of competitors, during announcements days, is something fundamentally wrong.
On the contrary it has made me appreciate antirez even more not just as a developer but as a true champion of open source who really wants open source to prosper and is willing to voice his opinions whenever required.
It's not just a fork, there have been two releases on Valkey that improved performance and memory efficiency. There is a lie that Redis likes to spread that only their own employees were working on the core engine at the time of the fork, but most of the engineers on Valkey came directly from having worked on Redis OSS. A recent example is we modernized the hash table a bit: https://valkey.io/blog/new-hash-table/.
Nobody wants to deny that Redis got from contribution from external developers. But it is fundamentally true that for like 8 years almost every substantial contribution was created by people working for Redis, and that later we got something that was still a small part compared to the total.
There are the commit histories, the GitHub contribution graphs. Everything is public. The current code base was written for the majority by a few single folks, for another small amount of the sum of all random people in the community, for a smaller part by people that now work at ValKey.
> It is also hard to reconcile the claims that cloud providers do not contribute with the actual commits to the Redis repository. A quick examination of the commits since the 7.0.0 release using gitdm shows 967 commits over that time period:
> Binbin Zhu, of Tencent, is responsible for nearly 25% of the commits to the project. Some of the contributors without a readily identifiable employer surely are Redis employees, but it's clear that the company has not been working alone.
If you go into the GitHub of any of the forks, and check the contribution page, you will see this data is not correct. Probably all my commits are into this "unknown", since I push with @gmail.com account without being part of any organization for most of the time.
This is likely some partial data of some specific fork or alike.
The LWN article is examining the 976 commits made after the 7.0.0 release. I don't think you had any commits during that time?
As is typical for software projects, early authors will be disproportionately represented in revision histories. I am still the #4 contributor to the Anaconda installer [1] originally used by Red Hat Linux, then RHEL, then Fedora, and others, despite not contributing to the code base for two decades.
Yeah, there has been a lot of stuff like performance [1] and efficiency improvements [2]. A lot of the contributors, that didn't work for Redis labs but worked on Redis OSS before the fork, moved to Valkey and they continued to contribute.
As one of those 12 other people, number of commits is not a good metric in this context. When you have a single maintainer, like antirez was for a long time, he could commit 10+ commits for a single small feature but I needed to get someone else to approve my commit before merging.
I know this doesn't mean much, but I spent a lot of time outside of work helping to maintain the Redis codebase. I attended our meetings and responded to issues on vacation because I cared about the community, it was more than just a paycheck for me. I don't regret doing any of that.
This code also looks generally fixed in Lua5.4, https://github.com/lua/lua/blame/9ea06e61f20ae34974226074fc6.... Valkey and Redis really need to move to Lua that isn't so old.