This is great intel - sorry it had to come with such painful experience.
Does anyone know if Open Distro for ElasticSearch (https://opendistro.github.io/for-elasticsearch/) has these problems? Or is it related to how AWS configures/maintains ES on their platform?
ODFE has a security plugin and supports auditing, RBAC and support for node to node encryption[1][2]. The security plugin is based on Search Guard[3].I have run Elastisearch clusters using both the Elastics Co opensource versions as well as ODFE and have personally found the security plugin of ODFE preferable to Elastic Co's X-pack. ODFE also has some additional features included like PerfAnalyzer and SQL interface. In terms of managing the Open Source Elasticsearch vs ODFE they are pretty much the same. I have not used the managed AWS Elasticsearch offering but I read the blog post and felt it had very little to do with ODFE. There is a good feature comparison matrix here:
Open Distro is the Amazon hard fork mentioned [1]. The missing features will most likely be a 1:1 issue.
The author does a disservice to their audience:
> As has happened before, Amazon took the open-source side of Elasticsearch, did a hard fork, and has been selling it as a hosted service, slowly implementing their own versions of features that have been available in one form or fashion in mainline Elasticsearch for years.
When what happened was Elasticsearch changed its licensing model after benefiting off of the open source community for years to be more restrictive, forcing the fork.
There's been no license change. For years, Elastic has had a set of commercially-licensed features on top of Apache2-licensed Elasticsearch. Within the last year or two, they made those commercially-licensed features source-available. Ironically enough, making the source available seems to have prompted a bunch of claims that they changed their licensing model.
Elastic mixed open source code with source available code, possibly as a landmine to sue large hosting providers, like Amazon. Amazon's fork includes removing these landmines. Elastic's hands aren't clean.
I have a hard time believing that Amazon's motives are pure. Companies that develop open source software have an incredibly hard time with profitability -- it is no surprise that elastic wants to reduce their own workload by maintaining a single repo for open source and source available code. The "land mines" are cordoned off in one directory. Hard to miss that.
For me the real frustration is Elastic's close tying of client features to server versions, making it impossible e.g. to buy the latest version of Kibana from Elastic and run it against a server managed by Amazon.
As I argued elsewhere in this thread this is completely untrue. Amazon ships an Elastic provided OSS build of Elasticsearch unmodified and as is. There is no fork repo in the opendistro account. There is no ambiguity in the upstream repo (it's a nicely documented code base).
Amazon is shipping and forking the elastic suite of tools, you are describing Elasticsearch itself.
When licensing is at a source file level that is ambiguous for sysadmins. I understand you are approaching this as a developer, where it might be very clear.
There is no Amazon fork of the Elasticsearch repository. I looked at their repositories. they are copying the oss build of elasticsearch unmodified, without patches.
As for the licensing; licensing is documented in a 10 line LICENSE.txt. Also, it would be hard to miss the license settings, when you set this up. Hard to miss unless you are a seriously dyslexic and negligent person. In which case, I'd argue the Elastic basic license is the least of your problems.
Does anyone know if Open Distro for ElasticSearch (https://opendistro.github.io/for-elasticsearch/) has these problems? Or is it related to how AWS configures/maintains ES on their platform?