Interesting, will keep an eye out as you mature it, looks like an awesome DSL for us to fill the runtime.
Advantages are:
1. mesos integration, with that comes containerization support, multi tenancy, QoS, proper pipeline supervision, etc
2. Scheduling of pipelines. i.e.: Schedule them on 100 computers.
3. the outputs of your DAG could be consumed by other systems immediately and even written in different programming languages. So your python DSL could be the source to the Scala DSL at some point. so language interop
4. Available KV storage
5. Tracing (Zipkin) - ala Google Dapper.
6. Fast networking - C++ backed runtime is 1 order of magnitude faster than the python one.
What would be involved, is not much from what I can tell.
Each concord 'operator' is like a networked function.
so given a DAG, you could generate many operators internally, or literally write them to a file, i.e.: operator_one.py etc. The code generation or internal scheduling would be the glue that's needed.
if you ever become interested, ping me! would love to collab alex@concord.io
Advantages are:
1. mesos integration, with that comes containerization support, multi tenancy, QoS, proper pipeline supervision, etc 2. Scheduling of pipelines. i.e.: Schedule them on 100 computers. 3. the outputs of your DAG could be consumed by other systems immediately and even written in different programming languages. So your python DSL could be the source to the Scala DSL at some point. so language interop 4. Available KV storage 5. Tracing (Zipkin) - ala Google Dapper. 6. Fast networking - C++ backed runtime is 1 order of magnitude faster than the python one.
What would be involved, is not much from what I can tell.
Each concord 'operator' is like a networked function.
so given a DAG, you could generate many operators internally, or literally write them to a file, i.e.: operator_one.py etc. The code generation or internal scheduling would be the glue that's needed.
if you ever become interested, ping me! would love to collab alex@concord.io