Issues around the sunset of Aperture or iPhoto, bad Photos, Final Cut Pro X, etc are most certainly not software quality problems, but product management problems.
On such large apps that need large teams of engineers, without good product management and also project management, even great engineers can do a terrible job despite all their efforts or heroics "on the ground".
> In most organizations, if someone wants to make a change to the website, they’ll want data to support that change.
So true and sad. In all the so called data-driven groups I have worked for, the tyranny of data makes metrics and numbers the justification for or counter to anything, however they have been put together.
> The sad truth is that most people aren’t being rigorous about their A/B testing and, in fact, one could argue that they’re not A/B testing at all, they’re just confirming their own hypotheses.
The sad truth is that most people aren’t being rigorous about anything.
With parametric polymorphism and generalized algebraic data types, you could:
(0) Give every process in the pipeline could its own input and output type, rather than forcing all processes to use strings.
(1) Enforce, statically, that only "compatible" processes can be composed. For instance, if `Foo` is a process that outputs integers, and `Bar` is another process that takes strings as input, then `Foo` can't feed its output to `Bar`.
However, this is involves some rather tricky type hackery (so-called "type-aligned sequences"). Hardly a good fit for a language with Go's design goals.
Well, I think unfortunately my sketchy code examples maybe did not cover the idea in enough detail.
What I'm doing in practice is to create different struct types for each type of process I use, which all are of a "base interface" called "process" (mainly for the Run() method, so that the pipeline component can call it).
The typing of channels for each process struct type should now enforce what components can be connected together (you will not be able to assign a "string chan" to a "[]byte chan" field, for example).
Did this answer the question, or do you see further problems here?
Ah, okay. I misunderstood the intention of your original example, then. My bad.
In order to prevent further embarrassing myself, I'm going to tell you how I (currently) understand your design, and, if I'm wrong, please tell me: Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping" (connecting processes to each other) is handled elsewhere. In fact, using your "pipeline" abstraction, you could perfectly well run a bunch of processes that aren't connected to each other at all.
> Your "pipeline" isn't actually connecting the processes to each other. You first connect the processes to each other, and only then add them to a "pipeline", whose primary (only?) responsibility is to ensure that each process runs in a different goroutine.
Exactly!
> This much doesn't really need parametric polymorphism, let alone generalized algebraic data types. However, calling it "pipeline" is kind of misleading, because the actual "piping"
Ah, yes, didn't think about that, but yeah, but that's a good point! I should consider a better name for this ... Thanks for pointing it out!
Catnaroek said it already: without parametric polymorphism you are forced to either: 1) commit to a single type upfront which limit the utility (string in your example I think), 2) use interface{} and suffer the boxing in every circonstances while not having any static type checking.
I actually like your pattern. I think it makes sense to have a process as a pair of input and output streams.
Is there any solution based on openvswitch ? If using vxlan to encapsulate containers L2 frames, the openvswitch kernel module should be able to route packets very well with much lower overhead than any userspace solution that do not access the nic memory directly.
OVN is 0.1 fairy dust and unicorn horns at the moment, looks very promising but you're most likely going to be waiting another 12 months before its solving your problems.
SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"
what could possibly go wrong ...