It helps to instrument the journey of your work from jira all the way to build, deployment, run and monitoring (observation).
From that you can get measurements on how long each stage takes and the duration of each transition.
From there you can compare team A and B. The transition times is where the human time cost usually sits.
Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight.
The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry.
Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed.
There's a wealth of data being left on the table that can help inform management decisions.
This feels all to familiar to a Java developer and similar to what they do in k8s with the genericapiserver component but not to the full extent described in the article. It's interesting to see how the service container pattern appears in other language ecosystems. The nice thing about go's module system is that it feels like a monorepo without the tooling feeling separate to the language. So splitting up a repo into different repos doesn't come with the versioning hell when working with maven
> So splitting up a repo into different repos doesn't come with the versioning hell when working with maven.
Damn you hurt me with this one. You can use 100 plugins and versioning with Maven in these scenarios is still a rat's nest. But using a monorepo can make CI/CD for individual services just as messy :(
What versioning hell is that? I've found maven versioning really nice.
If stuff is versioned together, keep it in the same repo, but in a multi-module project. If you want independent versioning you need to have an actual interface (meaning semver, backwards compatibility checking etc.) and that's the point where the pieces should live in separate repositories.
Kubernetes is becoming the lingua-franca of building infrastructure. Through CRDs and the kube api spec I can
- start an single application
- deploy a set of interconnected apps
- define network topologies
- define traffic flows
- define vertical and horizontal scaling of resources
And now I can define AWS resources.
This creates an interesting scenario where infrastructure can be defined by the k8s API resources and not necessarily have k8s build it. For example podman starting containers off a K8S deployment spec. It's an API first approach and its great for interoperability. The only downside is managing the yaml and keeping it consistent across the interdependencies.
I really wish fabric8, and more specifically kotlin k8 dsl[2] was getting more traction.
It removes the down side of yaml all over the place. It's missing the package management features of helm. But I have several jars acting as baseline deployments, and provisioning. It works really well, and I have an entire language. So I can map over a list of services, instead of do templating. The other big down side is a java run takes a minute or two to kick off
I was resilient to k8 for a long time. Complexity was secondary to cost, but Digital ocean has a relatively cheap implementation now. This commonality and perseverance of tooling is great.
I want metrics, a simple annotation. I want a secret injected from vault, just add an annotation. It's also cloud agnostic, so this logic can be deployed any where some one provides a k8 offering.
EKS was very powerful. As running service accounts via non managed clusters. Removed the need to pass an access key pair to the application. That service account just ran with a corresponding iam role.
It's been a while since I've looked at Fabric8 but it had good java -> k8s integration and was great for writing k8s tools.
It appears though that Fabric8 is useful for solo java projects without complex dependencies on non-java projects or small java shop. It overlaps with where jenkins-x is going, which has made major strides in the last 24 months.
The original team that worked on Fabric8 lead by James Strachan all moved on from Redhat and many of them are working on Jenkins-x.
I've been a big fan of brain storming tools for a long time so appreciate new tools with different takes on the problem.
I do have one observation and it maybe just me. I have no way of clustering a group of connected thoughts, or a way to 'chunk' the concepts. I see some attempt by using coloured lines but there should be a way to make these ideas hierchical and from that be able to zoom up and down various levels of conceptualisation. It would make managing a large network of ideas a lot easier, possibly a bit more ergonomic. Similar to the way the brain works.
It may just be me but was wondering if there's a way to achieve that with the current tooling or if you think there are workarounds?
The idea is that if you've got a bunch of cards you want to 'group' to isolate and expand on seperately, it's pretty convenient to be able to do so in a new space.
Let me know if this addresses grouping for you, thanks :)
Great article. I think you might be taking a subtle point in the article for granted. Namely, the ability to assemble a fairly complex distributed system from open source components.
Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.
Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.
You make a good counterpoint: it's awesome that this can be done concisely enough these days to fit in a hobby blog post. That's new and demonstrates the approachability of these tools.
At the same time, OP is right: a business doing this is making poor engineering decisions.
We don't know that yet! The post explains what they did but not the why of the problem. Maybe they have a good reason, maybe they don't.
But no need to jump to conclusions. I've seen plenty of bs fragile ducktape databases. Some of them exist because because someone wanted to hand-roll everything. And some exist because they were afraid to invent anything and ended up hacking together a bunch of garbage MySQL nodes. Because they didn't want "reinvent the wheel".
I agree my comment may be a knee-jerk reaction, and it is quite a good technical article with clear examples.
I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.
If AWS gets you going definitely go for it but if you're looking for existing tools type 'python speaker diarization' in github.'
Most of what you'll find would require downsampling your audio to 16khz and you'll find a combination of NN based diarizer and hmm based models pre-ML.
One thing to note a lot of the systems will work well for interviews, broadcast media and footage taken from a camera because the audio will tend to be clean.
Film and movies will be a challenge because of the background music being identified as a separate voice. It tends to to confuse it.
Haven't used the cloud based systems with audio the background sounds you tend to find in film movies.
iPAAS' are essentially ESBs remarketed for the cloud for organatisations without the in-house expertise to coordinate building their own integration hub with a combination of tools like Kafka and API management gateways.
Organizations lacking the
* Architectural leadership
* Coordination capabilities
* Engineering resources
* strong comms between teams
tend to go for these solutions to paper over their existing problems. Particularly those around engineering capabilities.
These solutions work great for a while until the edge cases around data arrival and consumption usages resurface.
It's important to note that these solutions shine when it comes to the API discoverability across teams and light data format translations but as soon as that has happened get your data out quick!
Don't be tempted by their in-house solutions that try to solve caching, scaling and the ETL languages provided because they will tend to be half-baked and not as battle tested and will lock you in. The commitment for a large org using these systems is a 5-10 horizon.
By all means check them out and be ready for alot upfront design work to find the sweet spot and alot of engineering effort and frustration (alot).
The poster is holding a line of bash to the standard of code and is illustrating that readability should be the goal and a way of bringing bash commands to a standard of readability for something like a PR. Readability is really there to show _intent_
I would say though that if you are bringing this to the code standards of today then this should really be wrapped up in some kind of unit test (https://github.com/sstephenson/bats )for it to pass the PR. That would make the code a bit more maintainable and can be integrated as a stage in your CI/CD pipeline.
If we do that then the intent would be clarified by the input and the expected output of the test. Then then the code would at least be maintainable and the readability problem becomes less of an issue when it comes to technical debt.
I've done this plenty of times with my teams and its certainly helped.
A software architect that can't code or refuses to has no business architecting the system.
A software architect who doesn't show up to stand ups really shouldn't be dictating a solution to the team who have to build it.
Ideally the software architect should be as hands on as possible with the team leads as the system is built up to the point that the architecture is proven or at least to see if the assumptions around the architecture actually hold up.
Really strictly on teams leads to accurately communicate the architecture before it has been proven to hold up is risky.
I get this issue and I share the frustration but is this really a comment on the article posted?
In this case we seem to be talking about huge military projects like GPS or Drones and that there is a divide between the hardware parts and the software parts that might have made sense in the past when the interfaces between the hardware were simpler and software was mostly internal. But now when software is such a big part you need to involve software people earlier looking at the bigger picture from the beginning. It will also drive how hardware will be developed, i.e. what you can do with software will drive how you should develop your autonomous military drone fleet.
Yea I don't get a huge number of the comments in this thread. The article was pretty specific in what it was talking about, which seems to be not what people are talking about here.
The examples given are all defense projects, but the article does not ever limit its claims to defense projects. Its thesis is stated and re-stated in general terms and it consistently makes broad claims like:
> Today and into the future the best way to ensure that a new system's software architecture supports needed capabilities and also system evolvability is to involve software-knowledgeable personnel--specifically software architects--in the early system decisions about breaking up a system into acquirable parts. At this early stage, many software problems are either caused or avoided, and software architects are needed to ensure the latter.
A similar problem I've seen a few times are the consultants that participate only in the first few months of a multi-year software project as architects, so never get to experience the long-term implications of their design and the mess they prescribe. Not only is this bad for the project, but also breaks the feedback loop for their own learning.
This is something I'm afraid of at my current company.
We just used 3-5 contractors to implement Zuora, Salesforce, and Netsuite in "record time". They're gone now, along with the biz ops person leading the rollout.
As an engineer, I feel like I'm eventually going to have to deal with that flaming pile of shit.
In the world of web development, and devops etc. this is probably very true.
However there are a lots of non-web orientated systems out there; banks, emergency services as an example. I fall into the latter, and have also worked in the former. In those kind of environments we often have an architect (who is indeed quite a talented and hands on coder), but the hairy implementation details of the architect's design tend to be worked out either by dedicated designers, or some other form of implementer (systems administrators/programmers).
It's unfortunate that this off-topic, general comment has derailed what could have been an interesting discussion about specific types of architecture (large HW/SW systems). And I have a feeling that like several other commenters you have some specific architecture work in mind and are incorrectly generalizing based on it.
Taking part in stand-ups for instance would be a complete waste of time for anything except application architecture and it's only relevant in projects which are using Scrum. Even then, it's possible to do the job by being available (at the risk of missing some unofficial or ephemeral info) and taking part in the sprint review.
I prefer involving the team in design decisions and giving them freedom to implement things as they see fit, as long as the architectural requirements are being met.
The usefulness of coding skills decreases the higher one climbs the abstraction ladder. Writing production code for a project is from my POV an anti-pattern unless we're talking application architecture and the architect is only part of that one project, which sounds like title inflation for anything except bigger applications.
In my experience there were two kinds of situations where it was worth it to write code: tackling less glamorous, but useful tasks that the team didn't have time for and solving trickier problems that the team couldn't yet work out.
Otherwise I expect and prefer mature dev teams which can implement on their own, while I am available for guidance if needed and to ensure that the business reqs are correctly translated and their piece of software integrates properly in the platform.
Personally, I found domain knowledge, connections and social skills invaluable for architecture roles. Technical skills are also valuable, but they're table stakes to be honest.
Totally agree. If you don't know to code and but are an architect you pretty much live in permanent la-la land. I bet on paper I could architect a whole company within a few hours. It would look good on paper but in practice it wouldn't work because there are a ton of implementation details that make my design impossible or hard to achieve. Much better to empower the devs to be involved in architecture from the start so they can give real-world input quickly. I have seen it so many times where managers and architects decided on an unsuitable stack because all they do is listen to sales reps and read whitepapers.
Software security architect here. Senior management strongly discourages writing code on my team. But this is because we work at-scale with hundreds of products comprised of thousands of microservices.
I don’t attend product teams’ standups , but our engagements run on the order of weeks to months.
We don’t micromanage development or design choices made by product teams. Yes, we review code and make (strong) suggestions, but we leave service teams free to make choices as long as the risk is low and commensurate with the threat model. It’s our job to guide major decisions and, occasionally, advise on the “least-bad” short-term solution, pointing out where, when, and why a re-architecture will be required in the future. Generally speaking, teams want our feedback on design proposals and concepts. It’s a healthy relationship.
Occasionally we have to force teams to reimplement, or block release, but that’s a very rare occurrence.
Unlike many comments on this thread, service teams leave us with pretty positive feedback. It’s very rare that teams leave negative feedback on our design engagements.
That’s definitely the case when thinking about security in a strictly limited scope. In contrast, my team is holistic — any issue that could affect ICA (integrity, confidentiality, availability) is in our purview.
My team engages in the earliest phases of design. Holistic security is our priority, but not a limiter for our engagements.
Oddly enough, I'm an architect for a company that makes scientific equipment. Mostly hardware rather than software. The software team is its own independent entity.
I'm usually present at every stand-up. Realistically, the engineers tend to be fairly specialized, not multidisciplinary, and are not up to speed on quantitative engineering, so my role in defending and adapting the architecture is continuous throughout the project.
They should also accept most of the blame for any problems, because anything higher level than "my code won't compile because the syntax is wrong" is an architecture problem.
You want to be the architect and me just a lowly coder? Ok, I'll fix my own compile errors, all other problems are on you.
Architects shouldn't be dictating your class structure or code choices. That's a code concern and not of the system. If the system has fundamental issues around performance, scalability, extension, or security, then it's probably on the architect.
This attitude seems to be the default "holier than though" attitude about architects. I think it's more likely that you would encounter pragmatism when the company culture is healthy. Unhealthy company or engineering team and there's no way to tell whether it would be good or bad
Only issue is, a bad dev can still make mistakes that compile and run but fail at the first hint of trouble.
A friend used to say, "every line of code you write, you make at least one, if not more decisions about the architecture of the program" and I totally agree. So I can still see a really good architect being betrayed by a below mediocre engineering team even if the architecture is sound.
> A software architect that can't code or refuses to has no business architecting the system
Well, in construction, you have both architects and civil engineers that never, ever place out rebars or pour the concrete. The architects know how to design the building, while the engineers materialize those designs according to the laws of physics and economy.
So I guess one could ask, why should it be any different in software?
In fact, it's like that in pretty much every other industry. I'm an electrical engineer by trade, and even though I know how to practically do the work myself (since I worked as a electrician apprentice before college), it is something that I never touch - nor have to touch. That's what electricians are for.
> So I guess one could ask, why should it be any different in software?
Since the software world does not generally work this way, we could also ask why the architect/brick layer model should be applied in this world. The decider/implementer model might be a common organizing principle in other industries; that does not mean that it's useful to software.
Brick layers cannot manipulate more than small chunks of a building at a time. Software engineers in contrast can perform the equivalent of turning the Empire State Building on its head.
Software architecture is difficult to get right and too important to leave to one or two senior folks who may or may not have the insight required to make the right decisions.
I agree, but as an architect who does all that, it is also a constant struggle to deal with developers that don't think more than one step ahead and need to be spoon fed concept to prevent them from e.g. writing custom code instead of preparing for reuse.
When you have developers who understand how to generalise and abstract, then the interactions can often be very simple, and deferring to team leads etc. can work just fine.
From that you can get measurements on how long each stage takes and the duration of each transition.
From there you can compare team A and B. The transition times is where the human time cost usually sits.
Just getting the time when a Jira or feature is raised, to the time it is picked up to the time of the first commit to the time of the first test and final build does already give you valuable insight.
The points you raised towards the end can be answered if observability of your CI/CD pipeline is actually in place or at least a place to start a line of inquiry.
Naturally the blockers will be aggregated into some of the values but as you work through the journey, they will start clustering at certain stages and maybe highlight a significant problem that needs to be addressed.
There's a wealth of data being left on the table that can help inform management decisions.