Everyone repeat after me. Makefiles are not scary. I can write a shell script. Do I really need to learn grunt/gulp/webpack/npm/rake/fake/maven/gradle/ant and on and on and on?

Probably somebody has released another one in the time it's taken me to write this comment.

I use Rake (or Gulp, or whatever) because then I can use Ruby (or JavaScript, or whatever). Shell plumbing is fine for informal and small-scale stuff, and I make my code conform if somebody down the line (who may be me) wants to get out their duct tape, but the world is more complex than what /bin/sh can see. Shell is the lowest common denominator. Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better. The profusion of tools happens because they iterate on each other to be better. If old tools were sufficient, people would use them because learning new ones is hard.

So, yes, you do need to learn those tools. Or invent a shell that isn't tooth-pullingly difficult to use with a JSON file (and do not say `jq`, I love `jq` as an inspector but it does not step to `JSON.parse` and a working subscript operator). Or change `make` so that a git checkout won't trigger a full rebuild. Lots of baseline, stump-simple things that `make` is just not going to do for you because it's built for a frankly outmoded method of development.

You can use Ruby in Make. I don't know Ruby but here's some Python:

  SHELL = python
  .SHELLFLAGS = -c
  .ONESHELL:
  .DELETE_ON_ERROR:

  foo.txt: bar.json
      import json
      with open('$<') as input, open('$@', 'w') as output:
        output.write(json.load(input)['the_text'])

You can even use different languages for different recipes with per-recipe variable settings.

At that point I think you might as well just use a language-native one.

Great pull, though.

I think you're (understandably) misinformed about what Makefiles do because you've run into some bad ones. The thing they're doing is managing a N-level deep dependency tree in a declarative way. So if A->B->C you can run something to generate C, then B can run, and finally A, and this can all be done in parallel for hundreds of files.

On the individual rule level this is really simple, e.g. just turning a .c file into a .o file, then finally some rule that depends on all *.o being generated creates a program out of them.

The language-native ones are usually much worse. They're easier to use at the outset because they don't make you create this dependency graph, but that also means that they can't run in parallel on your N cores, and they usually suck at incrementally updating the build.

I'm not sure what you mean? How would this allow you to use, say, Python as the language for recipes? Just having Make drop straight into Python kind of defeats the purpose of Make.

The point is that people conflate these two things. They open some random Makefile and see that it's mostly doing stuff with shellscripts, and think "oh I should do this all in Python", and then write some monstrosity that doesn't make a dependency DAG and thus can't run in parallel or be easily understood.

Instead they should have split the actual logic they found in shellscripts in to Python scripts the Makefile invokes.

Is it that hard to learn shell? Why is it so painful? What makes it the "lowest common denominator"? I use it all the time, but I admit at work I am one of the few.

> Expecting everything to at all times be written in and for that lowest common denominator is not reasonable. We're a tool-using species and we refine tools over time to make them better.

This is too vague. What makes a Ruby based build or JS based build a more "refined" tool? It sounds like familiarity is the real issue here.

> If old tools were sufficient, people would use them because learning new ones is hard.

How many people even know Makefiles these days anyway? The "modern" approach seems to be, learn a programming language and then try to do everything inside of it. Some languages are more interested in this cloistered philosophy than others (like JS).

If anything, I think the reason these build tools keep being proliferated is because nobody wants to learn anything more than the bare minimum to "be productive" (which, depending on what you're working on, can be anything from pushing out customer demos for a company that will never sell, to microservices operating at scale). Learning a language and never leaving its paradigms/comforts is easy.

That said, Maven is incredibly suck-tastic.

(You may be a wizard and have a reason to do them, for sure--but that's what writing Maven plugins is for. Or not using Maven. You've got choices.)

https://github.com/ingydotnet/json-bash

    source json.bash
    
    json='{"name":"Jason","friends":["Jimmy","Joe"]}'
    JSON.load "$json"
    joe=$(JSON.get /friends/1)
    JSON.put /friends/2 Jeff
    new_json=$(JSON.dump)

That is blatantly false, but I get the premise behind it. Writing and maintaining complex stuff in bourne or bash is not fun or easy.

I always use bourne until certain level of complexity or awkwardness is reached. It's pretty easy to write and dead simple to troubleshoot.

And yeah, I did say `sh`, because that is what you can practically be expected to have kicking around alongside `make` on a system where I can't just install something worthwhile. If I have them, then there's no reason to write much harder to troubleshoot shell scripts (and, thank you quotebarf, more likely to be incorrect) than to open Pry.

> --show me its arrays

  declare -a

  declare -A

See above

> Show me the basic building-block primitives of software,

  thing() {
     local localVar
     # Yes, it would be nice to have lexical scope as well as 
     # dynamic scope, but...
  }

And besides: if I can install bash, I can install something specifically better for whatever I'm doing instead.

What are these build systems where you need to install Bash? Bash 4 was released in 2009: it's been in every major Linux distribution for at least two major versions, ditto for FreeBSD... heck, even Solaris ships with it now.

And Busybox is commonplace. Systems that include Busybox usually do not include `bash`. And so, for my purposes, if I can specify `bash`, I can also specify, say, Ruby, which--while by no means perfect--makes life much, much easier.

"All of which are clunky, opaque, and harder to work with than any other language I have used in the last decade."

Wrong question. A better one: Is it more clunky, opaque, and harder to work with than every other language that's appeared in the last decade? Because no one seems to agree on what is specifically better.

(And I am what a current sysadmin would be if we did not call ourselves "devops engineers" now.)

Anyway, the core mission of anybody touching the stack is to enable the business to achieve its goals. Nothing, and I mean nothing, more. "Restricting that environment" is appropriate in some environments, and a number of my clients bring me in to help with that. Facilitating developer velocity--and, yes, developers do tend to like me, because I'm good at this while achieving goals around security and uptime--is appropriate in, probably, more. Pays better, too, even if it shouldn't.

1. Design. 2. Discipline.

Where these two values are dispensable long term devops and the new world shine through. I've worked in both worlds and the only mistake is assuming one size fits all.

There were guys like this that were there before and helped everyone along: https://www.computerhope.com/people/don_libes.htm

I'm still not certain what point you were trying to make with "neithere here nor there", however.

I love Bash and use it way more often than I should, but Bash is not a friendly language. Mocking and stubbing functions is really hard to do, which makes tests awkward, even with BATS. And you need to watch the file descriptor to which text is sent to avoid things get globes unexpectedly. And you need to properly scope your vars to avoid unexpected collisions. And everything regarding sourcing and importing dependencies. Etc.

This is an anti-feature. When you need this, your build is too complicated.

Million+ line codebases got errm made with make. In my experience people most sneering at it have far lesser demands on their tool...

The world advances.

How many features could people have implemented or bugs fixed or general progress made instead of obsessing over tooling?

My challenge to you: I want a makefile that has 20 third party dependencies and can be built on osx, linux, and windows.

I can do this within an hour with gradle, ant, or maven. The ecosystem doesn't exist for this in make, and anything I could come up with to make it possible would end up being a tool that would look like automake and the monstrosity that it entails.

Ugg... autotools. Yeah, lets use what feel like build tools created way back in the 60's or something. Sure they "work" for some value of "work" but oh boy are they ugly and nasty. Good luck hiring anybody under the age of 40 who is gonna be willing to work on such a clunky old tool.

There is a reason why the world is moving to build systems that replace the Make toolchain.

For something that isn't JVM-based?

The industry has moved to language-specific build tools and ecosystems, and the article and is complaining about this in part, no?

and you've already made sure they are running the same JRE version, right?

And can it do it all incrementally in parallel too please because enterprise shops tend to have a ton of code

Oh and it would be really nice if you could make it so if someone else somewhere in the org has compiled that thing then could we just use their computed binary to save the time compiling locally.

THANKS!

;)

> Everyone repeat after me.

I don't mean to pick on you specifically here because this attitude comes up a lot. In short, a lot of people are doing a thing, a thing that you aren't familiar with, and your gut reaction is to say "everyone: stop doing that, and do what I say!".

Wait a second and consider that maybe there is a reason why this incredibly large number of developers are using these tools. That perhaps they evaluated various different options and decided that what they are using is more suitable than make. Maybe you could find out.

Here we have another fundemental "problem" between dev and ops.

The inherent friction because of different areas of concern.

Dev's want to build fast and create new features, but sadly (even with the whole devops notion) are usually not thinking about viability in production.

Ops people need to keep thing stable, something which is sadly undervalued a lot of times in companies.

The rest of Make (which is 95% of what's in a Makefile) is its own language, which is actually not too bad for what it's intended for (munging around filenames), and has the flavor of a functional language.

https://maven.apache.org/pom.html#Dependency_Version_Require...

Don't forget to go the extra mile with your dependency versioning to ensure you actually get repeatable builds.

Have you considered that perhaps the previous commentor is familiar with the other tools? Or perhaps that the large number of developers have streamlined their particular workflows for their particular use case and have not considered the flexibility needed for other cases? Or perhaps that there is a cost associated simply with having 47 different build systems?

Its seems more likely that they’re frustrated about the sheer amount to learn and they probably know make quite well.

But there are real advantages to these tools. There’s a lot of vanity libraries out there but not many vanity build tools.

What we want to do is slurp some text files up, apply some transformations to them, glob them together, run them through the minifier, and dump them into a final output. This is exactly what a decent makefile would fit well for. Instead, I got this, because apparently nobody wants to use anything that isn't the hot new way to do things, and so very few people have even had enough exposure to know that there are tested tools for these kinds of things that existed. The last time I was using make beyond trivial uses was a decade ago in college (incidentally, I actually did build jars with make, easier than fighting Eclipse...). But just knowing that something exists is more than half the battle, although the problem then is that it's frustrating as all hell watching the same ideas cycle round and round and round every couple years.

And yet, those tools fill a need that would be very hard to replicate with the toolchains that came before it. Good luck doing half of what gulp / webpack do from, say, a Makefile.

It's just that this argument keeps getting used for every single new "reinvented wheel" (to borrow from the GP). Sometimes the argument is as strong as "it couldn't be done any other way" and sometimes it's as weak as "this one is just incrementally better," but it feels a little like crying wolf.

Was it really "very hard" to make the old wheel do what you needed, or perhaps somehow extend it or add a library, or was it just far more fun and exciting to build something from scratch?

I generally don't mind a proliferation of tools, except when they start to break or conflict with each other, which is, I believe the GP's main concern, and at least tangentially related to the article.

I've had success in several JS projects using make in combination with the shell version of a lot of these plugins and npm scripts.

There is value in both approaches.

If you're building in Elixir, I hope you're using Mix and not a Makefile. If you're building in Rust, I hope you're using Cargo, or some other Rust specific build tool.

And Makefiles do get stupid complicated when you need things portable and to work on Linux, Mac and FreeBSD; or allow them to have optional dependencies. That's why we have autoconf and all that ./configure && make && make install craziness.

For instance Make works based on timestamps and therefore works very poorly together with git. Switch to another branch and you can get weird effects based on what files were updated and not and often trigger needless rebuilds. And everyone uses git these days.

GNU Make, just using hashes instead of timestamps, would be a huge step forward.

Is ninja that tool?

Sounds like you're describing make on a system with ccashe installed. Hashing incurs a significant performance hit. The first build with ccashe is 20% slower than building without it[1]. Your modern make would likely be slower for people who just build something from source once and aren't doing incremental development.

https://ccache.samba.org/performance.html

Did you consider that it could be other things with ccache that makes it slow? E.g., the need to fully expand C headers and turn it into a text stream (NOT needed for a build tool).

Everything with git is lightning fast including operations that require a full directory tree diff / hash.

There are modern tools that work for many things. Gradle supports native compilation these days as well as any JVM language. Bazel supports compiling many kinds of languages, and there's still scons.

As to the cost of hashing: "git status" runs on 0.009s and requires a full diff of my working directory?..

it is using time stamp to check whether to hash to check for changes.

Writing correct Makefiles is also really hard, and good luck debugging them. Make is simple and beautiful for small, self-contained projects without many dependencies, but it does not scale well (and even then it's tricky, see [0][1]).

Having 47 different build tools with their own flaws is certainly bad, but they exist because of Make's shortcomings. Just saying "make is fine, use it" won't fix anything.

[0] https://cr.yp.to/redo/honest-script.html

[1] https://cr.yp.to/redo/honest-nonfile.html

* https://news.ycombinator.com/item?id=15060146

* https://news.ycombinator.com/item?id=15060146

* https://news.ycombinator.com/item?id=15044438

A lot of the Rake tasks I've encountered in my career would have been easier to write in bash. Not a majority, but a sizable minority. I suspect that in many cases, the gain was that the authors were more comfortable in Ruby than in bash.

(Just picking Python as an example because of the recent xkcd. The same is true for everything else, e.g. a Windows computer will rarely contain exactly one make.exe, it's usually either none or a whole bunch of them)

Writing good build systems is genuinely hard, and I think make is not a good build system.

https://gist.github.com/anonymous/d885c3bb66a319d22f9d60c1ef...

make is so bad you need automake to manage it.

there are much better tools. sadly, nothing LCD (least common denominator) so as to gain wide traction.

That said, for anyone distributing software, shame on them for not packaging their custom build so as to be runnable via ‘make all’ (just using make to drive everything else).

Autotools wasn't invented to overcome deficiencies in make, but deficiencies in C portability across Unix flavors.

Those deficiencies are greatly diminished today, both by POSIX standardization, and there being fewer viable surviving Unix variants that anyone cares to build for.

https://queue.acm.org/detail.cfm?id=2349257

[1] https://en.wikipedia.org/wiki/Pkg-config

I've seen plenty of messes using scons and ant, more so than I've seen with make. Make is a solid tool.

New way: we have a dozen tools, they collectively do less than the old one and integrating them is a full time job for someone since any upgrade breaks something. But we all get to put the names of all of these things on our CVs! And that’s what’s important.

I wish I was kidding.

Devs see the world as one where velocity and progress is among the most important, while 'sysadmins' (a term no longer used by companies and recruiters, unfortunately) have to worry about keeping the applications actually up so they can be used.

DevOps just seemed to have swept the sysadmin under the rug under some pretty words about breaking down barriers. It feels more that devs broke down the wall sometimes. The amount of infrastructure chaos and mess and confusion I see today (as a contractor bouncing around different places) seems higher than traditional infrastructure shops of 10+ years ago.

That's because only programmers get Peter Principled into the manager roles[1] that decide on whom to hire and how to run "Devops".

[1] or become startup founders

Also, a ton of people use abstraction as an excuse to not learn a body of knowledge that contains fundementals (especially on the systems side).

In comparison to network engineering for instance, where having (quite) low level knowledge of how protocols work and interact with each other is vital.

This is just not true, otherwise everyone would need to know ASM or be able to easily contract someone who does.

Pretty much. I know several admins who appear to be joining a growing pool of luddites who rail against anything new. They're particularly butt-mad about anyone drawing more salary than them. "DevOps" is their favored totem to direct their ire at.

I used to try and convince them otherwise, but it turned out to be a completely futile waste of time. At the end of the day, persistent FUD just means a more lucrative job market for the engineers who are pragmatic and fearless enough to give an honest try at making the newer paradigms work for their employers.

Every build system is like Make, but more friendly for their language (IIRC Make was originally for compiling C and C++). Make just so happened to become generic enough to build damn nearly anything and also get bundled into most Linux distros.

I think the author is arguing that having to install a shit ton of dependencies to use some other Make-like build system is garbage. That’s true in some cases. But I wouldn’t want to use a Makefile for packaging Node; npm is great for that and understands how Node works.