FWIW, a tokenizer is not really necessary. It (arguably) makes things cleaner / easier, but you can also just write a parser that operates directly on a byte stream, if you feel so inclined, and it turns out fine.
Yes I agree. The article actually says the purpose is
> To separate the logic that handles a language's microsyntax from the logic that handles regular syntax.
But everything is just syntax. This distinction between microsyntax and regular syntax is really just for convenience; and sometimes it's not that much convenience.
And isn’t this developer convenience at the cost of performance in a piece of software where performance is very important?
Today as my “end of the week fun experiment” I replaced Babel with esbuild on a years old 50k sloc production web application. It took 30 mins and builds now take 500ms instead of 63,000. Not quite apples to apples yet. But it’s such a compelling change that I can’t ignore it. I’m excited to see how Babel steps it up in the years ahead.
This is way more in depth of an evaluator than I've ever written, but it's a fun project. I'm a largely self taught dev and build an evaluator was an illuminating project for me. Mine was just a math parser but this is a cool project!
