Having spent the last 9 1/2 years working on systems for property and casualty insurers, I have to say that I don't really agree with everything.

I generally agree with the point about designing your architecture to accommodate whatever gets thrown at you, but doing that is hard. Immensely hard. Nothing is concrete, everything is some extensible meta-problem, where instead of writing code to do X, you have to think "how can I make it so my customers can do X, but can also change it to do Y instead." And of course, they want it to be easy to make it do Y instead. Do you just make them write custom code? Do you enumerate all possible options in a declarative fashion and let them choose between them? It's never easy.

The bits about finding structure in what's unstructured, though, I find to often be exactly the wrong mentality, though it's usually the mindset all engineers start with. The best approach in my experience is usually to just embrace the fact that everything is arbitrary, and recognize that you're just going to be writing a lot of if statements (or giving your customers ways to write if statements); otherwise, you'll inevitably A) run into some situation your nicely-structured algorithms can't handle and B) be endlessly frustrated by the exceptions. The processes being modeled by enterprise software are fundamentally irrational, illogical, inconsistent, and arbitrary. Accepting that, and building systems that can handle that, are hard things for engineers to do, since the tendency is to always try to find patterns and order and simpler, more general algorithms.

Not all problems in enterprise software fall into that category, so it's critical to be able to identify which sort of problem you have: is it one that's structured and makes sense? Then write code accordingly. Is it modeling some pre-existing human process? Then expect that the crazy exceptions you've heard about so far only comprise 1% of all the crazy exceptions out there, that you don't even know how to categorize what those exceptions could be, that there's no real underlying order, and that your job is to write the system in such a way that it can handle the fact that the process is arbitrary.

My first job out of college was with a consultancy that specialized in payroll integration projects. We were a partner with ADP (largest payroll provider in the country), and typically they sent us the clients with the trickier payroll processes, so know I'm speaking from experience.

Long story short, I agree with what you're saying, but you make it sound like it's really simple.

It's not.

Creating a single payroll solution to handle one company is very easy. Creating a single payroll solution to handle many companies is also quite doable.

But once you get to the scale of ADP, it's impossible to handle everyone elegantly. Between union contracts and non-standard practices, there are (what felt like) an infinite number of things a customer can throw at you, with one rule wackier than the next.

It's damn hard to write a generic payroll solution that handles most companies. Damn hard.

Basically, if you think it's simple, you haven't thought about it.

Not thinking about it is not the problem. A lack of domain knowledge is. If you don't know how payrolling actually works in practice, no amount of thinking will make you come up with the right solution. The real pitfall here, and the downfall of many startups, is thinking you can understand the domain merely by thinking about it. Engineers can usually think pretty well and can solve many problems in that way. It's crucial to realize not everything can be solved by thinking about it.

I seems like the general system is something like:

A) A set of tools to talk with external systems B) A set of tools for manipulating the internal representation of payroll data C) A GUI for interacting with / managing the system.

When their internal process is simple managing the payroll system should be simple. When their internal process is complex you get fat consulting checks. Which brings up a great point, it seems like large companies that simplify their internal systems could avoid writing a lot of consulting checks, so I suspect they must be out there and you just don't hear about.

You make two assumptions:

1) There is "one" way of representing payroll data. There's not. When you get a none trivial amount of clients, all their edge cases put together destroy any ability to fit the data into one conceptual model elegantly.

2) Companies have control over payroll rules. Large companies often can't simplify their systems due to contracts or the subsequent employee mutiny. Unless you want to get sued, lose the lawsuit, and pay out the nose (which I've seen happen), you can't just change how you do payroll on a dime.

Not to mention, there's a ton of disfunction too. Payroll managers aren't always the brightest tools in the shed. One project I worked on lead to the payroll manager getting fired over "bad stuff" I uncovered. And then as I uncovered more stuff his replacement was let go too.

...payroll can be a total clusterfuck. Avoid at all cost.

1) The only conceptual model I think needs to apply is the how things are taxed. Granted some organizations have edge cases in how they pay taxes (or how they try to hide things) but I think innate complexity dominates unique edge cases when it comes to the tax code.

2) I would argue that much of this complexity shows up because many companies negotiate while ignoring the cost of adding complexity. Company wants to minimize cost, workers want to maximize benefits, complexity is a dead weight that benefits nobody. Once things have been negotiated it's hard to change, but during negotiation they is no need to end up with yet another edge case.

PS: I don't think organizations end up spending 100+ million on their payroll systems because it's easy. I just think being proactive lets some companies minimize those costs. EX: Is Google's payroll system ugly?

Any simplified payroll system must result in no employees getting paycuts. That is not as easy as it sounds.

I totally agree. but, It is not impossible. The problem with Payroll is that it is like Security. When everything is working, no one notices its presence. Any issues are incredibly important to fix.

That said, there are ways to develop payroll software that keeps extension possibilities and accommodate the variations without disrupting the engine.

I came here to say the same thing, and you'd already said it.

I too love problems like this. A properly designed system like this one will account for every possible oddball situation you can come up with -- and if it can't support an especially weird solution out-of-the-box, it will be easily extensible such that it can.

Too many people code to the goal and don't try to model the underlying logical concepts.

> excellent developers

For some value of excellent. Simple, general rules, that are well understood, are better than complexity, even complexity your excellent developer can manage.

How about a developer, who says no to their customer, and comes up with something simpler, and better.

I just submitted this:

http://news.ycombinator.com/item?id=3245550

And declare that progress depends on the unreasonable person.

Are you paving a goat track?

> How can I design my architecture to accommodate just about anything my customer throws at me?

Poor attempts at this will result in very complex systems with unnecessarily many layers of indirection which still can't accommodate to the most peculiar business requirements thrown at them. It's like having the worst of both worlds, overengineering and dirty hacks.