It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends.
I can't stress this enough; claiming anything is "vastly superior", especially with no examples, isn't useful - things aren't this black and white in reality. Most tech-choices like this are somewhere between trade-offs, preferences, ignorance or wrongly held opinions about something being vastly superior > something else.
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing.
You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics.
I cannot stress this enough; you are flat wrong if you think Postgres is appropriate to use for queueing.
I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient.
If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations don't require the best tool for the job, and in that case, you are well served to use a tool you already have.
If you are using an abstraction layer over your queues (you almost certainly are) then you should be able to easily change to a different queue implementation later, should the need arise.
Except that PG and other RDMS don't offer "queue" commands in driver because the queue is built on top of other basic commands, when you use a system designed for that you can leverage the API offered by the driver directly.
So if my language doesn't have a library like Que ( no thanks using Ruby ) I can't do anything with PG whereas using Redis or any other solution you can use "subscribe" / "publish" ect ...
I'm not missing the point, I'm discounting the point, because the point that adding technology is additional complexity, and complexity is bad is a nonsense argument that actually stems from a fear of new things.
This has nothing to do with getting shit done, and everything to do with people who don't want to learn new things because they're afraid of not being smart enough to make the new thing work.
Every different technology added to an infrastructure adds potential failure points, raises the DevOps burden, and adds surface area subject to bugs and security attacks.
If a PG-based solution meets the needs then this reduces complexity in the large. And this has nothing to do with learning or not learning.
It only adds the burden to DevOps who can't figure out how to monitor/maintain a system, which is not rocket science, and if you're a DevOps person who doesn't know how to handle such a mainstream technology as, say, ZeroMQ or RabbitMQ or even Redis, then maybe you shouldn't be in the position you're in.
The only argument that doesn't hold up is "use one tool for everything". This ideology is for lazy people who want to anchor themselves in a period of time, and refuse to learn new things.
Nobody is saying use one tool for everything. In fact the argument is why not use PG until you grow out of it (most people won’t).
Agree on devops. Of course.
There’s still the security surface area argument and the overall complexity argument—complexity of the system of systems is NOT reduced by adding technologies.
By way of analogy, using military aircraft, take the F35 vs. the A-10. The F35 is insanely more complex than the nearly indestructible A10 in large part due to the massive number of subsystems, software, sensors, etc. Sure the F35 way more capable (this is NOT a head to head argument about the military capabilities or said aircraft) than the A10, but it also has orders of magnitude more failure modes. More parts. More complexity. More opportunities for failure. This is engineering fact.
People are literally saying to default to using PG for "everything". That's the attitude I'm arguing against.
What's frustrating about this conversation is that I'm literally, right now, supporting two different queueing systems based on PG and Redis, so I get on a very real level, the tradeoffs. I know in great detail the problems that come up, but HN is not conducive to talking at that level of detail. At this point every comment I make is flagged and downvoted, so why would I pay time into a system that has clearly decided my opinion isn't relevant?
I mean i think it’s mostly the blunt, binary right/wrong nature of your posts getting you this reaction. For most folks they’ve moved past that and almost all of us have non perfect world systems where we’ve made trade offs. It’s just not my (and I guess a lit of folks) reality to be so black/white on this.
Of course there are trade offs, of course we live in imperfect situations where we have to deal with shit that isn't ideal, but sitting around pretending like everything is alright is just lying to ourselves.
You're stuck in PG because your DevOps team is a bunch of incompetent dolts, so yeah, make the best out of the bad situation and go ahead and use something like Que.
Just don't develop Stockholm Syndrome while doing it.
People in this thread are protecting their own egos by expressing how great PG is as a message broker, but it's harmful to the industry to let that kind of attitude perpetuate.
But what about "box" products? We can't force clients have dedicated DevOps just to support our product. Less is better in this case.
To have same code in Cloud and offline installs is even better.
An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong.
Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which should also be blatantly obvious. You're applying some unstated subjective situation in your head, then dumping out something you've heard / done before. LOL.
Also "if you've ever used both" - I have, which is why I know. It's gray; sometimes you should, sometimes you shouldn't. You're assuming I haven't as I have differing point of view to you.
No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.
LOL; what problem domain? We've not talked about any specific situation - which is why you're wrong; you assume far too much. I was attempting to enlighten you a little, but after a few redeliveries, it seems it's gone to DLQ.
Earlier you said something about making excuses for learning and now, claim something subjective is 100% true and blatently obvious -- and that could block your own learning.
Like many things in this space there are a number of good ways - each with minor trade offs
Others posters have detailed those trades very clearly.
So now we can all be more educated about which one of the six good ways we'd choose.
I can't stress this enough; claiming anything is "vastly superior", especially with no examples, isn't useful - things aren't this black and white in reality. Most tech-choices like this are somewhere between trade-offs, preferences, ignorance or wrongly held opinions about something being vastly superior > something else.