How do you do testing out of interest? Whenever I have seen dbt used, it is usually data analysts creating new tables on the fly in data warehouse scenarios.

Maybe I am just too used to application developer workflows where models are defined in code and then there are ORMs and schema migration tools to help manage all that.

The pop-up is a little sneaky to be honest. I am quite enjoying the content sifted has been putting out recently so I decided in this instance I would create an account.

I then tried to view the article again and then the pop-up appears again with different wording, letting you know it is actually a paywall, so creating an account isn't enough.

I just used Firefox's reader mode. Half the time it cuts through modal paywalls, if they loaded the content behind.

Yeah can also just inspect, delete a few overlays, and then remove the overflow from the body element and your in.

Did not expect to see Apache Beam mentioned as part of this conversation. Mind me asking what you mean?

Apache beam is the underlying engine we use for fraud detection. Based on proprietary heuristics we can provide the application with a confidence score the user is legit. Its up to the sec/product teams to determine when specific actions need additional verification like one time token via email or captcha. Just two examples.

I started at a company using Shiny for their applications and R as part of their data pipelines.

A huge pain point for us is the packaging system. It is absolutely awful. Packages constantly get overridden so we have to install packages in a specific order. Whenever I have reached out to the community (including prominent members, which have written R books) I have always been told to just use the latest version of all packages and just get on with it, which as anybody knows, isn’t always possible, especially as there are constantly breaking API changes.

I understand R’s history and that in general, it is a lot better than it use to be, but I would only recommend R is used for notebook style work and to keep it well away from production.

We have migrated to Python, which isn’t perfect, but the difference in logging and packaging has been night and day.

I have also found R in production to be a nightmare. On packaging, the renv package seems to be the new way to try to manage things. It’s not perfect but seems to be a step up on what was around before. Have you tried it out at all?

I haven’t, thank you for the suggestion. I will give it a go.

At my old work we would “freeze” CRAN, by downloading a complete dump of everything and setup R to install from that version instead of the online version as a way of version controlling packages.

So instead of defining our app to use version 1.4.5 of a package, we would use “latest version from 3rd of May”.

Same experience here.

A lot of packages/functionality are not available in Python, however.

Did you always have in mind that you wanted to sell it to zoopla/rightmove? What was your initial pain point that caused you to develop your idea?

We had a rough plan of getting into the rental\sales listing game, using our map and search engine as the killer feature. Turns out to make any headway at all in that game you need big bucks as it's entirety marketing driven. We had way more features than this product and made exactly 0 money from it. We were pretty young and commercially incompetant though.

I actually think Jenkins is way too flexible for most use cases. We moved to GitLab CI which isn't perfect, but it provides safety rails/structure/opinions that pretty much provides an answer for everything you want to do, apart from maybe obscure corner cases that might not make sense for a CI/CD tool anyways.

Also you get the close integration of your CI tool and your git repos, which is very nice from a visibility point of view.

Having said that, GitLab is trying to own all parts of the build and deployment process, which from previous HN discussions, is of great annoyance to a lot of people who want to cherry pick what they use GitLab for.

Thanks for using GitLab. We want to be supportive of cherry picking GitLab features. For example we just release CI/CD with GitLab for GitHub https://about.gitlab.com/features/github/

Is there something we can add to GitLab to make it more composable?

Thanks for the question, greatly appreciated. We used GitLab on k8s when we first transitioned, but we found there were a few things we didn't quite like about the GitLab-Omnibus helm setup, so we moved it off the cluster and used the AWS EC2 AMI which was really easy to setup.

We are going to start experimenting with the new cloud native GitLab chart, but it would need to gain some maturity before we use it in production.

Do you know if the new GitLab cloud native helm chart will allow you to turn-off certain things like mattermost and prometheus? That was something that we didn't like about the omnibus chart because it exposed several extra services/ports that we didn't really want to manage/think about at the time.

samm we are indeed making all components that are not core gitlab services optional. You will be able to turn them off with a simple `prometheus.enabled=false`.

Thanks for giving the charts a try in alpha/beta, please pass along any feedback. We'd love to get it!

Thanks for the good work on GitLab, I tell everyone I can how great of an experience it was to use it at my last company. The CI integration was GREAT, the ui is pretty nice and the maintenance overhead was minimal.

Thanks, great to hear that.