Interesting post, but this perspective seems to be the main focus, like all the time. I find this statement to be completely wrong usage of AI:

“This is especially noteworthy because I don’t actually know Python. Yes, with 25+ years of software development experience, I could probably write a few lines of working Python code if pressed — but I don’t truly know the language. I lack the muscle memory and intimate knowledge of its conventions and best practices.”

You should not use AI to just “do” the hard job, since as many have mentioned, it does it poorly and sloppy. Use AI to quickly learn the advantages and disadvantages of the language, then you do not have to navigate through documentation to learn everything, just validate what the AI outputs. All is contextual, and since you know what you want in high level, use AI to help you understand the language.

This costs speed yes, but I have more control and gain knowledge about the language I chose.

I agree 100%, but in this very specific case, I really just wanted a working one-off solution that I'm not going to spend much time on going forward, AND I wanted to use it as an excuse to see how far I can go with AI tooling in a tech stack I don't know.

That being said, using AI as a teacher can be a wonderful experience. For us seniors, but also and probably more importantly, for eager and non-lazy juniors.

I have one such junior on my team who currently speed-runs through the craft because he uses AI to explain EVERYTHING to him: What is this pattern? Why should I use it? What are the downsides? And so on.

Of course I also still tutor him, as this is a main part of my job, but the availability of an AI that knows so much and always has time for him and never gets tired etc is just fantastic.

Excellent insight, and that explains a lot of your decisions. Your junior example is a prime example of why AI can be such an awesome tool, just used correctly. Just awesome!

This is so on point IMO, I feel like there is no better time than to learn more low level languages than now. Since the hype will in the end resolve around insanely technical people carrying all the major weight.

Where do you host your model? I am looking around on where I can deploy one without ruining me financially.

The easy answer here would be either pure CPU at Hetzner (e.g. a 24 core i9 with 64GB RAM for €84/month) or GPU at lambdalabs (starting at $360/month). Or maybe vast.ai, if you find a trustworthy offering with good uptime.

Though GPU workloads are still a point where building your own server and running it from your basement or putting it in colo can be very attractive.

You could easily host your model on https://beam.cloud (I'm a founder). You just add a decorator to your existing Python code:

    from beam import App, Runtime

    app = App(name="gpu-app", runtime=Runtime(gpu="T4"))

    @app.rest_api()
    def inference():
        print("This is running on a GPU")

Then run beam deploy {your-app}.py and boom, it's running on the cloud

An A10G for 1200$ per month will ruin me financially

I think the Beam website should be a lot clearer about how things work[0], but I think Beam is offering to bill you for your actual usage, in a serverless fashion. So, unless you're continuously running computations for the entire month, it won't cost $1200/mo.

If it works the way I think it does, it sounds appealing, but the GPUs also feel a bit small. The A10G only has 24GB of VRAM. They say they're planning to add an A100 option, but... only the 40GB model? Nvidia has offered an 80GB A100 for several years now, which seems like it would be far more useful for pushing the limits of today's 70B+ parameter models. Quantization can get a 70B parameter model running on less VRAM, but it's definitely a trade-off, and I'm not sure how the training side of things works with regards to quantized models.

Beam's focus on Python apps makes a lot of sense, but what if I want to run `llama.cpp`?

Anyways, Beam is obviously a very small team, so they can't solve every problem for every person.

[0]: what is the "time to idle" for serverless functions? is it instant? "Pay for what you use, down to the second" sounds good in theory, but AWS also uses per-second billing on tons of stuff... EC2 instances don't just stop billing you when they go idle, though, you have to manually shut them down and start them up. So, making the lifecycle clearer would be great. Even a quick example of how you would be billed might be helpful.

why did you decide to make such a bad pitch for your product like this?

I found gpu-mart.com but haven't tested yet.

An A4000 for 139$ x 12 is not terrible

Last year I did invest in a dual RTX 3090 Ryzen self-build tower. It runs fairly cool in the basement. So I literally self-host. I am confident that I have or soon will reach the cheaper to self host point of the cost curve, particularly as the two GPUs see very consistent use.

what are you consistently using it for?

I use Runpod, an A4000 is $0.31/hr

That’s really interesting, what kind of AI implementation have you decided to work with? I am making a similar project but more of an medieval approach. I went with behavior trees since it felt more appropriate and scalable for such a task

I went with a utility approach as I felt it would incur nice emergent behaviour.

Every so often, citizens and companies (I call the abstraction a "Soul") look through everything they can do and pick the on with the highest utility score.

If they're hungry, they'll get food. If it's the time of day where they should work, they'll go to work, except if they're too hungry!

I agree, it feels like coding is starting to become somewhat easier since AI can both explain and generate code snippets okay, and is improving over time. So the good old days of just creating an innovative program without major human interactions has passed.

Taking the customers requests and discussing into an suitable feature, without totally shooting down the idea and trying to explain why it won’t work is some of the harder parts of being a programmer these days, at least in my experience.