Gave it some existing python to modify, which it should be good at, at least I would expect it to be.
The first task seemed like it was heading the right way, but it just didn't finish up, left empty function stubs which didn't compile. Definitely tuned to be very lazy.
The second one was five files, tasked to fix a specific thing. It found the right function but changed unrelated parts of it so it used nonexistant values and in effect, broke it entirely.
I don't think I get the hype either tbh. Maybe the file upload is borked on their chat demo or just a classic case of long context IQ loss.
Sorry you had that experience. I used 3.5 Sonnet last night to merge two Python files and remove a "fake" loop I had made to simulate an activity and replace it with an actual loop to do what I wanted. It not only got it right on the first try, but saved tokens by telling me in the comments where to get the boilerplate and paste it into its generated code. I was impressed, at least.
The first task seemed like it was heading the right way, but it just didn't finish up, left empty function stubs which didn't compile. Definitely tuned to be very lazy.
The second one was five files, tasked to fix a specific thing. It found the right function but changed unrelated parts of it so it used nonexistant values and in effect, broke it entirely.
I don't think I get the hype either tbh. Maybe the file upload is borked on their chat demo or just a classic case of long context IQ loss.