Some people really don't get programming while others do.
When I studied at university many years ago, my course had a reputation for being tough. Before the course started properly, there was a three week intensive Java course with an exam at the end. They suggested that if you didn't pass the exam, then it probably wasn't the subject for you. A couple of people failed that exam and continued with the rest of the year long course anyway. Those people did struggle and I don't think any of them passed.
Most of the difference seem to be in motivation.