There's a hard limit on how frequently you can run trains, because you need to maintain safe separation distances. Try to pack too many trains into the timetable and the whole line is just one tiny error away from grinding to a complete halt, or worse. There are some tricks to pack in slightly more trains, but they generally require expensive and disruptive signalling upgrades. For commuter lines with lots of closely-spaced stations, the most effective way of increasing train throughput is to reduce dwell time at stations and ensure punctual departure, hence the often rather brisk attitudes at Japanese stations during rush hour.

There are several limits on the length of trains, but the primary one is platform length. It's no use running a 7 car train if all of your platforms are 6 cars long - anything you might gain in capacity per train is wiped out by increased dwell time. You can extend platforms, but it's expensive, disruptive and only works if you have sufficient space at all (or practically all) of the stations on the route.

Japan can't really justify major rail investment, because passenger demand has been steadily declining for decades. Peak-time trains get progressively less busy every year, simply because there are fewer commuters every year.

> higher-passenger-capacity trains

Length is limited by platform length and width is limited by tunnel loading gauge and platform sizes. To increase the platform length, you have to do it at most if not all stations. Crowded stations are usually in desirable areas which make it harder to dig or acquire land.

> run more trains

For lines at capacity, I believe it's usually limited by trains dwell time. Longer and bigger trains take longer for people to safely board. To improve this on the train side, you can have more doors and bigger doors. Station platforms also need to be bigger, have more stairs, bigger walkways, etc. Longer trains also make it tougher for train drivers and station staff to open and close the doors safely.

Tough but not impossible problems but many solutions contain trade offs. Only surefire way is to build another line but that costs tons of money.

I believe it's run absolutely maxed out during rush hour, like 12+ cars per train with 2-3 minute headway

It's just humans being humans. NYC has the same thing during rush hour for certain stations. A train might be 3mins away but people crush themselves into a subway car while I just wait for the next train which is pretty empty.

Often you are limited by dwell time + track separation.