And older structures have much less insulation, compared to the present building code, as put forth in the so called International (actually US) Residential Building Code, adopted by most states.
Energy use for heating and cooling is typically much higher in older buildings, over a lifetime of decades.
Older homes are usually well thought out in terms of natural cooling and heating. Being that many of these homes were built in an era where HVAC systems didn't exist, they can stay remarkably cool in the summer, and warm in the winter. They designed the homes to have proper airflow throughout the house (and underneath the house), and of course, shade from trees. So while their energy efficiency rating may be lower -- in how it can retain the heating or cooling put out by the HVAC, in practice, you don't use your AC or Heat as much as you would in a newer home (which for some reason new home builders seem hellbent on eliminating all trees, and placing the entire home in direct sunlight)
My house was built in 1927, and it stays ~75-78 even on a hot summer day, and due to the much thicker materials used, doesn't bottom out past 60 in the winter.
I have never seen a building from that era with foot-thick stone, you have to go back further for that here in Canada.
The century homes here in southern Ontario are double-wall brick with beefy 2x4 framing within the interior wall, then lath and plaster, and no insulation.
My 1930s UK house is foot-thick stone and brick on all walls, even internal. It doesn't have insulation except in the attic because it's one absolutely massive heat-sink mass - by the time it's warmed up from the summer it's already autumn and then it starts emitting the heat usefully instead.
Thermal mass as a tool is so very under utilized. When it's considered, it's often only on a 24hr cycle. As you point out, the 365 day cycle may be even more important. For a modern take (with for realz engineers and measurements even) have a look at Drake Landing community in southern Alberta. https://www.dlsc.ca/
edit: They've discontinued the 'Current Conditions' part of their website but I followed it with some regularity in the early years
I can't argue with your personal experience, but I pay less to heat a 17 year old, 3300 sq. ft. house than I did a 900 sq ft one that was built in 1950.
Insulation matters, especially when you have temperature swings from 100F in the summer to -25F in winter.
I think we need to settle on a description of "older" here. I do not consider a house from 1950 to be from the old generation of builders, but starting to be what was the budding mass building trends. The 1950s featured dirt cheap energy (also ironically, the US relied on renewables more in the 1950s than today), and so there wasn't much thought put into proper insulation because running your AC or heat all day wasn't that costly.
I consider older homes to be built in an era where HVAC either didn't exist or wasn't the norm, so they were designed as such -- thicker materials, proper airflow, and ample use of shade, even considering the house orientation relative to the sunrises and sunsets in the summer to ensure the primary living quarters would be out of direct sunlight the most amount of time.
Energy use for heating and cooling is typically much higher in older buildings, over a lifetime of decades.