I think it comes down to budget and what you're trying to develop. Developing your own full-blown engine with all the bells and whistles; scene management, physics, animation, ik solving, audio, video, asset importing, networking, scripting, UI, VR and so on is what used to be one of the biggest investments a game developer would make. ID software's engine generated a lot of revenue for for them back in the day - because was cheaper to pay 250k than do it yourself. At the end of the day as a gaming company you're trying to provide a unique experience - and as you say - a custom engine can be a way of doing that - but after you've created the engine, then you've got to create the game on top of that. So I think these generalised engines have satisfied a market need that wants to focus on the game itself rather than engine development.
As mentioned below in the comments - tools are also extremely important. If you're going to support standard tools (e.g. Maya etc.) for asset generation there's more work involved. The entire art pipeline has to be considered. It's a huge job. Making your own editors etc. again lots of work. If you have your own engine - you've probably got your own GUI and asset management system at this point for artists.
So this is a long way of coming around to "should you develop your own game engine?" If your use-case means that it's cheaper to do that than extend Unity or Unreal - and you've got the capital and talent - I guess it makes sense. But if you look at the majority of really successful games these days - they're using 3rd party engines. Even rockstar started with using Renderware - and now they have their internal engine with probably really good reason (super deep pockets and use case). But I guess if you're not Rockstar - and you're trying to make something AAA - it's expensive enough without trying for your own engine. Let's not even get started on making the engine cross-platform.
Of course, if you're making space invaders and know how to talk to the graphics API directly - none of this is required ;-)
Just my IMHO.
Something that i haven't seen mentioned much is that some people just prefer to work on custom engines (i'm willing to bet that the main reason John Carmack made his own engines was because he liked working on his own engines rather than the costs of licensing). I know of at least a case (i wasn't part of it myself) where after a developer decided to switch from their custom engine to Unreal, almost the entire engine team quit.
FWIW personally when judging potential workplaces i bias towards those using custom engines since there is way more room for having an actual impact on the project and i am more likely to learn something new from a custom codebase than the 482th Unreal fork.
The fact that people often feel the need to justify working (almost always in a defensive stance) with a custom engine shows that they simply prefer to work with a custom engine despite other concerns.
Also there isn't really a clear cut distinction between AAA and smaller studios using custom and generic engines - while indies are more likely to use something like Unity, that is mainly because there are way more indies than big teams and many indies do not have the knowledge for a custom engine. But there have been many successful indie and small studio games with custom engines and AAA studios with licensed engines to say that if you're Rockstar go with custom, if you're small go with 3rd party engine.
Honestly i think the people involved are way more important (and likely to affect the choice of an engine) than any budget or size concerns (i'm not even considering the game type as most custom engines are used to make games that could also be made with 3rd party engines and the game type is rarely the driving factor for that choice).
Back when Id started, games middleware still wasn't a thing regarding game engines and it was highly seen as an additional needless cost.
The industry was mostly focused on buying libraries for specific tasks instead.
It was around this time that companies like FMod and Bink started.
Nowadays with budgets and teams that rival Hollywood ones, it is a much higher risk to do everything from scratch, before starting to do the actual game given how middleware has turned the table and is now widely accepted, hence the defensive stance.
My point is that the defensive stance shows that many developers want to work on custom engines despite other factors and that is driven by personal preference rather than any sort of cost/benefit calculation.
And it makes perfect sense and you see it all the time in other fields too with people switching companies and/or start (side)projects so that they can work with specific frameworks, languages, etc.
As mentioned below in the comments - tools are also extremely important. If you're going to support standard tools (e.g. Maya etc.) for asset generation there's more work involved. The entire art pipeline has to be considered. It's a huge job. Making your own editors etc. again lots of work. If you have your own engine - you've probably got your own GUI and asset management system at this point for artists.
So this is a long way of coming around to "should you develop your own game engine?" If your use-case means that it's cheaper to do that than extend Unity or Unreal - and you've got the capital and talent - I guess it makes sense. But if you look at the majority of really successful games these days - they're using 3rd party engines. Even rockstar started with using Renderware - and now they have their internal engine with probably really good reason (super deep pockets and use case). But I guess if you're not Rockstar - and you're trying to make something AAA - it's expensive enough without trying for your own engine. Let's not even get started on making the engine cross-platform.
Of course, if you're making space invaders and know how to talk to the graphics API directly - none of this is required ;-) Just my IMHO.