For straight up modding: definitely the Xbox. The 007 and Mechwarrior bugs blew everything wide open, and the fact that it was just a PC with real (upgradeable!) storage spawned projects like XBMC, now known as Kodi: https://en.wikipedia.org/wiki/Kodi_(software)
And also piracy was rampant, but not the Swapmagic or Modchip kind. You could just upgrade the drive, _backup_ your games on there, and play 'em all of the drive.
The Wii and 3DS are also suuuuper open and hackable though. The homebrew scenes on both are incredibly impressive, not to mention the whole ecosystem of full blown launchers and shells and stuff. (Which, now that I think about it, was also a big deal on Xbox.)
On the XBOX, I wish PostmarketOS supported it. I know, x86, not x86_64, but it's still a nice platform to have. With the 128MB addon, Alpine/Linux/PmOS can do tons of things with the forked dillo (light HTTP/S | Gopher | Gemini client, a musis/video player with MPV, light office with Abiword/Gnumeric, a rescue system in case of something bad happens on the main PC, retrogaming with emulators, ScummVM (it will work with tinyGL)...
It was def the dreamcast - the first model didn't require any hardware, just a burned CD-ROM. It's demise and Segas departure from consoles is blamed on the amount of piracy. A real shame, because it had some great games
One thing I was found interesting about Dreamcast piracy was that everyone was burning them onto 700 MB CD-Rs. But the retail games were actually pressed onto 1GB GD-ROMs: https://en.wikipedia.org/wiki/GD-ROM
But for games that took advantage of the extra 300 MBs, pirates had to use all these tricks to get the game down to a CD-R size. They'd compress assets, compress or sometimes rip out the FMVs...I think they might have even split some games across multiple CDs.
That's why DRM cracks me up, the pirates will always figure a way around it one way or the other. (Especially in today's day and age where the live service model is so effective. I'd weep for the AAA single-player game, but I can't remember the last one I played and enjoyed. They've been dead for a long time. Long live the indie single-player game.)