Better said than practiced. I recently bought a Supermicro MBD-M11SDV-8CT-LN4F-O (AMD EPYC 3201 SoC). There are 5 NICs--4 Intel and 1 Realtek. The Realtek is set of by itself and obviously dedicated to the BMC controller.
However, reading around I learned that if the controller doesn't succeed in getting a DHCP address on the Realtek interface (unplugged, or apparently in Supermicro's case just a failed DHCP server), it will automatically switch to sharing the first Intel NIC.
You can disable it in the BIOS, but I've had problems in the past where I thought I disabled IPMI on the LAN but either didn't do it correctly or it reverted itself. It doesn't help that there are a ton of ambiguously named, poorly documented options.
To be on the safe side I've plugged both the Realtek and #1 Intel jack. I usually prefer to use the RS-232 serial port for management as it's more fool-proof, but unfortunately this MB doesn't have one. I could install one, but it doesn't make me any more confident that the BMC isn't (now or in the future) on the network.
I understand how useful these BMCs can be, especially for Windows machines, but they're so extremely dangerous. It sucks that they're so ubiquitous, and it sucks that SuperMicro's BMC seems to be the worst of them all in terms of security and bugginess. Unfortunately, a SuperMicro motherboard and case is the only way to get a low-power AMD EPYC server into a small, 1U, short-depth form factor. There are a few more options for Intel, but SuperMicro is still your best bet for do-it-yourself server hardware.
I really love my three PC Engines APUs. Simple serial connection, coreboot firmware, and other than AMD's PSP chip and perhaps the Intel NIC controller, no hidden vectors for remote access. And it has an RS-232 serial port, which because there's no GPU (or at least no video port) both the BIOS and most BSD and Linux distros will automatically use by default.
It'd be awesome if PC Engines built something using EPYC or even Ryzen, stripping everything down like they did for the APU and ALIX. But their niche is low-powered, passively cooled x86 devices using older, more reliable, cheaper, well-documented SoCs. Soekris tried to go upmarket with the net6501 and ended up folding. (They still sell custom DAC hardware from their EU outfit.) There's not enough of a market to make it worthwhile to take the risk. Lots of people love the idea in theory, but what people end up doing is sticking with traditional server suppliers, or in the home, hobbyist, and small business market just buying one of the many faster small form factor boxes that have flooded the market. The problem with those alternatives is that the boards are packed with too much crap, they're too diverse, and every model and family of models short-lived.
In tiny deployments, use a cheap dumb gigabit switch to connect up all the BMCs to your firewall. Give them a separate DHCP assigned subnet and allow access to it only from your management machine.
In larger deployments, you should have separate management networks anyway.
It's not that SuperMicro has the worst BMC. It's that all BMCs/IPMI/iLO/iDRAC are terrible at security, so you need to treat them just like console ports,
I think the large problem alluded to (if true), is that if DHCP fails it may switch which NIC it shares with as well. That problematic for many reasons, and many of them are still problems if you separate it to a different network.
You generally don't expect a DHCP server going down somewhere, possibly in a side-network where it may not be noticed, to cause a management system to open up on a server's publicly connected NIC when it reboots (man, I hope it requires a reboot!). That renders all your careful segregation of your control network completely useless with one service failure and a reboot. Sucks to be you if that DHCP server is on the same circuit as some other servers and they all power cycle at once, because I bet the BMC tries to DHCP an address before the DHCP server boots entirely.
To me, it seems like a very poor default behavior. I can see the benefit if you know what you're doing and want to enable it though.
However, reading around I learned that if the controller doesn't succeed in getting a DHCP address on the Realtek interface (unplugged, or apparently in Supermicro's case just a failed DHCP server), it will automatically switch to sharing the first Intel NIC.
You can disable it in the BIOS, but I've had problems in the past where I thought I disabled IPMI on the LAN but either didn't do it correctly or it reverted itself. It doesn't help that there are a ton of ambiguously named, poorly documented options.
To be on the safe side I've plugged both the Realtek and #1 Intel jack. I usually prefer to use the RS-232 serial port for management as it's more fool-proof, but unfortunately this MB doesn't have one. I could install one, but it doesn't make me any more confident that the BMC isn't (now or in the future) on the network.
I understand how useful these BMCs can be, especially for Windows machines, but they're so extremely dangerous. It sucks that they're so ubiquitous, and it sucks that SuperMicro's BMC seems to be the worst of them all in terms of security and bugginess. Unfortunately, a SuperMicro motherboard and case is the only way to get a low-power AMD EPYC server into a small, 1U, short-depth form factor. There are a few more options for Intel, but SuperMicro is still your best bet for do-it-yourself server hardware.
I really love my three PC Engines APUs. Simple serial connection, coreboot firmware, and other than AMD's PSP chip and perhaps the Intel NIC controller, no hidden vectors for remote access. And it has an RS-232 serial port, which because there's no GPU (or at least no video port) both the BIOS and most BSD and Linux distros will automatically use by default.
It'd be awesome if PC Engines built something using EPYC or even Ryzen, stripping everything down like they did for the APU and ALIX. But their niche is low-powered, passively cooled x86 devices using older, more reliable, cheaper, well-documented SoCs. Soekris tried to go upmarket with the net6501 and ended up folding. (They still sell custom DAC hardware from their EU outfit.) There's not enough of a market to make it worthwhile to take the risk. Lots of people love the idea in theory, but what people end up doing is sticking with traditional server suppliers, or in the home, hobbyist, and small business market just buying one of the many faster small form factor boxes that have flooded the market. The problem with those alternatives is that the boards are packed with too much crap, they're too diverse, and every model and family of models short-lived.