Intel needs everything to be NUMA-aware. They're betting a lot of money on Xeon Phi, and once the self-booting KNL machines are out nobody will want to deal with the pcie cards any more.
As far as I know, the Phi doesn't actually require NUMA-awareness at all (at least, the older models didn't; see https://arxiv.org/pdf/1310.5842v2.pdf). A Phi lives on a single socket with a coherent L2 cache, and remote L2 accesses are not much slower than main memory ones, nor does core distance along the interconnect seem to affect access time. The new models with lots of main memory are going to be used with six-DIMM slot DDR4 sockets (64 GB each of DDR4, in addition to 16 GB MCDRAM to get even more absurd bandwidth for pure FLOPS / benchmark / coprocessing workfloads; see http://www.intel.com/content/www/us/en/processors/xeon/xeon-...), in order to avoid having to split the Phi up into multiple NUMA domains.
So, I have no idea why Intel would care at all about making stuff NUMA-aware for the purpose of Phis. Cache-aware, sure, but that's pretty much required for good performance on modern machines already. What they would care about is making everything vectorize properly, since Phis do horribly if you aren't exploiting the VPU; hence, you'd think they'd be more interested in adding badly missing SIMD support to Go than NUMA-aware scheduling.
(Please let me know if I'm wrong and there's a multi-socket Phi announced, but I've been following it really carefully because I'm excited about the possibilities of using the new KNLs for main-memory databases, and I have yet to hear anything about that).
There is no multi-socket Phi - I asked about it at an Intel booth at a conference a while back and was told the delta between memory bandwidth and inter-socket bandwidth would be so great that it would not be a useful configuration.
I believe the talk of NUMA refers to the single socket behaving like a cluster with up to 4 NUMA domains, but I can't find any good references right now.
Ah, interesting; I hadn't read that anywhere. From the limited reading I just did, it does seem like that's a configuration they offer, but from the scant sources available I can't quite figure out to what extent it's actually necessary to extract maximum performance out of the machine (compared to just artificially pinning each core to disjoint memory). Either way, good information--thanks!
It's not about multi-socket configuration; selfbooting KNL machines can have two classes of memory. For brevity's sake you can think of them as "fast" and "huge." A regular malloc call gets you a piece of "huge", and there's a separate malloc function available to allocate "fast." This is the difference between the DDR4 and the MCDRAM you mentioned -- they're not accessed uniformly.
While Intel has done a ton of work to make sure you don't have to care about this, it's obviously in their best interest to have as much software as possible be able to care about this, especially because the KNL clock is so slow.
Sure, but it's not clear to me that the best way to deal with the two memory classes would be with NUMA-aware scheduling, unless you happen to have an application with "fast" and "slow" application threads (which I suspect describes relatively few applications in practice; and even if it does, wouldn't you have to tell the scheduler about it explicitly?) Seems to me like it will usually be much more efficient to use the MCDRAM either as L3 (default configuration) or as an explicit scratchpad (which a scheduler wouldn't really be able to exploit, given that if the scheduler has data structures that don't fit in L2 it's already probably screwed).
That being said, I did some more reading this morning and the sub-NUMA clustering configuration on the new Phi does provide tile-to-directory-to-MCDRAM affinity (via pin domains), which would make sense for maximizing its performance as either L3 or scratchpad; AFAICT this is not the case for the remote DDR4, though. So whether it's worth caring probably depends very much on your workload; I think KNL is most interesting for workloads with working datasets that are much larger than 16GB, since otherwise you could just use a GPU (you can get more usable working memory per second with much better bandwidth with something like a DGX-1 thanks to NVLink, but unless I'm missing something not at a remotely competitive pricepoint, and it's unclear to me whether it's sustainable for larger working sets since you can only transfer up to 80 GB/s from the CPU to the GPUs, which is lower than the 90 GB/s each Phi gets out of DDR4 on Triad [and a better comparison is probably the 115.2 theoretical peak for KNL anyway]).
In designing the second-generation Intel Xeon Phi chip, we created a massively multicore processor that is available in a self-boot socket. This eliminates the need to run an OS on a separate host and pass data across a PCIe* slot. (However, for those who prefer using the latest Intel Xeon Phi chip as a co-processor, a PCIe-card-version will be available shortly.)
