Its less about the actual amount of computing power at the actuating node and more about how much like a computer we treat it. If we have to sysadmin it like its a full-on server, that's probably already too much. Not picking on javascript in particular, but cramming a web-server-ish stack into a light switch feels like it might be heading in that direction.
Its not really a swarm being "controlled by a central server", its nodes discovering each other and creating a de facto, discoverable api amongst themselves without overt configuration. LivingRoom.Lights.Bright=5 will cause modules you've tossed in your living room to say "hey, I'm a light, I'm in the living room, I should be brightness 5 now!"
The thing is, these lights are actually more complicated than our old-fashioned web servers, because while they too have to communicate with other systems, update their codebases, maintain state, and so on, they also have to do all of that in unforeseen environments, surrounded by unknown services they have to discover and use. Plus they have sensors and whatnot. The sysadmin problems are still there, but now they need to be self-serviced, or at least serviced through some kind of autonomous negotiation with the swarm. If it's not allowed the conceptual bulkiness of an overtly-configured server, I don't see how it could be a self-configured one, which seems much more heavyweight.
A bit more generally, I'm just not sure how a light can be all the things you want it to be without having a pretty complicated stack underneath it. This is a hard problem, and there's going to need to be a bunch of technology in there. Even a common stack of technology. In fact, a web server seems like it would be exactly how you'd want here, because it presents a simple, consistent interface for communication. Generalized interfaces between diverse systems is exactly what HTTP has been so successful at. What are the alternatives and how would they make these devices less computery?
It sort of sounds like your objection is that many web servers aren't simple enough in the sense that they have the complicated configuration APIs which are used by their admins, who want to make really specific human-targeted applications out of generic but wide frameworks built specifically to make that possible. Routing and same-origin policy and access control and extension points and caching and whatnot. Perhaps you're about that, but it's not really a problem with the depth of the stack, and none of that is central to being a web server, or even part of NodeJS.
Its not really a swarm being "controlled by a central server", its nodes discovering each other and creating a de facto, discoverable api amongst themselves without overt configuration. LivingRoom.Lights.Bright=5 will cause modules you've tossed in your living room to say "hey, I'm a light, I'm in the living room, I should be brightness 5 now!"