Won’t you still need an NTP or a GPS time module unless the RTC has a battery backup?
How many applications would you have where timing is critical but you won’t have an external time reference that would be still viable without a guaranteed power backup?
Nobody said anything about not having NTP in the loop!
Here's the deal: RPi<5 runs at the speed of the voltage coming in. The OS creates a fake software clock that assumes it's a perfect 5v, but in reality not only does voltage fluctuate, it drops over the length of a cable run. If you have, I dunno, 70 Pis connected, the 5v on the first one is very different than the 5v on the last one, no matter how frequently you ask NTP to broadcast.
The first thing we did was deliver 12V down the wire with a high quality power supply. Then we used UBEC buck converter modules (intended for RC quadcopters) to deliver power directly into the Pi via its pins.
However, even now you're just chasing physics. There's no way to send a command to all of the Pis to do something right now. Instead, we tell all of them to do something at a very precise moment in about three seconds. In our case, this meant writing a very low-level C program to listen for raw UDP messages and trigger the camera module at the hardware register level.
The only way to verify that it was working as well as we believed it did was to create a reference clock. We had a friend build a 20 LED timer that would count 10ms and 1ms increments.
We achieved < 1ms syncronization across 70 Pis in this manner.
We needed it to be small, low power, low heat, provide hardware level access to a camera module, be ethernet accessible. Oh, and under $100 including a fixed M-mount lens.
At the time, it didn't seem like rolling our own hardware would have been faster.
I would argue - given we succeeded at sub 1ms syncronization - that the Pi was close to the perfect tool.
It just would have been even more perfect if it had an RTC.
You already solved your issue, but still I'll mention it:
For my Pi's I buy a separate RTC (uses GPIO pins). There's various available, the DS3231 is the most expensive but also the best. If you buy it in bulk (70+) I suppose you could shave some money off. If you're OK with a cheaper one (less than half the price) which functions less well (but still does the job) then PCF8523. I got mine from a local Adafruit reseller.
FWIW, the cheapest RTC module we could find that had kernel support was about $31 at the time. Things have gotten a lot better and cheaper since then. Adafruit didn't launch that DS3231 until four years after we did this.
We couldn't justify adding ~$2500 to the bottom line, but you can be confident that we sure did argue about it.
Missed this comment when asking why you didn't add a $2 RTC, but given that you mention Adafruit I assume making your own hardware to stick onto a Pi wasn't an option? The DS3231 you mention is a chip with mainline support since 2009. Cheaper options like a DS1307 cost less than a $0.50 a piece when you buy 100 of them, and they've been supported in Linux since 2006.
I'd say 50/50 that I didn't know what I2C was in 2014.
Still, if we had gone and designed our own RTC daughterboard, it would have replaced the UBEC in our architecture diagram. We were buying those for $3-4 in 1000 unit quantities.
So, had you been on hand a decade ago, you might have won the argument although if I put on my board member hat, fabbing our own PCB sounds a lot scarier than buying boxes of buck converters.
It's hard to express how wild it is that they just pop right on and guarantee close to perfect voltage to each device.
Fair enough! I'm not sure I caught that the project was from 2014.
From your URL, the second part made me wonder if the statement is correct: "UBEC 5V 3A/5A/7A/15A BEC 2-12S Lipo Step-down Module External Power Supply Full Shielding Antijamming For FPV Airplane" would this actually work?
This is an armchair hindsight, but sounds like that needed a way to broadcast trigger GPIO interrupts, which are like JavaScript event handler for hardware.
You can attach a function to a pin, and CPU force-jump into a preprogrammed address for a handler routine, which leads into your code, then return(return(return())) back to where it came from. The trigger can be delivered by, say, a Xenon flash pointed at the ceiling and a light sensor, or a dedicated pair of wires from a master Arduino, or a intentional glitch in power line, a drone receiver per each pi all on a same channel, etc.
> fixed M-mount lens
I assume this is meant to say metric threaded C/CS mount for surveillance cameras, as M-mount is boutique full-frame Leica mount.
Well, if you absolutely needed to use that specific rpi camera module (if I understood your other comment correctly) and you didn't have time to reverse engineer it, then rpi was probably the best option, sure. It just feels weird to assume that, or that every single module needed to be connected by ethernet. Rpi and low power in the same sentence also feels weird. I'd have tried to build little devices out of STM32. But again, I barely know anything about your project.
Ethernet was handy for moving the images off the Pis once they were captured.
The RP2 was indeed low powered compared to the Intel/AMD powered systems we were otherwise familiar with.
Outside of the bubble of people with EE skills, however big the bubble is, stopping to learn expert-level mcu programming from scratch is rarely the way things work in a software startup. Despite the obvious focus of this thread, the hardware we assembled and configured was all in support of our software.
I assume that the rather hostile reaction you've received in this thread stems from the combination of your insistence that you tried everything to "bend the law of physics", but dismissal of any suggestion of simpler methods because you're a software person and couldn't have possibly thought/known of everything and we should just all assume that you found the optimal solution. Said differently, it's a strange combination of "we were a team of geniuses so trust me our solution was the best one" and "we were all software people so stop suggesting hardware solutions, it was too complicated for us to learn".
I didn't perceive a hostile reaction, beyond one dude who declared that something hard would be easy.
On the contrary, I've had a number of solid back-and-forths with folks about GPS, GPIO and homebrew RTC daughterboards that we could likely find success with today - almost a decade after we built what we built.
I'm honestly not sure what else you expect from me. Should I apologize for our relative ignorance at the time? Are you upset that I'm proud we managed to succeed despite doing things differently than others might have?
Given that you can't share the unique details of your use case, what could have been an interesting discussion regarding design tradeoffs is moot.
(And I would politely ask that you don't make assumptions about what I might or might not have actually accomplished in similar spheres, you have zero idea.)
What's with the patching? It sounds like pretty much spot on for what NTP was designed for.
Each clock is monotonically increasing, with its individual drift. NTP will pretty soon find out exactly what the right polling interval is. Even if you run it in broadcast mode (why would you though, with only 70 devices).
It's been eight years, but if memory serves, the primary need was to force it to broadcast as frequently as possible.
Everything you said is totally legit unless you're addressing devices with software clocks that are running at speeds dictated by subtly shifting voltage.
Remember, according to the Pi itself, it's 100% in sync. No problem hereeeeeeee (robot voice as voltage dips to 4.9887 volts).
At some point, and considering how cheap they are these days, the simplest option is to add a GPS module if your use-case allows for satellites in decent view.
Would it have worked to add something like a single GPS module connected to an ESP8266 that sends a pulse every minute to all the Pis' GPIOs so they can sync?
Hm, yeah, you'd need a much more frequent method of synchronization to avoid that. I'm assuming the rate wasn't constant, huh? Because of voltage drops depending on draw?
I want to be a better conversation partner on this, but I am extremely constrained in what I can share and again, it was 8-9 years ago that this was all top of mind.
Can we mutually agree that in a year of not-dumb people trying everything we could manage to achieve the best outcome, we tried a lot of things and we often didn't know what we didn't know?
We cannot, because I'm not trying to find you a solution you didn't think of, I'm trying to learn from your failures so I can skip over them next time I need something similar to this.
First, our failures were not technical. We succeeded at creating a sub-$100 camera node that could be replicated and/or replaced quickly by relatively unskilled talent.
If I was going to do it all over again with the decade of engineering skills I've learned, I would look towards fabbing our own RTC I2C daughterboards and trying some GPIO-based approaches to triggering.
However, given our constraints, the solution I've described above worked incredibly well. I wouldn't be embarrassed to do it again.
I'm not saying you failed at the task, but that you tried a bunch of things along the way (that you've detailed in this thread) and they didn't work for whatever reason. That's valuable to know, it saves me time if I want to do something similar in the future. Plus, it's just nice to hear about the engineering of it all.
I got a u-blox NEO-6m module for a few bucks, add another few bucks for the active antenna. The newer GPS modules are much better (more satellites, faster lock etc) but this is sufficient for me.
We've synchronised multiple Pi's to fractions of a millisecond, with flash systems with accuracy in the microseconds for specialised motion capture. Happy to discuss if you're interested - I'm on <hn username> at 'tessierlabs . com' (without spaces).
Because moore's law is a proxy for RnD for an entire industry. if you want cutting edge features, you need to pay for them.
raspberrypi is about value engineering. Making stuff fast and cheap enough so that you can do useful work, but not worry too much if you break it (on the one hand) Or give access to someone who's never had a programmable computer for a price that's cheaper than most mid range lego sets.
The idea that even 10-20% of users would power their Pis with PoE seems wild to me. Seems like a much smaller niche to me (even though I'm one of those people).
Anyway, I have been wondering whether a headless-oriented SKU wouldn't make sense. Pay for the PoE BOM by jettisoning the video output.
I'm really surprised that PoE isn't more prevalent that is currently the case, but I guess it's because most people prefer WiFi devices. I'd add network jacks to everything and remove the power plugs if the option was presented. Most "normal" people seem to be the other way around.
"Wireless" speakers (they'd be networks speakers then) / home assistant devices / media players and whatever else should always be attached with a network cable, so just power them over PoE. I'm still annoying that the AppleTV isn't PoE enabled. That's not a device I'd use over WiFi anyway.
> I'd add network jacks to everything and remove the power plugs if the option was presented.
You could always use PLC to use your eletrical network as a data network. I use that for IP cameras. Unfortunately, I still haven't found a PCL adapter providing PoE power.
> The idea that even 10-20% of users would power their Pis with PoE seems wild to me. Seems like a much smaller niche to me (even though I'm one of those people).
I'd be surprised if the majority of RPi's userbase didn't used PoE. Having to cart around a power supply isn't a very attractive option when all you have to do to power the device is simply plugging in a yank-proof RJ45 cable. It's also cheaper as you don't have to buy a charger.
I've contemplated trying PoE for powering RPis, but just the HAT alone is more than the official power supply, so it's hardly cheaper and that's not even figuring in the extra cost of PoE network equipment.
I've been powering a Pi 4 for several years now with the official PoE hat (edit: not PoE+), without any problems (other than that the fan is starting to get noisy).
I liked that the Pi lacked features that were taken for granted. Made you think about things in a different way. Now it's just the same as everything else and much less about learning and education.
rPI went from armv6 to armv7 to armv8. CortexA76 does not even support using 32 bit kernels. Rp1 requires an armv6 32 bit kernel. There is NO kernel build on earth that would run on both an pi1 and a pi5 and no amount of efi would help. (I am ignoring the possibility of custom nasty hacks to make a polyglot kernel). We’ll have per-device images until we standardize on a single architecture or at least a subset of one.
And no, you really wouldn’t want a lowest common denominator build anyways. That would leave a lot on the table.
Yeah, I can live with images per architecture; the problem is needing an image specifically for the pi. I want the same thing as we get on x86; sure, there are separate images for i686 and x86_64, but there's not one image for Dell and one for HP and one for Lenovo and...
Well, now that the latest rPi is on the latest ARM arch, it might be possible for future rPis, unless ARM goes and changes the arch around in an incompatible way.
That's beside the point; you can build a UEFI bootloader for at least as far back as the Pi 3, but you have to stick it on an SD card because there's no on-board memory to hold it. I mostly just want enough flash built in to hold it.
rPI is quite cost optimized. That’s one of the stated goals. An extra component to buy and place on the board costs read money. Plus, an implementation of UEFI will not be tiny. It would likely would need a few MBytes of flash. That's a dollar or two already. If everybody’s requests, each of which would add a dollar or two to the board, were fulfilled, you’d be looking at a much more expensive board, and complaining about cost instead. :)
- a power button
- a real time clock
Crazy it took this long but finally, seriously.
Downsides here are that it might need more active cooling whereas previous generations worked without it.
Also, ugh:
> Gigabit Ethernet, with PoE+ support (requires separate PoE+ HAT)