Not sure, but I solved a similar problem many years ago, and ended up concluding it was silly to send all the data to the client when the client didn't have the visual resolution to show it anyway. So I sampled it adaptively client-side by precomputing and storing multiple zoom-levels. That way the client-side chart app would get the points and you could zoom in, but you'd only ever retrieve about 1000-2000 points at the time.
Yeah I agree, I'd like to get an idea of the order-of-magnitude of difference between the two approaches by trying it out but realistically I don't think there's an easy way to get a i16 raw array into the browser runtime with minimal overhead (WebRTC maybe?)
I once had to deal with many million data points for an application. I ended up mip-mapping them client-side.
But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours. When you tune it correctly, you can drop most points without the difference being noticeable.
I didn't find any else doing that at the time, and some people seemed to have trouble accepting it as a viable solution, but if you think about it, it doesn't actually make sense to plot say 1 million points in a line chart 1000 pixels wide. On average that would make 1000 points per pixel.
We routinely face this in the audio world when drawing waveforms. You typically have on the order of 10-100k samples per second, durations of 10s-1000s of seconds, and pixel widths of on the order of 1-10k pixels.
Bresenham's is one algorithm historically used to downsample the data, but a lot of contemporary audio software doesn't use that. In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
> In Ardour (a cross-platform, libre, open source DAW), we actually compute and store min/max-per-N-samples and use that for plotting (and as the basis for further downsampling.
I discovered flot during my academic research career circa 2008 and it saved my ass more times than I can count. I just wanted to say thank you for that. I wouldn't be where I am today without your help :)
> But regarding sampling, if it's a line chart, you can sample adaptively by checking whether the next point makes a meaningfully visible difference measured in pixels compared to its neighbours.
uPlot basically does this (see sibling comment), so hopefully that's some validation for you :)
Yes. DJGPP and Allegro was a great help, and a big step up from the old Borland Turbo Pascal I started out with. I remember trying to rotate an image pixel by pixel in Pascal. Allegro simply had a function to do it. And yes, the mailing list was great - Shawn Hargreaves and the couple of people in the inner circle (I seem to remember someone called George) were simply awesome, helpful people.
I eventually installed Red Hat, started at university and lost most of my free time to study projects.
Some years ago I bought a little pamphlet with a few mental exercises by Rudolf Steiner, known for the Waldorf school system.
One of them was about building a habit. You find a small meaningless thing to do, it must have no purpose at all, and then you do it once every day for as long as it takes to become a habit, probably a month or two. E.g. you could fill a glass with water and throw it out.
I did the exercise (I would kneel for a few seconds when taking a bath) for a couple of months, and I think it worked for me. I've recently used the same tactic to build a useful habit.
Now building a new habit is not necessarily the same as changing an old habit.
I also found out that kneeling changed my perspective. I could think about a situation with some level of tension, kneel, and then my perspective on the same situation would be more humble and appreciative. YMMV.
Taking out the emotion. Since the genesis of this discussion was "lifting weights", why uselessly lift weights instead of doing useful work like moving sacks of grain from point A to point B?
"If habit is a muscle that can be developed...", then being detached from the simple, useless habit being formed is good practice for being able to apply it to a productive or important situation.
I can see why; doing something useless requires conscious thought / effort. Doing something you want to do / achieve / change is, controversially, difficult because you think you can rely on just you "wanting it", that is, innate motivation. But that's usually not the case, especially not if it's something that causes discomfort at first (like going to the gym) or if it means giving up or cutting back on something pleasurable (mmm, donuts).
But intentionally doing something that you don't do normally, something you don't want to do, something that doesn't give you any kind of dopamine feedback can help you practice forming habits, practice self-discipline, etc. It's an interesting experiment.
The purpose is to focus on the process of building the habit by exercise of will. Having the habit be something that is useless makes the daily repetition an expression of pure will, rather than e.g. a sense of obligation or desire for a certain outcome.
My guess is that you get practice in habit-building that can be applied to useful habits. Sort of like having students do exercises that have solutions in the back of the textbook. It's not the solution that's needed, it's the practice.
Just FYI: Rudolf Steiner was a pusher of pseudoscientific nonsense. He was anti-vax and his views on race were problematic, to say the least. He was a fervent German nationalist and a critic of Einstein's theory of relativity. I could go on, but you get the idea.
More importantly in a modern context, I know that Waldorf schools seem harmless, but they are religious "Anthroposophy" schools at their core.
Greenland can become independent if they wish. There would be some things to work out, but the legal framework has been in place since the 1970es. And they seem to be working towards the goal.
The reason it hasn't happened yet is that they'd either have to increase tax income greatly, or reduce public spending greatly with financial support from Denmark. As I gather, infrastructure up there is really expensive.
Yeah, the only dumb thing about the digital mail is that they're not just using email with an official registry.
They could have started some kind of certification thing for email providers and even funded a couple of certified email providers much more effectively than the digital post monstrosity.
That would have been awesome and forward looking, and perhaps even helped ordinary people get better security for their personal emails.
One thing to watch out for with immutability is that if you're dealing with personal information about people, immutability is probably illegal. You must be able to forget information, and not just simulate you've forgotten it.
I don't how universal this is at the moment, but I think it's likely to be more universal in the future.
Yes, controlled excision is a (necessary) capability of all event-sourced systems and/or immutable object stores, i.e. it's quite universal already. All such systems have to build in some sort of special-case mechanism to excise facts, not merely redact (~tombstone) them.
But data-destruction superpower must be used sparingly, with care. See: git's documentation for --force-push, for example (and bemoan what popular git forges force us to do on a normal basis).
Excision is the complete removal of a set of datoms matching a predicate. Excision should be a very infrequent operation and is designed to support the following two scenarios:
Removing data for privacy reasons
Removing data older than some domain-defined retention period
Excision should never be used to correct erroneous data and is unsuitable for that task as it does not restore any previous view of the facts. Consider using ordinary retraction to correct errors without removing history.
Irrevocably erases documents from a table, for all valid-time, for all system-time.
While XTDB is immutable, in some cases it is legally necessary to irretrievably delete data (e.g. for a GDPR request). This operation removes data such that even queries as of a previous system-time no longer return the erased data.
The UK used to have very high subsidies for offshore wind for some reason. The last I've heard, subsidies for new plants are much lower today.
As for being cost effective, onshore wind is probably the cheapest option, and I think it's hoped that offshore will come close to that once more of the learning curve has been traversed. Perhaps fossil gas from the North sea is still cheaper for now, if you ignore the external cost.
I think solar power is even cheaper, but doesn't deliver much in the winter so far up north.
Backup: Batteries are cost effective for short term shortages. For long term shortages, you'd fire up thermal plants, either biomass or biogas (fossil gas for now).
It doesn't make sense to back up wind with nuclear. Nuclear has a high capital cost and relatively low running costs, so you don't save much from being standby but you still need to pay back the loans.
reply