As a Brazilian - Pix was a pleasant surprise, especially in that for once it feels like we're not lagging behind. It's convenient, free, instant transfers across banks. You can also easily create or programmatically generate QR codes or pastable codes with preset receivers and amounts. Great UX all around, and it quickly became the de-facto standard in how people send money.
It's technically quite impressive - it's a large scale thing and it works really well. I can think of maybe one or two times in these years where I saw downtime, and in both cases it was working again after a few minutes. The usual experience with the government building technical solutions is to have something that makes little sense, is slow, and goes down frequently with even the most predictable usage peaks, but with Pix they really seem to have nailed it.
It does feel a bit weird to have so many payments go through the government's systems, and it definitely feels like it puts them in a position of having more information than they should. There's a lot of Orwellian surveillance potential there, as any transfers are necessarily tied to both users' real identities. I don't think there's a realistic way around this, though.
Another concern is that people can expose some of their information without necessarily being aware of it. You can register e.g. emails and phone numbers as Pix "keys", and then anyone can initiate a transfer to those keys and your full name will pop up so you can confirm or cancel the transfer. I've seen some clever advice around this - "When using a carpooling app (often details are arranged off the platform using WhatsApp), put the driver's phone number on Pix. If a name comes up and it doesn't match the name or gender of the driver's profile, something is up". Obviously though there's potential for misuse and I'm sure the vast majority of people don't think about this when registering their Pix keys. You can, however, just use randomly generated uuids as keys as well, a different one for each transaction if you so desire, so this one can be a non-issue with more awareness.
Overall though it's a very convenient thing which works surprisingly well, and the downsides are theoretical at this point. IMO it's a rare case of our government nailing something.
> but most probably was training data to prevent deepseek from completing such prompts, evidenced by the `"finish_reason":"stop"` included in the span attributes
As I understand, the finish reason being “stop” in API responses usually means the AI ended the output normally. In any case, I don't see how training data could end up in production logs, nor why they'd want to prevent such data (a prompt you'd expect to see a normal user to write) from being responded to.
> [...] I'm guessing Wiz wanted to ride the current media wave with this post instead of seeing how far they could take it.
Security researchers are often asked to not pursue findings further than confirming their existence. It can be unhelpful or mess things up accidentally. Since these researchers probably weren't invited to deeply test their systems, I think it's the polite way to go about it.
This mistake was totally amateur hour by DeepSeek, though. I'm not too into security stuff but if I were looking for something, the first thing I'd think to do is nmap the servers and see what's up with any interesting open ports. Wouldn't be surprised at all if others had found this too.
Seems that you're right! Also, not that I doubted they were using OpenAI, but searching for `"finish_reason"` on the web all point to openai docs. Personally, I wouldn't say it's a very common attribute to see in logs generally.
> Now that you've generated your first chat completion, let's break down the response object. We can see the finish_reason is stop which means the API returned the full chat completion generated by the model without running into any limits.
Regarding how training data ends up in logs, it's not that far fetched to create a trace span to see how long prompts + replies take, and as such it makes sense to record attributes like the finish_reason for observability purposes. However the message being incuded itself is just amateur, but common nonetheless.
The OpenAI API is basically the gold-standard for all kinds of LLM companies and tools, both closed and open source, regardless of whether the underlying model is trained on OpenAI or not.
Not just the gold-standard, but also a de-facto standard - most of the proprietary and OSS tools I've seen that let you configure LLMs only implement support for OpenAI-compatible endpoints.
> [the extensive anti-reverse engineering measures are] more annoying than any financial app I've had, and I have 5 of them on my phone
Ah, this reminds me of the Tuya app.
I've done some ssl unpinning and mitm to see requests going in and out of my phone, it's pretty fun and there's often really nice and easy to use restful APIs underneath. Among them I've also done a couple of banking apps and they weren't particularly defensive either. That's great; as a user I'm empowered by it and like TFA says, it's totally fine from a security standpoint if you just don't trust the client to do anything they shouldn't be able to do. It shouldn't be your form validation that stops me from transferring a trillion dollars, and though I haven't tried, I'm sure that's not the case for those apps. All it does is allow me to get my monthly statements with a for loop rather than waiting for a laggy UI and clicking through each month.
Now, Tuya is a Chinese company offering a bunch of cheap IoT devices like smart power switches and IR motion detectors. You can interact with everything through their app. That app for some reason has spent by far the most resources on anti-RE of any apps I've seen. I already bought your hardware, mate. Please let me use it on my local network. My smart home infrared motion sensors were meant to turn lights on when I enter a room. But they don't feel very smart when I'm standing in the dark for 4 seconds while they check with a server in China. I don't even need a clean API; just let me see what you do, and I'll do something similar, no support or documentation necessary. But they go through extensive measures to prevent you from interacting with the hardware you bought and which is sitting in your home.
This was a while ago, but I think for the motion sensing in particular, I managed to just put them in a subnetwork with blocked internet access, and snooped on the network to catch their DHCP requests when they tried to call home. This would happen every once in a while presumably for settings/update checks, but crucially also when there was motion detected, and I didn't mind a few false positives. So in the end they were very quick, locally functioning, privacy-friendly little devices!
The problem with Tuya is that they don't manufacture the devices themselves. Instead, they provide a standardized interface for all those low-cost manufacturers and get paid by them. If it were easy to fake Tuya requests or set up your own account (trust me, I tried this to integrate a Fingerbot into Home Assistant, but you have to jump through countless hoops, and the developer account keeps expiring every few weeks), those manufacturers would simply automate this process through their own apps.
> they provide a standardized interface for all those low-cost manufacturers and get paid by them
As far as trends in IoT goes, I feel like Tuya is mostly positive. I bought some cheap smart plugs at Costco and the default app was worthless. When I learned that they were Tuya-compatible, I managed to get a half-decent (relative to cost) experience out of them. It seems to me that the alternative are a bunch of unmaintained one-off apps for each fly-by-night manufacturer. With a standard protocol and app I think old devices will live a bit longer at least.
Perfect (better) world it's all open source, but c'est la vie.
This sounds somewhat backwards to me but maybe missing something... We got a bunch of Tuya devices and was barely aware they even have an app. They paired out of the box to a zigbee2mqtt gateway on the local airgapped network without fuss. No apps, online servers, api keys, vendor signature checks, or such shenanigans at all. I don't think the motion sensors we have from them have the capability to send dhcp over ip even if they wanted.
The Fingerbot also seems to operate over zigbee? Why would you need a developer account in the first place? And why would anyone but Tuya themselves want to hook into their cloud?
Sorry for the late answer. My fingerbot just talks bluetooth and can be operated via ble locally. Just like your zigbee devices. That's because they are not "Tuya devices" they are simple hardware devices that can be remote controlled via the tuya cloud if you have a local bridge.
Tuya is just the cloud connector for those devices that communicates with the bridge (so the hardware manufacturers don't need to operate their own servers).
My first thought, looking at the webpage: "Huh, that's neat. I didn't know that painting software didn't even attempt to do color mixing beyond naive interpolation, though I guess it figures; the physics behind all the light stuff must be fairly gnarly, and there's a lot of information lost in RGB that probably can't be just reconstructed."
Scrolling down a bit: "Huh, there's some snippets for using it as a library. Wait, it does operations in RGB? What's going on here?"
Finally, clicking the paper link, I found the interesting bit: "We achieve this by establishing a latent color space, where RGB colors are represented as mixtures of primary pigments together with additive residuals. The latents can be manipulated with linear operations, leading to expected, plausible results."
That's very clever, and seems like a great use for modern machine learning techniques outside the fashionable realm of language models. It uses perceptual color spaces internally too, and physics based priors. All around very technically impressive and beautiful piece of work.
It rhymes with an idea that's been floating in my head for a bit - would generative image models, or image encoder models, work better if rather than rgb, we fed them with wavelength data, or at least a perceptually uniform color space? Seems it'd be closer to truth than arbitrarily using the wavelengths our cone cells happen to respond to (and roughly, at that).
Consider that just after the cone cells, there are other cells doing some computation / aggregate of cone signals. Don't forget that color is a brain construct.
For those reasons (and others), there's often a strong disconnect between stimuli and perception, which means there's no such thing as a perceptual uniform color space.
Sidestepping what is defined as an "edge", quite a lot of work is done in the retina, including differential computation across cones - some "aggregator" cells will fire when it detect lines, movement, etc.
You can read on ganglion cells, bipolar cells and amacrine cells and see that a lot of preprocessing is done before even reaching the brain!
AI and machine learning aren't necessary at all. You 'just' have to empirically measure a few constants that describe and bound the various nonlinearities of different real pigments, and then plug them into a relatively straightforward paint mixing equation.
Paints have predictable mathematical properties in terms of what color they produce when mixed; they just mix nonlinearly, which is counterintuitive for people who have not practiced mixing paint a lot.
Photoshop and the other comparison programs on the page illustrate the linear mixing that most people intuitively expect.
As far as I know, most painting apps mix color in sRGB space instead of the linear RGB space. Which means they're even "worse" than naive interpolation.
Here's an idea: have the LLM output each comment with a "severity" score ranging from 0-100 or maybe a set of possible values ("trivial", "small", "high"). Let it get everything off of its chest outputting the nitpicks but recognizing they're minor. Filter the output to only contain comments above a given threshold.
It's hard to avoid thinking of a pink elephant, but easy enough to consciously recognize it's not relevant to the task at hand.
That's certainly possible, but it reminds me a bit of a similar thing I've seen in their UI that rhymes in a way that makes me think otherwise. In the code interpreter tool, you have a little preview of the "steps" it's following as it writes code. This turns out to just be the contents of the last written/streamed comment line. It's a neat UI idea I think - pretty simple and works well. I wouldn't be surprised if that's what's going on with o1 too - the thought process is structured in some way, and they take the headings or section names and just display that.
For what it's worth, as a primarily backend dev having ~recently started getting more deeply into frontend web, I have specifically noted in my head that the box model isn't too intuitive and in my inexperienced opinion, the default was a bad one. I figured surely if it is the way it is, then it's for reasons I do not yet comprehend™, so it actually feels pretty validating that someone who knows what they're talking about agrees.
It does feel right to me, because it's not distilling the second model, and in fact the second model is not an image generation model at all, but a visual encoder. That is, it's a more "general purpose" model which specializes in extracting semantic information from images.
In hindsight it makes total sense - generative image models don't automatically start out with an idea of semantic meaning or the world, and so they have to implicitly learn one during training. That's a hard task by itself, and it's not specifically trained for this task, but rather learns it on the go at the same time as the network learns to create images. The idea of the paper then is to provide the diffusion model with a preexisting concept of the world by nudging its internal representations to be similar to the visual encoders'. As I understand DINO isn't even used during inference after the model is ready, it's just about representations.
I wouldn't at all describe it as "a technique for transplanting an existing model onto a different architecture". It's different from distillation because again, DINO isn't an image generation model at all. It's more like (very roughly simplifying for the sake of analogy) instead of teaching someone to cook from scratch, we're starting with a chef who already knows all about ingredients, flavors, and cooking techniques, but hasn't yet learned to create dishes. This chef would likely learn to create new recipes much faster and more effectively than someone starting from zero knowledge about food. It's different from telling them to just copy another chef's recipes.
The technique in this paper would still be rightly described as distillation. In this case it's distillation of "internal" representations rather than the final prediction. This a reasonably common form of distillation. The interesting observation in this paper is that including an auxiliary distillation loss based on features from a non-generative model can be beneficial when training a generative model. This observation leads to interesting questions like, eg, which parts of the overall task of generating images (diffusionly) are being learned faster/better due to this auxiliary distillation loss.
It's technically quite impressive - it's a large scale thing and it works really well. I can think of maybe one or two times in these years where I saw downtime, and in both cases it was working again after a few minutes. The usual experience with the government building technical solutions is to have something that makes little sense, is slow, and goes down frequently with even the most predictable usage peaks, but with Pix they really seem to have nailed it.
It does feel a bit weird to have so many payments go through the government's systems, and it definitely feels like it puts them in a position of having more information than they should. There's a lot of Orwellian surveillance potential there, as any transfers are necessarily tied to both users' real identities. I don't think there's a realistic way around this, though.
Another concern is that people can expose some of their information without necessarily being aware of it. You can register e.g. emails and phone numbers as Pix "keys", and then anyone can initiate a transfer to those keys and your full name will pop up so you can confirm or cancel the transfer. I've seen some clever advice around this - "When using a carpooling app (often details are arranged off the platform using WhatsApp), put the driver's phone number on Pix. If a name comes up and it doesn't match the name or gender of the driver's profile, something is up". Obviously though there's potential for misuse and I'm sure the vast majority of people don't think about this when registering their Pix keys. You can, however, just use randomly generated uuids as keys as well, a different one for each transaction if you so desire, so this one can be a non-issue with more awareness.
Overall though it's a very convenient thing which works surprisingly well, and the downsides are theoretical at this point. IMO it's a rare case of our government nailing something.