At ~9:50 Michael Moritz talks how the 100-year media history tells us that if you can gather a large audience in one place, you will be able to sell them advertising.
It's likely that Google will not be the last search company. What business model could a successor use to generate revenue without falling into the ad trap? Why didn't early search engines choose other models like subscription, revenue sharing with telecom companies, or turning queries into marketing signals for manufacturers/merchandisers?
While John talks about the period before Google, this quote brings up a remarkable déjà-vu:
[~14:30] Although the search companies were having great success, they were turning into something quite different than what they started out to be. They lost the sight of what brought users to them in the first place: The need to find things. The search engine companies stopped caring about search.
When it came to actually locating relevant information on the web, Yahoo, Excite, and the rest of the so called search companies, frankly, stunk. You could spend all day typing various combinations of keywords that you were looking for. Most of the results were links to sites trying to sell you something you didn't want. The world was hungry for a radically better way of searching the web.
The abstract says ..we present metrics from our large-scale deployment of CodeCompose that shows its impact on Meta's internal code authoring experience over a 15-day time window, where 4.5 million suggestions were made by CodeCompose. Quantitative metrics reveal that (i) CodeCompose has an acceptance rate of 22% across several languages, and (ii) 8% of the code typed by users of CodeCompose is through accepting code suggestions from CodeCompose. Qualitative feedback indicates an overwhelming 91.5% positive reception for CodeCompose.
In other terms, out of 4.5 million suggestions about 80% were off, yet there is 91% positive reception. That's 3.6 million rejected suggestions that potentially distracted programmers from doing their work. Yet users are happy. Is there a contradiction in these figures?
Reading these answers reminded me why I love HN - actually thoughtful perspectives :) Guess a lot boils down to two variables - (a) suggestion UX quality and (b) definition of 'rejection' event. I skimmed through the paper and it turns out that 91% figure is based on feedback from 70 people and anonymous feedback wasn't allowed. So, 'overwhelming 91% favorable' can be paraphrased to `64 people out of the total 16k user base said they liked it'. Would be interesting to see indirect metrics like retention on day 15.
Quite an insightful comment. In an institution that large it's surprising there were only 64 brown nosers. I expect out of 16k captive audience employees you could probably get 64 people to give a positive opinion of replacing paychecks with meta store scrip.
>It's a boilerplate generator and you're happy when it saves you tedious mental effort.
On the other hand the person trying to track down a subtle bug afterwards might be a little less happy at having to wade through oceans of boilerplate.
Sounds like you haven't tried copilot, basically scenarios like :
if(a) {...}
if(b) // here it predicts line by line from a once you start with similar logic
if(c) // here it will do a one shot generalization of a and b for c
Likewise method variations, enum mappings, inferring call parameters from names and signature, etc. - these things are trivial to check and test but take effort to type out. When you know what you want to do and someone suggests the solution you had in mind you're happy you saved half a min or min of typing.
I'm as, if not more, likely to zone out on tedium and introduce the subtle bug myself.
I’d say it’s hard to argue with the positive impression of the engineer using it. If they find it’s suggestions helpful it’s not a distraction, it’s helpful.
Using GitHub copilot daily I find it’s suggestions often nonsense but interesting to see regardless. Often for boilerplate it’s spot on and it saves me dozens of lines of typing. But it also suggests stuff on every key stroke many of which I just type through, similar to intellisense. Assuming Metas code thingy is better, I would find myself in that 91%, as I’m already there with what’s available to the general public.
My only gripe, fwiw, with copilot in vscode is it interferes with intellisense. Often I want to see the code completion from both, but copilot jumps in before intellisense and the intellisense never renders and I use it as an inline api reference. Sometimes it’s so frustrating I have to turn off copilot. But, copilot is generally useful enough that I reenable it once I’ve understood the api stuff I’m unsure of. There’s some escape backspace period dance I can do that sometimes let’s intellisense win. I’ve not dug deeply enough into vscode configuration to know if there’s some parameter to tweak the race conditions. I’d note that when intellisense renders first copilot still renders its suggestions but the other way doesn’t work.
I treat it the same way I do pre-LLM LSP suggestions, which is basically inline documentation lookup. ‘Oh what was that function name for inserting something at the end? PushB- no, InsertAft- no, App - end! Yea that’s it’
In this case it gave me 3 suggestions but I only accepted 1. I could see this taking 5-10 suggestions for an LLM to when it’s not something as straightforward as a function name. It’s still very useful despite this low acceptance rate
I think the 8% number better explains why users were so overwhelmingly happy. Assuming the suggestions in general are not distractingly wrong, then 8% of code automatically written is a decent amount of time saved researching solutions.
But only 22% are accepted for those 8%, which means that the 78% code suggestions that are not accepted correspond to an equivalent of over 28% of all code written. Not sure that having to spend the time evaluating an additional 28% of code in vain amounts to an overall win.
Though I guess the success rates when using Stack Overflow aren’t too dissimilar.
What it doesn't tell us though is how useful the rejected recommendations were.
Meaning, how many rejected solutions were sufficient to give the engineer enough context to turn a 30m task into a 5m task because they generated a recommendation, got an idea, rejected it, and rewrote it more efficiently or more correctly?
There's a lot of "devil in the details" likely buried in here.
Interesting that 91% find it useful but only 8% of the code is generated by LLM. This is even with a LLM tuned on the internal codebase. This will give a mild boost but not replace anyone.
Have you tried GitHub Copilot? You don't have to accept the code suggestions, so they don't really distract you or get in the way once you get used to the UX.
I find them extremely distracting. Evaluating a suggestion is, for me, an entirely different mental process from the creative process I’m in the middle of.
The tagline that copilot helps you stay in the flow is very much not my experience.
I am well aware that others are having a different experience with it.
I've found I am naturally ignoring the large complex suggestions because they usually have mistakes, and accepting the small easy suggestions. I respect your experience though, to each their own.
The thing can generate whole unit tests if you leave it a one-like description in a comment next to the function you want tested. It’s actually amazing.
The Industrial Challenges section of the paper addresses specific areas of flow disruption they focused on.
Some folks may never accept AI code completion / suggestions (like some prefer vim over modern IDEs) but at least people working on this stuff can describe points known to focus on.
It’s a different system, but it seems interesting to compare with what Google does for code review suggestions [1].
> The final model was calibrated for a target precision of 50%. That is, we tuned the model and the suggestions filtering, so that 50% of suggested edits on our evaluation dataset are correct. In general, increasing the target precision reduces the number of shown suggested edits, and decreasing the target precision leads to more incorrect suggested edits. Incorrect suggested edits take the developers time and reduce the developers’ trust in the feature. We found that a target precision of 50% provides a good balance.
Also, it seems like if the suggestions are too good then they’ll be blindly trusted and if they’re too bad they’ll be ignored?
Where to set the balance likely depends on the UI. For a web search, how many results do you click on?
Think of it like traditional code completion. It's mostly wrong but still useful. You either type through it, or tab/arrow to select the correct completion.
AI code completion (like Github Copilot) is like this. Still a time saver overall, even with a low acceptance rate.
I've had Bing provide me with code from SO that was from the question, which was code that was explicitly stated to not work and the poster wanted to know what was wrong with it. Bing's AI didn't understand this and claimed it was a solution.
The UX is really important, and the paper covers it, this is super spiffy tab completion, so even if it's wrong a lot, reading is faster than typing, and having something autofill `x if x is not None else ''` correctly even 5% of the time is nice.
What's the value of bringing Facebook/Instagram mechanics into HN? Wouldn't that skew social dynamics away from egalitarianism, giving rise to "influencers", social bubbles and rise in clickbait? I think that not having a 'follow your friends' mechanism is a feature of HN.
Some years ago I wanted to create LinkedOut. It was a privacy-conscious web site, where you couldn't spill any secrets because it wouldn't take any information. You got your page, which was blank, and stayed blank. A very anti-social network.
I'm pretty sure the domain name was taken already.
On slashdot you could mark people as friends and foes. You could then assign a penalty to posts by foes (and posts by foes of foes and posts of friends of foes, I think?) so they were unlikely to meet your 'show post' point threshold and would therefor be hidden.
Great idea! What about fade-away hnstyle with a color teint? Maybe updating à gist with Val Town and referencing that gist from a css file with Stylish? Or another way js only…
My impression is that many places that label themselves as hacker/maker spaces are some version of a commercial coworking space with equipment. They have the tools, but not the spirit. That's why I mentioned Noisebridge and CCL. E.g. in CCL you can join open projects like making cheese without milk. That's quite different than 'pay membership fee X, here's the equipment, do what you want'.
Do you know of specific places in Europe that capture that spirit? With both diversity and depth of knowledge in different disciplines?
+1 for IEEE Spectrum. I recently stumbled upon print issue archive from 2010-2012 and was delighted by their coverage of private space flight and 3D printing. To me, Spectrum is an early high-level overview of interesting technology.
Thanks for sharing. Cool to see someone from Aachen NLP group. I'll be visiting Aachen/Düsseldorf/Heidelberg area in spring. Do you know of any local ML meetups open to general (ML engineer/programmer) public?
Unfortunately, not really. We used to have some RWTH internal meetups, although that has been somewhat interrupted since Corona, and not really recovered afterwards.
Aachen has quite a few companies with activity on NLP or speech recognition, mostly due to my professor Hermann Ney. E.g. there is Apple, Amazon, Nuance, eBay. And lesser-known AppTek. And in Cologne, you have DeepL. In all those companies, you find many people from our group. And then, at the RWTH Aachen University, you have our NLP/speech group, and also the computer vision group.
It's likely that Google will not be the last search company. What business model could a successor use to generate revenue without falling into the ad trap? Why didn't early search engines choose other models like subscription, revenue sharing with telecom companies, or turning queries into marketing signals for manufacturers/merchandisers?