As a (former) ChatGPT plugins developer, our business has absolutely tanked due to GPTs. Discoverability is nonexistent because search is just buried by spam. At least with plugins, there were only a few hundred to sort through, and most had some unique API they would plug into.
For context, we built ChatOCR- an OCR tool that lets users extract text from photos and PDFs. We made roughly $20k from 39,000 users over 6 months on the plugins catalog
There's a conflict in value proposition here, which is more drastic than the discovery issue. That might help you better contextualize the failure, and avoid similar risks in the future.
Namely, GPT is now a multimodal trained llm capable of doing OCR over PDFs and images. Given the accessibility of that feature, we expect that users do not try to discover OCR plugins anymore- they feel no need.
Out of curiosity, what was the contingency plan in the case that OpenAI did this? What rationales did you use to estimate the likelihood and severity of that risk? Were there good reasons to discount that risk?
We knew the time would come, but we built ChatOCR in a week. If we overthought the time horizon problem, we'd have 0 users and $0.
But also, GPT-4-Vision is multi-modal but does not specifically use OCR. Our tool is used mostly to extract text from documents and load it into context, and we still saw growth after OpenAI built this feature into ChatGPT.
You can't get a large organization to move fast and ship something in a week. So this was either a solo developer en devour, or at max low single digits.
Thank you for sharing your risk thinking here. I like that you are risk hungry, not prone to hesitation, and you achieved revenue on a shoe string budget.
I'm curious about the idea that your OCR-generated context might give better recall than whatever GPT uses to parse PDFs into context. That would support the idea that it is discoverability, and not feature parity, that is killing the product. Can we discuss benchmarks that show conclusively that your solution does better on QA accuracy with respect to a PDF, as compared to GPT4's own PDF mode?
Also I'm curious about the trajectory of active users after the feature dropped from OAI. Was there an inflection point where new active users per month went negative? How long did it take before that inflection happened? Do you still have enough users to keep the service up and gathering revenue, or do you plan to mothball? Do you have any plans to pivot the offering to other markets?
My analysis of the platform risk of the ChatGPT plug-in store depends on
1) risk ideas like Failure Modes and risk hedging. The outcome in the plug-in store is an example of a well known failure mode of platforms that grow very fast.
2) remaining up to date with trends in the llm literature, so we can for instance predict that major vendors would release multimodal llms
3) being aware of trends in tech platform dynamics. Specifically the trend for "platform enshittification" - a term coined by Corey Doctorow - which is the tendency for tech monopolies to exploit their own users and vendors when they can.
I'll talk about each of these a little bit more in depth.
General Risk Analysis
First I recommend reading academic business and cyber risk analysis textbooks. Not just popular business books like "Anti fragile" or "The Black Swan" (which are good), but also academic books that introduced Bayesian networks and Failure Mode and Effect Analysis. You're not going to get away from some math, but thankfully there are techniques that let you draw situations as network diagrams, which I personally find helps with my natural difficulty with mathematics and symbolic processing. You can do pretty decent risk analysis just using "napkin math" - rough graphical and numerical estimates that approximate risks and can sharpen your intuition.
I like to recommend cyber risk analysis too, since it makes sense to be able to accurately assess your organizational cyber risks as an entrepreneur. It shows us how beautiful and interdisciplinary risk management can be, and ideas like adversarial modeling, the attack kill chain, attack sequences, and the attack graph are valuable in ordinary business risk management, like coping with insider threats or adversarial competitors.
Technical Observation
Observing the development of technology is also important, and with AI we have the advantage that most of the development is happening in publicly accessible research papers on arxiv.
By following the AI literature and evaluating LLMs and their capabilities, I made these two observations:
1. Multimodal and OCR assisted LLMs were hot in the AI literature early last year, and that approach was very helpful for reducing the hallucination/bullshitting problem - ai's biggest usability problem to date. Some of these papers were from openai themselves.
From this we could assume that the major AI vendors would work very hard to integrate this feature, and suppose that third party OCR solutions would get kicked out of the market very quickly once they did. We have seen that with GPT, Gemini, Claude, and more.
2. GPT4 continues to beat the pants off of every single competitor, in the hundred+ benchmark papers I've read, in the public benchmarks like Chatbot Arena, and in my own personal and business use case benchmarking. There's basically no contest on quality, and the result is 100% capture of the market of serious users who depend on GPT4 quality and are sophisticated enough to evaluate AI quality.
From this we can assume that OpenAI is well positioned to do everything it can to retain this advantage, even up to compromising developers who think it is safe to use the GPT store to build general improvements for GPT's weaknesses.
To gain this technical insight, I highly recommend reading many AI papers, internalizing their insights, and if possible, working hands on with their experimental techniques. Being aware of AI's current weaknesses and benchmarks, and being able to name papers working on overcoming specific problems, allows us to predict what might be coming next from big vendors.
Platform Risk Analysis
To gain insight into why large corporations tend to hurt their own developer communities and users when they gain the opportunity, I highly recommend Cory Doctorow's 2023 article, "The Enshittification of Tik Tok". It covers a great deal of other examples besides - including Apple and Google's app stores which are relevant to understanding the outcome for the GPT plug in store. And it does a very good analysis of user and vendor exploiting strategies on Tik Tok.
Doctorow also writes a freely available book/audiobook, "The Internet Con: Seizing The Means of Comoutation", where he analyzes the issue and its contributing factors, and proposes a solution based on regulations. While his solution - heavy regulation - doesn't seem that realistic, it's a good analysis, especially for entrepreneurs who might want to build on top of platforms that have this form of risk.
Finally, here's a paper that builds on this idea and analyzes how platforms are able to use their algorithmic control over the users attention to increasingly disadvantage a variety of stakeholders, including users, vendors, and advertisers:
I recommend a variety of sources including literature on risk management itself, literature having to do with the technology itself in a large variety of application domains, and being aware of market trends and forces, like the pressure to take advantage of non-financial stakeholders once a platform reaches a market dominance.
My general AI business strategy is to focus on my sub niche of technical work where AI has not really penetrated - automating 2d spline processing operations that have been ignored by AI researchers due to their focus on point clouds, radiance fields, bitmaps, and meshes, which are more important and applicable to a huge variety of domains. I focus on providing geometric processing services directly to manufacturers dealing with 2d vector data from CMM systems.
The CMM vendor I'm specializing in has been lax in adopting automation technology, and the customers I'm selling to are very reluctant to invest in improved CAD software, even when it could reduce their own labor by a factor of three or more. So between those two gaps I have a niche which I have exploited for 3 years now, and which I expect to exist for at least another 3 years. I plan to turn my CAD automation tools into an attractive product by that time, and also hope to advance the state of the art AI techniques in reverse engineering 2D splines from CMM data.
I hope that lets me hold on to the niche until 2030. But I also would not be very surprised if my niche gets disrupted by one of the major CAD or CMM vendors in that time.
My worst case outcome seems to be that I would have to join a more solidly established corporation as an research manager, strategic analyst, or software engineer. That could be an improvement in my quality of life, given that my current yearly revenues are still under 60k. However I'm going to fight that with all I've got, because I love owning my own company so much! I am an autistic entrepreneur, and owning my own company has been a huge breakthrough in my Independence and ability to avoid burnout. I can get by on as few as 10 hours of labor a week, which would be very hard to find in a corporation.
Anyways, I hope that this message has been beneficial to you even though it grew a bit long. I find Risk Management very fun and rewarding to study and apply, and hope you will as well! I wish you luck in all your endeavors!
> Is OpenAI going to use the Amazon playbook of compete with the more popular products?
They overtly target universal capability for the model itself, it’d be surprising if they didn’t, within that, prioritize functionality that has demonstrated that it is in demand for use with the other existing model functionalities, which successful GPTs or plugins clearly demonstrates.
I've heard it called "Platform risk". Also "Playing in someone else's walled garden", or something along those lines. I realize that's not a term for the inevitable rugpull, but that's the closest I can think of.
That one is a little different. That's when Apple clones your app into their OS as a core feature, thereby completely killing your market.
It usually implies Apple deliberately studied your specific app or replicated it based on details revealed in B2B licensing/acquisition meetings, similar to what MS pulled with Stacker back in the 90s.
Do end users have to pay you each time or buy access or credits or something, or do you just get a cut of ChatGPT paid subscriptions when a paying user uses your plugin?
We use a third party plugins manager called pluginlab.ai. It manages auth and subscriptions for plugins users by prompting them to sign in when they hit a paywall.
One would think that an company creating cutting edge AI tools, would dog-food that AI biggly. Like, using it for discoverability, here, or having a chat-with-docs system, like groq does. But that doesn't seem to be the case. Though they do use it for prompt generation.
I wonder.... The prompt generators can't really go wrong. I mean, they can look better or worse, subjectively. But a QnA bot answering questions about docs can be objectively wrong. Is this calculated to improve the optics, and avoid a lot of hallucination complaints?
Not sure what you mean: OpenAI's App Store doesn't charge and has discoverability. (hence the viability of the spam, hence TFA)
OP is likely affected by the change from plugins to GPTs
Now, users have to manually select an app. Whereas before, you'd have a list of, say, 5 apps and the AI would attempt to intelligently determine which app to use.
OpenAI said plugins didn't find product-market fit, and that's why they moved on.
It's worth opining there's an air of rushing around doing nothing with their product development. Like instead of "what should it be?", and building it, and sustaining it, it's a pastiche of every startup shibboleth you've ever heard - sprints, if 10-20% of the user base / 10^7 users aren't using it within 3-5 months, it's time to Pivot (throw away the working solution we invested in)
That makes sense when you're at 100 users in an uncertain market but at their scale, it reminds me more of how Google ended up with the reputation it has.
* forgive me for not air-quoting app and app store, these aren't __apps__ per se, but it was immensely distracting, with an air of condescention, when I air-quoted everything
> Not sure what you mean: OpenAI's App Store doesn't charge and has discoverability. (hence the viability of the spam, hence TFA)
I think the point was that the app store charge acts as an effective filter for spam, enough so as to determine the sustainability of the app store ecosystem.
I could very well be mistaken though, that was just how I interpreted it.
For context, we built ChatOCR- an OCR tool that lets users extract text from photos and PDFs. We made roughly $20k from 39,000 users over 6 months on the plugins catalog