I've thought about it and it would make the whole project even cooler with actual stats of "birdhouse regulars" not just in aggregated form but I don't know if it's possible and if bird faces have enough unique features to differentiate them?
Right now I only use it so that my thumbnails of pictures from the camera are centered on the head in the UI as I couldn't find a pre-existing model that does it for animals. I'm thinking that maybe having this data set of a few hundred bird faces will allow me to train a small one in the future to do it more automatically. If not...I at least learned something new about building models!
Yup, that's exactly what parent comment is saying.
Let's say your beverage LLM is there to recommend drinks. You once said "I hate espresso" or even something like "I don't take caffeine" at one point to the LLM.
Before recommending coffee, Beverage LLM might do a vector search for "coffee" and it would match up to these phrases. Then the LLM processes the message history to figure out whether this person likes or dislikes coffee.
But searching SQL for `LIKE '%coffee%'` won't match with any of these.
The basic idea is, you don't search for a single term but rather you search for many. Depending on the instructions provided in the "Query Construction" stage, you may end up with a very high level search term like beverage or you may end up with terms like 'hot-drinks', 'code-drinks', etc.
Once you have the query, you can do a "Broad Search" which returns an overview of the message and from there the LLM can determine which messages it should analyze further if required.
Edit.
I should add, this search strategy will only work well if you have a post message process. For example, after every message save/upddate, you have the LLM generate an overview. These are my instructions for my tiny overview https://github.com/gitsense/chat/blob/main/data/analyze/tiny... that is focused on generating the purpose and keywords that can be used to help the LLM define search terms.
That’s going to be incredibly fragile. You could fix it by giving the query term a bunch of different scores, e.g. its caffeine-ness, bitterness, etc. and then doing a likeness search across these many dimensions. That would be much less fragile.
You could instruct the LLM to classify messages with high level tags like for coffee, drinks, etc. always include beverage.
Given how fast interference has become and given current supported context window sizes for most SOTA models, I think summarizing and having the LLM decide what is relevant is not that fragile at all for most use cases. This is what I do with my analyzers which I talk about at https://github.com/gitsense/chat/blob/main/packages/chat/wid...
Honestly Gemini Flash Lite and models on Cerebras are extremely fast. I know what you are saying. If the goal is to get a lot of results where they may or may not be relevant, then yes, it is an order of a magnitude slower.
If you take into consideration the post analysis process, which is what inference is trying to solve, is it an order of a magnitude slower?
It has become fast enough that another call isn't going to overwhelm your pipeline. If you needed this kind of functionality for performance computing perhaps it wouldn't be feasible, but it is being used to feed back into an LLM. The user will never notice.
Your readmes did a great job at answering my question "why is this file called 1.md? What calls this?" when I searched for "1.md". (The answer is 1=user, 2=assistant, and it allows adding other analyzers with the same structure.)
The number is actually the order in the chat so 1.md would be the first message, 2.md would be the second and so forth.
If you goto https://chat.gitsense.com and click on the "Load Personal Help Guide" you can see how it is used. Since I want you to be able to chat with the document, I will create a new chat tree and use the directory structure and the 1,2,3... markdown files to determine message order.
If an LLM understands that coffee and expresso are both relevant, like the earlier comment suggests, why wouldn't it understand that it should search for something like `foo LIKE '%coffee%' OR foo LIKE '%expresso%'`?
In fact, this is what ChatGPT came up with:
SELECT *
FROM documents
WHERE text ILIKE '%coffee%'
OR text ILIKE '%espresso%'
OR text ILIKE '%latte%'
OR text ILIKE '%cappuccino%'
OR text ILIKE '%americano%'
OR text ILIKE '%mocha%'
OR text ILIKE '%macchiato%';
(I gave it no direction as to the structure of the DB, but it shouldn't be terribly difficult to adapt to your exact schema)
Implementations that use vector database do not use LLMs to generate queries against those databases. That would be incredibly expensive and slow (and yes there is a certain irony there).
Main advantages of a vector lookup are built-in fuzzy matching and the potential to keep a large amount of documentation in memory for low latency. I can’t see an RDMS being ideal for either. LLMs are slow enough already, adding a slow document lookup isn’t going to help.
You could ask an LLM to provide categorizarions for nouns and verbs, and store those. For ”I don’t like cappuccino”, you’d get back ”self”, ”human”, etc. for ”I”; ”negation” etc. for ”don’t”; ”preference”, ”trait” etc. for ”like”; ”coffee”, ”hot”, ”drink”, ”beverage” etc. for ”cappuccino”.
It would become unwieldy real fast, though. Easier to get an embedding for the sentence.
An actual use case I had for vector DBs was when users were using "credit card", "kredit kad", "kad kredit", "kartu" interchangeably.
If you're matching ("%card%" OR "%kad%"), you'll also match with things like virtual card, debit card, kadar (rates), akad (contract). The more languages you support, the more false hits you get.
Not to say SQL is wrong, but 30 year old technology works with 30 year old interfaces. It's not that people didn't imagine this back then. It's just that you end up with interfaces similar to dropdown filters and vending machines. If you're giving the user the flexibility of a LLM, you have to support the full range of inputs.
> The more languages you support, the more false hits you get.
Certainly you're at the mercy of what the LLM constructs. But if understands that, say, "debt card" isn't applicable to "card" it can add a negation filter. Like has already been said, you're basically just reinventing a vector database in 'relational' (that somehow includes MongoDB...) approach anyway.
But what is significant is the claim that it works better. That is a bold claim that deserves a closer look, but I'm not sure how you've added to that closer look by arbitrarily sharing your experience? I guess I've missed what you're trying to say. Everyone and their brother knows how a vector database works by this point.
A. Last month user fd8120113 said “I don’t like coffee”
B. Today they are back for another beverage recommendation
SQL is the place to store the relevant fact about user fd8120113 so that you can retrieve it into the LLM prompt to make a new beverage recommendation, today.
It’s addressing the “how many fucking times do I fucking need to tell you I don’t like fucking coffee” problem, not the word salad problem.
Just to be clear, are you looking for a provider to process these requests on your behalf for 3rd parties or are you wanting to ask Claude to process your Erasure request to Calude's owner Anthropic, for example?
The latter. They currently provide a method for requests related to account information but not for the information contained in their models or training datasets.
Figure out how AI can help the blind and the deaf. Take up ceramics. Write a travel book. Tinker with low-cost robotics. Volunteer for some environmental restoration project - the riverbank ones seem interesting.
We have several processes which involve the upload of multiple different images, which the user is walked through with baby steps, I.e. 'take this exact picture of the asset' -> take the next, etc. Etc.
We're currently working on an image classification model to automatically tag these images with what they are and just providing a visual sample gallery of 'take these pictures' with the sample image border turning green when that type of image is identified. We expect this will reduce a multi-step error prone process with a more streamline experience.
You probably don't want to hear this but that problem doesn’t require AI to solve. We've had asset tags and QR codes for decades and won't require re-training if a new equipment manufacturer comes into play.
I very much want to hear this! The issue we're facing is that the maintenance staff is very undertrained on a specific subset of equipment. But that equipment is actually very critical. Also I was vague in the first post - in reality this is for hospitals within a system, not one single building.
When you say "won't require retraining if a new equipment manufacturer comes into play" I don't really follow.
Edit: I get it now - you mean by using QR we wouldn't need to retrain the image recognition AI. It's a good concern, although in this specific area there is not a ton of innovation or newcomers. Worth keeping in mind though.
The most modern idea seems to be a Dyson swarm, which is not solid and consists of solar panels and habitats orbiting at a comfortable distance. Power is relayed using microwave transmitters and receivers and conceptually a large portion would be fed into particle accelerators for the manufacture of Anti-Matter on industrial scale.
