Yesterday I summarised 15 years of my web comments with Mistral for RAG augmentation, took a few hours. Great little model I can run even on a 2080ti. As small as it is, it understands English and executes instructions fluently.
It would be nice if they shared some before and after result
because I did some spot checks and I get the same output(using default Mistral) found in the python_code_instructions_18k_alpaca dataset.
Setting that aside, is there a guide on how to create good training dataset for fine tuning models like Mistral?
I'd say the original Alpaca paper is a good source of inspiration on how to create datasets, they even shared the script used to generate data using OpenAI API.
One thing that came to mind for creating datasets for other programming languages would be to start with this Python dataset and use GPT-4 to convert to equivalents in other languages. You can even automatically test each generates example and ask GPT-4 to correct any errors
Due to the default settings of `top_k`, `top_p`, and `temperature`, which introduce a probabilistic element to token selection, two runs of the same prompt with the same model may yield different outputs.
The first two parameters, `top_k` and `top_p`, define the subset of tokens eligible for selection, while `temperature` influences the probability distribution from which tokens are sampled.
Have you (or others!) done much research on using different subsets of target modules? Just doing q, k, v, o is dramatically less ram and much more accessible (mistral 7b on 3080ti)
They are different parts of the network. If you don't fine tune them, they don't change. They are "frozen"- if you're familiar with fine tuning neural networks
