Learn a thing about AI
LLM··2 min read

Making local llm based AI Assistants work in consumer grade devices

I thought fine tuning of models is to inject app specific data in to the model, but i found out recently that this could be used to make a model work well in consumer grade devices by injecting information to make the model more aware of app's system prompt specific expected outcomes.

llmfine-tuningagentsanjadhe
Ramachandran Bakthavachalam (Ram)Ramachandran Bakthavachalam (Ram)

I've been trying to improve the Anjadhe ai assistant to work with local models on a 16GB RAM Mac - my current MacBook.

Since none of the models worked well, I thought hosting models in the cloud and offering that as an option for smaller Macs was the only viable path.

But today I came across an alternative.

Fine-tuning a smaller model (Qwen 3.5 4B) with LoRA.

Until now, I assumed the purpose of fine-tuning with LoRA was to make the model aware of our app specific data - like giving the model app-specific context instead of feeding it through the prompt or RAG techniques. But today I learned an approach to fine-tune the model to make it learn how to respond to questions specific to system prompt of the app.

My assistant's system prompt is about 2,000+ characters of instructions and tool definitions. Instead of feeding all of that into the prompt on every call, if you add a few layers to the model itself using LoRA - trained on the system prompt plus question/answer combinations derived from a teacher model - small models do much better.

I validated this with evals. These were evals that passed on a bigger model - one too large for smaller machines (Qwen 3.6 35B) - but failed on smaller models. After this fine-tuning, the small model now passes the same evaluations.

The downside: users can't use any random small model for the assistant - only the small model I fine-tune with app-specific data. I believe this is fine, given the model stays fully local and the user's privacy is protected. It also creates a dependency between the system prompt and the model, which isn't ideal. But I believe fine-tuning needn't be repeated for small changes to the prompt - only for major ones.

This approach uses knowledge distillation to move from in-context learning to intrinsic learning.

So if you have app-specific use cases that aren't working well on the model of your choosing, you don't have to go to the next bigger model. Give fine-tuning a try: identify the evals that should work, have a bigger teacher model generate the answers for your use cases, and use them to add adapter layers to your existing model. One-time fine-tuning with LoRA needs extra compute for a few hours, but could save a lot more inference compute.