Model Distillation Services
When a large model does the job but costs too much or answers too slowly at your volume. We train a smaller model on your task from a teacher you're allowed to learn from, then measure quality, cost per call and speed against the large model on your own test set.
Bring your monthly volume and current cost per call. We'll tell you whether a smaller model is worth testing.
Every engagement starts with a written go/no-go, including which teacher models your licences allow.
We build and hand over the model. Running it in production is a separate engagement.
What is model distillation?
Model distillation trains a small “student” model to copy what a large “teacher” model does on one task. The teacher answers thousands of your real inputs; the student learns from those answers. Because the student only has to be good at that one job, it can be many times smaller, which makes it cheaper to run, faster to answer, and small enough for your own servers or devices. It works for language models, vision models and search rerankers. What it can't do is make the student better than the teacher at things it was never shown, so we measure exactly where it falls short. It's one of three methods in our custom AI model development work.
When a large model is more than the job needs
- “The bill grows with every call.” A narrow task such as classification, extraction or routing runs millions of times a month on a frontier model.
- “It answers too slowly.” The output arrives after the moment it's needed: autocomplete, live suggestions, routing a call.
- “It doesn't fit where it has to run.” A camera on a production line, a phone, an on-premises server without large GPUs.
- “Our search reranker is too slow.” Re-ordering results with a large model blows the latency budget.
If none of these applies, distillation probably isn't your next step.
Distillation, fine-tuning or quantization?
| Question | Distillation | Fine-tuning | Quantization |
|---|---|---|---|
| What it does | Trains a new, smaller model to copy a larger one on your task | Further trains a model on your examples so it learns a behaviour | Stores the same model's numbers at lower precision |
| Changes the model's size | Yes, a lot | No | Yes, moderately |
| Changes its behaviour | It learns the teacher's behaviour on your task | Yes | Very little, if done well |
| Needs labelled data | No: the teacher labels your inputs | Yes | No |
| Use it when | A large model already does the task well | No model does the task well yet | You want a cheaper run of the same model |
They're often combined: distil, then quantize the student. Pruning (removing parts of a model that contribute little) is a related technique we use less often. For behaviour no model has yet, see AI model fine-tuning. For cost work that doesn't involve training, such as caching, batching and routing, see AI inference cost reduction.
Is distillation legal? Which teachers are allowed
Distillation is legal when the licences allow it, and two licences always matter: the teacher's, and the base model the student is built on. DeepSeek is a clear example. DeepSeek-R1 is MIT-licensed and explicitly allows “distillation for training other LLMs”, yet DeepSeek's own distilled models stay under the licences of the models they were built on: Apache 2.0 for the Qwen-based ones, the Llama licences for the Llama-based ones (DeepSeek-R1 model card).
| Model | What its terms say | Use as a teacher? |
|---|---|---|
| OpenAI | Output may not be used “to develop models that compete with OpenAI” | Not for a competing model |
| Anthropic (Claude) | No training on outputs (“model scraping” or “model distillation”) without authorisation | Not without authorisation |
| DeepSeek-R1 | MIT; allows “distillation for training other LLMs” | Yes |
| Llama 3.3 | Allowed; a model trained on its outputs and distributed or made available must start its name with “Llama”, and show “Built with Llama” | Yes, with conditions |
| Gemma 1–3 | A distilled model is a “Model Derivative” under Gemma's terms and use policy | Yes; Gemma's terms carry over |
| Gemma 4 | Apache 2.0 | Yes |
| Your own labelled data | Yours | Yes |
We check the teacher and the student base in the Fit Sprint and write the conditions into the fit memo: what you can ship, under what name, and with what attribution. Licences change, so the memo records the version and the date checked. This is an engineering review, not legal advice; your legal team signs off. (Checked November 2026.)
What we distil
Language models
- Teacher → student: a permitted large model → a small open-weight model, then quantized.
- Measured: task accuracy vs the teacher on held-out inputs, cost per call, P50/P95 latency.
- Research: a fine-tuned 770M-parameter model outperformed the few-shot prompted 540B PaLM model on a benchmark, using 80% of the available data (Distilling Step-by-Step, ACL Findings 2023). Small fine-tuned models came close to GPT-4 quality at 1/5 to 1/29 of the cost (arXiv 2312.14972). Cleo cut median response time by 53%, from 1,310 ms to 631 ms, with a smaller fine-tuned model (Cleo engineering blog).
Vision models
- Teacher → student: large vision foundation models → a compact detector or segmenter that runs on a camera or edge device.
- Measured: recall and false alarms per class vs the teacher, frames per second on the target device.
- Research: one 2024 study (revised 2026) distilled compact detection and segmentation models from SAM2 and OpenCLIP for fruit detection “without manual annotation”, and they “nearly match” models trained with abundant labels (arXiv 2411.16196). That's one domain. We test it on yours.
Search rerankers
- Teacher → student: a slow, accurate reranker (a cross-encoder, which reads the query and each document together) → a faster, smaller one.
- Measured: retrieval scores (Recall@k, nDCG) vs the teacher, latency per query.
- Note: this is newer practice with less published evidence. The Fit Sprint measures whether it holds up on your queries. See also embedding model fine-tuning.
What changes when it works
- Lower cost per call, measured on your hardware next to the teacher's.
- Faster answers, reported as P50 and P95 latency.
- It runs where you need it: a smaller instance, your own servers, or a device.
- Clean to ship: the licence conditions are known and written down before training.
- A known trade-off: the report shows where the student is worse than the teacher, so you can route those cases to the large model if you want to.
We don't quote savings before measuring your current cost per call.
How a distillation runs
| Stage | What happens | You decide |
|---|---|---|
| 1. Fit | Measure the teacher's quality, cost and latency on your task; check both licences; size the student | Gate 1: sign the go/no-go |
| 2. Baseline | Build a held-out test set, including your hard cases, approved by your domain expert | Approve the bar the student must reach |
| 3. Data | Run the teacher on your real inputs to create training data; clean and de-duplicate it | — |
| 4. Train | Train two or three student candidates of different sizes, then quantize | — |
| 5. Prove | Score students against the teacher on the held-out set; measure cost and latency on your hardware | Gate 2: accept the report |
| 6. Hand over | Package, document, train your team; your team re-runs the pipeline once | — |
Generating teacher outputs costs money too, so the fit memo includes it.
How you buy it: free fit call → Fit Sprint → Build. The Fit Sprint and the Build are each priced per scope.
What you receive
Fit memo (with both licences and their conditions) · held-out test set and teacher baseline · teacher-generated dataset and pipeline · student model (weights, quantized files, configurations, run log) · evaluation report (student vs teacher: quality, where it loses, cost per call, latency) · deployment package and retraining runbook.
You own the trained model, the dataset and the evaluation set once the work is paid for; we keep our own reusable tooling, and the base model's licence still applies. Need it connected to your product? We can scope that integration into the Build.
What we need from you
A defined task the large model already does well · a sample of real inputs, and the right to use them · your current cost per call and volume, if known · a domain expert to approve the test set · where the student must run and its latency target · a cloud account for compute (including teacher calls), and an owner after handover.
Not included
Hosting and monitoring the student after handover (your team runs it, with the runbook we hand over) · distilling from a model whose terms forbid it · making the student better than the teacher at tasks it wasn't trained on · legal sign-off (your legal team decides; we supply the facts).
Managed alternative: AWS Bedrock offers distillation for Amazon Nova and Llama models, within one provider (e.g. Llama 3.3 70B → Llama 3.2 3B). The student then runs on Bedrock. The fit memo compares it with an open-weight build.
What can go wrong, and how we catch it
| Risk | How it's caught |
|---|---|
| The student handles typical inputs and fails on rare ones | Hard cases are kept in the held-out set and reported separately |
| The teacher was wrong, and the student learns its mistakes | Expert review of a sample of teacher outputs before training |
| A licence changes after training | The fit memo records the licence version and date |
| Quantization costs too much quality | Scored before and after quantization; you choose the trade-off |
How to judge us
| Question | Crescent AI | Managed distillation (e.g. Bedrock) | Distillation agency |
|---|---|---|---|
| Licence check on teacher and student | Yes, in writing | Within one provider's models | Rarely mentioned |
| Choice of teacher and student | Any permitted pair | Fixed pairs | Varies |
| Language, vision and reranker models | Yes | Language only | Usually language |
| Student runs in your environment | Yes | On the platform | Varies |
| Best when | You need the decision, the licence answer and a model you can run | You already run on that cloud and its pairs fit | You want a long-running retainer |
Who leads this work
Yash Amin, founder of Crescent AI. About Crescent AI
Common questions
Is distillation of AI models legal?
Yes, when both licences allow it: the teacher's and the student base model's. OpenAI's and Anthropic's terms restrict it for competing models; DeepSeek-R1's MIT licence allows it. We check yours before any data is generated.
Does DeepSeek allow distillation?
DeepSeek-R1's model card says its MIT licence allows "distillation for training other LLMs". The student's own base model licence still applies.
What's the difference between distillation and quantization?
Distillation trains a new, smaller model. Quantization stores the same model at lower precision. We often do both.
Will the small model be as good as the large one?
On the task it's trained for, it can come close, and published studies show cases where it matches or beats a prompted large model. On everything else, it won't. The report shows exactly where it falls short.
How do you distil an AI model?
Run the teacher on many real inputs, train the student on its answers, then test the student on inputs neither has seen. The process section above shows each stage.
Can you distil Claude or GPT?
Not into a model that competes with them, under their terms. We use a teacher whose licence allows it, or your own labelled data.
Do you host the distilled model?
No. We hand it over with a runbook; your team runs it.
Bring us your monthly volume and cost per call.
We'll tell you whether a smaller model is worth testing, and which teachers your licences allow.
Mutual NDA before you share data · Fixed-scope Fit Sprint · Licences checked before any data is generated
Part of Custom AI Model Development · AI Model Fine-Tuning · Embedding Model Fine-Tuning · AI Inference Cost Reduction