Capable AI on CPU-only, constrained hosts

There is a quiet assumption in most AI conversations: that useful models require GPUs, cloud budgets and a data centre. In our experience building learning tools for people on modest hardware, the opposite is increasingly true. A well-chosen small model, run on a CPU, solves a surprising number of real problems — faster, cheaper and more privately than the alternative.

What we actually run

Two of our projects are exercises in doing a lot with a little.

Our voice backend is a reusable HTTP voice backend — text-to-speech and streaming speech-to-text — designed for constrained hosts. It pairs a compact ONNX text-to-speech model with a streaming speech-recognition model, and it runs on CPU only. No GPU, no vendor account, no per-request fees. You can deploy it on a small VPS and it stays up.

A one-billion-parameter model on llama.cpp is our minimal, production-ready deployment of a one-billion-parameter language model. It ships a prebuilt CPU binary (no compiler needed on the host) and exposes an OpenAI-compatible API, so existing tooling just works. A 1B model will not write your novel. It will classify, summarise, extract, route and draft — reliably, locally, for free.

The trend is real

We are not alone in this. This week, researchers at NTU Singapore published a framework that uses ordinary language models to generate specialised decision-making software — port schedules, delivery routes, staff rosters — in about an hour and for under ten dollars in compute, work that used to take engineers months. Notably, the cheaper models did almost all of the value: a fast, low-cost model reached 0.95 of the best result for $1.59, while the strongest model added a few points for roughly 28 times the price.

That ratio is the whole story of practical AI right now. The last few per cent of quality is real, but it is expensive, and most tasks do not need it.

How to get there: a short playbook

• Pick the smallest model that clears your bar. Define the task narrowly, set an acceptance threshold, and test upward until it passes. Do not start at the top.
• Quantise. Four-bit quantised weights (the GGUF format, for example) cut memory and cost dramatically with a small quality loss. For narrow tasks, that trade is almost always worth it.
• Run it on the CPU you already have. With a good inference engine, a 1–3B model answers in interactive time on a modern CPU. No accelerator required.
• Keep it local. Local inference means no data leaves the machine, no per-token bill, and no dependency on someone else's uptime.
• Measure, then decide. Benchmark on your own inputs. Published scores are a starting point, not a verdict.

The point

The frontier gets the headlines, but the durable value is in small, local, well-scoped models that do one job perfectly and cost nothing to run. That is where we build — and it is where a lot of the next wave of software will be built too.