Member-only story
How I’m Choosing an Open Model Before Buying the Hardware
A practical way to select a local model when privacy is fixed but almost everything else is uncertain
I am planning a local AI inference system for confidential business data. The main workloads will combine documents and audio: transcribing recordings, summarizing them, extracting information, and eventually making the source material easier to search and use.
The privacy requirement makes one part of the decision simple. The data needs to remain inside an approved private environment. My preference is to start with models running on hardware we control, but that is not the only possible boundary. A managed model service such as Azure OpenAI or Gemini on Vertex AI could become a controlled fallback. A managed GPU platform such as Modal could also help me test an open model before I own the hardware.
My first instinct was to ask how large a model I could run within the hardware budget. Could I run a 70 billion parameter model? What about a large mixture of experts model? If some jobs can run overnight, perhaps I could accept slow inference in exchange for a more capable model.
I now think this might be looked at from a different perspective. The largest model that fits in memory may still be a poor fit for the work. It might take too long to process a long document, leave too little memory for the context I need, or support only one request at a time. A smaller model…
