If your data is sensitive, the advice is to host the model internally rather than send it to a public service. Reasonable enough. But what does that actually entail, and how big is too big?
First, what the thing actually is
The simplest analogy for a large language model is a sieve. A better one is a Pachinko machine, except the pins are of variable size and spacing, and there are dozens or hundreds or thousands of balls being poured in at the top. Those are your input. The balls that come out of the bottom are the output, and they are not the same balls you put in. They were released by the path the inputs took through the pins.
In reality there are billions of pins, and they were positioned by the training. That is where most of the heavy lifting happened. But processing an input still requires real computation at the moment of use, potentially for many people at once, and any private data set has to be processed as well.
So how big is too big?
The figures in this section are from March 2024, when this was written. Hardware and model efficiency have moved considerably since; treat them as the shape of the problem rather than a current quote.
At the top end, ChatGPT needed something like a terabyte of RAM in total and a decent-sized cluster of GPUs. Not something the average small or mid-sized business is going to deploy.
At the other end, a $3,000 single-user workstation could generate code quite well with a private instance of Llama.
Most real needs sit somewhere in between, and this is the part people miss: most in-house AI work does not require a general-purpose model at all. Even where the private data is large and complex, a number of techniques exist to cut the computing power needed for a useful deployment.
The middle option, and what it actually changes
There is also private hosting, for the initial intake of the data, for subsequent queries, or both. That lets you hand the heavy lifting to somebody else while keeping control of the data and of who can reach it.
Are there still security concerns? Of course. But they become the same concerns as any other private hosting arrangement. Handling the workload this way removes the AI data-leakage problem specifically and reduces what is left to a well-understood space. That is a genuine improvement, and it has a price attached.
The questions that decide it
- How much private or sensitive data is actually involved?
- Can the size of that data set be reduced?
- Do you have staff who understand the data well enough to curate it for ingestion?
- Will there be an ongoing intake of new data, or is this a one-off?
- How accurate do the outputs actually need to be?
That last question does more work than people expect. The accuracy you need determines the curation you need, and the curation is usually the expensive part.
And if that is all too involved and you simply want a number to plan against: in March 2024 the bare minimum for a private, secure deployment against your own internal data was around $5,000 per user. There is no maximum. As with the figures above, treat that as the order of magnitude it was at the time rather than a quote.
Itwerx Corp is a service-disabled veteran-owned small business providing IT services across Seattle, Bellevue, Everett and Snohomish County. This is the kind of thing our hybrid cloud work deals with – talk to us about yours.
