Local AI: when does it make sense?
I run models on a DGX Spark. I want to know which ones stay usable as documents get longer and several people ask questions at once.
Updated 14 September 2026
What actually runs locally?
On this page, local AI mainly means language models you run yourself. Speech recognition and image analysis can also run locally, but I do not test those here.
The model runs on your own computer or server. You can keep the documents it searches and the processing of questions there too. That could be a workstation, an office server or your own cluster.
That doesn’t mean everything is offline. An integration, log file or backup can still send data elsewhere. Check what runs around the model, who has access and where the data ends up.
Where does it run, and who manages it?
On-premises means the application runs on your organisation’s infrastructure. A private cloud is an isolated environment that may also be managed by a supplier. European hosting describes location, but does not by itself establish who has access or legal control.
Check the whole chain: model, document storage, search index, logging and administration. A Dutch language model can run in a foreign cloud. A foreign model can run on your own server.
Why this matters for organisations in the Netherlands
Accounting firms, municipalities and hospitals hold knowledge in documents that cannot simply be shared. AI can help retrieve that information or prepare a draft. Local processing can give you more control over access and the processing environment. Whether it provides sufficient protection depends on the implementation.
I want to make that confidential expertise available to staff who are allowed to use it. This does not require every office to buy its own server. A jointly managed environment can also work, provided responsibilities, access and the ability to switch providers are clearly agreed.
TNO identifies digital autonomy, transparency and Dutch language and context as principles for GPT-NL. This supports local expertise and alternatives to suppliers. It does not prove that a local model speaks better Dutch, costs less or is safer. Test those claims for each application.
When would I choose it?
Local AI becomes useful when you repeat the same tasks, need documents to stay in your environment or want a fixed model version. You still need a model that does the job well enough, and someone to maintain the installation.
If you’re still working out what to build, expect little use or have unpredictable demand, a cloud API is often more practical. You don’t have to provision capacity yourself. Quality, prices and limits do depend on the provider.
You can combine the two. A recurring task could run locally while a harder question goes to an external model. Decide which data may go with it and how you’ll check that the routing works as intended.
| Consideration | Local | Cloud API | Hybrid |
|---|---|---|---|
| Data | You manage the model and processing. Check logs and integrations too. | You send input to a provider. Check the processing agreement. | Define which data may leave for each task. |
| Kwaliteit | Test whether the chosen model does your task well enough. | Choose from the models the provider offers. | Use different models for different tasks. |
| Capacity | Your hardware limits how much can run at once. | Depends on quotas, limits and availability. | Split the load with explicit fallback rules. |
| Maintenance | You handle updates, monitoring and recovery. | The provider manages inference; your application remains your responsibility. | Maintain both the local setup and the external connection. |
| Kosten | Purchase, electricity, maintenance and replacement. | Usage and service, including variable demand. | Both cost categories, depending on how tasks are split. |
Where could you start?
These are tasks you could use for a small pilot. I have not tested these applications myself. Check whether AI helps with each task and whether running it locally adds anything.
| Organisation | Possible first application | Why consider local processing? | What needs checking? |
|---|---|---|---|
| Accountants | Collect information from a client file with references to the original documents. | More control over processing confidential client information. | Verify amounts and conclusions. Use software designed for calculations to do the arithmetic. |
| Lawyers | Compare contract versions or prepare a case chronology. | Control over confidential documents and file access. | Check legislation, case law and interpretations against the original sources. |
| Municipalities | Search internal policy documents and prepare draft replies. | Control over information services and processing residents’ data. | Start with assistance, not autonomous decisions about rights or benefits. |
| Hospitals | Search internal protocols; later, controlled drafting of documentation. | More control over sensitive information and integrations. | Use by healthcare professionals requires appropriate validation. Clinical decisions require more scrutiny than administrative assistance. |
These examples align with the NBA’s focus on control, the NOvA’s emphasis on professional responsibility and government guidance on responsible use. Nictiz investigates AI to reduce administrative burden in healthcare. These sources support careful use, not the superiority of local hosting.
A question in an accounting file
Illustrative scenario, not my own client case or a measured result.
Suppose an employee asks: what payment terms were agreed in client A’s file? The application first searches documents the employee is authorised to access. The search index must enforce those file permissions too.
The language model receives the retrieved passages and drafts an answer with references to the document, version and page. The employee opens the passages and checks the answer. Missing or contradictory documents must remain visible.
I would assess this pilot using documents checked in advance: does the system find the correct agreement, cite the right source and keep other clients’ information out? Include questions the file cannot answer. Only then measure whether total working time, including review and corrections, decreases.
I started measuring to understand the speed and capacity of my hardware. These measurements are in the Arena. They help with hardware choices, but you need your own examples to test whether a model does your work well.
The chart compares two precisions of one Gemma model on my DGX Spark. Precision concerns how model numbers are stored and processed. Lower precision can save memory and change speed, but may also affect answers. A token is a piece of text, not necessarily a whole word. This is neither a cloud-model comparison nor a quality test.
One model in two precisions
Gemma-4-26B-A4B-it on one DGX Spark, in a chat conversation. The two recorded profiles side by side.
Output speed per user
More tokens per second is faster.
Time to first response
Fewer seconds means less waiting.
These are speed measurements, not a quality test. Different precision can change the answers. Each run links to its source. The table shows the mean ± standard deviation.
All measurements and sources
My public measurements run on one DGX Spark. They show what happens on that machine under the stated conditions. These are speed measurements, not quality tests. A different precision can change the answers. The source for each run contains the details. The table shows the mean ± standard deviation.
| Workload | Profile | Output speed per user | Time to first response | Measured | Source |
|---|---|---|---|---|---|
| A chat conversation | BF16 | 10.55 ± 0.35 tokens/s/user | 1.56 ± 0.53 seconds | 2026-08-06 | Source |
| A chat conversation | NVFP4 | 21.1 ± 1.12 tokens/s/user | 1.28 ± 0.4 seconds | 2026-08-06 | Source |
| Searching documents | BF16 | 8.74 ± 0.68 tokens/s/user | 8.8 ± 4.57 seconds | 2026-08-06 | Source |
| Searching documents | NVFP4 | 16.06 ± 2.1 tokens/s/user | 8.17 ± 4.23 seconds | 2026-08-06 | Source |
| Writing long answers | BF16 | 11.36 ± 0.86 tokens/s/user | 0.51 ± 0.05 seconds | 2026-08-06 | Source |
| Writing long answers | NVFP4 | 22.94 ± 0.76 tokens/s/user | 0.38 ± 0.01 seconds | 2026-08-06 | Source |
What running locally does not solve
A language model can invent facts or miss a relevant exception. A citation helps with checking but does not guarantee that the conclusion follows from the source.
A document may contain instructions intended to manipulate the model. Treat retrieved text as data, restrict tools and test for these attacks. File permissions, updates and security remain necessary without an external API.
The GDPR still applies. The Dutch data protection authority has explicitly noted that the same legal requirements are relevant to locally run models. Assess the purpose, legal basis, data use, retention and need for a DPIA with the responsible privacy and security specialists. Applicable rules also depend on what the application does.
More control also means more work. A poorly managed local installation can be less secure than a well-configured service. Include maintenance and recovery costs, and assess whether the environment is available enough for the task.
Dutch DPA: legal requirements also apply to local use (2023 letter)
What I’d check before buying hardware
- Can the model do the work? Test with real examples you have permission to use. Define acceptable errors and who reviews the answers. A general benchmark won’t tell you whether your documents are processed correctly.
- What leaves your network? Follow the documents, questions and answers. Include retrieval sources, logs, backups and administration. A local model alone doesn’t guarantee privacy.
- Who keeps it running? Someone needs to handle updates, failures and model changes. Test what happens when the machine is at capacity or goes down, and how work continues.
- How much will you use it? Include the purchase, electricity, maintenance and replacement. An idle machine can cost more than an API. Predictable use makes the calculation easier.
- Can you change course later? Keep your configuration and make sure your data can move to another system. Owning hardware can still leave you dependent on a single model, engine or custom platform.
What I’ve investigated
My public measurements run on one DGX Spark. They show what happens on that machine under the stated conditions.
- Compare models in the ArenaThroughput, memory and individual measurements.
- My DGX Spark setupRunning models, context length and software choices.
- The cost of a DGX SparkPurchase, electricity and usage, with the assumptions included.
Start with one task
Choose a problem you need to solve now. Collect examples, define acceptable results and test long documents and concurrent questions. Check errors and recovery as well as speed. Then you can make an informed choice between local, cloud or a combination, and estimate the hardware needed.