Make a local LLM useful with your company data and workflows

Imagine a procurement manager asking an AI assistant whether a supplier meets company policy. The answer sounds convincing. It also uses last year's policy and a contract the manager should never have been able to open.

Putting the model on a company server would not, by itself, fix either problem.

A local LLM can give a business greater control over where inference happens, which documents it uses and how it connects to internal systems. The value appears when that control helps someone complete real work with less searching, fewer repeated explanations and a clear route for checking mistakes. Buying a GPU is the easier part.

Start with the question your employees cannot answer quickly

Consider a service team handling equipment questions. An experienced engineer knows which manual applies to each model, where the latest service notice lives and which exception needs a supervisor. A new colleague searches three folders and eventually interrupts that engineer.

A useful corporate AI assistant could retrieve the relevant passage, identify the equipment revision and link to the service notice. The engineer stays responsible for the procedure. The assistant handles the search.

That is a more testable business case than giving everyone another chat window. Measure time to find a usable answer, how often the correct document appears and how much checking remains. If staff still need to repeat the entire search, the assistant has added a step.

Start by collecting the questions people actually ask, including the awkward ones. Which version applies? Who can approve the exception? Is that price still valid? What information is missing? These questions expose the work that a general model cannot infer from its training.

Decide what you mean by local

A local LLM runs inference on a company workstation or an on-premises server. A private cloud deployment is a separate option for running a model on infrastructure you control. It is not the same as keeping the system on your premises. Neither arrangement automatically means the whole application is offline.

For example, Ollama documents a local-only mode that disables its cloud models and web search. That setting concerns Ollama. It does not determine where a separate document parser, embedding service or application sends information. Ollama documentation

Before describing a corporate AI system as private, trace its actual data path.

An air-gapped system needs additional controls and an operating process for updates. A private cloud system still involves infrastructure providers. An approved enterprise service may meet the business requirement without self-hosting. Decide which boundary you actually need before selecting hardware.

Connect company knowledge without teaching the model every document

For frequently changing policies, product manuals and operating procedures, retrieval-augmented generation is a useful starting architecture. Usually shortened to RAG, it retrieves relevant source material and gives it to the model as context for an answer.

This is different from retraining the model whenever a document changes. Open WebUI, for example, supports document retrieval and source citations in its chat workflow. Those features make a prototype possible, but do not establish that its answers are correct for your business. Open WebUI RAG documentation

Treat the document collection as a maintained business resource. Assign an owner to each source. Preserve revision dates, document identifiers and access rules. Remove superseded material from active retrieval, or clearly label why it remains available. A source link helps only if it leads to the right source.

Permissions belong in retrieval and application logic. If someone cannot access a contract in the original system, the assistant should not receive passages from it on that person's behalf. A prompt saying “do not reveal confidential information” is not an access-control system.

Fine-tuning may help a model follow a specialised format or perform a repeatable task. It is not a substitute for document permissions, nor a convenient way to keep yesterday's policy out of today's answer. Start with retrieval when the main problem is access to current knowledge.

Fix the documents before blaming the model

A scanned table can lose its columns during extraction. A heading can become detached from the paragraph it qualifies. A footnote can turn a seemingly clear rule into an exception.

Inspect the extracted content before putting it into a search index. Tools such as Docling can process documents into structured representations and support local execution. Their usefulness still depends on your files, especially scans, tables and unusual layouts. Docling project

Try a small, deliberately difficult sample. Include an old scan, a table with merged cells, a revised procedure and a document containing internal cross-references. Compare the extracted result with the original. If the parser associates the wrong number with a line item, a more eloquent model will not repair the evidence reliably.

For work involving invoices, separate document extraction from calculation and approval. Our invoice-to-Excel tool is a way to explore the extraction and review step. It is not a replacement for your accounting controls.

Give the assistant a bounded job

A company knowledge base becomes more useful when it supports a specific next action. The following are proposed workflow designs, not claims about client results.

In procurement, the assistant could assemble a supplier review from approved questionnaires and current purchasing policy. It should identify missing evidence and prepare a draft for the responsible buyer. Supplier approval remains a controlled decision.

In customer operations, it could draft a response using the customer's authorised contract terms and a current service record. A pricing exception should be routed to the right person rather than quietly invented in a helpful tone.

For maintenance teams, it could locate the relevant manual section and show its revision. Safety-critical instructions still need the established verification process. A plausible answer is not permission to restart a machine.

The boundary matters even for internal documents. OWASP describes how instructions hidden inside retrieved material can influence an LLM. RAG and fine-tuning do not remove that risk. Limit tool permissions, treat retrieved content as untrusted input and require approval for consequential actions. OWASP prompt-injection guidance

The useful handoff includes evidence and a named reviewer, not just a confident answer.

Compare operating costs, not just model prices

A downloadable model can have no usage fee and still be the more expensive business option. Hardware, hosting, maintenance, identity integration, document updates and support all need funding. Someone also has to handle the Monday morning incident.

Local deployment becomes more attractive when a clear control requirement justifies it, usage is substantial enough to use the infrastructure sensibly and the team can operate it. A managed service may be a better fit for occasional tasks or a business without that operating capacity.

Model size alone will not tell you how responsive the application feels. Test the intended number of simultaneous users with realistic document lengths. Ollama's documentation notes that concurrent requests and context length affect memory requirements and queueing. Ollama concurrency guidance

Compare options on the same work. Keep the questions, source documents and review criteria constant. Our AI model selector for business can help structure an initial shortlist, but a decision involving confidential data needs a separate deployment and security review.

Make a pilot earn the next investment

Use a four-week pilot as a planning shape, not a promise of production readiness. Its scope should be narrow enough that failure tells you something useful.

In the first week, choose one workflow and its owner. Gather a proposed test set of 50 to 100 representative questions, along with source-backed expected answers. Include questions the assistant must refuse, questions with no available answer and cases involving conflicting revisions. Record how the team handles this work today.

Next, build the smallest read-only version that can answer those questions. Test whether sources are retrieved for the correct user before polishing the interface.

Then ask actual users to work through representative cases. Record corrections, missing evidence, waiting time and the effort needed to verify each answer. Repeat the same tests after changing the model, prompt or document pipeline. Evaluation tools such as Promptfoo support explicit output checks that can help make these comparisons repeatable. Promptfoo assertions documentation

At the end, make the decision against criteria agreed beforehand.

• Can users reach a verified answer faster than with the existing process?
• Are citations correct and document revisions visible?
• Does the system withhold material outside the user's permissions?
• Does it admit when the evidence is missing?
• Are operating cost, ownership and recovery procedures acceptable?

Stop expansion if access boundaries fail or users cannot reliably check the output. A polished demo does not compensate for either problem.

Estimate the economics using observed review time rather than assuming every generated answer saves work. The AI agent ROI calculator can support scenario planning. Its result is an estimate, not evidence of realised savings.

Make knowledge easier to use without making authority ambiguous

The procurement manager's first question was reasonable. The system failed because it supplied the wrong evidence and ignored a permission boundary.

A local LLM can be part of a better answer. Its advantage is the control you can exercise over the complete workflow, provided you actually implement and maintain those controls.

Bring one expensive, repetitive knowledge task to an Amalgama assessment. Start with the documents, the people allowed to use them and the decision that follows. You can choose the model once you know what it has to do.