Open your supplied interface. Use the local address and sign-in instructions in your handover guide. Allow the model to finish loading.
Start with one small task. Provide the context, the result you need and the output format. Try one of the examples tested during handover.
Check the result. Review facts, calculations and any proposed file changes before using them. Refine the prompt with what needs to change.
Save a useful workflow. Keep the prompt and a known-good example so you can compare results after future changes.
Local inference runs on your workstation. Updates, web searches and connected services may use the internet; your handover guide lists the connections in your setup.
02 / CHOOSE THE RIGHT TOOLS
Model, runtime, agent. What’s the difference?
The model generates responses. The runtime loads it and serves requests. An agent harness manages the conversation, tools and steps around the model so it can work through a task.
A chat interface for everyday use
For drafting and asking questions, a chat interface can be enough. Open WebUI documents connecting to a local vLLM server. Read Open WebUI’s vLLM guide ↗
An agent harness for tool-based work
For tasks involving files, code or multiple steps, assess tool permissions, approvals and recovery alongside model quality. OpenHands is one option to explore for coding workflows. Explore the OpenHands project ↗
These are starting points for evaluation. We agree the interface and supported tools for each deployment, then test them with the selected local model.
For a short task, provide the relevant text with your question. For a larger collection, a document-retrieval workflow can find useful passages for the model to reference.
Begin with a small, representative set of documents. Ask questions with known answers, check the cited passages and include questions the documents cannot answer.
Document retrieval and model training are different approaches. Retrieval supplies context at request time; fine-tuning changes model weights. The right approach depends on the task and the data.
GPU memory holds model weights and working state. Model size, quantisation, conversation length and simultaneous requests all affect what fits. System RAM and SSD space serve different roles; extra RAM does not add GPU memory.
Start with the model and settings validated for your AI Station. Before adding a larger model or more users, compare response time and quality on your actual tasks.
