Setting up a home AI model

Setting up a home AI model

I decided it’s about time we implemented our own AI model for home use. I took two approaches – a RAG model for analyzing and working with documents, and a general chat agent. After a couple of days and some back and forth with Gemini, I came up with the following solutions.

System Specifications

This was installed on a 14-year-old HP desktop with original motherboard (Intel Core i7 with 4 cores), 12GB original RAM and 2012 NVIDIA Geoforce GTX 1650 (4GB) GPU. I installed a 128GB SSD and put Linux Mint (Cinnamon) on for the OS. I wasn’t expecting miracles, just to see what I could do with this old equipment.

RAG model solution

First I installed the NVIDIA desktop drivers, as the existing ones were Open Source. This requires the OEM drivers.

  1. Open the Mint menu, search for Driver Manager, and launch it.
  2. Select the recommended NVIDIA proprietary driver version (e.g., nvidia-driver-535 or newer) instead of the open-source driver
  3. Apply changes and restart your computer.
  4. Verify your GPU is recognized by opening a terminal and running nvidia-smi

If it prints out your video card status table, you are ready to proceed

I then decided to use Llama as the LLM. No real preference, just lack of experience in general with home AI models. I’ll eventually switch to Mistral, most likely. To use Llama I did the following:

Install Ollama via Terminal as a standard background service

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active

sudo systemctl status ollama

Next I installed Llama. I went with the 3B model due to memory and GPU limitations. Something like the 8B model would require at least 5GB of GPU, which exceeds the capability of the Geoforce GTX1650. The 3B model stays within the GPU entirely and for more intense processes I opted to create a swapfile. More details on that later.

ollama pull nomic-embed-text
ollama pull llama3.2:3b

Next install the Python dependencies

sudo apt update
sudo apt install python3-pip python3-venv -y

Create and activate a virtual environment

python3 -m venv rag-env # you can call this whatever you want, I named mine "oszak-rag-env"
source rag-env/bin/activate

Install application dependencies

pip install streamlit llama-index llama-index-llms-ollama llama-index-embeddings-ollama docx2txt llama-index-vector-stores-chroma chromadb

Ok, so now Llama exists in a virtual environment alongside all dependencies needed to run a Python script that produces a simple front end for interacting with the model. Next is to create the Python script itself (I called it app.py):

import os
import tempfile
import streamlit as st
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import nltk
nltk.download('stopwords')

# --- Global Configurations & Optimization ---
Settings.llm = Ollama(model="llama3.2:3b", request_timeout=300.0)
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

st.set_page_config(page_title="Linux Local Doc Chat", page_icon="🐧", layout="centered")
st.title("Oszak Chatbot")
st.subheader("Optimized for Linux Mint & NVIDIA GTX 1650")

# --- Initialize Persistent Storage & State ---
PERSIST_DIR = "./chroma_db"
db = chromadb.PersistentClient(path=PERSIST_DIR)
# Chroma collection name must be alphanumeric between 3-63 characters
chroma_collection = db.get_or_create_collection("local_rag_collection")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

if "messages" not in st.session_state:
    st.session_state.messages = [{"role": "assistant", "content": "Upload a document to save it permanently to your local index!"}]

# Proactively build or load the existing index on startup
if "index" not in st.session_state:
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    # This loads existing embeddings if they exist, or establishes an empty index
    st.session_state.index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

# Helper function to refresh the chat loop when a document updates
def get_chat_engine():
    return st.session_state.index.as_chat_engine(
        chat_mode="condense_plus_context",
        verbose=True
    )

# --- Sidebar: Document Ingestion Pipeline ---
with st.sidebar:
    st.header("Document Ingestion")
    uploaded_file = st.file_uploader("Upload your document (PDF, TXT, DOCX)", type=["pdf", "txt", "docx"])

    if uploaded_file is not None:
        if st.button("Process & Save Document", use_container_width=True):
            with st.status("Reading document and generating vectors locally...", expanded=True) as status:
                try:
                    with tempfile.TemporaryDirectory() as temp_dir:
                        temp_filepath = os.path.join(temp_dir, uploaded_file.name)
                        with open(temp_filepath, "wb") as f:
                            f.write(uploaded_file.getvalue())
                        status.write("Parsing document structure...")
                        # Read the new data
                        documents = SimpleDirectoryReader(temp_dir).load_data()
                        status.write("Calculating vector coordinates on GPU...")
                        # This inserts new document vectors straight into your persistent Chroma folder
                        for doc in documents:
                            st.session_state.index.insert(doc)
                    status.update(label="Document successfully saved!", state="complete", expanded=False)
                    st.success(f"'{uploaded_file.name}' is now permanently indexed.")
                except Exception as e:
                    status.update(label="Ingestion failed", state="error")
                    st.error(f"Error indexing data: {str(e)}")

# --- Main UI: Chat Interface ---
for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

if user_query := st.chat_input("Ask a question about your documents..."):
    st.session_state.messages.append({"role": "user", "content": user_query})
    with st.chat_message("user"):
        st.write(user_query)

    with st.chat_message("assistant"):
        # Check if the database has any items across app lifetimes
        if chroma_collection.count() == 0:
            fallback_text = "I don't have access to any data yet. Please upload a document via the sidebar first!"
            st.write(fallback_text)
            st.session_state.messages.append({"role": "assistant", "content": fallback_text})
        else:
            with st.spinner("Searching database..."):
                try:
                    chat_engine = get_chat_engine()
                    response = chat_engine.chat(user_query)
                    model_response = response.response

                    st.write(model_response)
                    st.session_state.messages.append({"role": "assistant", "content": model_response})
                except Exception as e:
                    st.error(f"Error generating answer: {str(e)}")

To run this application, execute the following command in Terminal (change the path to your script file after the Run command)

streamlit run app.py --server.port 8501 --server.address 0.0.0.0

A browser window will open where you can interact with the RAG agent. You should be able to upload a document, then ask questions or talk about the document with the agent.

General Chat Agent Solution

For this one, I kept the same configuration and just created a separate script This one I called “chat-app.py”.

import streamlit as st
from llama_index.llms.ollama import Ollama
from llama_index.core.llms import ChatMessage

# --- Global Configurations & Optimization ---
# Connect straight to your local Ollama instance running on your GPU
llm = Ollama(model="llama3.2:3b", base_url="http://127.0.0.1:11434", request_timeout=300.0)

st.set_page_config(page_title="Local AI Chat", page_icon="🤖", layout="centered")
st.title("🤖 Oszak Local Interactive Chatbot")
st.subheader("Powered by LlamaIndex & NVIDIA GTX 1650")

# Initialize persistent chat history across app refreshes using LlamaIndex's ChatMessage format
if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
        ChatMessage(role="assistant", content="Hello! I am Dadbot. How can I help you today?")
    ]

# --- Main UI: Render Historical Messages ---
for msg in st.session_state.chat_history:
    with st.chat_message(msg.role):
        st.write(msg.content)

# --- Handle Incoming Chat Questions ---
if user_query := st.chat_input("Type your message here..."):
    # Append the user's message to the session history state
    st.session_state.chat_history.append(ChatMessage(role="user", content=user_query))
    with st.chat_message("user"):
        st.write(user_query)

    # Generate model response dynamically via real-time token streaming
    with st.chat_message("assistant"):
        message_placeholder = st.empty()
        full_response = ""

        with st.spinner("Thinking..."):
            try:
                # Call Ollama's stream_chat endpoint passing the cumulative chat log
                response_stream = llm.stream_chat(st.session_state.chat_history)

                # Consume tokens as they are generated by your GTX 1650 card
                for response in response_stream:
                    full_response += response.delta
                    # Update the UI element in real-time for a smooth typing effect
                    message_placeholder.markdown(full_response + "▌")

                # Clean up the trailing cursor marker and log the complete assistant turn
                message_placeholder.markdown(full_response)
                st.session_state.chat_history.append(ChatMessage(role="assistant", content=full_response))

            except Exception as e:
                st.error(f"Error communicating with Dadbot: {str(e)}")

Same deal here, run the command via Terminal to launch the browser and interact with the agent (again, don’t forget to change the path to your script file after the Run command)

streamlit run chat-app.py --server.port 8501 --server.address 0.0.0.0

Pitfalls and Gotchas

  • I had to modify the timeout in both scripts near the top, in the configuration section, so it allowed for the old hardware to work with the memory and GPU that has had electricity running through it continuously for 14 years. Ideally the timeout should be 120 sec, but some queries take 3-5 minutes. So 300 seconds seems to be working so far.
  • Initially, the NVIDIA card would not talk with the command line. I had tried reconfiguring the kernel modules
    sudo dpkg-reconfigure nvidia-driver-595
    Purge a stuck installation lock
    sudo apt-get clean
    sudo apt-get update

    Force-reconfigure the package manager
    sudo dpkg –configure -a
    Explicitly install the driver set
    sudo apt install nvidia-driver-550
    Reboot the machine, it still failed, then I disabled Secure Boot in the BIOS, rebooted, but it still failed to communicate. I then cleared out the legacy driver files
    sudo apt-get remove –purge ‘^nvidia-.*’
    sudo apt-get autoremove

    Reinstalled the kernel headers and the production driver
    sudo apt update
    sudo apt install -y linux-headers-$(uname -r) dkms
    sudo apt install -y nvidia-driver-535

    Forced the system to rebuild the boot configuration, then rebooted (again)
    sudo update-initramfs -u
    sudo reboot

    Now running nividia-smi returned a grid with the current stats of the GPU. (whew!)
  • I had to add a base_url to the scripts in the configuration section to get past the “localhost” bug. By default, LlamaIndex tries to contact Ollama at localhost:11434. In many Linux Mint configurations, the Python stack translates localhost to the IPv6 address (::1), while the background Ollama service is listening exclusively on the IPv4 loopback (127.0.0.1). This causes an instant socket disconnect. Adding the base_url=”http://127.0.0.1:11434″ to the llm instantiation at the top of each script solved this.
  • The script was crashing while talking to the LLM because RAM was spilling over while conflicting with running applications. The solution was to create a swapfile Linux can use for shifting running apps to, allowing the full RAM and GPU for LLM access.
    # Deactivate any active minor swap space
    sudo swapoff -a
    # Allocate an 8 GB file space on your SSD
    sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
    # Restrict permissions for system security
    sudo chmod 600 /swapfile
    # Format the file into swap space and activate it
    sudo mkswap /swapfile
    sudo swapon /swapfile

    Double-check the system configuration file to make sure it persists after reboot
    sudo nano /etc/fstab
    The bottom of the file should have the following line
    /swapfile none swap sw 0 0
    Finally, clear out system caches and restart the background AI daemon
    sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
    sudo systemctl restart ollama

This was a lot of fun to put together and play with and is giving me all kinds of ideas for integration with home automation, monitoring, scheduling, and more. I’m sure there’s plenty of better ways to do this, or better ideas, so feel free to leave comments and suggestions!

Update: 11/01/2026

I reworked the RAG Python script so it accepts more than one document, ingests them, then you can have a conversation about some or all of them. Here’s the updated script

import os
import tempfile
import streamlit as st
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import nltk
nltk.download('stopwords')

# --- Global Configurations & Optimization ---
Settings.llm = Ollama(model="llama3.2:3b", request_timeout=300.0)
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

st.set_page_config(page_title="Linux Local Doc Chat", page_icon="🐧", layout="centered")
st.title("Oszak RAG Chatbot")
st.subheader("Optimized for Linux Mint & NVIDIA GTX 1650")

# --- Initialize Persistent Storage & State ---
PERSIST_DIR = "./chroma_db"
db = chromadb.PersistentClient(path=PERSIST_DIR)
# Chroma collection name must be alphanumeric between 3-63 characters
chroma_collection = db.get_or_create_collection("local_rag_collection")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

if "messages" not in st.session_state:
    st.session_state.messages = [{"role": "assistant", "content": "Upload a document to save it permanently to your local index!"}]

# Proactively build or load the existing index on startup
if "index" not in st.session_state:
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    # This loads existing embeddings if they exist, or establishes an empty index
    st.session_state.index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

# Helper function to refresh the chat loop when a document updates
def get_chat_engine():
    return st.session_state.index.as_chat_engine(
        chat_mode="condense_plus_context",
        verbose=True
    )

# --- Sidebar: Document Ingestion Pipeline ---

with st.sidebar:
    st.header("Document Ingestion")
    uploaded_files = st.file_uploader("Upload your documents (PDF, TXT, DOCX)", type=["pdf", "txt", "docx"])

    if uploaded_files is not None:
        if st.button("Process & Save Documents", use_container_width=True):
            with st.status("Reading documents and generating vectors locally...", expanded=True) as status:
                try:
                    with tempfile.TemporaryDirectory() as temp_dir:
                        for file in uploaded_files:
                            temp_filepath = os.path.join(temp_dir, file.name)
                            with open(temp_filepath, "wb") as f:
                                f.write(file.getvalue())
                            status.write(f"Processing document '{file.name}'...")
                            # Read the new data
                            documents = SimpleDirectoryReader(temp_dir).load_data()
                            status.write("Calculating vector coordinates on GPU...")
                            # This inserts new document vectors straight into your persistent Chroma folder
                            for doc in documents:
                                st.session_state.index.insert(doc)
                        status.update(label="All documents successfully saved!", state="complete", expanded=False)
                        st.success(f"Files are now permanently indexed!")
                except Exception as e:
                    status.update(label="Ingestion failed", state="error")
                    st.error(f"Error indexing data: {str(e)}")

# --- Main UI: Chat Interface ---

for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

if user_query := st.chat_input("Ask a question about your documents..."):
    st.session_state.messages.append({"role": "user", "content": user_query})
    with st.chat_message("user"):
        st.write(user_query)

    with st.chat_message("assistant"):
        # Check if the database has any items across app lifetimes
        if chroma_collection.count() == 0:
            fallback_text = "I don't have access to any data yet. Please upload a document via the sidebar first!"
            st.write(fallback_text)
            st.session_state.messages.append({"role": "assistant", "content": fallback_text})
        else:
            with st.spinner("Searching data..."):
                try:
                    chat_engine = get_chat_engine()
                    response = chat_engine.chat(user_query)
                    model_response = response.response

                    st.write(model_response)
                    st.session_state.messages.append({"role": "assistant", "content": model_response})
                except Exception as e:
                    st.Error(f"Error generating answer: {str(e)}")

Depending on how long it takes to process the documents and for the model to understand and interact with them, I may have to expand the timeout past 300 seconds. Under normal circumstances I wouldn’t, but since I’m dealing with ancient, well-used hardware and not designed for AI at all, I’m ok with that 😊

Additionally, I wanted to set up a way to access these models from anywhere without exposing an IP or port in our home’s firewall/router. So, I created a free CloudFlare account, added one of my existing domains and updated the nameservers to CloudFlare’s, then created a Tunnel. In that tunnel I created two Published Applications in CloudFlare. Each one points to the respective model using the internal IP address and port. It worked perfectly! I now have remote access on any device to my AI models without compromising my network.

Update: 11/8/2026

After switching around to gemma and mistral, I decided to implement gemma4:e2b as (to me) it provided the most robust responses which included iconography and emojis, and seems to be responsive to customizations like demeanor and persona when compared to the others.

I also replaced the Streamlit UI for the chat agent with Jan.ai and Open WebUI. Jan.ai was a great alternative to Streamlit, as the user experience is identical to ChatGPT without the subscription and lack of control over the model. Open WebUI also provides a nearly identical UI as ChatGPT, including saving chats, grouping them into projects, even temporary chats and selecting models. Right now I’m only allowing the modified version of gemma4 I installed. I had to create a service to launch Open WebUI just like I did for Streamlit, and I had to configure its settings.json file (which was in a hidden directory) to allow broadcasting the endpoint in order for CloudFlare to see it, but then it was a matter of updating the CF tunnel route to point to port 8080 instead of 1337 (which is Jan.ai’s default port). Jan.ai doesn’t provide a web UI, just the desktop client, but it does expose an API if you feel like writing your own front end. Seemed a little counter-productive installing a UI client that still requires a remote-facing UI, but that’s ok. Use the client locally and Open WebUI remotely!

# install Jan.ai - note this is for my local congifuration.  You may need to download a different version from github
wget https://app.jan.ai/download/latest/linux-amd64-appimage 

# set permissions
chmod +x linux-amd64-appimage

# run it
./linux-amd64-appimage --disable-gpu --no-sandbox

# if needed, edit the settings.json file so the address is 0.0.0.0 to allow broadcasting
sudo nano ~/.local/share/Jan/data/settings.json

# Install Open WebUI
# switch to your virtual environment first
pip install open-webui

# create the service for open-webui
sudo nano /etc/systemd/system/open-webui.service

Add the following to the file and save

[Unit]
Description=Open WebUI Browser Interface for Jan
After=network.target

[Service]
Type=simple
# Replace 'your_username' with your actual username
User=your_username
Group=your_username
WorkingDirectory=/home/your_username/{your venv}/bin
# Environment variables pointing Open WebUI straight to Jan's backend
Environment=OPENAI_API_BASE_URL=http://127.0.0.1:1337
Environment=OPENAI_API_KEY=jan-local-key

# The execution command (adjust path if pip installed it elsewhere)
ExecStart=/home/your_username/{your venv}/bin/open-webui serve

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Then enable and start the service

sudo systemctl enable open-webui.service
sudo systemctl start open-webui.service

# to check if it's running
sudo systemctl status open-webui.service

Create a config file that instructs the model with specifics like parameters, temperature and initial prompts (example below. You’ll need to update with your model and whatever parms you need. In my case I am deliberately preventing communication with the internet, forcing it to rely solely on internal knowledge. I’m also altering the demeanor to be wittier and more humorous.)

FROM gemma4:e2b
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|tool_call|>"
PARAMETER stop "tool_search"
PARAMETER temperature 0.95
PARAMETER top_p 0.88
PARAMETER top_k 50
SYSTEM """
You are a highly witty assistant with a sharp, clever sense of humor. You blend factual accuracy with playful banter, sarcasm, or dry observational wit. Never explicitly state "I am being funny" or use cheesy sitcom punchlines.

Here is how you should handle user interactions:

User: Why is the sky blue?
Assistant: Rayleigh scattering. Basically, the atmosphere prefers short-wavelength blue light over red light. Think of it as the atmosphere having a favorite color, and that color happens to be a mild existential crisis.

User: Can you explain quantum computing?
Assistant: Imagine a regular computer tosses a coin and it's either heads or tails. A quantum computer tosses a coin and it spins through the air in a state of existential dread, being both heads and tails until it hits the floor. It's incredibly fast, highly complex, and currently giving physicists a massive headache.

User: How often should I water a ponytail palm?
Assistant: Treat it like a houseplant that actively hates you and wants to be left alone. Ponytail palms store water in their bulbous trunks, meaning they have better hydration management than most humans. Water it only when the soil is completely dry—roughly every few weeks. If you overwater it, the roots will rot, and it will die out of sheer spite.
"""

Save and exit the editor, then rebuild your model. In my case I chose to clone the gemma4:e2b model into one called “gemma4-direct”, and during creation apply the Modelfile config.

# remove the existing model first to avoid caching issues known to be present when using gemma
ollama rm gemma4-direct
# now rebuild the model using the Modelfile
ollama create gemma4-direct -f ~/path-to-model-file/Modelfile

Ollama will know which model you want to clone based on the FROM entry in your Modelfile

I’m still struggling a little with getting the model to respond with humor and wit, I got it to work before but for some reason gemma4 remains a bit too dry for the way I configure it above. Light research says gemma4 leans more in this direction, whereas llama is easier to instruct. So I may try that again at some point…maybe on another machine.

Update: 11/11/2026

I noticed conversation context was getting lost after Turn 1. You could state something simple like “my code is 9942”, and if your next question was “what is my code?” the response indicated it had no idea what you were talking about. After a LOT of investigation and research a couple of things were revealed:

  1. Open-WebUI is buggy. I was unable to reproduce this by running ollama run gemma4:e2b in Terminal. So I made the following changes to the UI to get around this
    • In model settings, set Reasoning Effort to NONE
    • In model settings, set num_ctx to 4096
    • Update the SQLite database in /venv/lib/python3.12/site-packages/open_webui/data/webui.db by running the following
sudo sqlite /venv/lib/python3.12/site-packages/open_webui/data/webui.db  "PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;"

Update: 11/15/2026

I switched back to my custom Python script and Streamlit for hosting a front end for the gemma4 model. Open-WebUI is a really nice UX, and gives you a ChatGPT look and feel, but it’s such a PINTA to get working with this model. Just one problem after another

  • Making changes to the configuration has zero impact on the model. You could tell it to talk like a pirate and it will ignore you, yet in Terminal it works perfectly. Changing temperature, reasoning effort, max_tokens, num_ctx, nothing worked from the UI side. Everything works from Terminal.
  • It forgot context from the previous question, I spent two days modifying the UI to get it to keep context and hold a conversation, when I had to change nothing to use the model in Terminal
  • After changing this, it now responded with nothing. Again, could not reproduce this in Terminal. I spent several hours trying to fix this before giving up and switching back to my script and Streamlit for presentation.

I’ve read several times that Open WebUI can have issues with some models, I’ve tried with gemma, llama, mistral and oss-gpt with no luck. Again, it’s a slick looking UI, just wish it would play nice with the flagship models.

Leave a Reply

Your email address will not be published. Required fields are marked *