I decided it’s about time we implemented our own AI model for home use. I took two approaches – a RAG model for analyzing and working with documents, and a general chat agent. After a couple of days and some back and forth with Gemini, I came up with the following solutions.

System Specifications

This was installed on a 14-year-old HP desktop with original motherboard (Intel Core i7 with 4 cores), 12GB original RAM and 2012 NVIDIA Geoforce GTX 1650 (4GB) GPU. I installed a 128GB SSD and put Linux Mint (Cinnamon) on for the OS. I wasn’t expecting miracles, just to see what I could do with this old equipment.

RAG model solution

First I installed the NVIDIA desktop drivers, as the existing ones were Open Source. This requires the OEM drivers.

1. Open the Mint menu, search for Driver Manager, and launch it.
2. Select the recommended NVIDIA proprietary driver version (e.g., nvidia-driver-535 or newer) instead of the open-source driver
3. Apply changes and restart your computer.
4. Verify your GPU is recognized by opening a terminal and running nvidia-smi

If it prints out your video card status table, you are ready to proceed

I then decided to use Llama as the LLM. No real preference, just lack of experience in general with home AI models. I’ll eventually switch to Mistral, most likely. To use Llama I did the following:

Install Ollama via Terminal as a standard background service

curl -fsSL https://ollama.com/install.sh | sh

Confirm the service is active

sudo systemctl status ollama

Next I installed Llama. I went with the 3B model due to memory and GPU limitations. Something like the 8B model would require at least 5GB of GPU, which exceeds the capability of the Geoforce GTX1650. The 3B model stays within the GPU entirely and for more intense processes I opted to create a swapfile. More details on that later.

ollama pull nomic-embed-text
ollama pull llama3.2:3b

Next install the Python dependencies

sudo apt update
sudo apt install python3-pip python3-venv -y

Create and activate a virtual environment

python3 -m venv rag-env # you can call this whatever you want, I named mine "oszak-rag-env"
source rag-env/bin/activate

Install application dependencies

pip install streamlit llama-index llama-index-llms-ollama llama-index-embeddings-ollama docx2txt llama-index-vector-stores-chroma chromadb

Ok, so now Llama exists in a virtual environment alongside all dependencies needed to run a Python script that produces a simple front end for interacting with the model. Next is to create the Python script itself (I called it app.py):

import os
import tempfile
import streamlit as st
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import nltk
nltk.download('stopwords')

# --- Global Configurations & Optimization ---
Settings.llm = Ollama(model="llama3.2:3b", request_timeout=300.0)
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.chunk_size = 512
Settings.chunk_overlap = 50

st.set_page_config(page_title="Linux Local Doc Chat", page_icon="🐧", layout="centered")
st.title("Oszak Chatbot")
st.subheader("Optimized for Linux Mint & NVIDIA GTX 1650")

# --- Initialize Persistent Storage & State ---
PERSIST_DIR = "./chroma_db"
db = chromadb.PersistentClient(path=PERSIST_DIR)
# Chroma collection name must be alphanumeric between 3-63 characters
chroma_collection = db.get_or_create_collection("local_rag_collection")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

if "messages" not in st.session_state:
    st.session_state.messages = [{"role": "assistant", "content": "Upload a document to save it permanently to your local index!"}]

# Proactively build or load the existing index on startup
if "index" not in st.session_state:
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    # This loads existing embeddings if they exist, or establishes an empty index
    st.session_state.index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

# Helper function to refresh the chat loop when a document updates
def get_chat_engine():
    return st.session_state.index.as_chat_engine(
        chat_mode="condense_plus_context",
        verbose=True
    )

# --- Sidebar: Document Ingestion Pipeline ---
with st.sidebar:
    st.header("Document Ingestion")
    uploaded_file = st.file_uploader("Upload your document (PDF, TXT, DOCX)", type=["pdf", "txt", "docx"])

    if uploaded_file is not None:
        if st.button("Process & Save Document", use_container_width=True):
            with st.status("Reading document and generating vectors locally...", expanded=True) as status:
                try:
                    with tempfile.TemporaryDirectory() as temp_dir:
                        temp_filepath = os.path.join(temp_dir, uploaded_file.name)
                        with open(temp_filepath, "wb") as f:
                            f.write(uploaded_file.getvalue())
                        status.write("Parsing document structure...")
                        # Read the new data
                        documents = SimpleDirectoryReader(temp_dir).load_data()
                        status.write("Calculating vector coordinates on GPU...")
                        # This inserts new document vectors straight into your persistent Chroma folder
                        for doc in documents:
                            st.session_state.index.insert(doc)
                    status.update(label="Document successfully saved!", state="complete", expanded=False)
                    st.success(f"'{uploaded_file.name}' is now permanently indexed.")
                except Exception as e:
                    status.update(label="Ingestion failed", state="error")
                    st.error(f"Error indexing data: {str(e)}")

# --- Main UI: Chat Interface ---
for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

if user_query := st.chat_input("Ask a question about your documents..."):
    st.session_state.messages.append({"role": "user", "content": user_query})
    with st.chat_message("user"):
        st.write(user_query)

    with st.chat_message("assistant"):
        # Check if the database has any items across app lifetimes
        if chroma_collection.count() == 0:
            fallback_text = "I don't have access to any data yet. Please upload a document via the sidebar first!"
            st.write(fallback_text)
            st.session_state.messages.append({"role": "assistant", "content": fallback_text})
        else:
            with st.spinner("Searching database..."):
                try:
                    chat_engine = get_chat_engine()
                    response = chat_engine.chat(user_query)
                    model_response = response.response

                    st.write(model_response)
                    st.session_state.messages.append({"role": "assistant", "content": model_response})
                except Exception as e:
                    st.error(f"Error generating answer: {str(e)}")

To run this application, execute the following command in Terminal (change the path to your script file after the Run command)

streamlit run app.py --server.port 8501 --server.address 0.0.0.0

A browser window will open where you can interact with the RAG agent. You should be able to upload a document, then ask questions or talk about the document with the agent.

General Chat Agent Solution

For this one, I kept the same configuration and just created a separate script This one I called “chat-app.py”.

import streamlit as st
from llama_index.llms.ollama import Ollama
from llama_index.core.llms import ChatMessage

# --- Global Configurations & Optimization ---
# Connect straight to your local Ollama instance running on your GPU
llm = Ollama(model="llama3.2:3b", base_url="http://127.0.0.1:11434", request_timeout=300.0)

st.set_page_config(page_title="Local AI Chat", page_icon="🤖", layout="centered")
st.title("🤖 Oszak Local Interactive Chatbot")
st.subheader("Powered by LlamaIndex & NVIDIA GTX 1650")

# Initialize persistent chat history across app refreshes using LlamaIndex's ChatMessage format
if "chat_history" not in st.session_state:
    st.session_state.chat_history = [
        ChatMessage(role="assistant", content="Hello! I am Dadbot. How can I help you today?")
    ]

# --- Main UI: Render Historical Messages ---
for msg in st.session_state.chat_history:
    with st.chat_message(msg.role):
        st.write(msg.content)

# --- Handle Incoming Chat Questions ---
if user_query := st.chat_input("Type your message here..."):
    # Append the user's message to the session history state
    st.session_state.chat_history.append(ChatMessage(role="user", content=user_query))
    with st.chat_message("user"):
        st.write(user_query)

    # Generate model response dynamically via real-time token streaming
    with st.chat_message("assistant"):
        message_placeholder = st.empty()
        full_response = ""

        with st.spinner("Thinking..."):
            try:
                # Call Ollama's stream_chat endpoint passing the cumulative chat log
                response_stream = llm.stream_chat(st.session_state.chat_history)

                # Consume tokens as they are generated by your GTX 1650 card
                for response in response_stream:
                    full_response += response.delta
                    # Update the UI element in real-time for a smooth typing effect
                    message_placeholder.markdown(full_response + "▌")

                # Clean up the trailing cursor marker and log the complete assistant turn
                message_placeholder.markdown(full_response)
                st.session_state.chat_history.append(ChatMessage(role="assistant", content=full_response))

            except Exception as e:
                st.error(f"Error communicating with Dadbot: {str(e)}")

Same deal here, run the command via Terminal to launch the browser and interact with the agent (again, don’t forget to change the path to your script file after the Run command)

streamlit run chat-app.py --server.port 8501 --server.address 0.0.0.0

Pitfalls and Gotchas

• I had to modify the timeout in both scripts near the top, in the configuration section, so it allowed for the old hardware to work with the memory and GPU that has had electricity running through it continuously for 14 years. Ideally the timeout should be 120 sec, but some queries take 3-5 minutes. So 300 seconds seems to be working so far.
• Initially, the NVIDIA card would not talk with the command line. I had tried reconfiguring the kernel modules
  sudo dpkg-reconfigure nvidia-driver-595
  Purge a stuck installation lock
  sudo apt-get clean
  sudo apt-get update
  Force-reconfigure the package manager
  sudo dpkg –configure -a
  Explicitly install the driver set
  sudo apt install nvidia-driver-550
  Reboot the machine, it still failed, then I disabled Secure Boot in the BIOS, rebooted, but it still failed to communicate. I then cleared out the legacy driver files
  sudo apt-get remove –purge ‘^nvidia-.*’
  sudo apt-get autoremove
  Reinstalled the kernel headers and the production driver
  sudo apt update
  sudo apt install -y linux-headers-$(uname -r) dkms
  sudo apt install -y nvidia-driver-535
  Forced the system to rebuild the boot configuration, then rebooted (again)
  sudo update-initramfs -u
  sudo reboot
  Now running nividia-smi returned a grid with the current stats of the GPU. (whew!)
• I had to add a base_url to the scripts in the configuration section to get past the “localhost” bug. By default, LlamaIndex tries to contact Ollama at localhost:11434. In many Linux Mint configurations, the Python stack translates localhost to the IPv6 address (::1), while the background Ollama service is listening exclusively on the IPv4 loopback (127.0.0.1). This causes an instant socket disconnect. Adding the base_url=”http://127.0.0.1:11434″ to the llm instantiation at the top of each script solved this.
• The script was crashing while talking to the LLM because RAM was spilling over while conflicting with running applications. The solution was to create a swapfile Linux can use for shifting running apps to, allowing the full RAM and GPU for LLM access.
  # Deactivate any active minor swap space
  sudo swapoff -a
  # Allocate an 8 GB file space on your SSD
  sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
  # Restrict permissions for system security
  sudo chmod 600 /swapfile
  # Format the file into swap space and activate it
  sudo mkswap /swapfile
  sudo swapon /swapfile
  Double-check the system configuration file to make sure it persists after reboot
  sudo nano /etc/fstab
  The bottom of the file should have the following line
  /swapfile none swap sw 0 0
  Finally, clear out system caches and restart the background AI daemon
  sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
  sudo systemctl restart ollama
  

This was a lot of fun to put together and play with and is giving me all kinds of ideas for integration with home automation, monitoring, scheduling, and more. I’m sure there’s plenty of better ways to do this, or better ideas, so feel free to leave comments and suggestions!