Local AI App Builder (Dify + Ollama)

A private, self-hosted AI application platform. Dify gives you a visual drag-and-drop builder to create custom AI apps - chatbots, RAG pipelines, AI agents - without writing backend code. Connect it to Ollama for free local inference, and you get a full-stack AI platform that runs entirely on your own hardware.

No monthly fees, no data leaving your network, and you can build production-quality AI apps in minutes.

What you get

• Visual AI app builder - drag, drop, configure. Build chatbots, RAG pipelines, and agents in a web UI
• Private RAG - upload documents, Dify chunks and indexes them locally. Ask questions against your own knowledge base
• API endpoints - every app you build gets a REST API. Connect your frontend, Slack bot, or mobile app
• Multi-model - use any Ollama model per app, or mix local + cloud models with fallback routing
• Multi-tenant - invite team members, each with their own workspace
• $0/month - the GPU you already own is all you need

Architecture

Recommended GPU: RTX 3060 12GB for 14B models, or RTX 4060 Ti 16GB for more headroom.

Prerequisites

• A GPU with ≥12 GB VRAM for 14B models (8GB works for smaller 7B models)
• Docker + Docker Compose 2.24.0+
• ~10 GB free disk for Dify + models
• 4 GB RAM minimum for Dify's services

Setup

Step 1: Start Ollama

If you don't have Ollama running yet, launch it with Docker:

docker run -d --gpus all -p 11434:11434 --name ollama \
  -v ollama:/root/.ollama \
  ollama/ollama

Pull your default model:

docker exec ollama ollama pull qwen3:14b

Step 2: Start Dify

Clone and launch Dify:

git clone https://github.com/langgenius/dify.git
cd dify/docker
cp .env.example .env
docker compose up -d

This starts Dify's web UI (port 3000), API server, PostgreSQL, Redis, Weaviate vector DB, and a sandboxed code runner.

Step 3: Connect Ollama to Dify

1. Open http://localhost/install and create your admin account
2. Go to Settings → Model Provider in the top-right
3. Click Ollama and fill in:
   • Model Name: qwen3:14b
   • Base URL: http://host.docker.internal:11434 (Docker Desktop) or http://YOUR_HOST_IP:11434 (Linux)
4. Click Save

You can add multiple models - repeat for any model you've pulled in Ollama.

Use it

Build a Chatbot

1. Go to Studio → Create Application → Chatbot
2. Select your Ollama model
3. Add a system prompt like "You are a helpful assistant that answers from my documents"
4. Click Publish - your chatbot gets a public URL and API endpoint

Build a RAG Pipeline

1. Go to Knowledge → Create Knowledge
2. Upload PDFs, markdown, or text files
3. Choose chunking strategy and embedding model
4. Create an app in Studio that uses this knowledge base
5. Now your chatbot answers from your documents

Build an Agent

1. Go to Studio → Create Application → Agent
2. Add tools: web search, code interpreter, or custom API tools
3. Give the agent a goal, and Dify orchestrates the tool calls

Cost vs cloud

After about 5 months the GPU has paid for itself versus a mid-tier Dify Cloud plan with API usage - and you own the infrastructure.

Troubleshooting

• Dify can't reach Ollama → On Linux, use http://172.17.0.1:11434 (Docker host gateway). On Docker Desktop, host.docker.internal works.
• Slow model switching → Pre-pull models with ollama pull so they're cached locally.
• Dify startup fails → Check docker compose logs - common issues are port conflicts (port 80/443 already in use). Edit .env to change the NGINX port.
• No GPU acceleration → Install the NVIDIA Container Toolkit and confirm docker run --rm --gpus all nvidia/cuda:12.4.0-base nvidia-smi works.

Swap components

• Replace Weaviate with Qdrant by setting VECTOR_STORE=qdrant in .env
• Smaller model on 8GB → Llama 3.1 8B or Phi-4 mini
• More capable model → Qwen3 32B on an RTX 4090
• Prefer another builder → Try Flowise or Langflow instead of Dify