llm-app

1. Project Overview

llm-app (by Pathway) is a collection of ready-to-run, Docker-friendly templates for building production-grade RAG (Retrieval-Augmented Generation) pipelines and enterprise search applications that stay continuously synchronized with live data sources such as SharePoint, Google Drive, S3, Kafka, and PostgreSQL — without requiring a separate vector database or indexing service.

2. Background & Positioning

Most RAG tutorials assume a static, one-time-loaded document set. In real organizations, however, knowledge bases change constantly: new files are uploaded, old ones are deleted, permissions change. llm-app was built by Pathway, the company behind the Rust-powered Pathway data processing engine, to close that gap — every template ships with a live, incremental indexing pipeline that detects additions, updates, and deletions in the underlying data source and reflects them in retrieval results in real time, with no manual re-indexing step.

Compared to typical RAG example repositories that focus purely on prompt/chain logic, llm-app differentiates itself by:

• Bundling the data connectors, indexing engine, and serving API into a single deployable unit instead of gluing together separate vector-database, orchestration, and scheduling services.
• Offering an in-memory, cache-backed index (vector, hybrid, and full-text search via usearch and Tantivy) that scales to millions of pages without external infrastructure.
• Being template-driven — each use case is a complete, runnable application rather than a documentation snippet.

3. Feature Categories

• 📄 Question-Answering & Search (4 templates) — End-to-end RAG apps for querying document collections in natural language.

  • question_answering_rag — baseline document Q&A pipeline with an LLM.
  • document_indexing — a standalone live vector-store/indexing service usable by any downstream app.
  • slides_ai_search — multimodal search over PowerPoint/PDF slide decks.
  • document_store_mcp_server — exposes the document store as an MCP server for agent tooling.

• 🖼️ Multimodal RAG (2 templates) — Pipelines that go beyond plain text.

  • multimodal_rag — parses PDFs with embedded charts, tables, and images using GPT-4o.
  • video_rag_twelvelabs — question-answering over video content via the TwelveLabs API.

• ⚡ Cost & Privacy Optimized RAG (2 templates) — Variants tuned for cost or data-residency constraints.

  • adaptive_rag — dynamically adjusts the number of retrieved context chunks, cutting token usage by up to 4x.
  • private_rag — fully local/offline deployment using Mistral and Ollama, with no data leaving the premises.

• 🗄️ Structured Data & Automation (2 templates) — Pipelines that turn unstructured input into structured outputs or trigger actions.

  • unstructured_to_sql_on_the_fly — converts unstructured financial documents into a queryable SQL database on the fly.
  • drive_alert — monitors a live data source and raises alerts on relevant changes.

• 🧠 Agentic Patterns (1 collection) — Reference notebooks for advanced reasoning loops.

  • cookbooks/self-rag-agents — Self-RAG style agents that critique and refine their own retrieved context.

4. Key Highlights

• Live data sync — Pipelines watch SharePoint, Google Drive, S3, Kafka, PostgreSQL, and real-time APIs, automatically reflecting additions, edits, and deletions without a rebuild step.
• No extra infrastructure — Vector, hybrid, and full-text search are built into the Pathway engine (usearch + Tantivy) and run in-memory with caching, removing the need for a standalone vector database.
• Docker-first deployment — Every template runs as a container with an HTTP API, and most include an optional Streamlit UI for quick manual testing.
• Framework interoperability — Native integrations with LangChain and LlamaIndex let teams reuse existing chains and agents on top of Pathway's live index.
• Cost-aware retrieval — The Adaptive RAG template demonstrates retrieving only as much context as needed, reducing LLM token spend by up to 4x.
• Scale-tested — Designed to index and serve millions of document pages from a single pipeline.

5. Use Cases by Role

• General developers — Clone a template that matches your use case (Q&A, multimodal, SQL extraction) and adapt the data connector and prompt to your product.
• Data / research scientists — Use document_indexing or the self-rag-agents cookbook as a base for experimenting with retrieval strategies, chunking, and agentic self-correction.
• Project managers / enterprise search owners — Deploy question_answering_rag or slides_ai_search as an internal knowledge-base search tool that stays current without an indexing pipeline to maintain.
• Security/compliance-conscious teams — Use private_rag to keep document content and model inference entirely on-premises with Mistral/Ollama.

6. Getting Started

Find what you need — browse the templates/ directory on GitHub, or the curated list at the Pathway Templates page, to pick the closest match to your use case.

Install and run a template

git clone https://github.com/pathwaycom/llm-app.git
cd llm-app/templates/question_answering_rag
docker compose up

Each template folder contains its own README.md with the exact environment variables (e.g., API keys) and run instructions for that pipeline.

Contribute — read CONTRIBUTING.md; for anything beyond a small fix, coordinate first in the #get-help channel of the Pathway Discord.

7. Project Structure

llm-app/
├── templates/                    # Ready-to-run application templates
│   ├── question_answering_rag/
│   ├── document_indexing/
│   ├── multimodal_rag/
│   ├── adaptive_rag/
│   ├── private_rag/
│   ├── slides_ai_search/
│   ├── unstructured_to_sql_on_the_fly/
│   ├── video_rag_twelvelabs/
│   ├── document_store_mcp_server/
│   └── drive_alert/
├── cookbooks/
│   └── self-rag-agents/          # Self-RAG agent notebooks
├── assets/                       # Diagrams and demo media
├── pyproject.toml / setup.cfg    # Python packaging config
└── README.md

Each subfolder under templates/ is a self-contained app with its own Dockerfile/docker compose setup and configuration file, so templates can be run independently of one another.

8. Related Ecosystem

• Pathway — the underlying Rust-powered Python data processing engine that provides the streaming ETL, live indexing, and incremental computation this repo builds on.
• LangChain / LlamaIndex — supported as optional orchestration layers on top of Pathway's live vector index.
• usearch / Tantivy — embedded libraries powering vector search and full-text/hybrid search respectively.
• Ollama / Mistral — used by the private_rag template for fully local LLM inference.
• TwelveLabs — video understanding API used by the video_rag_twelvelabs template.

9. License

The project is distributed under the MIT License.

• ✅ Free to use, copy, modify, merge, publish, and distribute, including for commercial products.
• ✅ Can be embedded in proprietary/closed-source applications.
• ❌ Comes with no warranty; the authors are not liable for damages arising from its use.
• ℹ️ The MIT license text and copyright notice must be retained in copies or substantial portions of the software.

10. FAQ

Q: Do I need to run a separate vector database alongside these templates?
A: No — indexing (vector, hybrid, and full-text search) is built into the Pathway engine and runs in-memory with on-disk caching, so no external vector store is required.

Q: Can I use this fully offline, without sending data to a third-party LLM API?
A: Yes — the private_rag template runs with Mistral and Ollama locally, keeping documents and inference on-premises.

Q: Which data sources can these pipelines watch for live updates?
A: SharePoint, Google Drive, S3, Kafka, PostgreSQL, local filesystems, and generic real-time data APIs are supported out of the box.

Q: How do I reduce LLM token costs when scaling up retrieval?
A: See the adaptive_rag template, which dynamically sizes the retrieved context and reports up to 4x lower token usage.

Q: How do I get help or propose a larger contribution?
A: Open an issue on GitHub for bugs, or join the #get-help channel on the Pathway Discord before starting substantial work.

11. Quick Links

• Repository: https://github.com/pathwaycom/llm-app
• Official docs / templates catalog: https://pathway.com/developers/templates/
• Contributing guide: https://github.com/pathwaycom/llm-app/blob/main/CONTRIBUTING.md
• Community: https://discord.gg/pathway

12. Summary

llm-app packages Pathway's live-data engine into a set of ready-to-deploy RAG and enterprise-search templates that stay automatically in sync with changing document sources, eliminating the need for separate indexing infrastructure. It is best suited for developers and teams who want to move from a RAG prototype to a production pipeline quickly — whether the goal is document Q&A, multimodal search, cost-optimized retrieval, or fully private on-premises deployment.