Spark Documentation RAG Chatbot

An intelligent chatbot that answers questions about Apache Spark documentation using Retrieval-Augmented Generation (RAG). Built with Claude Sonnet 4.5, sentence-transformers (free embeddings), and ChromaDB.

Features

• Comprehensive Documentation Coverage: Scrapes and indexes the latest Apache Spark documentation
• Intelligent Q&A: Uses RAG pipeline to provide accurate, context-aware answers
• Source Citations: Always provides references to source documentation
• Free Embeddings: Uses sentence-transformers (no embedding API costs)
• Local Vector Storage: ChromaDB for fast, local similarity search
• Beautiful CLI: Interactive chat interface with rich formatting
• Streaming Responses: Real-time response generation

Architecture

User Question
     ↓
Query Embedding (sentence-transformers)
     ↓
Vector Search (ChromaDB)
     ↓
Context Retrieval (Top-K relevant chunks)
     ↓
Prompt Construction (Context + Question)
     ↓
Response Generation (Claude Sonnet 4.5)
     ↓
Formatted Answer + Sources

Prerequisites

• Python 3.8 or higher
• Anthropic API key (for Claude Sonnet 4.5)
• Internet connection (for scraping and API calls)

Installation

1. Clone the repository

git clone <repository-url>
cd spark-doc-chatbot

2. Create and activate virtual environment

python -m venv venv

# On macOS/Linux:
source venv/bin/activate

# On Windows:
venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Set up environment variables

cp .env.example .env

Edit .env and add your Anthropic API key:

ANTHROPIC_API_KEY=your_api_key_here

Get your API key from: https://console.anthropic.com/

Usage

Step 1: Index the Documentation (One-time setup)

This scrapes the Spark documentation, creates embeddings, and stores them in ChromaDB:

python setup_index.py

This process will:

1. Scrape the latest Spark documentation (~500 pages)
2. Chunk the content into manageable pieces
3. Generate embeddings using sentence-transformers
4. Store in ChromaDB for fast retrieval

Note: This may take 15-30 minutes depending on your internet connection and CPU.

Using Cached Data

If you've already scraped the documentation and want to re-index without scraping again:

python setup_index.py --use-cached

Step 2: Start the Chatbot

python chatbot.py

Example Questions

• "How do I create a DataFrame in PySpark?"
• "What is the difference between RDD and DataFrame?"
• "How do I perform a join operation in Spark SQL?"
• "What are transformations and actions in Spark?"
• "How do I optimize Spark job performance?"
• "How to read CSV files in Spark?"
• "What is lazy evaluation in Spark?"

Chatbot Commands

• /exit or /quit - Exit the chatbot
• /help - Show help message
• /clear - Clear the screen

Project Structure

spark-doc-chatbot/
├── scraper/
│   ├── __init__.py
│   ├── doc_scraper.py       # Web scraping with BeautifulSoup
│   └── chunker.py           # Text chunking logic
├── embeddings/
│   ├── __init__.py
│   └── generator.py         # Embedding generation (sentence-transformers)
├── vectordb/
│   ├── __init__.py
│   └── chroma_client.py     # ChromaDB operations
├── rag/
│   ├── __init__.py
│   ├── retriever.py         # Context retrieval
│   ├── generator.py         # Response generation (Claude)
│   └── pipeline.py          # Main RAG orchestration
├── config.py                # Configuration settings
├── setup_index.py           # One-time indexing script
├── chatbot.py               # Interactive CLI
├── requirements.txt         # Python dependencies
├── .env.example             # Environment variables template
├── .gitignore
└── README.md

Configuration

Edit config.py to customize:

• Scraper settings: Number of pages, request delay
• Chunking parameters: Chunk size, overlap
• Embedding model: Default is all-MiniLM-L6-v2
• RAG parameters: Top-K results, relevance threshold
• Claude settings: Model, temperature, max tokens

Technology Stack

Core Technologies

• Language Model: Claude Sonnet 4.5 (via Anthropic API)
• Embeddings: sentence-transformers (all-MiniLM-L6-v2) - FREE
• Vector Database: ChromaDB (local, persistent) - FREE
• Web Scraping: BeautifulSoup4 + Requests
• CLI Interface: Rich library

Key Libraries

anthropic         # Claude API
sentence-transformers  # Free embeddings
chromadb          # Vector database
beautifulsoup4    # Web scraping
requests          # HTTP client
rich              # CLI formatting
tiktoken          # Token counting
tqdm              # Progress bars

Cost Breakdown

• Embeddings: FREE (sentence-transformers runs locally)
• Vector Database: FREE (ChromaDB local storage)
• Web Scraping: FREE
• LLM (Claude API): Pay per use (~$3-5 per million input tokens)

Estimated cost for typical usage: $0.10-0.50 per day depending on query volume.

Performance

• Indexing Time: 15-30 minutes (one-time)
• Query Response Time: 2-5 seconds
  • Embedding generation: <0.1s
  • Vector search: <0.1s
  • Claude API call: 2-4s (depends on response length)
• Memory Usage: ~500MB (loaded model + embeddings)
• Disk Usage: ~100MB (ChromaDB + cached data)

Troubleshooting

Issue: "ANTHROPIC_API_KEY not found"

Solution: Make sure you've created a .env file and added your API key:

cp .env.example .env
# Edit .env and add your key

Issue: "Collection not initialized"

Solution: Run the indexing script first:

python setup_index.py

Issue: Scraping fails or is slow

Solution:

• Check your internet connection
• Increase request_delay in config.py to avoid rate limiting
• Use cached data: python setup_index.py --use-cached

Issue: Out of memory during indexing

Solution:

• Reduce max_pages in config.py
• Reduce batch_size in embedding generation
• Close other applications

Issue: Poor answer quality

Solution:

• Increase top_k in RAG_CONFIG to retrieve more context
• Lower min_relevance_score to include more results
• Adjust temperature in CLAUDE_CONFIG (lower = more deterministic)

Development

Running Individual Components

Test individual components:

# Test scraper
python scraper/doc_scraper.py

# Test chunker
python scraper/chunker.py

# Test embeddings
python embeddings/generator.py

# Test ChromaDB
python vectordb/chroma_client.py

# Test RAG pipeline
python rag/pipeline.py

Adding New Features

1. Multiple Spark Versions: Modify scraper to handle version selection
2. Web UI: Integrate Streamlit or Gradio
3. Advanced Filtering: Add metadata filters in retrieval
4. Conversation History: Add chat memory for follow-up questions
5. Code Execution: Integrate code interpreter for Spark examples

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

MIT License - see LICENSE file for details

Acknowledgments

• Apache Spark documentation: https://spark.apache.org/docs/latest/
• Anthropic Claude: https://www.anthropic.com/
• sentence-transformers: https://www.sbert.net/
• ChromaDB: https://www.trychroma.com/

Support

For issues and questions:

• Open an issue on GitHub
• Check existing issues for solutions
• Review the troubleshooting section

Roadmap

• Support for multiple Spark versions
• Web-based UI (Streamlit)
• Conversation memory
• Export conversation history
• Code execution sandbox
• Fine-tuned embeddings for better accuracy
• Multi-language support

---

Built with love for the Apache Spark community!