# Gemini FAQ Chatbot - Laravel + React + Python RAG

A sophisticated chatbot application that combines Laravel backend, React frontend, and Python RAG (Retrieval-Augmented Generation) service powered by Google's Gemini AI. This application allows users to ingest content (text and PDF files) into a knowledge base and query it using natural language with AI-powered responses.

## 🏗️ Architecture

```
React Frontend (Vite) → Laravel API → Python Flask RAG Service → Google Gemini AI
                                     ↓
                                  MySQL Database + Vector Embeddings
```

## 🚀 Features

-   **Multi-Format Content Ingestion**: Upload and process text files (.txt) and PDF documents (.pdf) into a searchable knowledge base
-   **PDF Processing**: Extract text from PDF documents using PyPDFLoader with intelligent chunking
-   **AI-Powered Chat**: Ask questions and get contextual answers using Google Gemini AI
-   **Vector Search**: Efficient similarity search using MySQL with sentence transformer embeddings
-   **Source Attribution**: See which documents were used to generate answers
-   **Content Management**: View, manage, and delete uploaded documents through a modern UI
-   **Real-time Processing**: Live feedback during document upload and processing
-   **Modern UI**: React-based interface with chatbot and content management views
-   **RESTful API**: Clean Laravel API endpoints for integration

## 📋 Prerequisites

Before you begin, ensure you have the following installed:

-   **DDEV** (for local development environment)
-   **Node.js** (v18 or higher) and **npm**
-   **Python** (3.10 or higher)
-   **Composer** (PHP dependency manager)
-   **MySQL** (8.0 or higher) with vector support capabilities
-   **Google AI Studio API Key** ([Get one here](https://aistudio.google.com/app/apikey))

## 🛠️ Installation & Setup

### Quick Setup (Recommended)

For a fully automated setup, use the provided scripts:

```bash
# Clone the repository
git clone <your-repo-url>
cd gemini-faq-app

# Run automated setup
./setup.sh

# Start all services
./start.sh
```

### Development Helper Script

Use the development helper for common tasks:

```bash
# Complete setup and start
./dev.sh setup
./dev.sh start

# Check service status
./dev.sh status

# Run tests
./dev.sh test

# Clean environment
./dev.sh clean

# View logs
./dev.sh logs
```

### Manual Setup (Alternative)

If you prefer manual setup, follow these steps:

#### Step 1: Clone and Setup DDEV

```bash
# Clone the repository
git clone <your-repo-url>
cd gemini-faq-app

# Start DDEV (if not already configured)
ddev start

# Install PHP dependencies
ddev composer install
```

#### Step 2: Environment Configuration

```bash
# Create environment file
cp .env.example .env

# Generate Laravel application key
ddev artisan key:generate
```

Edit the `.env` file and add your Google API key:

```bash
GOOGLE_API_KEY=your_google_api_key_here
```

#### Step 3: Frontend Setup (React + Vite)

```bash
# Install Node.js dependencies
ddev exec "npm install"

# Start Vite development server (in background)
ddev exec "npm run dev" &
```

#### Step 4: Python RAG Service Setup

```bash
# Install Python dependencies
cd services/rag-service
pip3 install -r requirements.txt

# Create local environment configuration (for DDEV setup)
cat > .env.local << EOF
RAG_DB_HOST=127.0.0.1
RAG_DB_PORT=32774
RAG_DB_USER=db
RAG_DB_PASSWORD=db
RAG_DB_DATABASE=db
GOOGLE_API_KEY=your_google_api_key_here
EOF
```

#### Step 5: Database Setup

```bash
# Run database migrations
ddev artisan migrate

# The migration will create the necessary tables:
# - rag_documents (for storing uploaded documents)
# - rag_chunks (for storing text chunks with embeddings)
```

#### Step 6: Start the Python RAG Service

```bash
# Start Flask service (in background)
cd services/rag-service
./start.sh background
```

#### Step 7: Verify Installation

Test that all components are working:

```bash
# Test Python service health
curl http://127.0.0.1:5002/health

# Test Laravel API
curl https://gemini-faq-app.ddev.site/api/content/status

# Test Gemini API connection
curl http://127.0.0.1:5002/test-gemini
```

## 🤖 Automation Scripts

The project includes several scripts to automate common development tasks:

### Setup Scripts

-   **`setup.sh`** - Complete automated setup script that handles all installation steps
-   **`start.sh`** - Starts all application services (Vite + Python RAG service)
-   **`dev.sh`** - Development helper with multiple commands

### Available Commands

```bash
# Complete setup from scratch
./setup.sh

# Start all services
./start.sh

# Development helper commands
./dev.sh setup     # Run setup
./dev.sh start     # Start services
./dev.sh status    # Check service status
./dev.sh test      # Run application tests
./dev.sh clean     # Clean development environment
./dev.sh logs      # View application logs
./dev.sh help      # Show help information
```

### Script Features

-   ✅ **Automated dependency installation** (PHP, Node.js, Python packages)
-   ✅ **Environment configuration** with interactive Google API key setup
-   ✅ **Database migration** and setup
-   ✅ **Service health testing** to verify everything works
-   ✅ **Graceful service management** with proper cleanup on exit
-   ✅ **Colored output** for better readability
-   ✅ **Error handling** with informative messages

## 🌐 Access Your Application

Once everything is running, you can access:

-   **Chatbot Interface**: `https://gemini-faq-app.ddev.site`
-   **Content Management**: `https://gemini-faq-app.ddev.site/content-management`
-   **Laravel API**: `https://gemini-faq-app.ddev.site/api/content/`
-   **Python RAG Service**: `http://localhost:5002` (internal)

## 📡 API Endpoints

### Content Management

#### Upload Text Content

```bash
POST /api/content/ingest
Content-Type: application/json

{
  "content": "Your content to add to the knowledge base",
  "clear_existing": false
}
```

#### Upload PDF File

```bash
POST /api/content/upload-pdf
Content-Type: multipart/form-data

{
  "file": <PDF file>,
  "clear_existing": false
}
```

#### Query Knowledge Base

```bash
POST /api/content/chat
Content-Type: application/json

{
  "question": "Your question about the ingested content"
}
```

#### Get System Status

```bash
GET /api/content/status
```

#### Get All Content

```bash
GET /api/content/all
```

#### Delete Document

```bash
DELETE /api/content/document/{id}
```

#### Clear Knowledge Base

```bash
POST /api/content/clear
```

## 🧪 Testing the Complete Workflow

### 1. Upload Text Content

```bash
# Add information about Laravel
curl -X POST https://gemini-faq-app.ddev.site/api/content/ingest \
  -H 'Content-Type: application/json' \
  -d '{"content": "Laravel is a free, open-source PHP web framework created by Taylor Otwell for web application development following the MVC pattern."}'
```

### 2. Upload PDF Document

```bash
# Upload a PDF file
curl -X POST https://gemini-faq-app.ddev.site/api/content/upload-pdf \
  -F 'file=@/path/to/your/document.pdf' \
  -F 'clear_existing=false'
```

### 3. Test Queries

```bash
# Ask about Laravel
curl -X POST https://gemini-faq-app.ddev.site/api/content/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "What is Laravel?"}'

# Ask about content from uploaded PDF
curl -X POST https://gemini-faq-app.ddev.site/api/content/chat \
  -H 'Content-Type: application/json' \
  -d '{"question": "Tell me about the document content"}'
```

### 4. Content Management

```bash
# View all uploaded documents
curl https://gemini-faq-app.ddev.site/api/content/all

# Check system status
curl https://gemini-faq-app.ddev.site/api/content/status

# Clear all content
curl -X POST https://gemini-faq-app.ddev.site/api/content/clear
```

## 🔧 Configuration

### Python RAG Service Configuration

Edit `services/rag-service/app.py` to customize:

-   **Chunk Size**: Default 500 characters (adjustable in `chunk_text()` function)
-   **Chunk Overlap**: Default 50 characters (prevents information loss)
-   **Gemini Model**: `gemini-1.5-flash` (AI model for response generation)
-   **Embedding Model**: `all-MiniLM-L6-v2` (sentence transformer for vector embeddings)
-   **Database**: MySQL with LONGTEXT support for large documents

### Supported File Types

-   **Text Files**: `.txt` (direct text ingestion)
-   **PDF Files**: `.pdf` (text extraction using PyPDFLoader)
-   **Size Limit**: 10MB maximum file size

### Environment Variables

```bash
# Required environment variables
GOOGLE_API_KEY=your_google_api_key_here

# Database configuration (auto-configured for DDEV)
RAG_DB_HOST=127.0.0.1
RAG_DB_PORT=32774
RAG_DB_USER=db
RAG_DB_PASSWORD=db
RAG_DB_DATABASE=db
```

## 🔍 Troubleshooting

### Common Issues

1. **Gemini API Errors**: Ensure your API key is valid and has sufficient quota
2. **Python Service Not Starting**: Check that all Python dependencies are installed
3. **Database Connection**: Verify DDEV is running and ports are correctly configured
4. **PDF Upload Errors**: Ensure database supports LONGTEXT and file size is under 10MB
5. **Frontend Not Loading**: Verify Vite dev server is running and build is up to date

### Debug Commands

```bash
# Check DDEV status
ddev status

# Check Python service health
curl http://127.0.0.1:5002/health

# View Laravel logs
tail -f storage/logs/laravel.log

# Test Python dependencies
cd services/rag-service
python3 -c "import langchain_community.document_loaders; print('PDF support OK')"
python3 -c "import sentence_transformers; print('Embeddings OK')"
python3 -c "import google.generativeai; print('Gemini OK')"

# Test database connection
ddev mysql -e "SHOW TABLES;"
```

### Database Schema Verification

```bash
# Verify tables exist
ddev mysql -e "DESCRIBE rag_documents;"
ddev mysql -e "DESCRIBE rag_chunks;"

# Check content column type (should be LONGTEXT)
ddev mysql -e "SHOW FULL COLUMNS FROM rag_documents WHERE Field='content';"
```

## 🏛️ Project Structure

```
├── app/Http/Controllers/Api/FaqController.php  # Laravel API controller
├── resources/js/                               # React frontend
│   ├── app.jsx                                # Main React app
│   └── components/ChatbotApp.jsx              # Chatbot component
├── services/                                  # Service layer
│   └── rag-service/                          # Python RAG service
│       ├── app.py                            # Flask application
│       ├── cli.py                            # CLI interface for Laravel
│       ├── requirements.txt                  # Python dependencies
│       ├── start.sh                          # Service startup script
│       ├── stop.sh                           # Service stop script
│       ├── test.py                           # Test suite
│       └── test_data.txt                     # Sample FAQ content
├── routes/
│   ├── web.php                               # Web routes
│   └── api.php                               # API routes
└── vite.config.js                            # Frontend build config
```

## 📝 How It Works

1. **Content Ingestion**:

    - User submits content via React frontend
    - Laravel API receives request and forwards to Python service
    - Python service processes content, creates embeddings using sentence transformers, stores in MySQL

2. **Question Answering**:

    - User asks question via chat interface
    - Laravel API forwards question to Python service
    - Python service searches MySQL for relevant content using cosine similarity
    - Gemini AI generates contextual answer using retrieved content
    - Response includes answer and source documents

3. **Vector Search**:
    - Content is chunked and converted to embeddings using sentence transformers
    - Questions are embedded and similarity search performed using MySQL cosine similarity
    - Most relevant chunks are retrieved for context

## 🔐 Security Notes

-   Keep your Google API key secure and never commit it to version control
-   The Python service runs locally and should not be exposed publicly
-   Use Laravel's built-in security features for production deployment

## 🆕 Current Architecture Status

**Active Service**: `app_mysql.py` running on port **5002**

-   ✅ MySQL database for vector storage with cosine similarity search
-   ✅ Google Gemini AI integration for answer generation
-   ✅ Sentence Transformers for embedding generation
-   ✅ Optimized prompt engineering to avoid redundant text in responses

**Legacy Service**: `app.py` (ChromaDB-based) - deprecated but available for reference

## 📄 License

This project is open-sourced software licensed under the [MIT license](https://opensource.org/licenses/MIT).