# RAG Service

A consolidated Python service for RAG (Retrieval-Augmented Generation) functionality, designed for staging and production deployment.

## 🚀 DevOps Quick Start

### Prerequisites

-   Python 3.8+ with pip3
-   MySQL/MariaDB database (configured in parent Laravel app)
-   Google API key for Gemini AI
-   4GB+ RAM recommended (for AI models)
-   Network access to download ML models on first run

### Production Deployment Commands

```bash
# 1. Navigate to RAG service directory
cd /path/to/project/services/rag-service

# 2. Copy and configure environment
cp .env.example .env.local
# Edit .env.local with your configuration

# 3. Start service in background (production mode)
./start.sh background

# 4. Verify service is running
curl http://localhost:5002/rag-service/health

# 5. Check service status
curl http://localhost:5002/rag-service/status
```

### Service Management

```bash
# Start service (foreground - for debugging)
./start.sh

# Start service (background - for production)
./start.sh background

# Stop service gracefully
./stop.sh

# View logs (when running in background)
tail -f rag_service.log

# Check if service is running
ps aux | grep app.py
```

## Features

-   **URL Path Routing**: Service runs under `/rag-service/` path for proper staging deployment
-   **Consolidated Architecture**: Single service combining all RAG functionality
-   **Production Ready**: Proper logging, health checks, and process management
-   **CLI Interface**: Command-line interface for Laravel integration
-   **Comprehensive Testing**: Built-in test suite
-   **Auto Virtual Environment**: Automatically manages Python virtual environment
-   **Process Management**: PID tracking and graceful shutdown

## Service Endpoints

All endpoints are available under the `/rag-service/` path:

-   `GET /rag-service/health` - Health check
-   `GET /rag-service/status` - Service status and statistics
-   `POST /rag-service/ingest` - Ingest content
-   `POST /rag-service/query` - Query knowledge base (now stores conversations)
-   `POST /rag-service/upload-pdf` - Upload and process PDF files
-   `POST /rag-service/clear` - Clear knowledge base

### Query Endpoint Enhancement

The `/rag-service/query` endpoint now automatically stores all conversations in the Laravel database for analytics and monitoring. The request now accepts additional optional parameters:

```json
{
    "query": "Your question here",
    "session_id": "optional_session_identifier"
}
```

**Response includes:**

-   `answer`: The generated response
-   `sources`: Context sources used
-   `processing_time`: Time taken to generate the response
-   `session_id`: Session identifier for tracking

**Conversation Storage:**
Each query automatically stores the following data:

-   Question and answer
-   Context chunks used
-   Processing time and similarity scores
-   User IP and session information
-   Technical metadata (models used, etc.)

## Quick Start

### Development Mode

```bash
# Install dependencies and start service
./start.sh

# Or run in background
./start.sh background
```

### Production Mode

```bash
# Set production environment variables
export RAG_SERVICE_DEBUG=false
export RAG_SERVICE_HOST=0.0.0.0
export RAG_SERVICE_PORT=5002

# Start service
./start.sh background
```

### Stop Service

```bash
./stop.sh
```

## Configuration

### Required Environment Variables

**Critical for Production:**

```bash
# Service Configuration
RAG_SERVICE_HOST=0.0.0.0          # Bind to all interfaces for production
RAG_SERVICE_PORT=5002             # Default port (ensure it's open in firewall)
RAG_SERVICE_DEBUG=false           # MUST be false in production

# Laravel Integration
LARAVEL_APP_URL=http://localhost:8000  # Laravel app URL for conversation storage

# Database Configuration (inherits from parent .env if not specified)
RAG_DB_HOST=db                    # Database host
RAG_DB_PORT=3306                  # Database port
RAG_DB_USER=db                    # Database username
RAG_DB_PASSWORD=db                # Database password
RAG_DB_DATABASE=db                # Database name

# AI Configuration - REQUIRED
GOOGLE_API_KEY=your_api_key_here  # Get from Google Cloud Console

# Optional
LOG_LEVEL=INFO                    # DEBUG, INFO, WARNING, ERROR
```

### Environment Setup

1. **Copy configuration template:**

    ```bash
    cp .env.example .env.local
    ```

2. **Edit configuration:**

    ```bash
    nano .env.local  # or your preferred editor
    ```

3. **Set Google API Key:**
    - Visit [Google AI Studio](https://aistudio.google.com/app/apikey)
    - Generate API key
    - Add to `.env.local`: `GOOGLE_API_KEY=your_actual_key_here`

### Database Requirements

The service requires these tables in your MySQL database:

-   `rag_documents` - Document storage
-   `rag_chunks` - Text chunks with embeddings

**Note:** These tables are created by Laravel migrations, not by this service.

## 💬 Conversation Storage & Analytics

### Automatic Conversation Logging

The RAG service now automatically stores all questions and answers in the Laravel database for analytics and monitoring purposes. Each query to the `/rag-service/query` endpoint creates a record in the `chatbot_conversations` table.

### Laravel Integration

**Database Schema:**
The service requires the `chatbot_conversations` table (created via Laravel migration):

-   `question` - User's question
-   `answer` - Generated response
-   `context_chunks` - JSON array of context sources used
-   `processing_time` - Response generation time
-   `similarity_score` - Average context similarity
-   `session_id` - Session tracking
-   `user_ip` - Request origin
-   `metadata` - Technical details

**API Endpoints:**
Laravel provides API endpoints for conversation management:

-   `POST /api/chatbot/conversations` - Store conversation (used by RAG service)
-   `GET /api/chatbot/conversations/stats` - Get statistics
-   `GET /api/chatbot/conversations/recent` - Get recent conversations
-   `GET /api/chatbot/conversations/search` - Search conversations
-   `GET /api/chatbot/conversations/session/{id}` - Get session history

**Admin Interface:**
Access the conversation analytics dashboard at:

-   `http://your-domain/admin/chatbot/dashboard` - Main dashboard
-   `http://your-domain/admin/chatbot/conversations` - Conversation browser
-   CSV export functionality for data analysis

### Configuration for Conversation Storage

Add to your RAG service `.env.local`:

```bash
# Laravel app URL for storing conversations
LARAVEL_APP_URL=http://localhost:8000
```

**Note:** If conversation storage fails, the query will still succeed - storage is non-blocking.

### Reverse Proxy Configuration

For staging/production deployment, the service should run behind a reverse proxy:

#### Nginx Configuration

```nginx
# Add to your Nginx server block
location /rag-service/ {
    proxy_pass http://localhost:5002/rag-service/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 300s;
    proxy_connect_timeout 300s;
}
```

#### Apache Configuration

```apache
# Add to your Apache virtual host
ProxyPass /rag-service/ http://localhost:5002/rag-service/
ProxyPassReverse /rag-service/ http://localhost:5002/rag-service/
ProxyPreserveHost On
```

### Firewall Configuration

```bash
# Allow port 5002 for local connections only (recommended)
ufw allow from 127.0.0.1 to any port 5002

# Or allow from specific server IP
ufw allow from YOUR_SERVER_IP to any port 5002
```

### SystemD Service (Optional)

For production deployment, you can create a systemd service:

```ini
# /etc/systemd/system/rag-service.service
[Unit]
Description=RAG Service for SACAP FAQ
After=network.target mysql.service

[Service]
Type=forking
User=www-data
Group=www-data
WorkingDirectory=/path/to/your/project/services/rag-service
ExecStart=/path/to/your/project/services/rag-service/start.sh background
ExecStop=/path/to/your/project/services/rag-service/stop.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

```bash
# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable rag-service
sudo systemctl start rag-service
sudo systemctl status rag-service
```

## CLI Interface

The service provides a CLI interface for integration with Laravel:

```bash
# Ingest content
echo '{"operation": "ingest", "content": "Your content here"}' | python3 cli.py

# Query knowledge base
echo '{"operation": "query", "query": "Your question here"}' | python3 cli.py

# Upload PDF
echo '{"operation": "upload_pdf", "file_path": "/path/to/file.pdf"}' | python3 cli.py

# Clear knowledge base
echo '{"operation": "clear"}' | python3 cli.py
```

## Testing

Run the comprehensive test suite:

```bash
# Make sure service is running first
./start.sh background

# Run tests
python3 test.py

# Stop service
./stop.sh
```

## 📊 Service Architecture

### Components Overview

```
RAG Service Architecture:
┌─────────────────────────────────────────┐
│                Browser                  │
└──────────────┬──────────────────────────┘
               │ HTTP Requests
┌──────────────▼──────────────────────────┐
│         Nginx/Apache Proxy              │
│    Location: /rag-service/*             │
└──────────────┬──────────────────────────┘
               │ Proxy to localhost:5002
┌──────────────▼──────────────────────────┐
│         Flask App (app.py)              │
│    Blueprint: /rag-service/*            │
│    - Health checks                      │
│    - Content ingestion                  │
│    - Query processing                   │
│    - PDF upload                         │
└──────────┬───────────────┬──────────────┘
           │               │
┌──────────▼──────────┐   ┌▼──────────────┐
│   MySQL Database    │   │  AI Models    │
│   - rag_documents   │   │  - Gemini AI  │
│   - rag_chunks      │   │  - Embeddings │
└─────────────────────┘   └───────────────┘
```

### File Structure

```
services/rag-service/
├── app.py              # Main Flask application
├── cli.py              # CLI interface for Laravel
├── start.sh            # Production startup script
├── stop.sh             # Graceful shutdown script
├── test.py             # Comprehensive test suite
├── requirements.txt    # Python dependencies
├── .env.example        # Configuration template
├── .env.local          # Local configuration (not in git)
├── rag_service.log     # Service logs (when running in background)
├── rag_service.pid     # Process ID file (when running)
├── venv/              # Python virtual environment (auto-created)
└── README.md          # This documentation
```

### Process Flow

1. **Startup Process:**

    - `start.sh` creates/activates virtual environment
    - Installs Python dependencies from `requirements.txt`
    - Loads environment variables from `.env.local`
    - Downloads and initializes AI models (first run only)
    - Starts Flask server with PID tracking

2. **Request Processing:**

    - Nginx/Apache proxies requests to Flask app
    - Flask routes requests via Blueprint (`/rag-service/*`)
    - Database operations via MySQL connector
    - AI processing via Google Gemini and sentence transformers

3. **Shutdown Process:**
    - `stop.sh` sends SIGTERM to process
    - Graceful shutdown with cleanup
    - PID file removed

### Key Features

1. **URL Path Routing**: All endpoints under `/rag-service/` for proper staging deployment
2. **Virtual Environment Management**: Automatic creation and management of Python virtual environment
3. **Process Management**: PID tracking and graceful shutdown
4. **Logging**: Structured logging with configurable levels
5. **Health Checks**: Comprehensive health and status endpoints
6. **Error Handling**: Robust error handling and recovery

### Database Schema

The service uses the existing Laravel database schema:

**rag_documents table:**

-   `id` - Primary key
-   `content` - Full document content (LONGTEXT)
-   `session_id` - Session identifier
-   `content_type` - Type of content
-   `source` - Content source identifier
-   `metadata` - JSON metadata
-   `created_at`, `updated_at` - Timestamps

**rag_chunks table:**

-   `id` - Primary key
-   `document_id` - Foreign key to rag_documents
-   `content` - Chunk text content
-   `embedding` - JSON array of vector embeddings
-   `chunk_index` - Position in original document
-   `session_id` - Session identifier
-   `metadata` - JSON metadata
-   `created_at`, `updated_at` - Timestamps

## 🚨 Monitoring & Alerting

### Health Check Endpoints

**Primary Health Check:**

```bash
curl -f http://localhost:5002/rag-service/health || echo "Service DOWN"
```

**Detailed Status Check:**

```bash
curl -s http://localhost:5002/rag-service/status | jq '.'
```

### Monitoring Script Example

```bash
#!/bin/bash
# monitor-rag-service.sh
SERVICE_URL="http://localhost:5002/rag-service/health"
LOG_FILE="/var/log/rag-service-monitor.log"

if ! curl -f -s "$SERVICE_URL" > /dev/null; then
    echo "$(date): RAG Service is DOWN" >> "$LOG_FILE"
    # Send alert (email, Slack, etc.)
    # systemctl restart rag-service  # Auto-restart if desired
else
    echo "$(date): RAG Service is UP" >> "$LOG_FILE"
fi
```

### Cron Job for Monitoring

```bash
# Add to crontab (crontab -e)
# Check every 5 minutes
*/5 * * * * /path/to/monitor-rag-service.sh

# Daily log rotation
0 0 * * * find /path/to/rag-service -name "*.log" -mtime +7 -delete
```

### Metrics to Monitor

**Service Metrics:**

-   Response time for health checks
-   Memory usage (should be < 4GB)
-   CPU usage
-   Disk space usage
-   Process uptime

**Application Metrics:**

-   Number of documents in database
-   Number of chunks processed
-   Query response times
-   Error rates in logs

### Alerting Conditions

**Critical Alerts:**

-   Service health check fails
-   Memory usage > 90%
-   Disk space < 1GB
-   Database connection failures

**Warning Alerts:**

-   Response time > 30 seconds
-   Memory usage > 80%
-   High error rate in logs
-   Model loading failures

## 🔄 Deployment Checklist

### Pre-Deployment

-   [ ] Database tables created via Laravel migrations
-   [ ] Google API key obtained and tested
-   [ ] Server resources meet requirements (4GB RAM, 2+ CPU cores)
-   [ ] Firewall configured
-   [ ] Reverse proxy configured

### Deployment

-   [ ] Code deployed to server
-   [ ] `.env.local` configured with production values
-   [ ] `RAG_SERVICE_DEBUG=false` set
-   [ ] Service started: `./start.sh background`
-   [ ] Health check passes: `curl http://localhost:5002/rag-service/health`

### Post-Deployment

-   [ ] Service accessible via reverse proxy
-   [ ] Test query returns valid response
-   [ ] Logs are being written correctly
-   [ ] Monitoring/alerting configured
-   [ ] Performance benchmarks established

### Rollback Plan

1. Stop service: `./stop.sh`
2. Restore previous version code
3. Restore previous `.env.local` if needed
4. Start service: `./start.sh background`
5. Verify health check passes

## 📚 API Documentation

## 🔧 Troubleshooting & Operations

### Health Monitoring

**Basic Health Check:**

```bash
curl http://localhost:5002/rag-service/health
```

**Detailed Status:**

```bash
curl http://localhost:5002/rag-service/status
```

**Expected Healthy Response:**

```json
{
    "status": "healthy",
    "service": "rag-service",
    "database": "connected",
    "embedding_model_loaded": true,
    "gemini_available": true,
    "documents": 5,
    "chunks": 150
}
```

### Common Issues & Solutions

#### 1. Service Won't Start

```bash
# Check Python and dependencies
python3 --version
pip3 --version

# Check if port is in use
netstat -tlnp | grep :5002
# Or: lsof -i :5002

# Check virtual environment
ls -la venv/
./start.sh  # Run in foreground to see errors
```

#### 2. Database Connection Issues

```bash
# Test database connection
mysql -h $RAG_DB_HOST -P $RAG_DB_PORT -u $RAG_DB_USER -p$RAG_DB_PASSWORD $RAG_DB_DATABASE

# Check if required tables exist
mysql -h $RAG_DB_HOST -P $RAG_DB_PORT -u $RAG_DB_USER -p$RAG_DB_PASSWORD $RAG_DB_DATABASE -e "SHOW TABLES LIKE 'rag_%';"
```

#### 3. AI Model Loading Issues

```bash
# Check Google API key
curl -H "Authorization: Bearer $GOOGLE_API_KEY" https://generativelanguage.googleapis.com/v1beta/models

# Check internet connectivity for model downloads
ping -c 3 huggingface.co

# Check disk space (models need ~500MB)
df -h
```

#### 4. Performance Issues

```bash
# Check memory usage
free -h
ps aux | grep python | grep app.py

# Check CPU usage
top -p $(pgrep -f "python.*app.py")

# Check logs for errors
tail -f rag_service.log | grep -i error
```

### Log Analysis

**Log Locations:**

-   **Background mode:** `rag_service.log`
-   **Foreground mode:** Console output
-   **System logs:** `/var/log/syslog` (if using systemd)

**Common Log Patterns:**

```bash
# Check for startup issues
grep -i "error\|failed\|exception" rag_service.log

# Monitor real-time activity
tail -f rag_service.log | grep -E "(query|ingest|error)"

# Check model loading status
grep -i "model.*loaded" rag_service.log
```

### Performance Monitoring

**Resource Requirements:**

-   **RAM:** 2-4GB (models loaded in memory)
-   **CPU:** 2+ cores recommended
-   **Disk:** 1GB+ for models and logs
-   **Network:** Stable internet for initial model downloads

**Monitor Key Metrics:**

```bash
# Memory usage
ps aux | grep python | awk '{print $6}' | head -1  # RSS memory

# Response time
time curl -X POST http://localhost:5002/rag-service/query -H "Content-Type: application/json" -d '{"query":"test"}'

# Active connections
netstat -an | grep :5002 | grep ESTABLISHED | wc -l
```

### Backup & Recovery

**Important Files to Backup:**

-   `.env.local` - Configuration
-   `rag_service.log` - Operational logs
-   Database tables: `rag_documents`, `rag_chunks`

**Recovery Process:**

1. Restore configuration file
2. Restart service: `./stop.sh && ./start.sh background`
3. Verify health: `curl http://localhost:5002/rag-service/health`
4. Re-ingest content if database was restored

### Security Considerations

**Production Checklist:**

-   [ ] `RAG_SERVICE_DEBUG=false` in production
-   [ ] Firewall configured to block external access to port 5002
-   [ ] Google API key stored securely (not in version control)
-   [ ] Database credentials secured
-   [ ] Logs rotation configured
-   [ ] Service runs as non-root user
-   [ ] Regular security updates applied

## Migration from Old Structure

This service replaces the old `python_rag_service` and `python_scripts` directories. The main differences:

1. **URL Structure**: Endpoints now under `/rag-service/` path
2. **Process Management**: Improved startup/shutdown scripts
3. **Configuration**: Centralized environment configuration
4. **Testing**: Built-in comprehensive test suite
5. **CLI Interface**: Standalone CLI for Laravel integration

### Migration Steps

1. Stop old services
2. Update Laravel integration to use new endpoints
3. Deploy new service structure
4. Test functionality
5. Remove old service directories

## 👥 Development & Contributing

### Development Setup

```bash
# Clone and navigate to service directory
cd services/rag-service

# Copy environment template
cp .env.example .env.local

# Edit configuration for development
nano .env.local

# Start in development mode (foreground)
./start.sh
```

### Adding New Endpoints

Add new routes to the `rag_bp` Blueprint in `app.py`:

```python
@rag_bp.route('/new-endpoint', methods=['POST'])
def new_endpoint():
    """New endpoint description"""
    try:
        # Your implementation here
        return jsonify({'status': 'success'})
    except Exception as e:
        return jsonify({'error': str(e)}), 500
```

### Modifying AI Behavior

Customize the AI behavior in the `generate_answer()` function:

```python
def generate_answer(query: str, context_chunks: List[Dict[str, Any]]) -> str:
    """Generate answer using Google Gemini AI"""
    # Modify the prompt or logic here
    # Update the persona in the prompt
    # Add new response formatting
    pass
```

### Testing Changes

```bash
# Run comprehensive test suite
python3 test.py

# Test specific functionality
curl -X POST http://localhost:5002/rag-service/query \
  -H "Content-Type: application/json" \
  -d '{"query": "test question"}'
```

### Contributing Guidelines

1. Make changes to the service
2. Run tests: `python3 test.py`
3. Test with Laravel integration
4. Update documentation as needed
5. Follow existing code style and patterns

## 📞 Support & Contact

### For DevOps Issues:

-   Check logs: `tail -f rag_service.log`
-   Verify configuration: `cat .env.local`
-   Test database connection
-   Check service status: `curl http://localhost:5002/rag-service/health`

### For Development Issues:

-   Review application logs for errors
-   Test individual endpoints
-   Check AI model loading status
-   Verify Python dependencies

### Common Commands Reference

```bash
# Service Management
./start.sh background    # Start in production mode
./start.sh              # Start in development mode
./stop.sh               # Stop service

# Monitoring
curl http://localhost:5002/rag-service/health
curl http://localhost:5002/rag-service/status
tail -f rag_service.log
ps aux | grep app.py

# Testing
python3 test.py
curl -X POST http://localhost:5002/rag-service/query -H "Content-Type: application/json" -d '{"query":"test"}'

# Debugging
python3 app.py          # Run directly for debugging
./start.sh              # Run in foreground to see errors
```

---

**Last Updated:** July 2025  
**Version:** 2.0 (Consolidated Service)  
**Maintainer:** Development Team