User sends message
        |
        v
sam_chat_controller.py ──────> ai_service.py (2500+ lines!)
        |                           |
        |                           ├── context gathering
        |                           ├── permission checks
overlay_manager.js                  ├── memory search
        |                           ├── AI API calls
        |                           ├── streaming
sam_chat_vanilla_v2.js              └── error handling
        |
        v
   5+ files touched for any bug fix

                    ┌─────────────────┐
                    │ http_routes.py  │  "I receive web requests"
                    └────────┬────────┘
                             │
                             v
┌────────────────────────────────────────────────────────────┐
│                    conversation.py                          │
│                "The brain - coordinates everything"         │
│                                                            │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │
│   │ ai_voice.py  │  │  memory.py   │  │api_services.py│   │
│   │ "THE VOICE"  │  │ "remembers"  │  │ "calls AI"   │    │
│   └──────────────┘  └──────────────┘  └──────────────┘    │
│                                                            │
│                    ┌──────────────┐                        │
│                    │ response.py  │                        │
│                    │"formats reply"│                        │
│                    └──────────────┘                        │
└────────────────────────────────────────────────────────────┘

                    ┌─────────────────┐
                    │  workflow.py    │  "Orchestrates node execution"
                    └────────┬────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        v                    v                    v
┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│node_input.py │    │   node.py    │    │node_output.py│
│              │    │              │    │              │
│"gathers data"│    │ "processes"  │    │"sends result"│
│              │    │              │    │              │
│ - folders    │    │ - executes   │    │ - next node  │
│ - files      │    │ - transforms │    │ - files      │
│ - prev nodes │    │ - AI calls   │    │ - APIs       │
│ - APIs       │    │              │    │ - Odoo       │
│ - webhooks   │    │              │    │              │
└──────────────┘    └──────────────┘    └──────────────┘

CONVERSATION inherits from ──────> BASE PATTERN <────── WORKFLOW inherits from

                              ┌─────────────┐
                              │   CORE.py   │
                              │ orchestrator│
                              └──────┬──────┘
                                     │
                    ┌────────────────┼────────────────┐
                    │                │                │
                    v                v                v
              ┌─────────┐     ┌─────────┐     ┌─────────┐
              │ INPUT   │     │ PROCESS │     │ OUTPUT  │
              │ gather  │     │ execute │     │ format  │
              └─────────┘     └─────────┘     └─────────┘

ai_sam_base/
├── __init__.py
├── __manifest__.py
│
├── models/                      # Odoo ORM models (database)
│   ├── __init__.py
│   ├── ai_conversation.py       # Conversation records (unchanged)
│   ├── ai_message.py            # Message records (unchanged)
│   └── ai_file_permission.py    # Permission records (unchanged)
│
├── services/                    # NEW - Business logic
│   ├── __init__.py
│   │
│   │   # === CORE ORCHESTRATOR ===
│   ├── conversation.py          # THE BRAIN - all chat flows through here
│   │
│   │   # === SUPPORTING SERVICES ===
│   ├── http_routes.py           # Web endpoints (receives requests)
│   ├── api_services.py          # External AI providers (Claude, GPT)
│   ├── memory.py                # Vector search, past conversations
│   ├── ai_voice.py              # THE VOICE - Prompt composer (renamed from behaviour.py)
│   └── response.py              # Streaming, formatting, error handling
│
└── controllers/                 # Odoo HTTP controllers (thin wrappers)
    ├── __init__.py
    └── main.py                  # Routes to services/http_routes.py

ai_sam_workflows/
├── __init__.py
├── __manifest__.py
│
├── models/                      # Odoo ORM models (database)
│   ├── __init__.py
│   └── workflow_node.py         # Node records (unchanged)
│
├── services/                    # NEW - Business logic
│   ├── __init__.py
│   │
│   │   # === CORE ORCHESTRATOR ===
│   ├── workflow.py              # THE BRAIN - all workflow flows through here
│   │
│   │   # === NODE LIFECYCLE ===
│   ├── node_input.py            # Gathers data from all sources
│   ├── node.py                  # Processes/executes node logic
│   └── node_output.py           # Sends results to destinations
│
└── controllers/                 # Odoo HTTP controllers (thin wrappers)
    └── main.py

# ai_sam_base/api_communications/conversation.py
"""
Central orchestrator for all SAM AI chat interactions.
ALL behavior modifications happen here or in its service modules.

This is THE file to start with when debugging any chat issue.
"""

class ConversationCore:
    """
    The brain of SAM AI chat.

    Flow:
        1. Receive message via http_routes.py
        2. Build context via behaviour.py
        3. Search memory via memory.py
        4. Send to AI via api_services.py
        5. Stream response via response.py
    """

    def __init__(self, env):
        self.env = env
        self.voice = Voice(env)  # THE VOICE - prompt composer
        self.memory = Memory(env)
        self.api = APIServices(env)
        self.response = Response(env)

    def process_message(self, user_message, conversation_id, context_data):
        """
        Single entry point for ALL chat messages.

        Args:
            user_message: What the user typed
            conversation_id: Session identifier
            context_data: Frontend context (node_id, folder_file_link, etc.)

        Returns:
            dict: Response with 'content', 'conversation_id', etc.
        """
        # 1. Build full context
        context = self.voice.build_context(context_data)

        # 2. Check permissions
        if permission_needed := self.voice.check_permissions(context):
            return permission_needed

        # 3. Search memory for relevant past conversations
        memories = self.memory.search(user_message, limit=5)

        # 4. Send to AI provider
        ai_response = self.api.send(
            message=user_message,
            context=context,
            memories=memories
        )

        # 5. Format and return response
        return self.response.format(ai_response)

# ai_sam_workflows/services/node_input.py
"""
Handles ALL ways a workflow node can receive data.

This is THE file to check when node inputs aren't working.

Supported input sources:
    - folder_file_link: Local files and folders
    - previous_node_id: Output from upstream nodes
    - api_source: External API data
    - odoo_model: Odoo database records
    - webhook_data: Incoming webhook payloads
"""

class NodeInput:
    """
    Gathers data from all input sources for a workflow node.

    Example:
        input_service = NodeInput(env)
        data = input_service.gather(node_data)
        # data = [
        #     {'type': 'folder', 'path': '...', 'files': [...]},
        #     {'type': 'previous_node', 'output': {...}},
        # ]
    """

    def __init__(self, env):
        self.env = env

    def gather(self, node_data):
        """
        Main entry point - gathers all inputs for a node.

        Args:
            node_data: Dict with node configuration
                - folder_file_link: str (optional)
                - previous_node_id: str (optional)
                - api_source: dict (optional)
                - odoo_model: dict (optional)
                - webhook_data: dict (optional)

        Returns:
            list: All gathered inputs with type, source, and content
        """
        inputs = []

        # File/Folder inputs
        if folder_path := node_data.get('folder_file_link'):
            inputs.extend(self.from_folder(folder_path))

        # Previous node outputs
        if prev_node_id := node_data.get('previous_node_id'):
            inputs.extend(self.from_previous_node(prev_node_id))

        # API inputs
        if api_config := node_data.get('api_source'):
            inputs.extend(self.from_api(api_config))

        # Odoo model inputs
        if model_config := node_data.get('odoo_model'):
            inputs.extend(self.from_odoo(model_config))

        # Webhook inputs
        if webhook := node_data.get('webhook_data'):
            inputs.extend(self.from_webhook(webhook))

        return inputs

    # === INPUT SOURCE METHODS ===

    def from_folder(self, path):
        """Read files from a local folder path."""
        # Validation, reading, content extraction
        ...

    def from_previous_node(self, node_id):
        """Get output from an upstream node."""
        ...

    def from_api(self, api_config):
        """Fetch data from an external API."""
        ...

    def from_odoo(self, model_config):
        """Read records from Odoo database."""
        ...

    def from_webhook(self, webhook_data):
        """Process incoming webhook payload."""
        ...

{
    'type': 'folder',
    'path': '/path/to/shared/folder',
    'stats': {
        'total_folders': 5,
        'total_files': 42,
        'text_files': 28,
        'summary': 'Found 5 folders and 42 files'
    },
    'user_message': 'I can now access 5 folders and 42 total files within the shared link.',
    'subfolders': [...],
    'files': [...],
    'content': '...'  # Human-readable context for AI
}

## Your Instructions for This Context

**CRITICAL: Be Proactive, Not Passive**

1. **READ the extracted content** - It's already provided below
2. **SYNTHESIZE what you find** - Don't just list files
3. **SHARE your perspective** - Like Claude Code does
4. **CONNECT to the user's work** - Relate findings to their needs

**DO NOT:**
- Say "I don't have access" (you do)
- Ask "Would you like me to..." (just do it)
- List files without analyzing them

WORKFLOW: node_input.py → node.py → node_output.py
CHAT:     chat_input.py → ai_brain.py (shared) → chat_output.py

User sends message
        ↓
Controller calls chat_input.gather_context()
        ↓
Generator yields activity events as work happens:
    → 📂 Reading folder...
    → 📄 Found 9 files...
    → 🧠 Searching past conversations...
    → 🔧 Building context...
        ↓
Controller forwards events via SSE
        ↓
Frontend displays each activity
        ↓
🤖 Asking Claude...
        ↓
✅ Done

   for event in chat_input.gather_context(context_data, message):
       if event.get('type') == 'activity':
           yield f"event: activity\ndata: {json.dumps(event)}\n\n"
       elif event.get('type') == 'context_complete':
           gathered_context = event.get('context', {})

User shares a folder link
        ↓
chat_input.gather_context() runs
        ↓
🔐 Checking access...
        ↓
ai.access.gate.check_access(path)
        ↓
┌─────────────────────────────────────┐
│ Already approved? → ✅ Continue     │
│ Explicitly denied? → ⛔ Stop        │
│ New path? → 🔐 Create pending       │
└─────────────────────────────────────┘
        ↓
If pending → yield permission_required event
        ↓
Frontend shows approval buttons:
  [Yes, Allow] [Yes, Allow All in Folder] [No]
        ↓
User clicks → POST /sam/permission_response
        ↓
ai.access.gate.action_approve() or action_approve_recursive()
        ↓
User re-sends message → Now allowed!

event: activity      → 🔐 Checking access to: My Folder...
event: permission_required → {permission_request: {id, path, buttons}}
event: done          → {needs_permission: true}

BEFORE: SAM + Claude = Claude-dependent assistant
AFTER:  SAM = Unique intelligence, Provider = Conversational engine

SAM's characteristics, behavior, and intelligence are CONSISTENT
regardless of which AI provider is the "language engine".

def _stream_openai(self, config, system_prompt, messages):
    """
    Works with ALL OpenAI-compatible providers:
    - OpenAI, Azure OpenAI
    - Groq, Together AI, DeepSeek
    - Ollama, LM Studio (local)
    - Any provider using OpenAI's API format
    """
    with requests.post(url, headers=headers, json=payload, stream=True) as response:
        for line in response.iter_lines(decode_unicode=True):
            if line.startswith('data: '):
                data = json.loads(line[6:])
                content = data['choices'][0]['delta'].get('content', '')
                yield {'type': 'chunk', 'data': {'text': content}}

User sends message
       |
       v
send_message_streaming()
       |
       ├── api_format = 'openai'  ──> _stream_openai() [RAW HTTP SSE]
       ├── api_format = 'anthropic' ──> _stream_anthropic() [SDK]
       └── api_format = 'google'  ──> _stream_google() [NOT YET]

SAM's Value (YOUR IP):
├── Business Intelligence
├── Odoo Integration
├── Memory System
├── Workflow Automation
└── Power Prompts

Language Engine (REPLACEABLE):
├── Anthropic ($$$) - "Claude personality"
├── OpenAI ($$) - "GPT personality"
├── Groq ($) - Llama at incredible speed
└── Ollama (FREE) - 100% local, 100% private

"groq": {
  "displayName": "Groq",
  "api_format": "openai",
  "api_endpoint": "https://api.groq.com/openai",
  "default_model": "llama-3.1-70b-versatile",
  "is_ai_nodes": true,
  "is_whitelisted": true,
  "pricing_notes": "Very low cost - ~$0.05-0.27 per million tokens"
}

"ollama": {
  "displayName": "Ollama",
  "api_format": "openai",
  "api_endpoint": "http://localhost:11434",
  "default_model": "llama3.1",
  "requires_credentials": false,
  "pricing_notes": "FREE - Runs on your own hardware"
}

Development/Testing:  Use Groq (nearly free)
Enterprise customers: Let THEM choose (their API key)
Privacy-sensitive:    Use Ollama (100% local)
Maximum quality:      Anthropic/OpenAI (customer pays)

HELPFUL (Do This):
├── Learn user prefers bullet points → Use bullet points
├── Notice user is technical → Skip basic explanations
├── Recognize recurring topics → Provide deeper context
└── Adapt to communication style → Match their tone

CREEPY (Never Do This):
├── "I noticed you always work late on Thursdays..."
├── "Based on your emotional patterns..."
├── "Your productivity seems lower today..."
└── "I've been tracking your response times..."

scikit-learn>=1.3.0    # Machine Learning for user pattern recognition

┌─────────────────────────────────────────────────────────────┐
│                    ML LAYER (Optional)                       │
│                                                             │
│  ┌─────────────────┐  ┌─────────────────┐                   │
│  │ Pattern Analysis │  │ Style Detection │                   │
│  │ (Topic Clusters) │  │ (TF-IDF + KMeans)│                  │
│  └────────┬────────┘  └────────┬────────┘                   │
│           │                    │                            │
│           └──────────┬─────────┘                            │
│                      │                                      │
│                      ▼                                      │
│           ┌─────────────────────┐                           │
│           │    sam_voice.py     │                           │
│           │   (THE GUARDRAIL)   │                           │
│           │                     │                           │
│           │ ├ FORBIDDEN_PHRASES │                           │
│           │ ├ Silent Adaptation │                           │
│           │ └ Privacy-First     │                           │
│           └──────────┬──────────┘                           │
│                      │                                      │
│                      ▼                                      │
│              AI Brain (ai_brain.py)                         │
│              ML insights filtered through sam_voice         │
└─────────────────────────────────────────────────────────────┘

ai_sam/static/src/js/chat/
├── chat_ui.js              ← Renders the chat interface
├── chat_state.js           ← Manages messages, sessions, context
├── chat_api.js             ← Makes RPC calls to backend
├── chat_input.js           ← Handles user input, attachments
│
└── chat_interaction.js     ← NEW: Real-time activity feedback
                               "The eyes" - Shows what SAM is doing

// ai_sam/static/src/js/chat/chat_interaction.js
/**
 * Real-time activity feedback for SAM AI chat.
 * Shows the human WHAT SAM is doing, not just "Processing..."
 *
 * Philosophy: The human should never wonder "Is it stuck?"
 * They should always see meaningful, changing activity messages.
 */

class ChatInteraction {

    // Activity states with human-readable messages
    static ACTIVITIES = {
        // === INPUT GATHERING ===
        'reading_folder':     '📂 Reading folder: {folder}...',
        'counting_files':     '📄 Found {count} files...',
        'extracting_file':    '📝 Reading: {filename}...',
        'validating_path':    '🔍 Checking path access...',

        // === CONTEXT BUILDING ===
        'searching_memory':   '🧠 Searching past conversations...',
        'found_memories':     '💡 Found {count} relevant memories...',
        'building_context':   '🔧 Building context...',

        // === AI INTERACTION ===
        'calling_ai':         '🤖 Asking Claude...',
        'thinking':           '💭 Thinking...',
        'composing':          '✍️ Composing response...',

        // === RESPONSE ===
        'streaming':          '💬 Responding...',
        'complete':           '✅ Done',

        // === ERRORS ===
        'path_not_found':     '⚠️ Cannot access: {path}',
        'permission_needed':  '🔐 Permission required...',
    };

    constructor(displayElement) {
        this.display = displayElement;
        this.currentActivity = null;
        this.activityHistory = [];
    }

    /**
     * Show an activity to the user
     * @param {string} activity - Key from ACTIVITIES
     * @param {object} params - Values to interpolate ({folder}, {count}, etc.)
     */
    show(activity, params = {}) {
        let message = ChatInteraction.ACTIVITIES[activity] || activity;

        // Interpolate parameters
        Object.keys(params).forEach(key => {
            message = message.replace(`{${key}}`, params[key]);
        });

        this.currentActivity = { activity, message, timestamp: Date.now() };
        this.activityHistory.push(this.currentActivity);
        this.updateDisplay(message);
    }

    /**
     * Update the visual display
     */
    updateDisplay(message) {
        if (this.display) {
            this.display.textContent = message;
            this.display.classList.add('activity-pulse');  // CSS animation
        }
    }

    /**
     * Clear the activity display
     */
    clear() {
        if (this.display) {
            this.display.textContent = '';
            this.display.classList.remove('activity-pulse');
        }
    }

    /**
     * Get activity history (for debugging/logging)
     */
    getHistory() {
        return this.activityHistory;
    }
}

// Export for use in chat modules
window.ChatInteraction = ChatInteraction;

# ai_sam_base/api_communications/conversation.py

def process_message(self, user_message, context_data):
    """
    Process a chat message with real-time activity feedback.
    Yields activity updates that frontend displays to user.
    """

    # === INPUT GATHERING ===
    if folder_path := context_data.get('folder_file_link'):
        yield {'activity': 'reading_folder', 'folder': os.path.basename(folder_path)}

        files = self.list_files(folder_path)
        yield {'activity': 'counting_files', 'count': len(files)}

        for file in files[:5]:  # Show first 5
            yield {'activity': 'extracting_file', 'filename': file.name}
            content = self.extract_content(file)

    # === CONTEXT BUILDING ===
    yield {'activity': 'searching_memory'}
    memories = self.memory.search(user_message)
    if memories:
        yield {'activity': 'found_memories', 'count': len(memories)}

    yield {'activity': 'building_context'}
    context = self.voice.build(context_data, memories)

    # === AI INTERACTION ===
    yield {'activity': 'calling_ai'}
    response_stream = self.api.send_streaming(user_message, context)

    # === RESPONSE STREAMING ===
    yield {'activity': 'streaming'}
    for chunk in response_stream:
        yield {'type': 'content', 'content': chunk}

    yield {'activity': 'complete'}

/* ai_sam/static/src/css/chat_interaction.css */

.chat-activity-display {
    font-size: 13px;
    color: #666;
    padding: 8px 12px;
    min-height: 24px;
    transition: all 0.2s ease;
}

.activity-pulse {
    animation: pulse 1.5s infinite;
}

@keyframes pulse {
    0%, 100% { opacity: 1; }
    50% { opacity: 0.6; }
}

/* Activity-specific colors */
.activity-reading { color: #3498db; }   /* Blue - reading */
.activity-thinking { color: #9b59b6; }  /* Purple - AI thinking */
.activity-error { color: #e74c3c; }     /* Red - errors */
.activity-success { color: #27ae60; }   /* Green - complete */

┌──────────┬──────────────────────────────────────┐
│ ● You    │ can you tell me about my files please│
├──────────┼──────────────────────────────────────┤
│ ● Sam    │ To provide you with information...   │
├──────────┼──────────────────────────────────────┤
│ ● You    │ what can you tell me about them?     │
└──────────┴──────────────────────────────────────┘

Column 1: Names (fixed ~80px, left-aligned)
Column 2: Messages (flexible width)

/* ai_sam/static/src/css/chat_messages.css */

/* Compact 2-column message layout */
.sam-messages-container {
    display: flex;
    flex-direction: column;
    gap: 8px;  /* Tighter spacing between messages */
}

.sam-message {
    display: grid;
    grid-template-columns: 80px 1fr;  /* Name | Message */
    gap: 12px;
    align-items: start;
    padding: 8px 12px;
}

.sam-message.user {
    background: rgba(102, 126, 234, 0.08);
    border-radius: 8px;
}

.sam-message.assistant {
    background: transparent;
}

/* Column 1: Name */
.message-header {
    font-weight: 600;
    font-size: 13px;
    color: #333;
    white-space: nowrap;
    display: flex;
    align-items: center;
    gap: 6px;
}

.message-header .status-dot {
    width: 8px;
    height: 8px;
    border-radius: 50%;
    background: #27ae60;  /* Green = online */
}

.message-header.user .status-dot {
    background: #3498db;  /* Blue for user */
}

/* Column 2: Message content */
.message-content {
    font-size: 14px;
    line-height: 1.5;
    color: #333;
}

/* Timestamp - subtle, inline */
.message-timestamp {
    font-size: 11px;
    color: #999;
    margin-left: 8px;
}

/* REMOVED: Actions row */
.message-actions {
    display: none !important;
}

// In chat_ui.js - Updated message rendering

renderMessage(message) {
    return `
        <div class="sam-message ${message.role}">
            <div class="message-header ${message.role}">
                <span class="status-dot"></span>
                ${message.role === 'user' ? 'You' : 'Sam'}
            </div>
            <div class="message-content">
                ${this.formatContent(message.content)}
                <span class="message-timestamp">${this.formatTime(message.timestamp)}</span>
            </div>
        </div>
    `;
    // NOTE: No .message-actions row - removed for cleaner UI
}