AI Model Configuration by Extraction Type

This document explains the new AI model configuration system that allows you to specify different AI models for different types of document extraction.

Overview

The system now supports:

  • Claude 3.5 Haiku (Latest) as the default model across the system
  • Extraction-type-specific model configuration for permits, 203k loan files, etc.
  • Centralized UI for managing AI models and extraction configurations
  • Admin access control to protect configuration from unauthorized changes

Security & Access Control

⚠️ IMPORTANT: AI model configuration is protected by system administrator access control.

Access Requirements

To access the AI Models configuration (/dashboard/ai-models), you need:

System Admin Password: Valid ADMIN_PASSWORD environment variable

Note: This is a SYSTEM-WIDE configuration that affects all organizations. Organization admin/owner roles are NOT sufficient for access.

Setting Up System Admin Access

Set the ADMIN_PASSWORD environment variable in your .env file:

ADMIN_PASSWORD=your_secure_system_admin_password

Access Hierarchy

  • System Admin Password: Full access to AI model configuration
  • Organization Admins/Owners: NO access (this affects all organizations)
  • Regular Users: Denied access with clear error message

What Changed

1. Default Model Updated

  • Previous default: Claude 3.5 Haiku
  • Current default: Claude 3.5 Haiku (Latest) (claude-3-5-haiku-latest)
  • Available models: Multiple Claude and Gemini models for different use cases

2. New Database Schema

A new table extraction_type_config has been added to manage AI model configurations per extraction type:

CREATE TABLE extraction_type_config (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  organization_id uuid REFERENCES organizations(id),  -- NULL = global config
  extraction_type text NOT NULL,                      -- 'permits', '203k', etc.
  ai_model_id uuid REFERENCES ai_models(id) NOT NULL,
  is_default boolean DEFAULT false,
  created_at timestamp DEFAULT now(),
  updated_at timestamp DEFAULT now()
);

3. New Functions

  • getAIModelForExtractionType(extractionType, organizationId?) - Gets the appropriate AI model for a specific extraction type
  • Fallback hierarchy: Organization-specific → Global configuration → General active model
  • checkAdminAccess(password?) - Verifies admin access via role or password
  • requireAdminAccess(password?) - Throws error if admin access is not available

Supported Extraction Types

  • permits - Permit document analysis (building, electrical, plumbing, etc.)
  • 203k - 203k loan file analysis (property details, renovation costs, etc.)
  • general - General document analysis
  • invoices - Invoice processing
  • inspection - Inspection report analysis

Setup Instructions

1. Set System Admin Password

Set the ADMIN_PASSWORD environment variable:

ADMIN_PASSWORD=your_secure_system_admin_password

2. Ensure Claude 3.5 Haiku is Available

Make sure you have the ANTHROPIC_API_KEY environment variable set with a valid Anthropic API key that has access to Claude 3.5 Haiku.

3. Access the AI Models Dashboard

Navigate to /dashboard/ai-models in your application. You'll be prompted for the system admin password.

4. Add Claude 3.5 Haiku Model (if not already present)

If Claude 3.5 Haiku isn't already in your system:

  1. Go to the "Add New AI Model" section
  2. Fill in:
    • Name: Claude 3.5 Haiku (Latest)
    • Provider: claude
    • Model ID: claude-3-5-haiku-latest
    • API Key Reference: ANTHROPIC_API_KEY
  3. Click "Add Model"
  4. Click "Set as Fallback" to make it the default

5. Configure Extraction Types

For each extraction type you want to configure:

  1. Go to the "Add New Extraction Type Configuration" section
  2. Select:
    • Extraction Type: Choose from permits, 203k, general, etc.
    • AI Model: Select Claude 3.5 Haiku (or another model)
    • Scope: Choose "Global (System-wide)" for system-wide configuration
  3. Click "Add Configuration"

6. Alternative: Run Setup Script (if DATABASE_URL is available)

If you have direct database access, you can run:

# Set your DATABASE_URL environment variable first
export DATABASE_URL="your_database_connection_string"

# Run the setup script
node scripts/setup-claude-sonnet-4.js

This script will:

  • Deactivate all existing models
  • Add/activate Claude Sonnet 4
  • Set up default extraction type configurations for all supported types

Security Features

Protected Routes

All AI model configuration routes are protected:

  • /dashboard/ai-models - Main configuration page
  • /dashboard/ai-models/create - Create new AI model
  • /dashboard/ai-models/toggle/[id] - Toggle model status
  • /dashboard/ai-models/create-extraction-config - Create extraction config
  • /dashboard/ai-models/delete-extraction-config/[id] - Delete extraction config

Error Handling

  • 403 Forbidden: Returned when user lacks admin privileges
  • Clear error messages: Users see helpful messages explaining access requirements
  • Graceful fallbacks: System continues to work for regular users

Audit Trail

  • All admin access attempts are logged
  • User roles and access patterns are tracked
  • Failed access attempts are recorded for security monitoring

How It Works

API Integration

The system now automatically selects the appropriate AI model based on the extraction context:

  1. Permit Extraction: /api/permits/extract-from-document uses getAIModelForExtractionType('permits')
  2. 203k Loan Files: /api/loan-files/create-from-document uses getAIModelForExtractionType('203k')

Fallback Strategy

  1. Organization-specific configuration (if org ID provided)
  2. Global configuration for the extraction type
  3. General active model as final fallback

Example Usage in Code

// Get model for permit extraction
const model = await getAIModelForExtractionType('permits', organizationId);

// Get model for 203k analysis  
const model = await getAIModelForExtractionType('203k', organizationId);

// This automatically handles fallbacks if no specific config exists

Benefits

  1. Optimized Performance: Use different models optimized for different document types
  2. Flexibility: Organizations can customize models per extraction type
  3. Future-Proof: Easy to add new extraction types and models
  4. Cost Control: Use more efficient models for simpler extractions
  5. Quality: Use more capable models for complex extractions
  6. Security: Prevent unauthorized changes to critical AI model configurations

UI Features

The new AI Models dashboard (/dashboard/ai-models) includes:

Admin Access Control

  • Role-based access for organization admins/owners
  • Password-based override for system administrators
  • Clear error messages for unauthorized users
  • Secure password input (if configured)

Model Configuration per Extraction Type

  • Visual display of which models are configured for each extraction type
  • Easy-to-use forms for adding new configurations
  • Global vs. organization-specific scope options
  • Remove configurations with one click

Available AI Models Management

  • View all available AI models
  • Add new models from different providers
  • Set fallback/general model
  • See model status and details

Migration Notes

  • Existing functionality is preserved - if no extraction-type-specific configuration exists, the system falls back to the general active model
  • No breaking changes - all existing API endpoints continue to work
  • Automatic model selection - the system automatically uses the best available model for each extraction type
  • Backward compatibility - non-admin users continue to use the system normally

Troubleshooting

Access Denied

If you get "Admin access required" errors:

  1. Check if you have 'admin' or 'owner' role in your organization
  2. Contact your organization administrator for role promotion
  3. Ask system administrator for the admin password (if configured)
  4. Verify the ADMIN_PASSWORD environment variable is set correctly

Model Not Found

If you get "API key not found" errors:

  1. Verify the ANTHROPIC_API_KEY environment variable is set
  2. Check that the model has the correct apiKeyRef field
  3. Ensure the API key has access to Claude Sonnet 4

No Extraction Type Configuration

If an extraction type doesn't have a specific configuration:

  1. It will fall back to the general active model
  2. You can add a configuration via the dashboard (if you have admin access)
  3. Global configurations work for all organizations

Database Migration Issues

If you encounter migration issues:

  1. The extraction_type_config table should be created automatically
  2. If not, you may need to run database migrations manually
  3. Foreign key constraints should be added to link to ai_models and organizations tables

Environment Variables

# Required for Claude 3.5 Haiku
ANTHROPIC_API_KEY=your_anthropic_api_key_here

# Optional: Admin password for override access
ADMIN_PASSWORD=your_secure_system_admin_password

# Required: Database connection
DATABASE_URL=your_database_connection_string

Future Enhancements

  • Per-user model preferences
  • Usage analytics and cost tracking
  • A/B testing between models
  • Automatic model selection based on document complexity
  • Integration with additional AI providers
  • Multi-factor authentication for admin access
  • Granular permission system

API Reference

New Functions

checkAdminAccess(providedPassword?)

Checks if the current user has admin access.

Parameters:

  • providedPassword?: string - Optional admin password for override access

Returns: Promise<boolean> - True if user has admin access

requireAdminAccess(providedPassword?)

Throws an error if the current user doesn't have admin access.

Parameters:

  • providedPassword?: string - Optional admin password for override access

Throws: Error with message "Admin access required"

getAIModelForExtractionType(extractionType, organizationId?)

Returns the appropriate AI model for the specified extraction type.

Parameters:

  • extractionType: string - The type of extraction ('permits', '203k', etc.)
  • organizationId?: string - Optional organization ID for org-specific configs

Returns: AI model configuration object

createExtractionTypeConfig(formData)

Creates a new extraction type configuration. Requires admin access.

deleteExtractionTypeConfig(configId)

Deletes an extraction type configuration. Requires admin access.

getAllExtractionTypeConfigs()

Retrieves all extraction type configurations with joined AI model data. Requires admin access.

Making Hardcoded Models Editable

The system includes several hardcoded AI models that are used as fallbacks. To make these easily editable through the admin interface:

Option 1: Web-Based Sync (Recommended)

  1. Access /dashboard/ai-models with system admin password
  2. Click the "Sync System Models" button in the top-right corner
  3. This will automatically add all hardcoded models to the database

Option 2: Command Line Sync

If you have direct database access:

npm run ai:sync

What Gets Synced

The sync process adds these hardcoded models to the database:

  • Claude 3.5 Haiku (Latest) (claude-3-5-haiku-latest) - Primary default model for all document analysis
  • Claude Sonnet 4 (claude-sonnet-4-20250514) - Advanced analysis model (backup option)
  • Claude 3.7 Sonnet (claude-3-7-sonnet-20250219) - PDF processing with native support
  • Claude 3 Sonnet (claude-3-sonnet-20240229) - Permit analysis
  • Gemini 2.5 Flash Preview (gemini-2.5-flash-preview-05-20) - Alternative document processing

After Syncing

  • All models become visible and editable in the admin interface
  • Claude 3.5 Haiku is automatically set as the active fallback model
  • Default extraction type configurations are created using Claude 3.5 Haiku
  • No more hardcoded fallbacks - everything is database-driven

Setup Instructions