# AI-Powered Data Sources (Version 2.1+)

> AI-driven data extraction and transformation using natural language prompts.

---

## Datasource Types

| Type | Location | Purpose | Architecture |
|------|----------|---------|--------------|
| **Web Scraper** | External | Extract data from websites | URL → Headless browser → JS render → AI → JSON |
| **AI Datasource** | System | Transform/aggregate datasources | 0+ Inputs → AI → JSON |

### Web Scraper (External)

**Key capability:** Handles JS-heavy sites (React, Vue, Angular).

**Configuration:**

| Field | Required | Description |
|-------|----------|-------------|
| URL | Yes | Target webpage |
| Wait time (sec) | Yes | JS render completion time (default: 5s) |
| Prompt | Yes | Extraction instructions |
| Output format | No | AI decide / Example JSON / JSON schema |

**Prompt examples:** "Extract product names, prices, availability" / "Get event title, date, location from upcoming events" / "Find contact: name, email, phone"

### AI Datasource (System)

**Key capability:** Multi-datasource intelligence + built-in internet tools.

**Configuration:**

| Field | Required | Description |
|-------|----------|-------------|
| Source datasources | No | 0+ input datasources |
| Prompt | Yes | Processing instructions |
| Output format | No | AI decide / Example JSON / JSON schema |

**Built-in tools (0 inputs):** Web search, X (Twitter) search, Image analysis, Google Maps

**Prompt examples:** "Combine weather + calendar. For each event, include weather" / "Analyze sales + inventory. List low stock + high demand" / "Search Twitter for brand mentions (24h)"

---

## Output Formats

| Format | Behavior | Use Case |
|--------|----------|----------|
| **AI decide** | AI determines structure | Exploratory, flexible |
| **Example JSON** | Match user sample | Specific format required |
| **JSON Schema** | Follow schema (best-effort) | Structured output |

**Schema presets:** Feed (RSS-like), Calendar, Multi table, Custom

**Validation:** No strict validation, guidance only.

---

## Rate Limits & API Keys

### API Key Resolution

Priority order — first non-empty wins:
1. **Customer key** — per-customer entity fields (`aiXaiApiKey` / `aiGeminiApiKey`), write-only
2. **System key** — server admin global key
3. **Central key** — platform built-in key (fallback)

### Rate Limiting

Limits depend on which key is in use:

| Key Source | Rate Limits |
|------------|-------------|
| Customer key | **None** — unlimited usage |
| System key | Configurable by server admin |
| Central key | Fixed — protects shared platform infrastructure |

**Design target:** Each customer should be able to comfortably run frequent AI datasource refreshes throughout the day (~$1/month worth of AI usage at full utilization). Both image generation and text AI count toward the budget.

**Behavior:** Exceeding limits → hard block (refresh fails with error). Customers can upgrade to their own API key for unlimited usage.

### Logging

Every AI datasource and web scraper refresh is logged (visible in UI):
- **Success:** tokens used, model, duration, tool calls
- **Blocked:** rate limit warning
- **Failed:** error details

**Refresh:** Manual or scheduled only (no auto-refresh on input changes)

---

## Use Cases

**Web Scraper:** E-commerce pricing/reviews, news aggregation, real estate listings, social content, research data  
**AI Datasource:** Multi-source fusion (weather+events, sales+inventory), translation, pattern detection, web enrichment, data normalization

---

## Best Practices

**Prompts:** Be specific, structure requests, provide context  
**Performance:** Adjust wait time (simple sites 2-3s, JS-heavy 5-10s, dynamic 15+s)  
**Output consistency:** Use Example JSON/Schema for predictable structure

---

## Integration

**Template + Web Scraper:** Single template, multiple URLs (quick-edit per location)  
**Multi-source fusion:** Combine multiple datasources via AI processing  
**AI datasource chaining:** AI Datasource A → AI Datasource B (A's output as B's input)  
**Hybrid approach:** Real-time search + cached data

---

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Empty results | Increase wait time, verify URL, simplify prompt |
| Inconsistent output | Use Example JSON/Schema, more specific prompt |
| Rate limit | Strategic scheduling, combine extractions |
| Invalid JSON | Clarify prompt, simplify structure |