# Wallboard Data Sources - Core Architecture

> Fundamental architecture principles and data source type overview for the Wallboard platform.

---

## Core Architecture Principle

**Universal JSON Storage**: All data sources, regardless of their origin or type, are ultimately stored as JSON in the database. This creates a unified data model where:
- Internal data sources → JSON storage
- External API responses → JSON storage
- System-generated data → JSON storage
- All content widgets consume the same JSON format

---

## Data Source Types Overview

### 1. **Internal Data Sources**
**Manual control and webhook-driven updates:**
- No automatic refresh mechanism
- Multi-table structure or custom JSON format
- Editable via UI, API, or sophisticated webhook actions
- Real-time distributed state management across devices
- Team-based permission system with strongly typed fields

### 2. **External Data Sources**
**Live integration with external systems:**
- Automatic refresh via polling or cron scheduling
- Read-only data access from external services
- Comprehensive authentication via Cloud Credentials system
- Advanced caching and resource management
- Error handling with automatic deactivation/reactivation

### 3. **System Data Sources**
**Platform-generated data:**
- Device information, file system metadata
- Screenshot generation service
- Auto-refresh when underlying platform data changes
- Dependency management for computed data sources

*For detailed specifications of external data source types, see: [External Data Source Types Documentation]*
*For authentication and credential management, see: [Cloud Credentials Management Documentation]*

---

## Data Source Refresh Strategies & Lifecycle

### Refresh Strategy Matrix:

| **Type** | **Refresh Method** | **Configuration** | **Use Case** |
|----------|-------------------|-------------------|--------------|
| **Internal** | Event-driven only | No refresh options | User/API controlled state |
| **External** | HTTP polling | Interval/Cron | External service integration |
| **System (Auto)** | Platform-triggered | Auto-managed | Device info, file changes |
| **System (Screenshot)** | Scheduled generation | Interval/Cron | Heavy content optimization |

### Universal Lifecycle Management:

#### **Usage-Based Deactivation (All External Types):**
- **30-day threshold**: Unused data sources automatically deactivated
- **Usage detection**: API requests + active WebSocket subscriptions from devices
- **Automatic reactivation**: When content references deactivated source → instant reactivation
- **Transparent to users**: Background performance optimization, no manual intervention

#### **Error Handling (External Only):**
- **Error counter**: 25 consecutive failures → automatic deactivation
- **Ignore error counter**: On-prem/unreliable services can disable auto-deactivation
- **Manual reactivation**: Refresh button resets counter

#### **Ad-Hoc Lifecycle (Aggressive):**
- **Days not weeks**: Much shorter deactivation period than standard
- **Automatic cleanup**: Quick removal of unused user-created resources
- **Immediate reactivation**: Seamless restoration when referenced

---

## Performance & Scalability Architecture

### Data Size Limits:
- **1 MB maximum**: Per data source JSON size (server configurable)
- **No pagination**: All data loaded into memory for widget consumption
- **Memory efficiency**: Device-side processing for optimal performance

### Real-time Synchronization:
- **WebSocket distribution**: Instant updates to all subscribed devices
- **Server arbitration**: Conflict resolution for concurrent edits
- **Network resilience**: Data sources work offline after initial load

### Enterprise Scalability:
- **Multi-tenant isolation**: Complete data separation per tenant
- **Team-based access**: Scalable permission model within tenants
- **Resource monitoring**: Usage tracking and performance optimization
- **Centralized management**: Global admin visibility with local team control

---

## Integration Architecture

### External Data Source Categories:
- **Business & Productivity**: Google Sheets, Microsoft Excel, SharePoint Lists, Calendar systems
- **Communication & Content**: RSS feeds, notification systems, file downloads
- **Technical Integration**: JSON APIs, XML services, JDBC database connections
- **Specialized Services**: Weather data, emergency alerts (CAP), directory services

### Authentication Integration:
- **Cloud Credentials system**: Centralized authentication management
- **Scoped permissions**: Minimal access principle for security
- **Multi-tenant isolation**: Credentials scoped per tenant
- **Enterprise compliance**: Support for OAuth, API keys, certificates

### Performance Optimization:
- **Intelligent caching**: External resource caching and proxy functionality
- **HTTP optimization**: Cache-Control header respect, conditional GET support
- **Error resilience**: Fallback data mechanisms and retry logic
- **Resource efficiency**: Automatic deduplication and cleanup

# Advanced Data Source Features

> Advanced capabilities of Wallboard's data source system including transformations, performance optimization, distributed state management, and revolutionary permission architecture.

---

## Data Source Permissions - Revolutionary Architecture

### Multi-Table Structure & Strongly Typed System:
- **10 different field types**: String, Number, Boolean, Text, Color, URL, Date, Time, File picker, Folder picker
- **Data validation**: Type system prevents invalid data entry based on column type
- **Media integration**: File/Folder picker types connect directly to Wallboard media library
- **UI optimization**: Different input widgets based on field type (color picker, date picker, etc.)

### Data-Centric Permission Model:

**Team-Based Data Source Permissions:**
- **View permissions**: Teams that can see the data source
- **Edit permissions**: Teams that can modify data in the data source
- **Binary access control**: Users either have EDIT or VIEW-ONLY access
- **Team inheritance**: Users inherit permissions through team membership

### Revolutionary Role Separation:

**Edit Rights = Global Edit Capability:**
```
Internal Data Source: "Restaurant Menu Prices"
  -  Team Permission: Marketing Team = EDIT
  -  Data appears everywhere:
     -  Manager Dashboard (user has no content access)
     -  Drive-through Display (user has no content access)
     -  Lobby Kiosk (user has no content access)
      - Kitchen Display (user has no content access)
  - Result: Marketing team member can edit prices on ANY screen
```

**Content Permissions Are Irrelevant for Data Editing:**
- **Data ownership transcends content boundaries**: Edit data source = edit anywhere it appears
- **No content access required**: Don't need permission to the content itself
- **Device-agnostic editing**: Walk to any screen → edit data directly
- **Unknown deployment transparency**: Data editors don't need to know where data is used

### Complete Role Separation:

**Data Managers (Domain Experts):**
- **Role**: Manage business data (prices, schedules, inventory)
- **Permissions**: EDIT rights to relevant data sources only
- **Workflow**: See data on any screen → edit directly → changes propagate everywhere
- **Knowledge**: Domain expertise only (pricing, scheduling, etc.)
- **System knowledge**: Zero - no need to understand content/devices/deployments

**Content Creators (Technical/Design):**
- **Role**: Design presentations and bind data sources to content
- **Permissions**: Content creation rights, typically VIEW-ONLY on data sources
- **Workflow**: Create content → select data sources → deploy to devices
- **Knowledge**: Design skills, technical content creation
- **Data management**: Cannot modify actual business data

### Real-World Example - Restaurant Chain:

```
Data Source: "Daily Specials" (Kitchen Manager = EDIT)
Content everywhere:
  -  Lobby Menu Boards (IT manages)
  -  Drive-through Displays (IT manages)  
  -  Kitchen Display System (IT manages)
  -  Mobile App (App team manages)
  - Website Menu (Marketing manages)

Kitchen Manager:
  -  Walks to ANY screen showing daily specials
  -  Clicks "Fish Special $18.99" → inline editor
  -  Changes to "$16.99" → saves
  -  ALL locations update instantly
  - No IT/marketing/app team coordination needed
```

**Key Principle**: If you have EDIT rights to a data source, you can edit that data on any screen showing it, regardless of content permissions or who created the content.

---

## Data Source Transformation Architecture

> Post-processing transformations applied to datasource data before consumption.

---

### Universal Post-Processing Transformers (Version 2.1+)

**Architecture:** Datasource fetch/update → Transformer 1 → Transformer 2 → ... → Final JSON

**Execution:** Server-side, sequential pipeline, applies to all datasource types (Internal, External, System).

**Transformer Types:**

| Type | Purpose | Configuration |
|------|---------|---------------|
| **JSON query** (JSONata) | Complex transformation, filtering, aggregation, restructuring | Query expression (textarea) |
| **Randomize arrays** | Shuffle array order | Array selectors (dot-notation paths) |

**Common Configuration:**

| Field | Options | Behavior |
|-------|---------|----------|
| **Run condition** | Every update, On data change | When transformer executes |
| **Execution order** | 1, 2, 3... | Sequential processing |

**Chaining Behavior:**
- Sequential execution (T1 output → T2 input → T3 input)
- **Output handling:** Each transformer overwrites entire datasource JSON
- **On data change:** Hash-based comparison, transformer skipped if no change
- **Chain independence:** If T1 skips (no change) + T2 "Every update" → T2 runs with unchanged input
- **Performance:** 5 second timeout per transformer

**JSONata Transformer:**

JSONata query language for JSON transformation. XPath-like syntax for complex data manipulation.

**Use cases:**
```jsonata
# Filtering
items[price < 100 and inStock = true]

# Mapping/Transform
{
  "products": items.{
    "name": title,
    "cost": price * 1.1,
    "available": inStock
  }
}

# Aggregation
{
  "total": $sum(items.price),
  "average": $average(items.price),
  "count": $count(items)
}

# Nested extraction
categories.products[featured = true].{
  "id": productId,
  "category": $parent.name
}
```

**Randomize Arrays Transformer:**

Shuffle array elements for random order display.

**Configuration:** Array selectors (dot-notation, multiple paths)
- Example: `items`, `data.products`, `categories.items`
- Multiple arrays can be randomized in single transformer
- **Run condition:** Every update = shuffle each refresh, On data change = shuffle only when data changes

---

### Content-Level Transformers (Data Source Mapper)

**JavaScript-Based Processing:** Custom transformation functions within content editor.

**Architecture:** Content base data path or one or more connected datasources -> JavaScript function -> property path, mock datasource, or internal datasource output

**Execution:** Device-side, real-time during content rendering.

**Core Functionality:**
- **Input:** Content base data path, one datasource, or at least two selected datasources for Multiple Datasources
- **Processing:** Custom JavaScript with full execution environment
- **Output:** Result written to a property path in the selected datasource, a selected mock datasource, or an internal datasource using SET (replace) or MERGE; changed results are published to bound widgets
- **Built-in templates:** Calendar processing, filtering, emergency alert processing (copy-based, user-modifiable)

**Security & Performance:**
- **Editor restrictions:** Custom JS blocked in logged-in sessions (security)
- **Device execution:** Full JavaScript capabilities on device (user trusts own content)
- **Performance isolation:** Crashes/infinite loops affect only that device
- **No debugging:** Difficult to troubleshoot custom transformations

**Advanced Features:**
- **Recalculate periodically:** Automatic refresh of transformations
- **Chained transformations:** Multiple calculators on same data source
- **Dynamic business logic:** Real-time calculations (KPIs, filtering)

---

## Screenshot Service - Performance Optimization Strategy

**Cloud-Based Image Generation**: Screenshots serve as performance optimization for heavy content delivery.

### Infrastructure Reality:
- **Wallboard's own cloud service**: Dedicated screenshot rendering infrastructure
- **Queue-based processing**: Sequential rendering, not parallel
- **Bandwidth limitations**: Finite cloud capacity affects processing speed
- **Queue depth impact**: More requests = longer processing times

### Practical Limitations:
- **Hourly/daily refresh recommended**: Queue delays make frequent updates impractical
- **Not real-time**: Due to queue processing and bandwidth constraints
- **Resource planning**: High-frequency requests can overwhelm service

### Revolutionary Use Cases:

#### **Heavy Content → Lightweight Delivery:**
- **PowerBI widgets**: Crash old devices → Screenshot alternative works everywhere
- **Complex playlists**: 50 heavy slides → Screenshot gallery instead
- **Server-side rendering**: Complex computation moved from device to cloud
- **Performance guarantee**: Screenshots work on ANY device capability

#### **E-Ink Display Perfect Match:**
- **Static technology + dynamic data**: E-Ink shows only images, but screenshots contain live data
- **Low refresh alignment**: E-Ink's infrequent updates match screenshot service limitations
- **Battery optimization**: Perfect for solar-powered outdoor displays
- **Content versatility**: Any Wallboard content → E-Ink compatible

#### **Hybrid Architecture:**
- **Screenshot backgrounds**: Heavy dashboard as background image
- **Live overlays**: Real-time widgets layered on top
- **Performance balance**: Visual richness + guaranteed compatibility
- **Seamless integration**: Screenshot URLs update → widgets auto-refresh

---

## Internal Data Source - Distributed Operational Memory

**Real-time Shared State**: Internal data sources = persistent, distributed memory across all devices.

### Architecture:
- **Event-driven updates**: Changes via API, webhook, or UI trigger instant distribution
- **WebSocket propagation**: Sub-second updates to all subscribed devices
- **Persistent state**: Survives device restarts and network interruptions
- **Conflict resolution**: Server-side sequential processing prevents race conditions

### Interactive Control Patterns:

**Scoreboard Example:**
```
Mobile Control Device:
  -  Touch interface triggers API calls
  -  Updates "Live Scoreboard" data source
└── Changes propagate instantly

Stadium Displays:
  -  Subscribe to same data source
  -  Auto-refresh when data changes
  -  No direct user interaction
└── Pure display consumption
```

### Enterprise Use Cases:
- **Queue management**: Staff tablet updates → Customer displays update
- **Inventory control**: Warehouse updates → Store displays update
- **Event coordination**: Mobile control → Venue screens update
- **Emergency alerts**: Control room → All facility displays update

### Business Value:
- **Operational memory**: One device changes state, all devices see update
- **Role separation**: Control devices vs display devices
- **Real-time coordination**: No polling, immediate state synchronization

### Technical Benefits:
- **Cross-device interaction**: One device can control content on multiple other devices
- **Instant propagation**: Changes distributed to all subscribed devices immediately
- **Network resilience**: Data persists through connectivity issues
- **Scalable architecture**: Supports hundreds of devices sharing same data source

### Shared Internal Datasource Editor (Version 2.1+)

**Purpose:** Share a browser link to a table-type internal datasource so non-Wallboard users can view or update controlled table data without a Wallboard account.

**Access Control:**
- **Share source**: Internal Datasources -> Actions -> Copy shared editor URL
- **API key selection**: Link uses the selected API key token; the key's team access list limits which datasources can be opened
- **Team scope**: Empty API key team access list = customer-wide datasource access; otherwise only datasources assigned to listed teams
- **UI/backend scope mapping**: UI shows "Manage internal datasource"; backend separates datasource data read from internal datasource write
- **Read-only link**: API key has internal datasource read access only; shared editor opens but cannot save changes
- **Read-write link**: API key has internal datasource read + write access; shared editor can save table data changes
- **Inactive/restricted key**: Shared link is blocked

**Shared Editor:**
- **Supported**: Table-type internal datasources only
- **Read-only mode**: View/search data; cells and row actions are disabled
- **Read-write mode**: Edit unlocked cells, use configured dropdown values, add/insert/duplicate/remove unlocked rows, save table data
- **Desktop view**: Shared route can use the table editor in shared mode and allows switching to mobile view
- **Mobile/card view**: Phone-sized shared editor uses row cards/forms with search, refresh, save for write links, row add/remove where allowed, and desktop-view toggle
- **Mobile field support**: String/number/boolean/text/color/URL/date/time/dropdown and file/folder picker cells are editable with write link + unlocked row; scheduling status is visible on mobile, while scheduling edits use the desktop editor
- **Restricted**: No Wallboard navigation, datasource rename, column/schema changes, dropdown configuration changes, lock/unlock changes, or drag reorder
- **Locks**: Locked rows/cells remain protected in shared editor

### Dropdown Manager & Row Locking (Version 2.1+)

**Dropdown Manager:**
- **Datasource-wide definitions**: Named fixed value lists can be reused by multiple columns
- **Column assignment**: Table connectors map a definition ID to one or more column keys
- **Internal only**: Definitions are stored as stringified JSON in datasource config (no external source)
- **Enforcement**: Freetext disabled, prevents invalid data entry
- **Shared editor**: Values usable, cannot modify dropdown config

**Row Locking:**
- **Lock/unlock**: Row actions menu in normal datasource editor
- **No role control**: Any EDIT permission user can lock/unlock
- **Shared editor**: Respects locked rows but cannot lock/unlock
- **Use case**: Protect template rows, recurring events, reference data

**Example:**
```
Customer Service Queue:
Row 1: "VIP Template" (LOCKED, dropdown: High priority)
Row 2: "Standard Template" (LOCKED, dropdown: Normal)
Rows 3+: Active queue (UNLOCKED, staff can edit/delete)
```

---

## Advanced Integration Patterns

### Multi-Device Coordination:
- **Control interfaces**: Dedicated control devices (tablets, phones)
- **Display devices**: Pure consumption devices (large screens, kiosks)
- **Hybrid devices**: Can both control and display (interactive kiosks)

### Real-Time Business Operations:
- **Dynamic content updates**: Content changes based on business state
- **Live data synchronization**: Instant updates across all touchpoints
- **Interactive experiences**: User actions affect multiple displays simultaneously
- **Operational efficiency**: Eliminate manual content update processes

### Performance Optimization Strategies:
- **Client-side processing**: JavaScript transformations on devices
- **Server-side rendering**: Screenshot generation for complex content
- **Hybrid approaches**: Combine real-time updates with performance optimization
- **Resource management**: Automatic cleanup and optimization

### Permission Architecture Benefits:
- **No bottlenecks**: Data updates don't require technical team involvement
- **Real-time agility**: Changes happen when business needs change
- **Domain expertise**: Business experts manage data without intermediaries
- **Clean separation**: Data ownership completely separate from content management

# Data Source Management Systems

> User-friendly data source creation and management features including ad-hoc creation and global resource sharing.

---

## Ad-Hoc Data Sources - User Experience Revolution

### Zero-Configuration Philosophy:
**Traditional**: Admin → Datasources → Complex Configuration → Content Editor
**Ad-Hoc**: Quick Editor → Minimal Input → Automatic Creation

### User Workflow Transformation:
```
Quick Editor → Template Selection → Data Source Configuration → Automatic Creation
```

1. **Open Quick Editor** on template content
2. **Select data source type** directly in widget (RSS, Weather, etc.)
3. **Enter minimal information** (URL for RSS, location for Weather)
4. **System automatically creates** optimally configured data source

Quick Editor ad-hoc creation offers compatible external datasource types only; it cannot create an internal TABLE datasource.

### Intelligent Deduplication:
**Global Resource Sharing**: System automatically consolidates identical requests
- **Weather example**: Multiple users requesting same location → single shared data source
- **RSS example**: Same RSS feed URL → consolidated automatically
- **Efficiency**: Reduces external API calls and system resource usage

**Tenant Isolation**: Credential-based data sources remain tenant-specific
- **Calendar data**: Each tenant's calendar data sources remain separate
- **SharePoint lists**: No cross-tenant sharing for authenticated resources
- **Security boundary**: Tenant data isolation maintained

### Automatic Configuration:
- **Optimal defaults**: System applies best-practice settings automatically
- **Performance optimized**: Refresh rates and caching configured for typical use cases
- **Zero technical knowledge**: Users don't need to understand data source internals

### Lifecycle Management:
- **Aggressive deactivation**: Deactivated within days (not 30 days like standard)
- **Performance optimization**: Quick cleanup of unused ad-hoc resources
- **Automatic reactivation**: Immediate reactivation when referenced by content
- **User transparency**: Users never see deactivation/reactivation process

### Administrative Visibility:

**User Perspective:**
- **Completely hidden**: Ad-hoc data sources not visible in user interfaces
- **Seamless experience**: Users work with data without seeing underlying infrastructure
- **Simplified workflow**: Focus on content creation, not data source management

**Administrator Perspective:**
- **Global visibility**: Admins can see all ad-hoc data sources across tenants
- **Usage tracking**: Monitor which users create which data sources
- **Resource management**: Understand system resource utilization
- **Debug capability**: Troubleshoot data source issues across organization

### Business Benefits:
- **Template-based content**: Non-technical users manage content independently
- **Resource efficiency**: Automatic deduplication reduces external API calls
- **Rapid deployment**: Content created and deployed in minutes
- **User experience**: Complex infrastructure hidden behind simple interface

---

## Global Data Sources - Cross-Tenant Resource Sharing

### Concept & Administration:
Global data sources are **server-wide shared resources** created exclusively by Super Administrators, accessible across all tenants.

#### Super Admin Exclusive:
- **Creation privilege**: Only Super Admins can create global data sources
- **Management control**: Only Super Admins can modify or delete global resources
- **Server-wide scope**: Available to all tenants without individual configuration

#### Tenant Access:
- **Read-only usage**: All tenants can use global data sources in their content
- **No modification rights**: Tenants cannot alter global data source configuration
- **Seamless integration**: Global data sources appear alongside tenant-specific ones

### Primary Use Cases:

#### Demo & Testing Resources:
- **News feeds**: CNN, BBC, Reuters RSS feeds for content demonstration
- **Sample data**: Pre-configured data sources for testing and proof-of-concept
- **Training materials**: Educational content for user onboarding
- **Template support**: Data sources that work with demo templates

#### Common Public Resources:
- **Weather services**: Popular location weather data for multiple tenants
- **News aggregation**: Major news outlets' RSS feeds
- **Public APIs**: Commonly used public data sources
- **Industry-standard feeds**: Widely applicable data sources across industries

### Business Benefits:

#### Rapid Deployment:
- **Zero configuration**: New tenants can immediately access useful data sources
- **Quick demonstrations**: Sales teams can show functioning content immediately
- **Reduced onboarding time**: New customers don't need to find/configure common data sources
- **Template readiness**: Demo templates work immediately with pre-configured data

#### Resource Efficiency:
- **Shared infrastructure**: Single data source serves multiple tenants
- **Reduced API calls**: Consolidated requests to external services
- **Server optimization**: Centralized management of common resources
- **Cost reduction**: Shared external service costs across tenants

### Technical Implementation:

#### Data Source Hierarchy:
```
Data Source Selection Interface:
  -  Tenant-Specific Data Sources (full control)
  -  Ad-Hoc Data Sources (user-created, hidden)
└── Global Data Sources (Super Admin managed, read-only)
```

#### Isolation & Security:
- **Data isolation**: Each tenant's usage tracked separately
- **No cross-tenant visibility**: Tenants cannot see each other's usage
- **Configuration protection**: Global settings cannot be modified by tenants
- **Usage analytics**: Super Admins can monitor global data source utilization

### Operational Characteristics:

#### Limited Usage:
- **Niche feature**: Not frequently used in production environments
- **Demo-focused**: Primarily for demonstration and testing scenarios
- **Specialized deployment**: Mainly used in hosted/demo environments

#### Management Overhead:
- **Super Admin responsibility**: Requires Super Admin involvement for changes
- **Global impact**: Changes affect all tenants using the resource
- **Careful curation**: Limited set of high-value, broadly applicable resources

### Example Global Data Sources:

#### News & Media:
- **CNN RSS Feed**: International news for demo content
- **BBC News Feed**: Alternative news source for testing
- **Reuters Business**: Financial news for corporate demos

#### Reference Data:
- **Major city weather**: New York, London, Tokyo weather for location demos
- **Currency exchange rates**: Financial data for business applications
- **Stock market indices**: Market data for dashboard demonstrations

#### Integration Examples:
- **Template galleries**: Demo templates automatically work with global data
- **Training environments**: Educational content with pre-populated data
- **Sales demonstrations**: Immediate working examples without customer data setup

---

## Data Source Management Best Practices

### User Experience Design:
- **Progressive complexity**: Start simple (ad-hoc), advance to complex (admin-managed)
- **Role-appropriate interfaces**: Different UI complexity for different user types
- **Invisible infrastructure**: Technical complexity hidden from end users

### Resource Optimization:
- **Intelligent deduplication**: Automatic consolidation of identical resources
- **Lifecycle management**: Automatic cleanup of unused resources
- **Performance monitoring**: Track usage and optimize accordingly

### Enterprise Deployment:
- **Global resources**: Shared demo/training data across organization
- **Tenant isolation**: Private data remains completely separate
- **Administrative oversight**: Super Admin visibility into all resource usage