# External Data Source Types - Complete Specification > Comprehensive documentation of all external data source types in Wallboard, including configuration options, authentication methods, and business applications. --- ## Universal External Data Source Features ### **Error Handling & Reliability** #### **Error Counter System:** - **Automatic failure tracking**: Failed refresh attempts increment error counter - **Deactivation threshold**: Data source deactivated after 25 consecutive failures - **Manual reactivation**: Refresh button resets error counter and reactivates - **Ignore error counter**: Optional setting to disable automatic deactivation **Use Cases for Ignore Error Counter:** - **On-premise deployments**: Internal services may have planned downtime - **Intermittent services**: External APIs with known reliability issues - **Long-term outages**: Services unavailable for extended periods #### **Usage-Based Deactivation:** - **Usage tracking**: Monitor API requests and active WebSocket subscriptions - **Device activity**: Track active subscriptions from displayer/content - **30-day threshold**: Unused data sources automatically deactivated after 30 days - **Automatic reactivation**: Seamless reactivation when content references data source ### **HTTP Request Configuration** Available for all HTTP-based data sources (JSON, XML, RSS, File from URL, etc.): #### **HTTP Method Support:** - **GET**: Standard data retrieval - **POST**: Submit data with request body - **PUT**: Update/replace resource data - **PATCH**: Partial resource updates #### **Authentication Options:** - **Basic Authentication**: Username/password authentication - **Custom Headers**: API keys, Bearer tokens, custom authentication - **Header flexibility**: Support for any required HTTP headers ### **External Resource Caching System** #### **Automatic Resource Download & Proxying:** - **URL detection**: Identify media file URLs within data source JSON - **Whitelisted file types**: Download images, videos, documents referenced in data - **Internal URL replacement**: Replace external URLs with internal cached URLs - **Proxy functionality**: Serve cached resources from Wallboard servers #### **Device Connectivity Benefits:** - **Limited internet access**: Devices often only reach Wallboard servers - **Performance optimization**: Local resource serving faster than external requests - **Reliability**: Cached resources available even if original source unavailable #### **Advanced Caching Configuration:** **Remove Broken External Resource References:** - **Broken link handling**: Remove URLs that fail to download - **Data integrity**: Prevent broken references in data source JSON - **Clean data**: Ensure only valid resources remain in data **Rotate Cache on Every Update:** - **Cache management**: Remove unused resources after data source refresh - **Default behavior**: Keep resources for potential reuse across updates - **Storage optimization**: Enable for data sources with frequently changing resources **Exchange Internal Resource References:** - **Internal file referencing**: Reference Wallboard media library files by name - **Advanced feature**: Complex internal resource management - **Specialized use cases**: Internal file system integration ### **HTTP Caching & Performance Optimization** #### **Respect Cache-Control Header:** - **HTTP standard compliance**: Honor external service cache directives - **Cache-Control respect**: Follow Max-Age headers from external services - **Override refresh frequency**: Cache headers take precedence over configured refresh - **Example**: If external service sets Max-Age: 1 year, data won't refresh regardless of configured frequency #### **Conditional GET Support:** - **Efficient requests**: Use If-None-Match and If-Modified-Since headers - **ETag support**: Handle ETag-based conditional requests - **Last-Modified support**: Use Last-Modified headers for conditional requests - **Bandwidth optimization**: Avoid unnecessary downloads when data unchanged ### **Data Processing Features** #### **Randomize Arrays:** - **Post-processing step**: Applied after data retrieval and caching - **Array selector**: Target specific arrays within JSON structure using selectors - **Randomization timing**: Configure when randomization occurs - **On data change**: Only randomize when underlying data changes - **On every update**: Randomize on every refresh cycle regardless of data changes - **Use cases**: Rotating promotional content, randomized product displays, varied content presentation #### **Run Condition:** - **Conditional processing**: Execute data source refresh based on conditions - **Flexible triggers**: Configure when data source should actually update - **Resource optimization**: Prevent unnecessary processing during specific conditions --- ## Simple HTTP-Based Data Sources ### **JSON Data Source** - **Purpose**: Simple HTTP requests returning JSON data - **Process**: HTTP GET/POST → JSON response → Store as-is in database - **Use cases**: REST API integration, simple web service data - **Authentication**: Custom headers, Basic Auth, API keys ### **XML Data Source** - **Purpose**: HTTP requests returning XML data with automatic conversion - **Process**: HTTP GET/POST → XML response → Convert to equivalent JSON → Store - **Use cases**: SOAP services, legacy XML APIs, RSS-like feeds - **Authentication**: Custom headers, Basic Auth, API keys ### **File from URL** - **Purpose**: Simple file download from URL with caching - **Process**: Downloads file from URL → Caches file → Returns cached URL in data source JSON - **Data format**: JSON containing the cached file URL - **Use cases**: Media file caching, document downloads, external asset management - **Features**: Automatic caching, URL replacement with internal cached version --- ## Calendar Integration ### **Google Calendar Integration:** - **Personal accounts only**: No Google Workspace support - **Authentication**: Google OAuth credential with Calendar scope - **Simple configuration**: Credential selection + calendar picker ### **Microsoft Calendar Integration:** **Calendar Access Types:** 1. **User Calendar**: Authenticated user's personal calendar 2. **Group Calendar**: SharePoint group-based calendars - **Automatic discovery**: Dropdown selection of available SharePoint groups - **Group calendar access**: Access SharePoint group calendars after group selection 3. **User Principal Calendar**: Any user's calendar by email address - **Manual email entry**: Must specify email address for target user - **Meeting room use case**: Perfect for room booking displays (conference-room-a@company.com) - **Permission-based**: Only accessible if user has calendar viewing permissions **Enterprise Applications:** - **Meeting room solutions**: Real-time room availability and booking displays - **Resource management**: Equipment and facility scheduling - **Dedicated calendar widgets**: Specialized widgets for calendar display ### **iCal (iCalendar)** - **Purpose**: Calendar integration via iCalendar format (.ics) - **Process**: HTTP request to iCal URL → Parse iCalendar format → Convert to JSON - **Configuration**: URL + formatting options for event display - **Data format**: Calendar events converted to JSON structure - **Use cases**: Public calendars, event listings, schedule imports, meeting room displays - **Features**: Standard iCalendar parsing, recurring event support, timezone handling --- ## Spreadsheet Integration ### **Google Sheets Integration** #### **Authentication Architecture:** **Scoped Credential System**: Google credentials require specific permission scopes - **Google Sheets scope**: Access only to Google Sheets API - **Security benefit**: Minimal permission principle - only necessary access granted #### **Configuration Process:** 1. **Select Google credential**: Choose credential with Google Sheets scope 2. **Sheet selection**: Pick specific Google Sheet from accessible sheets 3. **Refresh configuration**: Set polling frequency for data updates 4. **Automatic integration**: Sheet data becomes available as JSON data source #### **Empty ranges (as of 2026-09-08):** Valid empty sheet ranges are treated as successful refreshes. They must **not** accumulate refresh failures or trip the stop-after-25-failures path. Merged to `release/2.1` — [wb-backend-core#609](https://github.com/WallboardPlatform/wb-backend-core/pull/609). #### **Advanced Resource Caching:** - **URL processing**: Automatically download files referenced by URLs in sheet cells - **Image caching**: Download and cache images linked in sheets - **Asset optimization**: Optimize cached resources for digital signage delivery #### **Business Use Cases:** **Non-Technical User Management Workflow:** ``` Content Creation (One-time): ├── Developer creates content with Google Sheets data binding ├── Content configured with sheet-based widgets └── Content deployed to devices Ongoing Management: ├── Non-technical users edit Google Sheet ├── Changes automatically sync to Wallboard ├── Digital signage content updates automatically └── No Wallboard system access required for content updates ``` **Target Applications:** - **SME solutions**: Small/medium enterprises without dedicated IT staff - **Distributed management**: Multiple locations with local content managers - **Price list management**: Store managers update pricing in familiar interface - **External vendor integration**: Third-party data providers using Google Sheets ### **Microsoft Excel** - **Authentication**: Microsoft credential with Excel scope - **Source selection**: - Personal account: Direct workbook selection - Work account: SharePoint group → Workbook selection - **Advanced capabilities**: - **Multi-workbook support**: Select multiple workbooks - **Multi-sheet support**: Select multiple sheets per workbook - **All sheets option**: Import all sheets from a workbook - **Multi-table data source**: Results in data source with multiple tables - **Range selection**: Specify exact cell ranges (e.g., A2:F3) - **Header row configuration**: Designate first row as headers - **Data format**: Tabular data with optional multi-table structure - **Use cases**: Financial reports, inventory management, complex spreadsheet integration --- ## Database Integration ### **JDBC Database Integration** - **Architecture**: Plugin-based system with unlimited database support via JDBC drivers - **Server Admin control**: Admin uploads JDBC drivers (security-sensitive operation) - **Microservice isolation**: JDBC drivers run in isolated microservice for security - **Driver detection**: Automatic driver selection based on JDBC URL format - **Configuration**: - JDBC URL (determines which driver to use) - Username/Password authentication - SQL query (returns tabular data) - **Database support**: Any database with JDBC driver (MySQL, PostgreSQL, SQL Server, Oracle, etc.) - **Data format**: Query results as tabular JSON structure - **Use cases**: Direct database integration, real-time reporting, legacy system connectivity --- ## Content Syndication ### **RSS Feeds** - **Purpose**: RSS/Atom feed integration for content syndication - **Process**: Fetch RSS XML → Convert to JSON (structure similar to original XML) - **Data format**: Feed items with titles, descriptions, dates, links in JSON - **Use cases**: News tickers, blog feeds, announcements, internal communications - **Features**: Standard RSS/Atom parsing, maintains familiar RSS structure --- ## Microsoft Ecosystem Integration ### **SharePoint Lists** - **Purpose**: Microsoft SharePoint list data integration - **Authentication**: Microsoft credential with SharePoint Lists scope - **Data format**: SharePoint list items as tabular JSON - **Special features**: - **Image handling**: Automatic caching of images in list items - **Metadata preservation**: Maintains SharePoint metadata - **Use cases**: Corporate lists, task tracking, inventory systems, employee directories - **Limitations**: Lists only (no document libraries or workflows) ### **User Active Directory** - **Purpose**: Microsoft Active Directory user information (similar to Teams user list) - **Data includes**: - User names and profiles - Status information - Profile pictures (automatically cached) - **Authentication**: Microsoft credential with User AD scope - **Data format**: User directory as JSON with profile information - **Use cases**: Employee directories, organizational charts, user status boards - **Features**: Profile image caching, status synchronization --- ## Specialized Services ### **Weather Integration** - **Purpose**: Weather data integration from meteorological services - **Location selection methods**: - City name - ZIP code - Country code - Map selection (geo-position) - **Configuration**: Display units (metric/imperial), location, update frequency - **Data format**: Current conditions, forecasts, weather alerts - **Use cases**: Digital signage weather widgets, outdoor displays, event planning ### **Screenshot Service** - **Purpose**: Capture screenshots of web pages as data source - **Infrastructure**: Headless browser on centralized screenshot service - **Configuration options**: - **URL**: Target webpage to capture - **Resolution**: Screenshot dimensions - **Wait time**: Critical - time to wait for page load completion - **Timezone**: For time-sensitive content - **Crop settings**: Define specific page regions - **Process**: Request → Queue → Headless browser render → Image generation → URL in data source - **Use cases**: Dashboard captures, web content monitoring, visual status boards - **Performance note**: Subject to queue delays, not real-time ### **Toast POS Integration** - **Purpose**: Restaurant management system integration (Toast POS) - **Industry focus**: Restaurant menu and inventory management - **Configuration**: - Location ID (from Toast system) - Group ID (from Toast system) - **Data complexity**: Extensive menu data with multiple attributes - **Pre-built format templates**: - **Menu only**: Just menu items and prices - **Menu + Stock**: Include availability status - **Restaurant info**: Basic restaurant information - **With modifiers**: Include menu modifiers and options - **Custom combinations**: Fine-grained control over data inclusion - **Data includes**: - Menu items and categories - Real-time availability (what's in stock/sold out) - Spice levels and dietary information - Modifiers and customization options - Pricing and specials - **Use cases**: Digital menu boards, QSR displays, kitchen displays, drive-through menus - **Features**: Real-time menu updates, out-of-stock notifications, modifier management ### **Amadeus Event Integration** - **Purpose**: Hospitality event listings from Amadeus Sales & Catering Delphi Event API. - **Configuration**: Amadeus cloud credential, property/location selector, upcoming day count. - **Data format**: `events` array with event/group names, room/location fields, start/end timestamps, attendance values, posted flag, and alternate labels. - **Behavior**: Fetches definite posted events for the selected location, normalizes local event times to UTC timestamps using the location timezone, and caches integration results briefly. - **Limits**: Number of days is 1-7 in UI/API validation; Amadeus requests are chunked by the integration service. - **Use cases**: Hotel lobby boards, meeting-room directories, concierge displays, venue reader boards. ### **Notion Integration** - **Configuration**: **Settings > Datasources > Notion**; select **Credentials**, then an accessible **Datasource** inside a Notion database. One Wallboard datasource imports one Notion data source. - **Data format**: `{dataSourceId: {header, rows}}`; property names become columns for ordinary table binding. Title/rich text becomes plain text; numbers and checkboxes retain their types; select/status becomes its name; dates use the start value. - **Property conversion**: People and multi-select become comma-separated text; relations become page IDs; files become URL arrays. Formula/rollup support is partial and unsupported property types become null. Validate complex properties and media separately. - **Refresh**: Read-only scheduled import of row properties. **Simple interval** defaults to daily (86,400 seconds); **Advanced schedule (cron)** also has **Time zone**. The configured customer/server minimum applies even to administrators (default minimum: 300 seconds). - **Row limit**: At most 500 rows per refresh, fetched in batches of 100. Truncation is logged by the integration service but is not signaled in the datasource UI. - **Schema caveats**: Columns are inferred from returned rows; an empty source produces empty `header` and `rows`. Property renames or selecting another source can require repairing bindings. Notion page bodies, layouts, and saved-view filters/sorts are not imported. --- ## Emergency Alert Integration ### **CAP (Common Alerting Protocol)** - **Purpose**: Emergency alert system integration with automatic device state management - **Complexity**: Most sophisticated data source type with extensive configuration - **Two operation modes**: #### **1. Poll Mode (Default)** - **Process**: Regular polling of CAP feed URL based on refresh frequency - **Feed URL**: Source of CAP alerts (XML format) - **Alert URL selector**: XML path (dot-notation) to locate individual alert URLs within feed - **Alert default expiry**: Auto-expire alerts after specified minutes if not in feed #### **2. Callback Mode** - **Process**: External system pushes CAP data via webhook - **No polling**: System waits for webhook triggers - **Use case**: Integration with systems like RaveCAP that push updates - **Webhook delivery**: External system sends CAP data to specific data source ID #### **Advanced Caching Configuration** - **Feed last updated selector**: XML path to feed's last update timestamp - **Feed item last updated selector**: XML path to individual alert's update timestamp - **Purpose**: Optimization to avoid unnecessary fetching of unchanged alerts - **Embedded elements**: CAP alerts may contain embedded items requiring separate fetching - **Smart fetching**: Only fetch items when timestamps indicate changes #### **Device Identification & Emergency State** - **Tag-based matching**: Automatically determine which devices respond to alerts - **Tag attribute selection**: Extract device tags from alert attributes (e.g., "Addresses", "Audience", "Area desc") - **Tag separator character**: Define how multiple tags are separated in alert - **Tag filter type**: "Any match" or "All match" for tag matching logic #### **Emergency State Management** - **Set device emergency state automatically**: Toggle devices to emergency mode when alert matches - **Workflow**: 1. CAP alert arrives with location/audience tags 2. System matches alert tags to device tags 3. Matching devices automatically enter emergency state 4. Emergency content (pre-assigned) becomes active 5. Emergency content uses same CAP data source to display alert details 6. Device-side logic filters relevant alerts for display #### **Ignore error counter**: Disable automatic deactivation for unreliable CAP feeds #### **Complete Emergency Alert Flow Example**: ``` CAP Alert: Tornado warning for "County-A, County-B" ├── Tag extraction: Addresses = ["County-A", "County-B"] ├── Device matching: All devices tagged with County-A or County-B ├── Emergency activation: Matched devices enter emergency state ├── Content switch: Emergency content template activates ├── Alert display: Template shows tornado warning details └── Auto-clear: When alert expires, devices return to normal ``` #### **Business Value**: - **Public safety**: Instant emergency notifications across all relevant displays - **Automatic targeting**: Geographic/demographic alert routing without manual intervention - **Compliance**: Meets emergency communication requirements - **Zero-touch operation**: Fully automated from alert to display to clear --- ## Custom Integrations ### **Other/Custom Data Sources** - **Purpose**: Custom integrations developed for specialized requirements - **Development triggers**: Complex authentication, multi-API coordination, legacy systems - **Use cases**: On-premise integrations, university systems, complex OAuth flows - **Implementation**: Project-specific development for unique business requirements **Common Use Cases:** - **On-premise integrations**: Custom authentication for internal systems - **Educational institutions**: University-specific authentication systems - **Complex OAuth flows**: Multi-step authentication processes - **Data aggregation**: Combining data from multiple APIs requiring coordinated authentication - **Legacy system integration**: Authentication for older systems without modern OAuth support --- ## Integration Patterns & Best Practices ### **Authentication Strategy:** - **Credential reuse**: Single credential for multiple data sources of same type - **Scoped permissions**: Minimal access principle for security - **Centralized management**: Cloud Credentials system for all authentication ### **Performance Optimization:** - **Appropriate refresh rates**: Match refresh frequency to data change frequency - **Resource caching**: Enable for data sources with media references - **Error handling**: Configure ignore error counter for unreliable services - **HTTP optimization**: Enable cache-control respect for bandwidth efficiency ### **Enterprise Deployment:** - **Team-based access**: Assign data sources to appropriate teams - **Usage monitoring**: Track data source utilization and performance - **Lifecycle management**: Regular review of unused data sources - **Security compliance**: Follow organizational authentication and data handling policies