# Wallboard Backend Infrastructure Documentation

## 🏗️ Deployment Architecture

### Version 2.1+: Kubernetes Only

Wallboard deploys via **Helm charts** on Kubernetes. Single-node deployments use **k3s** or **microk8s**.
Helm charts hosted in **private Docker Hub repository**.

---

## 🎯 Kubernetes Deployment Models (Version 2.1+)

| Model | K8s Expertise | Infrastructure | ArgoCD & Updates | Monitoring | Secret Mgmt | Backup |
|-------|---------------|----------------|------------------|------------|-------------|--------|
| **1: Self-Managed** | High | Customer cluster | Manual (customer choice) | Customer deploys | Customer choice | Customer choice |
| **2: On-Prem Limited** | Low-Medium | Single node VM (k3s/microk8s) | Manual `helm upgrade` | Separate wallboard-monitoring chart | Auto-generated K8s secrets | Helm chart backup CronJobs (if enabled), VM/host snapshots customer responsibility |
| **3: Managed GitOps** | Low | Single node VM (k3s/microk8s) | Automatic (ArgoCD in customer cluster, WB Git) | Separate wallboard-monitoring chart | Auto-generated or external | Helm chart backup CronJobs (if enabled), VM/host snapshots customer responsibility |
| **4: WB SaaS** | None | WB-managed (DigitalOcean primary, AWS planned) | Automatic (WB-managed ArgoCD) | WB-managed | External Secrets + Azure KeyVault | Disaster Recovery is WB responsibility |
| **5: Air-Gapped** | Low-Medium | Single node VM (k3s/microk8s) | Manual offline packages | Separate wallboard-monitoring chart | Auto-generated K8s secrets | Helm chart backup CronJobs (if enabled), VM/host snapshots customer responsibility |

### Model Details

**Model 1 - Self-Managed:**
Customer has existing cluster and expertise. Wallboard provides Helm chart, documentation. Customer manages all operations.

**Model 2 - On-Prem Limited K8s:**
Single node VM (k3s/microk8s). Customer has limited K8s knowledge. Wallboard provides installation assistance (optional), documentation for setup. Monitoring via separate wallboard-monitoring chart. No ArgoCD. Customer executes `helm upgrade` with WB support. Backup via Helm chart CronJobs (if enabled); VM/host snapshots are customer responsibility. Version control via changelogs (no prescriptive guidance).

**Model 3 - Managed GitOps:**
Single node VM (k3s/microk8s). Customer has limited K8s expertise. Wallboard deploys ArgoCD (in customer cluster, connects to WB Git), separate wallboard-monitoring chart, ingress controller (Contour recommended). Automatic updates via GitOps. Backup via Helm chart CronJobs (if enabled); VM/host snapshots are customer responsibility.

**Model 4 - Wallboard SaaS:**
Fully managed service. Three isolation options:
- **Shared multi-tenant cluster** (default): Namespace-based isolation, cost-optimized
- **Isolated namespace model**: Dedicated namespace within shared cluster with separate pods, services, and persistent volumes. Provides full runtime process isolation and data separation at namespace level while leveraging shared infrastructure. Balance between isolation and operational efficiency.
- **Dedicated cluster** (on request): Full cluster isolation for stricter compliance. Entire system provisioned exclusively including compute, storage, databases, and network layer.

Node pools: shared (default) or dedicated (larger customers). Infrastructure: DigitalOcean (current primary), AWS (planned). Minimum one large cluster per region. All operations WB-managed. Disaster Recovery is WB responsibility. Architecture is multi-tenant whether 1 or 100 customers.

**Model 5 - Air-Gapped Offline:**
Single node VM (k3s/microk8s), completely offline. All dependencies pre-downloaded (images, charts, packages). Monitoring via separate wallboard-monitoring chart. Backup via Helm chart CronJobs (if enabled); VM/host snapshots are customer responsibility. No WB operational support after installation. Customer responsible for all management and manual updates.

---

## 🗄️ Database Deployment (Version 2.1+)

### Managed vs In-Cluster

| Database | Managed Service Availability | In-Cluster Sub-Chart |
|----------|------------------------------|----------------------|
| MySQL | All major providers | Yes (WB-maintained) |
| Redis | All major providers | Yes (WB-maintained) |
| TimescaleDB | AWS only (official) | Yes (WB-maintained) |
| Blob Storage | All (S3-compatible) | Yes (MinIO, WB-maintained) |

**Sub-Chart Reason:** Bitnami image restrictions → All database sub-charts are WB-maintained.

**Managed Databases (HA Preferred):**
- Automatic failover, replication, backups
- Provider SLAs, automatic minor updates
- Encryption: AES-256 (provider-managed)

**In-Cluster Databases:**
- Not HA by default (single instance), not auto-scaled
- Can pin to dedicated nodes for isolation
- Encryption: Relies on encrypted block storage volumes provided by infrastructure (AES-256 volume-level encryption)

---

## 💾 Backup (Version 2.1+)

**Managed Databases:** Cloud provider handles backups, retention, point-in-time recovery.

**In-Cluster Databases:**
- Wallboard-developed backup containers (included in Helm chart, enable via configuration)
- K8s CronJob triggers backup → database dump → rsnapshot storage (hard-linked snapshots) → PersistentVolume
- Single-node: hostPath PV (customer must add VM snapshots/external backup)
- Multi-node: PVC with storage class
- Manual restore (WB documentation provided)

**Deployment Model Specifics:**
- **Models 2, 3, 5:** Helm chart includes backup CronJobs (if enabled). VM/host snapshots are customer responsibility.
- **Model 4 (WB SaaS):** Disaster Recovery is Wallboard responsibility.
- **Model 1:** Customer choice (customer manages all backup strategy).

---

## 📈 Scaling & High Availability (Version 2.1+)

**HPA (Horizontal Pod Autoscaling):**
- Configurable via Helm chart
- Default: 80% CPU/memory, 3-5 replicas, **disabled by default**
- Enable only for **2000+ device deployments**

**HA Strategy:**
- Services: Multiple replica pods, rolling updates, auto-rescheduling
- Databases: Managed DBs preferred (failover + replication); in-cluster not HA

**Cluster Size Recommendations:**

Single-Node:
| Devices | CPU | RAM | Storage |
|---------|-----|-----|---------|
| 1-500 | 4 core | 16GB | 250GB SSD + additional |
| 500+ | 8+ core | 32GB | 250GB SSD + additional |

Multi-Node (HA):
- Minimum: 2 nodes (basic HA)
- Recommended: 3 nodes (allows maintenance)

**Version Policy:**
- **Major versions (2.0, 2.1, 2.2...):** Contain breaking changes or risky data migrations (e.g., blob storage migration in 2.1). Always require manual update coordination with Wallboard for data safety. **Cannot skip major versions** - must update sequentially (2.0→2.1→2.2).
- **Minor versions (2.1.0, 2.1.1, 2.1.2...):** Can be automated (ArgoCD in Model 3/4). Helm chart version increments with any component update.

**Architecture & Migration Behavior:**
- **Master (monolith) + MySQL:** Primary migration challenge. Master runs MySQL migrations on startup. Deployment strategy: **Recreate** (not RollingUpdate) - only one instance runs during update (prevents concurrent migration conflicts). After update completes, multiple replicas can run.
- **Microservices:** 99% backward compatible within major version. Most microservices have independent databases with independent migrations. Deployment strategy: **RollingUpdate** - zero-downtime updates (multiple versions can run simultaneously during rollout).
- **MySQL migrations:** 90% backward compatible (typically additive: ADD COLUMN, not DROP). Breaking changes exist but are exceptions.

**Minor Version Update Strategy:**
- **Cloud (sequential deployment):** Minor versions deployed one-by-one (2.1.0→2.1.1→2.1.2). Breaking schema changes use **two-phase migration pattern**:
    - Phase 1: ADD new_col (backward compatible)
    - Deploy and verify
    - Phase 2: DROP old_col (cleanup after all pods updated)
    - Rolling update achievable, most migrations zero-downtime
- **On-prem (large minor jump):** Customers may skip 20+ minor versions (2.1.0→2.1.20). All migrations bundled into single deployment (migrations #1-#20 run sequentially on startup), preventing two-phase pattern separation. **Requires downtime** (stop all → migrate all sequentially → start all).

**Trade-off:** Cloud sequential deployment enables two-phase migration pattern for zero-downtime. On-prem large jumps bundle migrations, eliminating wait phase between additive and cleanup migrations.

---

## 🔐 Security Architecture (Version 2.1+)

### Secret Management
- **WB SaaS:** External Secrets Operator + Azure Key Vault
- **On-prem Model 1:** Customer choice (HashiCorp Vault, AWS Secrets Manager, etc.)
- **On-prem Models 2,3,5:** Auto-generated K8s secrets (default)

### Network Policies
- **Namespace isolation:** Enabled by default (blocks cross-namespace traffic)
- **Pod-to-pod:** Currently unrestricted within namespace (future enhancement if needed)

### Service Mesh (Linkerd)
- **WB SaaS:** Enabled by default (mTLS between services)
- **On-prem:** Optional (Model 1 implements own; Models 2/5 only if requested; single-node provides minimal benefit)
- **Istio:** Customers can deploy if preferred (WB doesn't provide but not blocked)

### Ingress Controller
- Any standard ingress controller supported
- **Wallboard preference:** Contour
- Simple architecture: All traffic → internal proxy → backend services

### TLS Configuration
- Supported: TLS 1.2, TLS 1.3
- Cipher suites: ECDHE-ECDSA/RSA-AES128/256-GCM-SHA256/384, CHACHA20-POLY1305
- Forward secrecy enabled

---

## 🏢 Multi-Tenant Architecture

Applies to all deployment models.

**Namespace Isolation (Kubernetes):**
- Each tenant in dedicated namespace
- NetworkPolicies enforce namespace-level isolation
- Resource quotas per namespace
- **Optional isolated namespace model**: For customers with separation requirements but cost-sensitive constraints. Dedicated namespace within shared cluster with separate pods, services, and persistent volumes. Ensures full runtime process isolation and data separation at namespace level.

**Database-Level:**
- Customer records in shared database tables
- Each record tagged with `customerId`
- Strict logical separation at application/database layer
- No separate physical databases per customer (multi-tenant)
- Private cloud model: dedicated cluster with separate databases/services for stricter compliance

---

## 🗂️ Object Storage

All user-uploaded media and generated content stored in object storage:

**S3-Compatible Cloud Storage:**
- AES-256 encryption at rest
- Providers: AWS S3, DigitalOcean Spaces, Azure Blob, GCS, any S3-compatible storage

**In-Cluster MinIO (Optional):**
- Available for on-prem/isolated deployments
- S3-compatible object storage running in-cluster
- Provides blob storage without external dependencies

---

## 🧮 Process and Container Separation

### Kubernetes (Version 2.1+)
- Containerized microservices in separate pods
- Components: web server pods, backend API pods, worker service pods, optional DB pods
- No traditional server segregation (separate VMs per component)
- Logical isolation at pod/orchestration level
- NetworkPolicies for security boundaries
- Resource requests/limits per pod
- Node affinity/anti-affinity for placement
- **External dependencies** (databases, object storage, message brokers) provisioned via cloud-managed services (e.g. AWS RDS, GCP Cloud SQL, MongoDB Atlas) on provider infrastructure with stronger SLAs, HA, and backup options
- **Self-managed/on-prem**: Databases deployed as separate containers with volume isolation and service-level network policies for segregation

## ☁️ Hosting Infrastructure

### Wallboard-Managed SaaS (Model 4)
- **Primary Platform:** DigitalOcean (current)
    - DigitalOcean Kubernetes (DOKS)
    - Regional deployments (min 1 large cluster/region)
- **Planned:** AWS (future expansion)

### Customer-Managed (Models 1,2,3,5)
- Any standards-compliant Kubernetes distribution
- Supported: DigitalOcean DOKS, AWS EKS, Azure AKS, GCP GKE, on-prem K8s
- Provider-agnostic