feat: add comprehensive documentation for Pixel Streaming setup, deployment, and integration with Unreal Engine and React

Author: Schlomoh

docs/DEPLOYMENT.md

@@ -0,0 +1,69 @@
+# Deployment & Scaling: Pixel Streaming with Docker & Cloud
+
+## 1. Docker Compose Example
+
+```yaml
+version: "3.8"
+services:
+  ue_streamer:
+    image: myproject-pixelstream:latest
+    runtime: nvidia
+    network_mode: host
+    environment:
+      - NVIDIA_DRIVER_CAPABILITIES=all
+
+  signaling:
+    image: ghcr.io/epicgames/pixel-streaming-signalling-server:5.6
+    network_mode: host
+
+  turn:
+    image: coturn/coturn:latest
+    command: ["turnserver", "-n", "--log-file=stdout"]
+    ports:
+      - "3478:3478/udp"
+      - "3478:3478/tcp"
+```
+
+> Note: Use network_mode: host on Linux for lowest latency. On Windows, map ports manually.
+
+## 2. Cloud Hosting
+
+- **AWS**: EC2 GPU (G4/G5), ECS/EKS for containers, Auto Scaling Group for orchestration.
+- **Azure**: NV-series VMs, VM Scale Sets, Azure Pixel Streaming Marketplace Template.
+- **Google Cloud**: GPU instances, GKE for containers.
+
+## 3. Staging vs Production
+
+- **Staging:** Single instance, protected by simple auth or VPN. Use for testing/QA.
+
+- **Production:**
+  - Use autoscaling (ASG/Scale Sets) for demand-based scaling.
+  - Implement a matchmaker or routing backend to assign users to sessions.
+  - Terminate idle sessions after timeout to control costs.
+  - Consider GPU quotas and encoder limits per node.
+
+## 4. Authentication & Security
+
+- Protect signaling server with HTTPS and valid TLS certs.
+- Gate frontend via web auth (OAuth, SSO, etc).
+- Optionally, require API token for WebSocket upgrade or use reverse proxy for access control.
+- Secure TURN/STUN credentials.
+
+## 5. Scaling Scenarios
+
+- 1:1 Streaming: One UE instance per user/session for max interactivity.
+- SFU-based: One UE instance, SFU forwards to many viewers (spectators only, single controller).
+- Use horizontal scaling for high concurrency.
+
+## 6. Best Practices
+
+- Use health checks to detect and restart failed containers.
+- Monitor GPU/CPU usage for scaling triggers.
+- Regularly update all images (UE, signaling, TURN) for security.
+- Log all connections, user actions, and errors for traceability.
+
+## 7. Troubleshooting
+
+- If video is black, check GPU drivers and encoding settings.
+- If clients can't connect, check port forwarding, signaling logs, and TURN config.
+- For high latency, prefer host networking and regional cloud instances close to users.

docs/DEVELOPMENT_WORKFLOW.md

@@ -0,0 +1,70 @@
+# Development Workflow for UE + React Pixel Streaming
+
+## 1. Prerequisites
+
+- Unreal Engine 5.6 (with Pixel Streaming enabled)
+- Node.js (for signaling server)
+- Docker & NVIDIA Docker (for containerized workflows)
+- React (latest)
+- (Optional) TURN/STUN server for public access
+
+## 2. Local Development
+
+### Unreal Side
+
+- Develop and test interaction logic using PIE (Play-In-Editor) and Standalone.
+- Use `-PixelStreamingIP=127.0.0.1 -PixelStreamingPort=8888` for local signaling server.
+- Set up a test Blueprint with `OnPixelStreamingInputEvent`.
+
+### Signaling Server
+
+- Start signaling server from Epic’s Pixel Streaming Infrastructure.
+
+```sh
+./start_with_stun.sh
+```
+
+- For Docker: use Epic’s Docker image or build your own.
+
+### React Side
+
+- Run React dev server as usual (`npm run dev` or `npm start`).
+- Proxy or CORS may be needed if React and signaling run on different origins.
+
+### Integration
+
+- Open React app, connect to signaling server, and verify video stream.
+- Use browser DevTools and UE logs for debugging event handling.
+
+## 3. Version Control
+
+- Use a mono-repo or separate repos for React and UE.
+- Store all config files, protocol definitions, and Dockerfiles.
+- Maintain documentation in `/docs` or as markdown in repo.
+
+## 4. Testing
+
+- Test all UI commands (color, part, camera, environment).
+- Test error paths and unexpected JSON.
+- Simulate poor network conditions (using Chrome devtools, etc.).
+- Test multi-user and session handover if needed.
+
+## 5. Build Artifacts
+
+- For deployment, package UE project for production.
+- Build React app with `npm run build`.
+- Containerize both UE and signaling server for cloud deployment.
+
+## 6. CI/CD (Optional)
+
+- Use GitHub Actions or similar for:
+- Lint/build React app
+- Package UE builds
+- Build Docker images
+- Push to container registry
+- Deploy with Docker Compose or Kubernetes manifests.
+
+## 7. Documentation
+
+- Keep all event protocols, endpoints, and deployment guides up to date.
+- Provide architecture diagrams for clarity.

docs/EVENT_HANDLING.md

@@ -0,0 +1,61 @@
+# Event Handling: Pixel Streaming UI Protocol
+
+## 1. Bi-Directional Communication
+
+- **emitUIInteraction:** Send JSON events from web to UE via WebRTC data channel.
+- **SendPixelStreamingResponse:** UE can reply/notify web UI.
+
+## 2. Blueprint Integration (Unreal Side)
+
+- Add **PixelStreamingInputComponent** to PlayerController (or UI manager Actor).
+- In Event Graph:
+  1. On BeginPlay, **Bind Event to OnPixelStreamingInputEvent**.
+  2. Use the **Descriptor** parameter (JSON string).
+  3. Parse JSON using Blueprint nodes (`Get JSON Field`, etc).
+  4. Switch logic based on `"command"` field.
+
+## 3. Example Protocol
+
+| Command        | Parameters                      | Action                         |
+| -------------- | ------------------------------- | ------------------------------ |
+| ChangeColor    | part, color (hex or RGB)        | Set dynamic material on mesh   |
+| SwapPart       | part, variant                   | Show/hide or attach new mesh   |
+| FocusCamera    | target (part/camera preset)     | Play camera animation/sequence |
+| SetTimeOfDay   | hour (0-23)                     | Adjust directional light/sky   |
+| SetEnvironment | preset (day, night, rainy, etc) | Switch HDRI, sublevel, etc     |
+
+**Example JS sent from React:**
+
+```js
+emitUIInteraction({
+  command: "ChangeColor",
+  part: "Body",
+  color: "#FF0000",
+});
+```
+
+**Blueprint Pseudocode:**
+
+```c++
+OnPixelStreamingInputEvent(Descriptor)
+  Parse Descriptor as JSON
+  Switch (Descriptor.command)
+    "ChangeColor"  -> Set Material Color
+    "SwapPart"     -> Swap Mesh
+    "FocusCamera"  -> Play Animation
+    ...
+```
+
+## 4. Returning Data to Web
+
+Use `SendPixelStreamingResponse` Blueprint node to send JSON to web.
+
+In React, register a response event listener to handle UE responses (e.g., completion, errors).
+
+## 5. Tips
+
+- Always validate inputs in Blueprint.
+
+- Define a versioned protocol if you expect evolution.
+
+- Log or display unrecognized commands for debugging.

docs/REACT_INTEGRATION.md

@@ -0,0 +1,73 @@
+# React Integration: Pixel Streaming Configurator UI
+
+## 1. Pixel Streaming Library
+
+- Install Epic’s frontend library (example):
+
+```bash
+  npm install @epicgames-ps/lib-pixelstreamingfrontend-ue5.6
+```
+
+- Use or adapt the provided Pixel Streaming React components.
+
+## 2. Basic Integration
+
+```tsx
+import { useEffect, useRef } from "react";
+import { PixelStreaming } from "@epicgames-ps/lib-pixelstreamingfrontend-ue5.6";
+
+export default function PixelStreamingView() {
+  const videoRef = useRef<HTMLVideoElement>(null);
+  useEffect(() => {
+    const stream = new PixelStreaming({
+      videoElement: videoRef.current,
+      signallingUrl: "ws://your-signal-server:80/signalling",
+    });
+    return () => stream.close();
+  }, []);
+  return <video ref={videoRef} autoPlay playsInline />;
+}
+```
+
+## 3. UI Overlay Design
+
+- Build React UI overlay: buttons, color pickers, sliders, etc.
+- When user interacts, call:
+
+```js
+pixelStreaming.emitUIInteraction({
+  command: "ChangeColor",
+  part: "Roof",
+  color: "#00FF00",
+});
+```
+
+- For camera, part, environment, repeat with suitable protocol.
+
+## 4. Receiving Responses
+
+```js
+pixelStreaming.addResponseEventListener((msg) => {
+  // Handle events from UE (e.g., completion, error, state sync)
+});
+```
+
+## 5. Touch & Mouse Controls
+
+- Use `controlScheme` config for pointer lock/hovering as needed.
+- Prevent click-through from UI elements to UE video when overlay is active.
+- For mobile, ensure touch events are mapped (Pixel Streaming lib does this by default).
+
+## 6. Example UI Features
+
+- Color Picker (for materials)
+- Dropdown/Selector (for variants/parts)
+- Environment Presets (buttons/sliders)
+- Camera Jump buttons (for part highlights)
+- Day/Night slider
+
+## 7. Advanced
+
+- Maintain state in React (current config, sync from UE on load).
+- Support multi-user with role-based UI (e.g., only one controller).
+- Responsive layout for mobile/desktop.

docs/UNREAL_SETUP.md

@@ -0,0 +1,61 @@
+# Unreal Engine 5.6 Pixel Streaming Setup
+
+## 1. Enable Pixel Streaming Plugin
+
+- Open your project in Unreal Engine 5.6.
+- Go to **Edit > Plugins > Graphics**.
+- Enable the **Pixel Streaming** plugin.
+- Restart the editor.
+
+## 2. Project Settings
+
+- In **Edit > Project Settings > Pixel Streaming**:
+  - Set default resolution (e.g., 1920x1080).
+  - Adjust bitrate (e.g., 10 Mbps) for quality/performance.
+  - Set framerate (e.g., 60 FPS for smoothness).
+  - (Optional) Enable AV1 codec for higher quality at lower bitrate (requires supported GPU).
+
+## 3. Packaging the Project
+
+- Go to **File > Package Project > [Windows/Linux]**.
+- Select a folder to save your build.
+- Ensure Pixel Streaming plugin is included.
+- For containerized or cloud use, package for **Linux** when possible (preferred for Docker).
+
+## 4. Pixel Streaming Startup Parameters
+
+- When launching your packaged build, use:
+
+```bash
+MyProject.exe -PixelStreamingIP=127.0.0.1 -PixelStreamingPort=8888 -RenderOffscreen -Windowed -ForceRes -ResX=1920 -ResY=1080
+```
+
+- `-RenderOffscreen` is crucial for headless operation.
+- Adjust IP and port to match your signaling server.
+- For Linux, launch the `.sh` wrapper with the same flags.
+
+## 5. Signaling Server (Official)
+
+- Download the Pixel Streaming Infrastructure (signaling server) from Epic’s GitHub or use prebuilt Docker images.
+- Run with:
+
+```bash
+> cd Engine/Plugins/Media/PixelStreaming/Resources/WebServers/SignallingWebServer/platform_scripts/cmd
+
+> ./start_with_stun.bat (or .sh)
+```
+
+- Listens on **port 8888** for streamer, **80** for web clients.
+- Modify `config.json` for TURN/STUN settings if needed.
+
+## 6. Networking
+
+- Open required ports: **80** (HTTP/WS), **443** (HTTPS/WSS), **8888** (WebSocket for streamer).
+- If internet-facing, use HTTPS with valid certificates.
+- Configure TURN/STUN for NAT traversal in production.
+
+## 7. Best Practices
+
+- For multi-user, consider one UE instance per session.
+- For spectators, use SFU (Selective Forwarding Unit) in UE 5.5+ (experimental).
+- Monitor GPU encoding session limits (NVIDIA cards typically allow 2–5 concurrent streams per GPU).