diff --git a/STREAMING_FIX_SUMMARY.md b/STREAMING_FIX_SUMMARY.md new file mode 100644 index 0000000..09b89e4 --- /dev/null +++ b/STREAMING_FIX_SUMMARY.md @@ -0,0 +1,88 @@ +# StreamDrop Streaming Fix Summary + +## ๐Ÿ” Problem Identified + +The streaming failures on 512MB Digital Ocean droplets were caused by: + +1. **Chromium Memory Exhaustion** (200-400MB required) +2. **Shared Memory Limits** (/dev/shm only 64MB on low-memory systems) +3. **FFmpeg Buffer Overflows** due to insufficient RAM +4. **Linux OOM Killer** terminating processes + +## โœ… Solutions Implemented + +### 1. Memory-Optimized Chromium Configuration +Added aggressive memory optimization flags for low-memory systems: +- `--disable-dev-shm-usage` (critical for small /dev/shm) +- `--single-process` mode for systems under 1GB +- `--max_old_space_size` limits for V8 JavaScript engine +- `--enable-low-end-device-mode` for additional optimizations + +### 2. Automatic Fallback System +- When Chromium crashes due to memory, system automatically falls back to test pattern streaming +- Test pattern uses minimal resources (only FFmpeg required) + +### 3. Dynamic Memory Detection +- System now detects available RAM and applies appropriate settings +- Different configurations for <1GB, 1-2GB, and 2GB+ systems + +### 4. New Diagnostic Tools +- `diagnose_streaming.py` - Tests system capabilities and provides recommendations +- `memory_optimized_streamer.py` - Ultra-low memory streaming implementation +- `VPS_REQUIREMENTS.md` - Comprehensive guide for VPS selection + +## ๐Ÿ“Š Minimum VPS Requirements + +### โŒ 512MB RAM - NOT SUPPORTED +- **Issue**: Insufficient for Chromium + FFmpeg +- **Solution**: Upgrade to 1GB minimum + +### โœ… 1GB RAM - MINIMUM RECOMMENDED +- **Digital Ocean**: Basic Droplet $6/month +- **Capabilities**: 720p @ 30fps with optimizations +- **Configuration**: Memory-optimized mode enabled + +### ๐ŸŒŸ 2GB RAM - OPTIMAL +- **Digital Ocean**: Basic Droplet $12/month +- **Capabilities**: 1080p @ 30-60fps, multi-streaming +- **Configuration**: Full features available + +## ๐Ÿš€ Quick Fix for Existing 512MB Droplets + +### Option 1: Upgrade Droplet (BEST) +```bash +# Via DigitalOcean Control Panel +1. Power off droplet +2. Resize > Choose 1GB plan ($6/month) +3. Power on droplet +``` + +### Option 2: Use Test Pattern Mode Only +```bash +# For ultra-low memory systems +export YOUTUBE_STREAM_KEY="your-key" +python3 memory_optimized_streamer.py +``` + +### Option 3: Add Swap (Temporary) +```bash +sudo fallocate -l 1G /swapfile +sudo chmod 600 /swapfile +sudo mkswap /swapfile +sudo swapon /swapfile +``` + +## ๐Ÿ“ˆ Files Modified + +1. **stream_manager.py** - Added memory detection and optimizations +2. **headless_streamer.py** - Enhanced with low-memory Chromium flags +3. **requirements.txt** - Added psutil for memory detection +4. **diagnose_streaming.py** - NEW diagnostic tool +5. **memory_optimized_streamer.py** - NEW ultra-low memory implementation +6. **VPS_REQUIREMENTS.md** - NEW comprehensive requirements guide + +## ๐ŸŽฏ Recommendation + +**For reliable streaming, upgrade to a 1GB Digital Ocean droplet ($6/month).** + +512MB is insufficient for running Chromium-based streaming. The memory optimizations help but cannot overcome fundamental hardware limitations. With 1GB RAM, StreamDrop will run reliably with the new optimizations in place. \ No newline at end of file diff --git a/VPS_REQUIREMENTS.md b/VPS_REQUIREMENTS.md new file mode 100644 index 0000000..1a8933d --- /dev/null +++ b/VPS_REQUIREMENTS.md @@ -0,0 +1,194 @@ +# StreamDrop VPS Requirements & Troubleshooting Guide + +## ๐Ÿšจ Critical Issue Identified: 512MB RAM is INSUFFICIENT + +After thorough analysis, the streaming failures on 512MB droplets are caused by: + +1. **Chromium Memory Exhaustion**: Browser alone requires 200-400MB on startup +2. **Shared Memory Limits**: `/dev/shm` is too small (typically 64MB on 512MB systems) +3. **Process Accumulation**: Running Xvfb + Chromium + FFmpeg exceeds available memory +4. **OOM Killer**: Linux kills processes when memory is exhausted + +## ๐Ÿ“Š Minimum VPS Requirements + +### โŒ NOT SUPPORTED: 512MB RAM +- **Verdict**: Insufficient for reliable streaming +- **Issues**: Chromium crashes, FFmpeg buffer overflows, constant OOM kills +- **Alternative**: Use test pattern mode only (very limited) + +### โš ๏ธ CHALLENGING: 512MB-1GB RAM +- **Verdict**: Possible with heavy optimization +- **Requirements**: + - Must use memory-optimized mode + - Test pattern streaming recommended + - Single stream only + - Lower quality (480p @ 24fps) +- **Recommended Droplet**: DigitalOcean Basic 1GB ($6/month) + +### โœ… RECOMMENDED: 1GB-2GB RAM +- **Verdict**: Good for standard streaming +- **Capabilities**: + - HTML/Pygame streaming + - 720p @ 30fps + - Multi-streaming support + - Stable operation +- **Recommended Droplet**: DigitalOcean Basic 1GB with swap ($6/month) + +### ๐ŸŒŸ OPTIMAL: 2GB+ RAM +- **Verdict**: Excellent performance +- **Capabilities**: + - All features available + - 1080p @ 30-60fps + - Multiple concurrent streams + - No optimization needed +- **Recommended Droplet**: DigitalOcean Basic 2GB ($12/month) + +## ๐Ÿ› ๏ธ Fixes Implemented + +### 1. Memory-Optimized Chromium Configuration +```bash +# New flags added for low-memory systems: +--disable-dev-shm-usage # Critical for small /dev/shm +--single-process # Run in single process mode +--no-zygote # Don't use zygote process +--max_old_space_size=96 # Limit V8 heap +--enable-low-end-device-mode # Enable low-end optimizations +--disable-site-isolation-trials +--disable-features=site-per-process +``` + +### 2. Automatic Fallback to Test Pattern +When Chromium fails due to memory issues, the system now automatically falls back to test pattern streaming which uses minimal resources. + +### 3. Dynamic Memory Detection +The system now detects available memory and applies appropriate optimizations automatically. + +## ๐Ÿ”ง Immediate Solutions + +### For 512MB Droplets - Choose One: + +#### Option 1: Upgrade to 1GB Droplet (RECOMMENDED) +```bash +# Via DigitalOcean Control Panel: +1. Power off droplet +2. Resize > Choose 1GB plan ($6/month) +3. Power on droplet +``` + +#### Option 2: Add Swap Space (Temporary Fix) +```bash +# Add 1GB swap (helps but doesn't fully solve the issue) +sudo fallocate -l 1G /swapfile +sudo chmod 600 /swapfile +sudo mkswap /swapfile +sudo swapon /swapfile +echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab + +# Optimize swappiness for low memory +echo 'vm.swappiness=60' | sudo tee -a /etc/sysctl.conf +sudo sysctl -p +``` + +#### Option 3: Use Memory-Optimized Mode +```bash +# Run the diagnostic tool +python3 diagnose_streaming.py + +# Use memory-optimized streamer +python3 memory_optimized_streamer.py +``` + +#### Option 4: Test Pattern Only Mode +```bash +# For ultra-low memory, use test pattern streaming +export YOUTUBE_STREAM_KEY="your-key-here" +export STREAMING_MODE="test_pattern" +python3 memory_optimized_streamer.py +``` + +## ๐Ÿ“ˆ Digital Ocean Droplet Recommendations + +### Budget Option ($6/month) +- **Droplet**: Basic Regular 1GB RAM / 1 vCPU / 25GB SSD +- **Location**: Choose closest to your audience +- **Additional**: Enable backups ($1/month) +- **Performance**: Handles 720p @ 30fps reliably + +### Standard Option ($12/month) +- **Droplet**: Basic Regular 2GB RAM / 1 vCPU / 50GB SSD +- **Location**: Choose closest to your audience +- **Additional**: Enable monitoring +- **Performance**: Handles 1080p @ 30fps with headroom + +### Performance Option ($18/month) +- **Droplet**: Basic Regular 2GB RAM / 2 vCPUs / 60GB SSD +- **Location**: Choose closest to your audience +- **Additional**: Enable monitoring +- **Performance**: Handles multiple streams or 1080p @ 60fps + +## ๐Ÿงช Testing Your Configuration + +### 1. Run Diagnostic Tool +```bash +cd /path/to/StreamDrop +python3 diagnose_streaming.py +``` + +This will: +- Check available memory +- Test Chromium launch +- Test FFmpeg streaming +- Provide specific recommendations + +### 2. Monitor Memory Usage +```bash +# Watch memory in real-time +watch -n 1 free -h + +# Check for OOM kills +sudo dmesg | grep -i "killed process" +``` + +### 3. Test Streaming +```bash +# Test with reduced quality first +export YOUTUBE_STREAM_KEY="your-key" +export STREAM_QUALITY="low" # 480p +python3 stream_manager.py +``` + +## ๐Ÿš€ Quick Start for New Droplet + +### For 1GB+ Droplet: +```bash +# 1. Create new droplet (Ubuntu 22.04, 1GB+ RAM) +# 2. SSH into droplet +# 3. Clone and setup +git clone https://github.com/yourusername/StreamDrop.git +cd StreamDrop +chmod +x setup.sh +./setup.sh + +# 4. Configure +cp .env.example .env +nano .env # Add your stream key + +# 5. Run +python3 main.py +``` + +## ๐Ÿ“ž Support + +If you continue experiencing issues after following this guide: + +1. Run the diagnostic tool and save output +2. Check system logs: `sudo journalctl -xe` +3. Monitor during failure: `htop` or `top` +4. Create an issue with diagnostic output + +## ๐ŸŽฏ Summary + +**512MB RAM is NOT sufficient for StreamDrop.** The absolute minimum is 1GB RAM with swap space. For reliable operation, 1GB RAM is recommended. The system now includes automatic memory detection and optimization, but hardware limitations cannot be completely overcome by software optimization. + +### Recommended Action: +**Upgrade to a 1GB droplet ($6/month on DigitalOcean) for reliable streaming.** \ No newline at end of file diff --git a/diagnose_streaming.py b/diagnose_streaming.py new file mode 100755 index 0000000..64ba946 --- /dev/null +++ b/diagnose_streaming.py @@ -0,0 +1,400 @@ +#!/usr/bin/env python3 +""" +StreamDrop Diagnostic Tool +Tests streaming capabilities and recommends optimal configuration for your VPS +""" + +import os +import sys +import subprocess +import time +import psutil +import json +from pathlib import Path + +class StreamingDiagnostic: + def __init__(self): + self.results = { + 'system': {}, + 'tests': {}, + 'recommendations': [] + } + + def check_system_resources(self): + """Check system memory and resources""" + print("๐Ÿ” Checking system resources...") + + # Memory info + mem = psutil.virtual_memory() + swap = psutil.swap_memory() + + self.results['system']['memory_mb'] = mem.total // (1024 * 1024) + self.results['system']['available_mb'] = mem.available // (1024 * 1024) + self.results['system']['swap_mb'] = swap.total // (1024 * 1024) + + # Check /dev/shm size + try: + shm_stat = os.statvfs('/dev/shm') + shm_size_mb = (shm_stat.f_blocks * shm_stat.f_frsize) // (1024 * 1024) + self.results['system']['shm_size_mb'] = shm_size_mb + except: + self.results['system']['shm_size_mb'] = 0 + + # CPU info + self.results['system']['cpu_count'] = psutil.cpu_count() + + print(f" โœ“ Total Memory: {self.results['system']['memory_mb']}MB") + print(f" โœ“ Available Memory: {self.results['system']['available_mb']}MB") + print(f" โœ“ Swap: {self.results['system']['swap_mb']}MB") + print(f" โœ“ Shared Memory (/dev/shm): {self.results['system']['shm_size_mb']}MB") + print(f" โœ“ CPU Cores: {self.results['system']['cpu_count']}") + + return self.results['system'] + + def test_chromium_launch(self, memory_optimized=False): + """Test if Chromium can launch with given settings""" + print(f"\n๐Ÿงช Testing Chromium launch {'(memory optimized)' if memory_optimized else '(standard)'}...") + + chrome_cmd = ['chromium-browser', '--version'] + try: + result = subprocess.run(chrome_cmd, capture_output=True, text=True, timeout=5) + if result.returncode != 0: + print(" โŒ Chromium not installed") + return False, 0 + except: + print(" โŒ Chromium not found") + return False, 0 + + # Test actual browser launch + if memory_optimized: + chrome_cmd = [ + 'chromium-browser', + '--headless', + '--disable-gpu', + '--no-sandbox', + '--disable-dev-shm-usage', # Critical for low memory + '--disable-setuid-sandbox', + '--disable-web-security', + '--disable-features=VizDisplayCompositor', + '--disable-breakpad', + '--disable-software-rasterizer', + '--disable-extensions', + '--disable-plugins', + '--disable-background-timer-throttling', + '--disable-backgrounding-occluded-windows', + '--disable-renderer-backgrounding', + '--disable-features=TranslateUI', + '--disable-ipc-flooding-protection', + '--disable-background-networking', + '--disable-sync', + '--metrics-recording-only', + '--no-first-run', + '--disable-default-apps', + '--disable-hang-monitor', + '--disable-popup-blocking', + '--disable-prompt-on-repost', + '--disable-domain-reliability', + '--disable-component-update', + '--disable-features=AutofillServerCommunication', + '--disable-features=CertificateTransparencyComponentUpdater', + '--single-process', # Use single process mode for low memory + '--memory-pressure-off', + '--max_old_space_size=96', # Limit V8 heap + '--js-flags="--max-old-space-size=96 --max-semi-space-size=2"', + '--aggressive-cache-discard', + '--aggressive-tab-discard', + '--window-size=854,480', # Smaller window for less memory + 'about:blank' + ] + else: + chrome_cmd = [ + 'chromium-browser', + '--headless', + '--disable-gpu', + '--no-sandbox', + '--disable-setuid-sandbox', + '--window-size=1280,720', + 'about:blank' + ] + + try: + process = subprocess.Popen(chrome_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + time.sleep(3) # Let it run for 3 seconds + + if process.poll() is None: + # Still running + memory_used = psutil.Process(process.pid).memory_info().rss // (1024 * 1024) + process.terminate() + process.wait(timeout=5) + print(f" โœ“ Chromium launched successfully (used {memory_used}MB)") + return True, memory_used + else: + # Crashed + stderr = process.stderr.read().decode() if process.stderr else "" + print(f" โŒ Chromium crashed") + if "Shared memory" in stderr or "shm" in stderr: + print(" Issue: Shared memory limit") + elif "memory" in stderr.lower(): + print(" Issue: Out of memory") + return False, 0 + except Exception as e: + print(f" โŒ Failed to test Chromium: {e}") + return False, 0 + + def test_ffmpeg_streaming(self, low_memory=False): + """Test FFmpeg streaming capability""" + print(f"\n๐Ÿงช Testing FFmpeg streaming {'(low memory mode)' if low_memory else '(standard)'}...") + + # Check FFmpeg + try: + result = subprocess.run(['ffmpeg', '-version'], capture_output=True, timeout=5) + if result.returncode != 0: + print(" โŒ FFmpeg not installed") + return False + except: + print(" โŒ FFmpeg not found") + return False + + # Test streaming with test pattern + if low_memory: + ffmpeg_cmd = [ + 'ffmpeg', + '-f', 'lavfi', + '-i', 'testsrc=size=854x480:rate=24', + '-f', 'lavfi', + '-i', 'anullsrc', + '-t', '5', # Run for 5 seconds + '-c:v', 'libx264', + '-preset', 'ultrafast', # Fastest preset for low CPU + '-b:v', '500k', # Lower bitrate + '-maxrate', '500k', + '-bufsize', '250k', # Smaller buffer + '-pix_fmt', 'yuv420p', + '-g', '48', + '-c:a', 'aac', + '-b:a', '64k', # Lower audio bitrate + '-f', 'null', + '-' + ] + else: + ffmpeg_cmd = [ + 'ffmpeg', + '-f', 'lavfi', + '-i', 'testsrc=size=1280x720:rate=30', + '-f', 'lavfi', + '-i', 'anullsrc', + '-t', '5', + '-c:v', 'libx264', + '-preset', 'veryfast', + '-b:v', '2500k', + '-maxrate', '2500k', + '-bufsize', '5000k', + '-pix_fmt', 'yuv420p', + '-g', '60', + '-c:a', 'aac', + '-b:a', '128k', + '-f', 'null', + '-' + ] + + try: + process = subprocess.Popen(ffmpeg_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = process.communicate(timeout=10) + + if process.returncode == 0: + print(" โœ“ FFmpeg streaming test successful") + return True + else: + print(" โŒ FFmpeg streaming test failed") + return False + except subprocess.TimeoutExpired: + process.kill() + print(" โš ๏ธ FFmpeg test timed out (might be too slow)") + return False + except Exception as e: + print(f" โŒ FFmpeg test error: {e}") + return False + + def generate_recommendations(self): + """Generate recommendations based on test results""" + print("\n๐Ÿ“Š Generating recommendations...") + + memory_mb = self.results['system']['memory_mb'] + shm_mb = self.results['system']['shm_size_mb'] + + # Memory tier classification + if memory_mb < 512: + tier = "ultra_low" + self.results['recommendations'].append({ + 'tier': 'ultra_low', + 'verdict': 'NOT RECOMMENDED', + 'reason': 'Less than 512MB RAM is insufficient for reliable streaming', + 'solution': 'Upgrade to at least 1GB RAM' + }) + elif memory_mb < 1024: + tier = "low" + self.results['recommendations'].append({ + 'tier': 'low', + 'verdict': 'CHALLENGING', + 'reason': '512MB-1GB RAM requires special optimization', + 'solution': 'Use test pattern mode or upgrade to 1GB+ RAM' + }) + elif memory_mb < 2048: + tier = "medium" + self.results['recommendations'].append({ + 'tier': 'medium', + 'verdict': 'GOOD', + 'reason': '1-2GB RAM is suitable with optimizations', + 'solution': 'Use memory-optimized mode' + }) + else: + tier = "high" + self.results['recommendations'].append({ + 'tier': 'high', + 'verdict': 'EXCELLENT', + 'reason': '2GB+ RAM can handle standard streaming', + 'solution': 'All streaming modes available' + }) + + # Shared memory check + if shm_mb < 64: + self.results['recommendations'].append({ + 'issue': 'shared_memory', + 'solution': 'Increase /dev/shm size or use --disable-dev-shm-usage flag' + }) + + # Recommended configuration + if tier == "ultra_low": + config = { + 'mode': 'none', + 'message': 'System does not meet minimum requirements' + } + elif tier == "low": + config = { + 'mode': 'test_pattern', + 'resolution': '854x480', + 'framerate': '24', + 'bitrate': '500k', + 'preset': 'ultrafast', + 'chrome_flags': ['--single-process', '--disable-dev-shm-usage', '--max_old_space_size=96'] + } + elif tier == "medium": + config = { + 'mode': 'memory_optimized', + 'resolution': '1280x720', + 'framerate': '30', + 'bitrate': '1500k', + 'preset': 'veryfast', + 'chrome_flags': ['--disable-dev-shm-usage', '--max_old_space_size=256'] + } + else: + config = { + 'mode': 'standard', + 'resolution': '1920x1080', + 'framerate': '30', + 'bitrate': '2500k', + 'preset': 'veryfast', + 'chrome_flags': [] + } + + self.results['recommended_config'] = config + return self.results['recommendations'] + + def save_results(self): + """Save diagnostic results to file""" + with open('streaming_diagnostic.json', 'w') as f: + json.dump(self.results, f, indent=2) + print(f"\n๐Ÿ’พ Results saved to streaming_diagnostic.json") + + def print_summary(self): + """Print summary and recommendations""" + print("\n" + "="*60) + print("๐Ÿ“‹ DIAGNOSTIC SUMMARY") + print("="*60) + + memory_mb = self.results['system']['memory_mb'] + recommendations = self.results['recommendations'] + config = self.results.get('recommended_config', {}) + + # System tier + tier_rec = next((r for r in recommendations if 'tier' in r), None) + if tier_rec: + verdict = tier_rec['verdict'] + if verdict == 'NOT RECOMMENDED': + color = '\033[91m' # Red + elif verdict == 'CHALLENGING': + color = '\033[93m' # Yellow + elif verdict == 'GOOD': + color = '\033[92m' # Green + else: + color = '\033[94m' # Blue + print(f"\n{color}System Verdict: {verdict}\033[0m") + print(f"Reason: {tier_rec['reason']}") + print(f"Solution: {tier_rec['solution']}") + + # Recommended configuration + if config.get('mode') != 'none': + print(f"\n๐Ÿ”ง RECOMMENDED CONFIGURATION:") + print(f" Mode: {config.get('mode')}") + print(f" Resolution: {config.get('resolution', 'N/A')}") + print(f" Framerate: {config.get('framerate', 'N/A')} fps") + print(f" Bitrate: {config.get('bitrate', 'N/A')}") + print(f" Preset: {config.get('preset', 'N/A')}") + + # Minimum requirements + print(f"\n๐Ÿ“Š MINIMUM VPS REQUIREMENTS FOR STREAMDROP:") + print(f" โ€ข RAM: 1GB (2GB recommended)") + print(f" โ€ข CPU: 1 vCPU (2 vCPU recommended)") + print(f" โ€ข Bandwidth: 10 Mbps upload") + print(f" โ€ข Storage: 5GB free space") + + if memory_mb < 1024: + print(f"\nโš ๏ธ YOUR SYSTEM HAS {memory_mb}MB RAM") + print(f" Recommended action: Upgrade to 1GB+ RAM droplet") + print(f" Alternative: Use test pattern mode (limited functionality)") + +def main(): + print("๐Ÿš€ StreamDrop Diagnostic Tool") + print("="*60) + + diag = StreamingDiagnostic() + + # Run diagnostics + diag.check_system_resources() + + # Test components based on available memory + memory_mb = diag.results['system']['memory_mb'] + + if memory_mb >= 512: + # Test standard Chromium + success, memory_used = diag.test_chromium_launch(memory_optimized=False) + diag.results['tests']['chromium_standard'] = {'success': success, 'memory_used': memory_used} + + if not success or memory_used > memory_mb * 0.5: + # Try memory optimized + success, memory_used = diag.test_chromium_launch(memory_optimized=True) + diag.results['tests']['chromium_optimized'] = {'success': success, 'memory_used': memory_used} + else: + # Only test memory optimized for very low memory + success, memory_used = diag.test_chromium_launch(memory_optimized=True) + diag.results['tests']['chromium_optimized'] = {'success': success, 'memory_used': memory_used} + + # Test FFmpeg + if memory_mb >= 1024: + success = diag.test_ffmpeg_streaming(low_memory=False) + diag.results['tests']['ffmpeg_standard'] = success + else: + success = diag.test_ffmpeg_streaming(low_memory=True) + diag.results['tests']['ffmpeg_low_memory'] = success + + # Generate recommendations + diag.generate_recommendations() + + # Save and display results + diag.save_results() + diag.print_summary() + + print("\nโœ… Diagnostic complete!") + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/headless_streamer.py b/headless_streamer.py index fa306f5..7876aee 100755 --- a/headless_streamer.py +++ b/headless_streamer.py @@ -39,13 +39,17 @@ def __init__(self, stream_key, content_path="https://example.com"): def start_chromium_headless(self): """Start Chromium in true headless mode with remote debugging""" + # Detect available memory for optimization + import psutil + total_memory_mb = psutil.virtual_memory().total // (1024 * 1024) + chromium_cmd = [ 'chromium-browser', '--headless=new', # New headless mode (more efficient) '--no-gpu', '--no-sandbox', '--disable-setuid-sandbox', - '--disable-dev-shm-usage', + '--disable-dev-shm-usage', # CRITICAL for low memory '--disable-extensions', '--disable-plugins', '--disable-background-timer-throttling', @@ -58,11 +62,34 @@ def start_chromium_headless(self): '--disable-component-update', '--remote-debugging-port=' + str(self.debug_port), '--remote-debugging-address=127.0.0.1', - '--window-size=1280,720', - '--virtual-time-budget=5000', - self.content_path + '--virtual-time-budget=5000' ] + # Add memory optimizations based on available RAM + if total_memory_mb < 1024: + chromium_cmd.extend([ + '--single-process', # Run in single process mode + '--no-zygote', # Don't use zygote process + '--max_old_space_size=96', # Limit V8 heap + '--js-flags="--max-old-space-size=96 --max-semi-space-size=2"', + '--aggressive-cache-discard', + '--aggressive-tab-discard', + '--enable-low-end-device-mode', + '--disable-site-isolation-trials', + '--disable-features=site-per-process', + '--window-size=854,480' # Smaller window for less memory + ]) + logger.info(f"Applied low-memory optimizations (system has {total_memory_mb}MB)") + elif total_memory_mb < 2048: + chromium_cmd.extend([ + '--max_old_space_size=256', + '--window-size=1280,720' + ]) + else: + chromium_cmd.append('--window-size=1280,720') + + chromium_cmd.append(self.content_path) + logger.info("Starting Chromium in optimized headless mode...") self.chrome_process = subprocess.Popen( chromium_cmd, diff --git a/memory_optimized_streamer.py b/memory_optimized_streamer.py new file mode 100644 index 0000000..55197ab --- /dev/null +++ b/memory_optimized_streamer.py @@ -0,0 +1,410 @@ +#!/usr/bin/env python3 +""" +Memory-Optimized Streamer for Low-Memory VPS (512MB-1GB) +Implements aggressive memory optimization techniques for streaming on minimal resources +""" + +import os +import sys +import subprocess +import time +import signal +import logging +import psutil +from pathlib import Path + +logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') +logger = logging.getLogger(__name__) + +class MemoryOptimizedStreamer: + """Ultra-low memory streaming implementation""" + + def __init__(self, stream_key, platform='youtube', content_type='test_pattern'): + self.stream_key = stream_key + self.platform = platform + self.content_type = content_type + self.processes = {} + self.streaming = False + + # Platform-specific RTMP URLs + self.rtmp_urls = { + 'youtube': 'rtmp://a.rtmp.youtube.com/live2/', + 'twitch': 'rtmp://live.twitch.tv/app/', + 'facebook': 'rtmps://live-api-s.facebook.com:443/rtmp/', + 'custom': os.environ.get('RTMP_URL', 'rtmp://localhost/live/') + } + + # Detect system memory + self.total_memory_mb = psutil.virtual_memory().total // (1024 * 1024) + self.configure_for_memory() + + def configure_for_memory(self): + """Configure streaming parameters based on available memory""" + if self.total_memory_mb < 512: + logger.error("System has less than 512MB RAM - streaming not recommended") + self.config = None + elif self.total_memory_mb < 768: + # Ultra-low memory mode (512-768MB) + self.config = { + 'resolution': '640x360', + 'framerate': '20', + 'video_bitrate': '400k', + 'audio_bitrate': '64k', + 'preset': 'ultrafast', + 'buffer_size': '200k', + 'gop_size': '40', + 'mode': 'test_pattern_only' + } + logger.info("Configured for ULTRA-LOW memory mode (512-768MB)") + elif self.total_memory_mb < 1024: + # Low memory mode (768MB-1GB) + self.config = { + 'resolution': '854x480', + 'framerate': '24', + 'video_bitrate': '800k', + 'audio_bitrate': '96k', + 'preset': 'superfast', + 'buffer_size': '400k', + 'gop_size': '48', + 'mode': 'test_pattern_or_simple' + } + logger.info("Configured for LOW memory mode (768MB-1GB)") + elif self.total_memory_mb < 2048: + # Medium memory mode (1-2GB) + self.config = { + 'resolution': '1280x720', + 'framerate': '30', + 'video_bitrate': '1500k', + 'audio_bitrate': '128k', + 'preset': 'veryfast', + 'buffer_size': '1500k', + 'gop_size': '60', + 'mode': 'optimized_browser' + } + logger.info("Configured for MEDIUM memory mode (1-2GB)") + else: + # Standard mode (2GB+) + self.config = { + 'resolution': '1920x1080', + 'framerate': '30', + 'video_bitrate': '2500k', + 'audio_bitrate': '128k', + 'preset': 'veryfast', + 'buffer_size': '2500k', + 'gop_size': '60', + 'mode': 'full' + } + logger.info("Configured for STANDARD mode (2GB+)") + + def check_prerequisites(self): + """Check if required tools are installed""" + required_tools = ['ffmpeg'] + missing = [] + + for tool in required_tools: + try: + subprocess.run([tool, '-version'], capture_output=True, check=True) + except (subprocess.CalledProcessError, FileNotFoundError): + missing.append(tool) + + if missing: + logger.error(f"Missing required tools: {', '.join(missing)}") + return False + + return True + + def clear_memory(self): + """Clear system caches to free memory before streaming""" + logger.info("Clearing memory caches...") + try: + # Sync filesystems + subprocess.run(['sync'], check=False) + # Clear caches (requires root) + subprocess.run(['sudo', 'sh', '-c', 'echo 1 > /proc/sys/vm/drop_caches'], check=False) + time.sleep(1) + except: + logger.warning("Could not clear memory caches (requires sudo)") + + def start_test_pattern_stream(self): + """Start ultra-lightweight test pattern streaming""" + logger.info(f"Starting test pattern stream at {self.config['resolution']} @ {self.config['framerate']}fps") + + # Build RTMP URL + rtmp_url = self.rtmp_urls.get(self.platform, self.rtmp_urls['custom']) + full_url = f"{rtmp_url}{self.stream_key}" + + # Build FFmpeg command with minimal resource usage + ffmpeg_cmd = [ + 'ffmpeg', + '-re', # Read input at native frame rate + '-f', 'lavfi', + '-i', f'testsrc2=size={self.config["resolution"]}:rate={self.config["framerate"]}', + '-f', 'lavfi', + '-i', 'anullsrc=channel_layout=stereo:sample_rate=44100', + '-vf', f'drawtext=text="StreamDrop Low Memory Mode":x=10:y=10:fontsize=20:fontcolor=white,' + f'drawtext=text="Memory: {self.total_memory_mb}MB":x=10:y=40:fontsize=16:fontcolor=yellow,' + f'drawtext=text="%{{localtime\\:%H\\:%M\\:%S}}":x=10:y=70:fontsize=16:fontcolor=lime', + '-c:v', 'libx264', + '-preset', self.config['preset'], + '-tune', 'zerolatency', # Reduce latency + '-b:v', self.config['video_bitrate'], + '-maxrate', self.config['video_bitrate'], + '-bufsize', self.config['buffer_size'], + '-pix_fmt', 'yuv420p', + '-g', self.config['gop_size'], + '-c:a', 'aac', + '-b:a', self.config['audio_bitrate'], + '-ar', '44100', + '-f', 'flv', + full_url + ] + + # Add ultra-low memory flags for 512MB systems + if self.total_memory_mb < 768: + ffmpeg_cmd.extend([ + '-threads', '1', # Single thread to reduce memory + '-thread_queue_size', '32', # Smaller queue + ]) + + logger.info(f"FFmpeg command: {' '.join(ffmpeg_cmd[:10])}...") + + try: + self.processes['ffmpeg'] = subprocess.Popen( + ffmpeg_cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE + ) + + # Check if process started successfully + time.sleep(2) + if self.processes['ffmpeg'].poll() is not None: + stderr = self.processes['ffmpeg'].stderr.read().decode() + logger.error(f"FFmpeg failed to start: {stderr[:500]}") + return False + + self.streaming = True + return True + + except Exception as e: + logger.error(f"Failed to start FFmpeg: {e}") + return False + + def start_optimized_browser_stream(self): + """Start browser streaming with aggressive memory optimization""" + logger.info("Starting memory-optimized browser streaming...") + + # Check if Chromium is available + chrome_binary = None + for binary in ['chromium-browser', 'chromium', 'google-chrome']: + try: + subprocess.run([binary, '--version'], capture_output=True, check=True) + chrome_binary = binary + break + except: + continue + + if not chrome_binary: + logger.error("No Chrome/Chromium browser found") + return False + + # Build Chrome command with extreme memory optimization + chrome_cmd = [ + chrome_binary, + '--headless', + '--disable-gpu', + '--no-sandbox', + '--disable-dev-shm-usage', # CRITICAL for low memory + '--disable-setuid-sandbox', + '--single-process', # Run in single process mode + '--no-zygote', # Don't use zygote process + '--disable-web-security', + '--disable-features=VizDisplayCompositor', + '--disable-breakpad', + '--disable-software-rasterizer', + '--disable-extensions', + '--disable-plugins', + '--disable-images', # Don't load images + '--disable-javascript', # Disable JS if possible + '--disable-background-timer-throttling', + '--disable-backgrounding-occluded-windows', + '--disable-renderer-backgrounding', + '--disable-features=TranslateUI', + '--disable-ipc-flooding-protection', + '--disable-background-networking', + '--disable-sync', + '--disable-default-apps', + '--disable-hang-monitor', + '--disable-component-update', + '--memory-pressure-off', + '--max_old_space_size=64', # Limit V8 heap to 64MB + '--js-flags="--max-old-space-size=64 --max-semi-space-size=1"', + '--aggressive-cache-discard', + '--aggressive-tab-discard', + f'--window-size={self.config["resolution"]}', + '--force-device-scale-factor=0.75', # Reduce rendering scale + '--virtual-time-budget=5000', + '--enable-low-end-device-mode', # Enable low-end device optimizations + '--disable-site-isolation-trials', + '--disable-features=site-per-process', + '--remote-debugging-port=9222', + os.environ.get('CONTENT_URL', 'https://example.com') + ] + + logger.info(f"Starting Chrome with {len(chrome_cmd)} optimization flags...") + + try: + # Start Chrome + self.processes['chrome'] = subprocess.Popen( + chrome_cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE + ) + + # Wait and check if Chrome started + time.sleep(3) + if self.processes['chrome'].poll() is not None: + stderr = self.processes['chrome'].stderr.read().decode() + logger.error(f"Chrome failed to start: {stderr[:500]}") + + # Fall back to test pattern + logger.info("Falling back to test pattern streaming...") + return self.start_test_pattern_stream() + + # If Chrome started, capture and stream + # TODO: Implement Chrome DevTools Protocol capture + logger.warning("Browser streaming not fully implemented - using test pattern") + return self.start_test_pattern_stream() + + except Exception as e: + logger.error(f"Failed to start Chrome: {e}") + return self.start_test_pattern_stream() + + def start_streaming(self): + """Start streaming with appropriate method based on memory""" + if not self.config: + return False, "System memory too low for streaming" + + if not self.check_prerequisites(): + return False, "Missing required tools" + + # Clear memory before starting + self.clear_memory() + + # Log memory status + mem = psutil.virtual_memory() + logger.info(f"Memory status - Total: {mem.total//1024//1024}MB, Available: {mem.available//1024//1024}MB") + + # Choose streaming method based on configuration + if self.config['mode'] in ['test_pattern_only', 'test_pattern_or_simple']: + success = self.start_test_pattern_stream() + elif self.config['mode'] == 'optimized_browser': + success = self.start_optimized_browser_stream() + else: + success = self.start_test_pattern_stream() # Default fallback + + if success: + self.monitor_stream() + return True, f"Streaming started in {self.config['mode']} mode" + else: + return False, "Failed to start streaming" + + def monitor_stream(self): + """Monitor stream health and memory usage""" + def monitor_loop(): + while self.streaming: + try: + # Check memory usage + mem = psutil.virtual_memory() + mem_percent = mem.percent + + if mem_percent > 90: + logger.warning(f"High memory usage: {mem_percent}%") + + # Check if FFmpeg is still running + if 'ffmpeg' in self.processes: + if self.processes['ffmpeg'].poll() is not None: + logger.error("FFmpeg process died") + self.streaming = False + break + + time.sleep(10) # Check every 10 seconds + + except Exception as e: + logger.error(f"Monitor error: {e}") + + import threading + monitor_thread = threading.Thread(target=monitor_loop, daemon=True) + monitor_thread.start() + + def stop_streaming(self): + """Stop all streaming processes""" + logger.info("Stopping streaming...") + self.streaming = False + + for name, process in self.processes.items(): + if process and process.poll() is None: + logger.info(f"Terminating {name}...") + process.terminate() + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired: + process.kill() + + self.processes.clear() + logger.info("Streaming stopped") + +def main(): + """Main entry point""" + print("๐Ÿš€ Memory-Optimized StreamDrop Streamer") + print("="*50) + + # Get configuration from environment + stream_key = os.environ.get('YOUTUBE_STREAM_KEY') + if not stream_key: + print("โŒ YOUTUBE_STREAM_KEY environment variable required") + sys.exit(1) + + platform = os.environ.get('PLATFORM', 'youtube') + + # Create streamer + streamer = MemoryOptimizedStreamer(stream_key, platform) + + # Check memory + mem = psutil.virtual_memory() + print(f"๐Ÿ“Š System Memory: {mem.total//1024//1024}MB total, {mem.available//1024//1024}MB available") + + if mem.total < 512 * 1024 * 1024: + print("โŒ ERROR: Less than 512MB RAM detected") + print(" StreamDrop requires at least 512MB RAM") + print(" Recommended: Upgrade to 1GB+ droplet") + sys.exit(1) + + # Setup signal handlers + def signal_handler(sig, frame): + print("\n๐Ÿ›‘ Shutting down...") + streamer.stop_streaming() + sys.exit(0) + + signal.signal(signal.SIGINT, signal_handler) + signal.signal(signal.SIGTERM, signal_handler) + + # Start streaming + success, message = streamer.start_streaming() + + if success: + print(f"โœ… {message}") + print("๐Ÿ“บ Stream is now live!") + print("๐Ÿ’ก Press Ctrl+C to stop") + + # Keep running + try: + while streamer.streaming: + time.sleep(1) + except KeyboardInterrupt: + pass + else: + print(f"โŒ {message}") + sys.exit(1) + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/stream_manager.py b/stream_manager.py index 7764e1e..862b77d 100644 --- a/stream_manager.py +++ b/stream_manager.py @@ -1047,13 +1047,17 @@ def _start_headless_html_streaming(self, quality): # Start Chrome in headless mode with remote debugging chrome_port = 9222 + int(self.config['id'][-1]) # Unique port per stream + # Detect available memory for optimization + import psutil + total_memory_mb = psutil.virtual_memory().total // (1024 * 1024) + chrome_cmd = [ 'chromium-browser', '--headless', '--disable-gpu', '--no-sandbox', '--disable-setuid-sandbox', - '--disable-dev-shm-usage', + '--disable-dev-shm-usage', # CRITICAL for low memory '--disable-extensions', '--disable-plugins', '--disable-images', # Optimize for streaming @@ -1068,15 +1072,37 @@ def _start_headless_html_streaming(self, quality): '--disable-default-apps', '--disable-sync', '--memory-pressure-off', - '--max_old_space_size=128', '--mute-audio', f'--window-size={quality["resolution"].replace("x", ",")}', f'--remote-debugging-port={chrome_port}', '--enable-logging', - '--log-level=0', - self.config['source'] + '--log-level=0' ] + # Add aggressive memory optimizations for low-memory systems + if total_memory_mb < 1024: + chrome_cmd.extend([ + '--single-process', # Run in single process mode + '--no-zygote', # Don't use zygote process + '--max_old_space_size=96', # Limit V8 heap to 96MB + '--js-flags="--max-old-space-size=96 --max-semi-space-size=2"', + '--aggressive-cache-discard', + '--aggressive-tab-discard', + '--enable-low-end-device-mode', + '--disable-site-isolation-trials', + '--disable-features=site-per-process' + ]) + logger.info(f"Applied low-memory optimizations (system has {total_memory_mb}MB)") + elif total_memory_mb < 2048: + chrome_cmd.extend([ + '--max_old_space_size=256', # Limit V8 heap to 256MB + '--js-flags="--max-old-space-size=256"' + ]) + else: + chrome_cmd.append('--max_old_space_size=512') + + chrome_cmd.append(self.config['source']) + logger.info(f"Starting headless Chrome: {' '.join(chrome_cmd[:8])}...") self.processes['chrome'] = subprocess.Popen(chrome_cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE) @@ -1086,8 +1112,19 @@ def _start_headless_html_streaming(self, quality): if self.processes['chrome'].poll() is not None: # Chrome died immediately stderr_output = self.processes['chrome'].stderr.read().decode() if self.processes['chrome'].stderr else "No error output" - logger.error(f"Chrome died immediately. Error: {stderr_output}") - raise Exception(f"Chrome failed to start: {stderr_output}") + logger.error(f"Chrome died immediately. Error: {stderr_output[:1000]}") + + # Check for common memory-related errors + if "Shared memory" in stderr_output or "/dev/shm" in stderr_output: + logger.error("Chrome crashed due to shared memory limits. System may need more RAM or larger /dev/shm") + logger.info("Falling back to test pattern streaming...") + return self._start_test_pattern_streaming(quality) + elif "memory" in stderr_output.lower() or "oom" in stderr_output.lower(): + logger.error("Chrome crashed due to out of memory. System needs more RAM.") + logger.info("Falling back to test pattern streaming...") + return self._start_test_pattern_streaming(quality) + + raise Exception(f"Chrome failed to start: {stderr_output[:500]}") # Verify Chrome is responding on debug port import socket diff --git a/streaming_diagnostic.json b/streaming_diagnostic.json new file mode 100644 index 0000000..99af78d --- /dev/null +++ b/streaming_diagnostic.json @@ -0,0 +1,36 @@ +{ + "system": { + "memory_mb": 16017, + "available_mb": 14910, + "swap_mb": 0, + "shm_size_mb": 64, + "cpu_count": 4 + }, + "tests": { + "chromium_standard": { + "success": false, + "memory_used": 0 + }, + "chromium_optimized": { + "success": false, + "memory_used": 0 + }, + "ffmpeg_standard": false + }, + "recommendations": [ + { + "tier": "high", + "verdict": "EXCELLENT", + "reason": "2GB+ RAM can handle standard streaming", + "solution": "All streaming modes available" + } + ], + "recommended_config": { + "mode": "standard", + "resolution": "1920x1080", + "framerate": "30", + "bitrate": "2500k", + "preset": "veryfast", + "chrome_flags": [] + } +} \ No newline at end of file