Cron Scheduler Plugin

[butschster]

Implemented: https://github.com/roadrunner-plugins/cron

Current Pain Point

PHP applications using RoadRunner often need to execute scheduled tasks (like Laravel's task scheduler, periodic maintenance jobs, or data synchronization). Currently, this requires external cron daemons or separate orchestration tools, which adds operational complexity and deployment overhead.

Proposed Solution

A native RoadRunner plugin that manages scheduled command execution within the RoadRunner process itself. The plugin will parse cron expressions, execute shell commands at scheduled intervals, and provide comprehensive observability through Prometheus metrics and debug logging.

Benefits

• Simplified Deployment: No need for system-level cron configuration
• Unified Observability: Metrics and logs integrated with RoadRunner ecosystem
• Precise Scheduling: Support for sub-minute (second-level) precision
• Resource Management: Built-in overlap prevention and graceful shutdown

---

2. Technical Requirements

2.1 Configuration Structure

The plugin reads configuration from .rr.yaml under a cron section. Configuration is a collection of scheduled jobs, where each job defines:

Job Definition Fields:

• name (string, required): Unique identifier for the job (used in metrics and logs)
• command (string, required): Shell command to execute (e.g., php artisan cache:clear)
• schedule (string, required): Cron expression defining when to run
• allow_overlap (boolean, optional, default: false): Whether to allow concurrent executions of the same job
• timeout (duration, optional, default: no timeout): Maximum execution time before SIGTERM is sent

Global Settings:

• timezone (string, optional, default: UTC): Timezone for cron schedule interpretation
• grace_period (duration, optional, default: 30s): Time to wait for running jobs during shutdown

2.2 Cron Expression Support

The plugin must support standard cron syntax with extended precision:

Standard 5-field format:

* * * * *
│ │ │ │ │
│ │ │ │ └─── Day of week (0-6, Sunday=0)
│ │ │ └───── Month (1-12)
│ │ └─────── Day of month (1-31)
│ └───────── Hour (0-23)
└─────────── Minute (0-59)

Extended 6-field format for second precision:

* * * * * *
│ │ │ │ │ │
│ │ │ │ │ └─── Day of week (0-6)
│ │ │ │ └───── Month (1-12)
│ │ │ └─────── Day of month (1-31)
│ └───────── Hour (0-23)
└─────────── Minute (0-59)
Second (0-59)

Special expressions:

• @yearly / @annually - Run once a year (0 0 1 1 *)
• @monthly - Run once a month (0 0 1 * *)
• @weekly - Run once a week (0 0 * * 0)
• @daily / @midnight - Run once a day (0 0 * * *)
• @hourly - Run once an hour (0 * * * *)
• @every <duration> - Run at fixed intervals (e.g., @every 5m, @every 30s)

2.3 Command Execution Behavior

Execution Environment:

• Commands execute as shell commands in the host system environment
• Inherit RoadRunner process environment variables
• Working directory is the RoadRunner process working directory
• Standard output (stdout) and standard error (stderr) are captured

Overlap Prevention:

• Each job maintains an execution mutex
• If allow_overlap: false (default) and job is already running:
  • Skip the scheduled execution
  • Log a debug message: "Skipping job '{name}' - previous execution still running"
  • Increment roadrunner_cron_skipped_total metric
• If allow_overlap: true:
  • Allow concurrent executions without limits

Timeout Handling:

• If timeout is configured and exceeded:
  • Send SIGTERM to the command process
  • Wait 5 seconds for graceful shutdown
  • If still running, send SIGKILL
  • Log a warning: "Job '{name}' timed out after {duration}"
  • Increment roadrunner_cron_timeout_total metric

Exit Code Handling:

• Exit code 0: Success, update success metrics
• Non-zero exit code: Failure, log error with exit code and stderr output, update failure metrics
• No automatic retries (failures are logged and counted only)

2.4 Lifecycle Management

Initialization (Init phase):

• Parse configuration from .rr.yaml
• Validate all cron expressions
• Initialize Prometheus metrics (if metrics plugin available)
• Prepare scheduler data structures

Service Start (Serve phase):

• Start background goroutine for each configured job
• Each goroutine calculates next execution time and sleeps until scheduled
• Begin accepting shutdown signals

Graceful Shutdown (Stop phase):

1. Stop scheduling new job executions
2. Send SIGTERM to all running command processes
3. Wait up to grace_period duration for commands to complete
4. After grace period, send SIGKILL to remaining processes
5. Clean up resources and return

2.5 Metrics Integration (Prometheus)

The plugin exposes the following metrics when the metrics plugin is available:

Counter Metrics:

• roadrunner_cron_executions_total{job_name, status} - Total executions (status: success, failure)
• roadrunner_cron_skipped_total{job_name} - Skipped executions due to overlap prevention
• roadrunner_cron_timeout_total{job_name} - Executions terminated due to timeout

Histogram Metrics:

• roadrunner_cron_execution_duration_seconds{job_name} - Command execution duration
  • Buckets: 0.1, 0.5, 1, 5, 10, 30, 60, 120, 300 seconds

Gauge Metrics:

• roadrunner_cron_running_jobs{job_name} - Number of currently running instances per job

2.6 Debug Logging

When debug logging is enabled, the plugin outputs:

Initialization:

• "Loaded {count} cron jobs"
• For each job: "Scheduled job '{name}': {schedule} -> {command}"

Execution Events:

• "Executing job '{name}': {command}"
• "Job '{name}' completed successfully in {duration}ms"
• "Job '{name}' failed with exit code {code}: {stderr_snippet}"
• "Skipping job '{name}' - previous execution still running"
• "Job '{name}' timed out after {duration}"

Shutdown:

• "Stopping cron scheduler, waiting for {count} running jobs"
• "Sending SIGTERM to job '{name}' (PID {pid})"
• "Graceful shutdown completed, {count} jobs terminated forcefully"

---

3. Architecture Design

3.1 Concurrency Model

Per-Job Goroutines:

• Each configured job runs in its own dedicated goroutine
• Goroutine lifecycle: calculates next execution time → sleeps until scheduled time → executes command → repeats
• Uses time.NewTimer() for efficient sleep scheduling (not busy-waiting)

Execution Isolation:

• Command executions spawn separate OS processes via os/exec package
• Each command gets isolated stdin/stdout/stderr streams
• Process group management for proper signal propagation

Mutex-Based Overlap Prevention:

• Each job maintains a sync.Mutex for execution serialization
• On scheduled execution, TryLock() is used (non-blocking)
• If lock fails, execution is skipped and counted

Context Propagation:

• Root context created at plugin initialization
• Child contexts with cancellation for each command execution
• Timeout contexts when timeout is configured
• Context cancellation triggers on RoadRunner shutdown signal

3.2 Error Handling Strategy

Configuration Validation:

• Invalid cron expressions: fail plugin initialization with descriptive error
• Missing required fields: fail initialization
• Duplicate job names: fail initialization

Runtime Errors:

• Command not found: log error, increment failure metric, continue scheduling
• Command execution errors: log stderr, increment failure metric, continue scheduling
• Timeout exceeded: terminate process, log warning, increment timeout metric
• System errors (fork failure, etc.): log error, increment failure metric, continue scheduling

No Panic Policy:

• All errors are handled gracefully
• Failed jobs don't affect other scheduled jobs
• Plugin remains operational even if all jobs fail

3.3 Resource Management

Memory Management:

• Minimal memory footprint per job (goroutine stack + scheduler state)
• Command output buffers are limited (e.g., last 1KB of stderr for error logging)
• No unbounded buffering of command output

Process Management:

• Process table cleanup after command completion
• Zombie process prevention via cmd.Wait()
• Proper signal handling for child processes

File Descriptor Management:

• Stdio pipes properly closed after command completion
• No file descriptor leaks

---

4. Performance Considerations

4.1 Scalability Characteristics

Number of Jobs:

• Target: Support 100+ concurrent scheduled jobs efficiently
• Each job consumes: ~4KB goroutine stack + negligible CPU when sleeping
• Memory overhead: O(n) where n = number of jobs

Execution Frequency:

• Support second-level precision (jobs running every second)
• Timer precision: ~milliseconds (limited by OS scheduler)
• No drift accumulation (each execution recalculates next run time)

Concurrent Executions:

• If allow_overlap: true, theoretically unlimited (limited by system resources)
• Recommended: Use allow_overlap: false for resource-intensive commands
• System load depends on commanded process behavior, not plugin overhead

4.2 Bottlenecks and Mitigations

Potential Bottleneck: Many jobs scheduled at exact same time (e.g., all at midnight)

• Mitigation: Add small random jitter (0-1000ms) to identical schedule times
• Alternative: Document best practice of staggering schedules manually

Potential Bottleneck: Long-running commands blocking shutdown

• Mitigation: Configurable grace_period with forceful termination fallback
• Best Practice: Design commands to handle SIGTERM gracefully

Potential Bottleneck: Large command output causing memory pressure

• Mitigation: Stream and discard stdout, only buffer stderr for error reporting (limited to 1KB)

---

5. Integration Points

5.1 RoadRunner Plugin System

Dependencies:

• config.Provider (required): Configuration access
• log.Logger (required): Logging interface
• metrics.Provider (optional): Prometheus metrics registration

Interfaces Implemented:

• endure.Named: Returns plugin name "cron"
• endure.Service: Implements Serve() and Stop() for lifecycle management
• endure.Weight: Returns weight for dependency resolution (low priority, no dependencies on worker pools)

5.2 Configuration Example

version: "3"

cron:
  # Global settings
  timezone: "UTC"
  grace_period: "30s"
  
  # Scheduled jobs
  jobs:
    - name: "cache-warmup"
      command: "php artisan cache:warmup"
      schedule: "0 */6 * * *"  # Every 6 hours
      allow_overlap: false
      timeout: "5m"
    
    - name: "queue-cleanup"
      command: "php artisan queue:prune-failed --hours=48"
      schedule: "@daily"
      allow_overlap: false
      timeout: "10m"
    
    - name: "health-check"
      command: "php artisan app:health-check"
      schedule: "@every 30s"  # Every 30 seconds
      allow_overlap: true  # Allow concurrent checks
      timeout: "5s"
    
    - name: "reports-generator"
      command: "/usr/local/bin/generate-reports.sh"
      schedule: "0 2 * * *"  # Daily at 2 AM
      allow_overlap: false
      timeout: "1h"

# Optional: metrics plugin for observability
metrics:
  address: "0.0.0.0:2112"

5.3 Prometheus Query Examples

Job Success Rate:

rate(roadrunner_cron_executions_total{status="success"}[5m]) 
/ 
rate(roadrunner_cron_executions_total[5m])

Jobs Timing Out:

increase(roadrunner_cron_timeout_total[1h])

Average Execution Duration:

rate(roadrunner_cron_execution_duration_seconds_sum[5m]) 
/ 
rate(roadrunner_cron_execution_duration_seconds_count[5m])

Jobs Currently Running:

sum(roadrunner_cron_running_jobs) by (job_name)

---

6. Acceptance Criteria

6.1 Functional Requirements

Must Have:

• ✓ Execute shell commands based on cron expressions
• ✓ Support standard 5-field and extended 6-field cron syntax
• ✓ Support special expressions (@daily, @hourly, @every, etc.)
• ✓ Prevent overlapping executions when allow_overlap: false
• ✓ Capture and log command exit codes and stderr
• ✓ Graceful shutdown with configurable grace period
• ✓ Expose Prometheus metrics for executions, durations, and errors
• ✓ UTC timezone for all schedule calculations
• ✓ Debug logging for execution events

Should Have:

• ✓ Command execution timeouts with SIGTERM/SIGKILL progression
• ✓ Skip overlapping executions without blocking scheduler
• ✓ Minimal memory footprint (efficient goroutine usage)

Won't Have (Out of Scope):

• Job persistence across RoadRunner restarts
• Dynamic job registration via RPC (runtime job management)
• Job execution history storage
• Retry logic for failed commands
• Multi-instance coordination (distributed locking)
• Different timezones per job

6.2 Performance Targets

• Scheduler Overhead: < 1% CPU usage with 50 jobs scheduled
• Memory per Job: < 10KB overhead per configured job
• Schedule Precision: ±100ms for second-precision schedules
• Startup Time: < 1 second for 100 configured jobs
• Shutdown Time: Respect grace_period with ±500ms accuracy

7. Implementation Considerations

7.1 Cron Library Selection

Recommend using github.com/robfig/cron/v3 for cron expression parsing:

• Mature, well-tested library
• Supports second-precision schedules
• Handles special expressions (@every, @daily, etc.)
• Good timezone support

7.2 Command Execution

Use os/exec.CommandContext for command execution:

• Context-aware cancellation
• Process group management for signal propagation
• Stream handling for stdout/stderr

7.3 Metrics Registration

Follow existing RoadRunner plugin patterns:

• Check if metrics plugin is available via dependency injection
• Gracefully degrade if metrics plugin not present
• Use descriptive metric names with roadrunner_cron_ prefix

7.4 Configuration Validation

Validate during Init() phase before Serve():

• Parse all cron expressions to detect syntax errors early
• Check for duplicate job names
• Validate duration formats (timeout, grace_period)
• Fail fast with clear error messages

---

8. Future Enhancements (Not in Initial Scope)

Potential V2 Features:

• Distributed locking for multi-instance RoadRunner deployments
• RPC interface for runtime job management (add/remove/trigger jobs)
• Job execution history with SQLite storage
• Webhook notifications on job failures
• Retry policies with exponential backoff
• Per-job timezone configuration
• Job dependencies (run Job B after Job A succeeds)

These features are explicitly out of scope for initial implementation to maintain simplicity and focus on core scheduling functionality.