Feature Request: ETL Testing Harness

[chriselsen]

Overview

This proposal outlines the implementation of a standardized testing harness for the @tak-ps/etl library. The goal is to provide a reusable testing framework that simplifies the process of writing unit tests for ETL implementations that rely on external API calls.

Problem Statement

ETL implementations built on the @tak-ps/etl library frequently interact with external APIs, making them challenging to test without:

1. Making actual API calls (which can be slow, unreliable, and potentially costly)
2. Writing extensive mocking code for each ETL project
3. Duplicating testing patterns across multiple repositories

Currently, there is no standardized approach to testing ETL implementations, leading to inconsistent test coverage and duplicated effort.

Proposed Solution

Add a testing module to the etl-base repository that provides utilities for mocking the core functionality of the ETL library, particularly the fetch function and TypedResponse class.

Implementation Details

1. Create a new testing directory in the etl-base repository
2. Implement a createETLTestHarness function that:
   • Mocks the fetch function to return predefined responses
   • Mocks the TypedResponse class to handle schema validation
   • Provides utilities for creating mock ETL tasks with predefined environment variables

Code Structure

etl-base/
├── src/
│   ├── ...
│   └── testing/
│       └── index.ts  # Main testing utilities
├── package.json
└── ...

Implementation

// src/testing/index.ts
import { jest } from '@jest/globals';
import ETL from '../index';
import { SchemaType, DataFlowType } from '../index';

/**
 * Creates a reusable ETL test harness
 * @param mockResponses - Map of URL patterns to mock responses
 * @returns Object containing testing utilities
 */
export function createETLTestHarness(mockResponses = new Map<RegExp, any>()) {
  // Mock the @tak-ps/etl module
  jest.mock('@tak-ps/etl', () => {
    const originalModule = jest.requireActual('@tak-ps/etl');
    
    // Create a mock TypedResponse class
    const MockTypedResponse = jest.fn().mockImplementation((response) => {
      return {
        typed: jest.fn().mockImplementation((schema) => {
          return Promise.resolve(response);
        })
      };
    });
    
    // Create a mock fetch function
    const mockFetch = jest.fn().mockImplementation((url, options) => {
      // Find matching response based on URL pattern
      const urlString = url.toString();
      for (const [pattern, response] of mockResponses.entries()) {
        if (pattern.test(urlString)) {
          return Promise.resolve(new MockTypedResponse(response));
        }
      }
      
      // Default response if no pattern matches
      return Promise.resolve(new MockTypedResponse({}));
    });
    
    // Return the modified module
    return {
      ...originalModule,
      fetch: mockFetch,
      TypedResponse: MockTypedResponse
    };
  });
  
  // Return utilities for testing
  return {
    /**
     * Create a mock environment for an ETL task
     * @param TaskClass - The ETL task class to test
     * @param mockEnv - Mock environment variables
     * @returns Configured task instance
     */
    createMockTask: <T extends ETL>(TaskClass: new () => T, mockEnv: any = {}) => {
      const task = new TaskClass();
      jest.spyOn(task, 'env').mockResolvedValue(mockEnv);
      return task;
    },
    
    /**
     * Add a mock response for a specific URL pattern
     * @param urlPattern - RegExp pattern to match URLs
     * @param response - Mock response to return
     */
    addMockResponse: (urlPattern: RegExp, response: any) => {
      mockResponses.set(urlPattern, response);
    }
  };
}

// Re-export types that are commonly needed in tests
export { SchemaType, DataFlowType };

Package.json Updates

{
  "name": "@tak-ps/etl",
  "version": "x.y.z",
  "exports": {
    ".": "./dist/index.js",
    "./testing": "./dist/testing/index.js"
  },
  "files": [
    "dist"
  ],
  "devDependencies": {
    "@jest/globals": "^29.0.0",
    "jest": "^29.0.0",
    "ts-jest": "^29.0.0"
  }
}

Usage Example

Here's how the testing harness would be used in an ETL implementation:

// task.test.ts in an ETL project
import { jest, describe, test, expect, beforeEach } from '@jest/globals';
import { createETLTestHarness, SchemaType, DataFlowType } from '@tak-ps/etl/testing';
import Task from './task';

// Create test harness with mock responses
const testHarness = createETLTestHarness(new Map([
  [/api\.example\.com/, {
    // Mock response data
    data: [
      { id: 1, name: 'Test Item' }
    ]
  }]
]));

// Import the mocked fetch after setting up the test harness
import { fetch } from '@tak-ps/etl';

describe('Example ETL Task', () => {
  let task;
  
  beforeEach(() => {
    jest.clearAllMocks();
    
    // Create a mock task with environment variables
    task = testHarness.createMockTask(Task, {
      'API_URL': 'https://api.example.com/data',
      'API_KEY': 'test-key'
    });
  });
  
  test('control method processes data correctly', async () => {
    // Call the method under test
    await task.control();
    
    // Verify fetch was called with correct URL and headers
    expect(fetch).toHaveBeenCalledWith(
      expect.objectContaining({
        href: expect.stringContaining('api.example.com')
      }),
      expect.objectContaining({
        headers: {
          'Authorization': 'Bearer test-key'
        }
      })
    );
  });
});

Benefits

1. Standardization: Provides a consistent approach to testing ETL implementations
2. Reusability: Eliminates the need to rewrite mocking code for each ETL project
3. Simplicity: Makes it easier to write comprehensive tests for ETL implementations
4. Maintainability: Centralizes testing utilities in one location
5. Efficiency: Reduces the time required to set up testing for new ETL projects

Implementation Plan

1. Create the testing directory in the etl-base repository
2. Implement the createETLTestHarness function
3. Update the package.json to export the testing module
4. Add documentation and usage examples
5. Create example tests for the etl-template repository

Conclusion

Adding a standardized testing harness to the etl-base repository will significantly improve the testability of ETL implementations and promote consistent testing practices across all ETL projects. This will lead to more reliable ETL implementations and reduce the effort required to maintain them.