Daily Test Coverage Improver - Research and Plan

[github-actions[bot]]

Executive Summary

I've analyzed the backend.ai-webui repository with a focus on the /react and /packages/backend.ai-ui directories. The repository has significant testing infrastructure in place but currently has very low test coverage (~4% file coverage) with only 19 test files covering 482 source files.

Current State Analysis

Test Infrastructure

Testing Framework:

• Jest 30.1.3 with React Testing Library
• Tests located alongside source files with .test.ts or .test.tsx extensions
• Coverage configuration already in place

Coverage Steps Configuration:

• Configuration exists at .github/actions/daily-test-improver/coverage-steps/action.yml
• Builds both React project and backend.ai-ui package
• Generates coverage reports for each project separately
• Attempts to merge coverage reports into combined report

Current Test Coverage

Existing Test Files (19 total):

React Project (14 tests):

• react/src/hooks/ - 6 test files (useMemoWithPrevious, backendai, useBackendAIImageMetaData, useControllableState, useResourceLimitAndRemaining, index)
• react/src/components/ - 3 test files (EnvVarFormList, SessionFormItems/ResourceAllocationFormItems, MyResourceWithinResourceGroup)
• react/src/helper/ - 5 test files (resultTypes, csv-util, graphql-transformer, big-number, index)

backend.ai-ui Package (11 tests):

• packages/backend.ai-ui/src/hooks/ - 1 test file (useIntervalValue)
• packages/backend.ai-ui/src/components/ - 9 test files (BAIBulkEditFormItem, BAIButton, BAIBackButton, BAIUnmountAfterClose, BAIStatistic, fragments/BAIDomainSelect, BAIFlex, BAIPropertyFilter, BAITag)
• packages/backend.ai-ui/src/helper/ - 1 test file (index)

Test Coverage Ratio:

• 482 source files (React + backend.ai-ui combined)
• 19 test files
• Coverage: ~4% of files have test coverage

Testing Challenges Encountered

1. Environment Setup Issues:

   • pnpm dependency installation requires --no-frozen-lockfile due to lockfile mismatch
   • Electron postinstall fails (SSL/network issues) but doesn't affect testing
   • Test runs fail due to mock configuration issues (jest not defined in matchMedia.mock.js)

2. Module Resolution:

   • Some tests failing due to Relay compiler not being run (relay-compiler: not found)
   • Jest/Vitest mock compatibility issues in test setup files

3. Configuration:

   • matchMedia.mock.js uses Jest API (jest.fn()) instead of Vitest API (vi.fn())
   • May indicate transition from Jest to Vitest not fully complete

Testing Strategy and Plan

Phase 1: Fix Test Environment (Priority: HIGH)

Actions Needed:

1. Fix matchMedia.mock.js to use proper mock API (determine if using Jest or Vitest)
2. Ensure Relay compiler runs before tests or tests can run without it
3. Verify coverage report generation works end-to-end
4. Document required environment setup in coverage-steps.log

Phase 2: Low-Hanging Fruit - Utility Functions (Priority: HIGH)

Target: Helper/Utility Functions (Easy to test, high impact)

React helper/ directory candidates (0% coverage):

• bui-language.ts - Simple object exports, easy to test
• customThemeConfig.ts - Theme loading and configuration
• react-to-webcomponent.tsx - Web component wrapper
• const-vars.ts - Constants file (likely trivial but should verify exports)

backend.ai-ui helper/ directory candidates:

• useDebouncedDeferredValue.ts - Custom hook combining debounce and deferred value
• reactQueryAlias.ts - React Query wrappers with custom logic

Estimated Impact:

• Adding 4-6 new test files
• Coverage increase: ~1-2%
• Risk: LOW
• Effort: LOW-MEDIUM

Phase 3: Component Testing - BAI UI Library (Priority: MEDIUM)

Target: backend.ai-ui Components (Already has some test patterns established)

High-Value Candidates:

• BAIModal - Modal component (widely used)
• BAITable - Table component with custom features
• BAIText - Text component with ellipsis features
• BAISelect - Select component variations
• BAICard - Card component with status

Estimated Impact:

• Adding 5-10 new component test files
• Coverage increase: ~2-3%
• Risk: MEDIUM
• Effort: MEDIUM

Phase 4: Hook Testing - React Project (Priority: MEDIUM)

Target: React hooks/ (Many uncovered hooks)

High-Value Candidates:

• useBAINotification.tsx - Notification management
• useBackendAIAppLauncher.tsx - App launcher logic
• useCurrentProject.tsx - Project context
• useWebUIMenuItems.tsx - Menu generation
• useStartSession.tsx - Session creation

Estimated Impact:

• Adding 5-8 new hook test files
• Coverage increase: ~2-3%
• Risk: MEDIUM-HIGH (may require mocking Relay queries)
• Effort: MEDIUM-HIGH

Phase 5: Page Testing (Priority: LOW)

Target: Pages (Most complex, requires extensive mocking)

Note: Pages are integration-heavy and may be better covered by E2E tests (Playwright tests exist in e2e/ directory). Recommend focusing on unit/component tests first.

Commands for Build, Test, and Coverage

Install dependencies:

cd /home/runner/work/backend.ai-webui/backend.ai-webui
npx pnpm@latest install --no-frozen-lockfile --ignore-scripts

Build React GraphQL types (required before testing):

cd react
npx pnpm run relay

Run React tests with coverage:

cd react
NODE_OPTIONS='--no-deprecation --experimental-vm-modules' npx jest --coverage --coverageDirectory=coverage

Run backend.ai-ui tests with coverage:

cd packages/backend.ai-ui
NODE_OPTIONS='--no-deprecation --experimental-vm-modules' npx jest --coverage --coverageDirectory=coverage

Generate combined coverage report:

# After running both test suites
mkdir -p coverage/combined
npx nyc merge coverage/temp coverage/combined/coverage-final.json
npx nyc report --temp-dir coverage/combined --reporter=html --reporter=lcov --reporter=text-summary --report-dir coverage/combined

Test Organization

Tests follow colocation pattern:

• Test files sit alongside source files
• Naming: {SourceFileName}.test.{ts|tsx}
• Structure mirrors source code organization

Example:

src/
  components/
    BAIButton.tsx
    BAIButton.test.tsx
  hooks/
    useInterval.ts
    useInterval.test.ts

Opportunities for Significant Coverage Increase

1. Automated Test Generation:

   • Use AI to generate basic test scaffolds for utility functions
   • Create test templates for common patterns (components, hooks)

2. Snapshot Testing:

   • Add snapshot tests for presentational components
   • Quick wins for visual regression protection

3. Integration with CI:

   • Enforce minimum coverage thresholds
   • Fail builds if coverage decreases
   • Generate coverage badges

4. Test Utilities:

   • Create test helpers for common Relay mocking patterns
   • Build custom render utilities for testing with providers

5. Documentation:

   • Document testing patterns and best practices
   • Create examples for testing common scenarios (hooks with Relay, components with Ant Design)

Questions for Maintainers

1. Test Framework: Is the project using Jest or Vitest? The matchMedia.mock.js uses Jest API but some configurations suggest Vitest.

2. Priority Areas: Which components/features are most critical and should be prioritized for test coverage?

3. Coverage Goals: What is the target coverage percentage? Should we aim for 80%, or is there a different goal?

4. Relay Testing: What's the preferred approach for mocking Relay queries in tests? Should we use relay-test-utils or another approach?

5. Breaking Changes: Are breaking changes acceptable if tests reveal bugs? Or should bugs be reported separately?

How to Control this Workflow

You can control this workflow using the following commands:

Disable the workflow:

gh aw disable daily-test-improver --repo lablup/backend.ai-webui

Enable the workflow:

gh aw enable daily-test-improver --repo lablup/backend.ai-webui

Run the workflow manually (with optional repeats):

# Run once
gh aw run daily-test-improver --repo lablup/backend.ai-webui

# Run multiple times (useful for testing or making rapid progress)
gh aw run daily-test-improver --repo lablup/backend.ai-webui --repeat 5

View workflow logs:

gh aw logs daily-test-improver --repo lablup/backend.ai-webui

Provide Feedback:

• Add comments to this discussion with feedback or priorities
• Close this discussion if you don't want automated test improvements
• React to specific items in the plan to indicate priority

What Happens Next

The next time this workflow runs, Phase 2 will be performed:

• The workflow will analyze the codebase to create coverage steps configuration
• It will attempt to validate the configuration by running the coverage steps
• A pull request will be created with the configuration file
• If running in "repeat" mode, the workflow will automatically proceed after PR creation

After Phase 2 completes and the PR is merged, Phase 3 will begin on subsequent runs:

• The workflow will select specific areas to improve test coverage
• It will write new tests following the established patterns
• Draft pull requests will be created with coverage improvements
• Coverage metrics will be tracked and reported

Humans can review this research and add comments before the workflow continues to Phase 2.

AI generated by Daily Test Coverage Improver