Skip to content

REFACTORING_NOTES.md

Latest commit

 

History

History
379 lines (302 loc) · 11.5 KB

REFACTORING_NOTES.md

File metadata and controls

379 lines (302 loc) · 11.5 KB
Raw

Project Structure Improvements - November 12, 2025

This document summarizes the structural improvements made to the aika-cpp project.

Changes Made

1. Build System Improvements

.gitignore Updates

  • Added comprehensive coverage for build artifacts
  • Added activation_tests, debug_test, simple_activation_tests executables
  • Added node_modules/ directory
  • Added install_manifest.txt
  • Added Python packaging artifacts (.egg-info/, dist/, *.egg)
  • Consolidated duplicate entries and improved organization

CMakeLists.txt Enhancements

  • Added warning for in-source builds (recommends out-of-source builds)
  • Made Python site-packages path dynamic and configurable:
    • Checks PYTHON_SITE_PACKAGES environment variable first
    • Falls back to detected Python installation path
    • Uses .venv as final fallback
  • Updated test paths to reflect new directory structure
  • Improved comments and documentation

Build Documentation (CLAUDE.md)

  • Added comprehensive build instructions for both out-of-source and in-source builds
  • Documented recommended workflow: 
    mkdir -p build && cd build
cmake ..
cmake --build . --target install
  • Updated all test commands to reflect new directory structure
  • Added notes about build configuration options

2. Python Package Organization

New Structure

The python/ directory now has a proper package structure:

python/
├── __init__.py (enhanced with proper package documentation)
├── networks/
│   ├── __init__.py
│   ├── standard_network.py
│   └── transformer.py
├── types/
│   ├── __init__.py
│   ├── softmax_types.py
│   └── dot_product_types.py
├── utils/
│   ├── __init__.py
│   └── aika_debug_utils.py
└── examples/
    ├── __init__.py
    ├── activation_function_examples.py
    └── debug_example.py

Benefits

  • Clear separation of concerns
  • Easier imports: from python.networks import transformer
  • Better discoverability of functionality
  • Follows Python packaging best practices
  • Ready for future setup.py if needed

3. Test Directory Standardization

Old Structure

tests/              # Mixed C++ tests
├── haslink_test.cpp
└── network/
python-tests/       # All Python tests
├── (test files)
└── fields/

New Structure

tests/
├── cpp/
│   ├── haslink_test.cpp
│   └── network/
│       ├── test_runner.cpp
│       ├── activation_test.cpp
│       ├── abstract_activation_test.cpp
│       └── link_latent_test.cpp
└── python/
    ├── (all python test files)
    └── fields/
        └── (field-specific tests)

Benefits

  • Consistent naming convention (no hyphens vs underscores)
  • Clear separation between C++ and Python tests
  • Single tests/ directory as the entry point
  • Easier to navigate and understand
  • Aligns with common project conventions

4. Node Modules Investigation

Findings

  • Found orphaned node_modules/ directory (140+ packages, ~67KB package-lock)
  • No package.json in project root
  • Contains packages including claude, async, broadway, etc.
  • Last modified: May 5, 2025
  • Appears to be leftover from a previous tool or experiment

Status

  • Already covered by updated .gitignore
  • Safe to remove if not needed: rm -rf node_modules/
  • No impact on C++/Python build or functionality

Updated Workflows

Building the Project

# Recommended: Out-of-source build
mkdir -p build && cd build
cmake ..
cmake --build . --target install

# Alternative: In-source build (legacy)
cmake . && make clean && cmake --build . --target install

Running Tests

# Python tests
python -m unittest discover tests/python

# C++ tests
cd build && ctest
# or
./build/activation_tests

Importing Python Modules

# Old way (still works but less organized)
import standard_network

# New way (recommended)
from python.networks import standard_network
from python.types import softmax_types
from python.utils import aika_debug_utils

Migration Notes

For Developers

  1. Test Scripts: Update any scripts that reference python-tests/ to use tests/python/

    # Old
python python-tests/addition-test.py

# New
python tests/python/addition-test.py

  2. Python Imports: Update imports to use new package structure (optional but recommended)

  3. Build Process: Consider switching to out-of-source builds for cleaner project root

Breaking Changes

  • Test file paths have changed (from python-tests/ to tests/python/)
  • Python module organization changed (files moved to subdirectories)
  • CMakeLists.txt test paths updated

Non-Breaking Changes

  • .gitignore improvements (only adds new patterns)
  • CMakeLists.txt build improvements (backward compatible)
  • Documentation updates

Next Steps (Optional)

Short Term

  1. Clean up node_modules/ if not needed: rm -rf node_modules/
  2. Test the new structure with a clean build
  3. Run test suite to ensure everything still works

Medium Term

  1. Consider adding setup.py for Python package installation
  2. Split large C++ implementation files (neuron.cpp ~17K lines, activation.cpp ~10K lines)
  3. Complete Sphinx documentation in /docs
  4. Clean up or populate /ai_docs directory

Long Term

  1. Add API documentation generation (Doxygen)
  2. Set up CI/CD to use new test structure
  3. Consider containerization (Docker) for consistent build environments

Files Modified

  1. .gitignore - Enhanced build artifact coverage
  2. CMakeLists.txt - Out-of-source build support, dynamic paths
  3. CLAUDE.md - Updated build and test commands
  4. python/__init__.py - Enhanced package documentation
  5. python/networks/__init__.py - New file
  6. python/types/__init__.py - New file
  7. python/utils/__init__.py - New file
  8. python/examples/__init__.py - New file

Files/Directories Moved

  • python/*.py → python/*/ (organized into subdirectories)
  • tests/haslink_test.cpp → tests/cpp/haslink_test.cpp
  • tests/network/ → tests/cpp/network/
  • python-tests/ → tests/python/ (entire directory)

Testing Recommendations

After these changes, verify:

  1. Clean build works:

    rm -rf build .venv
python3 -m venv .venv
source .venv/bin/activate
mkdir build && cd build
cmake ..
cmake --build . --target install

  2. Python tests pass:

    python -m unittest discover tests/python

  3. C++ tests pass:

    cd build && ctest

  4. Git status is clean (no untracked build artifacts):

    git status

CI/CD and Obsolete Artifacts Cleanup

Removed obsolete conda packaging and CI workflows:

Removed /conda.recipe directory:

  • Leftover from pybind11 example template
  • Wrong package name: "cmake_example" instead of "aika"
  • Wrong module import: "cmake_example" (doesn't exist)
  • Generic template description not updated for project
  • Last modified April 2025, never customized

Removed .github/workflows/conda.yml:

  • Conda-based CI workflow
  • Referenced removed pybind11 submodule (would fail)
  • Tried to install non-existent "cmake_example" package
  • Not needed - project uses pip-based distribution

Removed .appveyor.yml:

  • AppVeyor CI configuration (obsolete)
  • Referenced removed pybind11 submodule (line 16: git submodule update)
  • Wrong package name: "cmake_example" instead of "aika"
  • Redundant - GitHub Actions workflows already cover Windows builds
  • AppVeyor no longer needed since GitHub Actions provides all CI

Removed .venv/ from git tracking:

  • Virtual environment was incorrectly committed to repository
  • Removed from git with git rm -r --cached .venv
  • Directory still exists locally for development use
  • Already covered by .gitignore (line 60)
  • Virtual environments should never be version controlled

Fixed remaining CI workflows:

  • pip.yml: Removed submodules: true (line 23)
  • wheels.yml: Removed submodules: true (lines 20, 45)
  • format.yml: No changes needed

Current CI Strategy:

  • pip-based installation via setup.py
  • Works with removed pybind11 submodule (uses system pybind11)
  • PyPI wheels generation via cibuildwheel
  • Pre-commit formatting checks

External Dependencies Cleanup

Removed pybind11 git submodule and properly declared it as an external dependency:

Removed pybind11 git submodule:

  • Was vendored as git submodule (outdated v2.9.2)
  • Removed from repository (14MB including all history)
  • Removed .gitmodules file (no longer needed)
  • Build already used system pybind11 v2.13.6 via find_package()

Updated dependency management:

  • Added pybind11>=2.10.0 to setup.py dependencies
  • Will be installed automatically via pip install -e .
  • Can also be installed via system package managers (brew, apt, etc.)
  • CMake uses find_package(pybind11 REQUIRED) - standard approach

Benefits:

  • Smaller repository size
  • No git submodule maintenance
  • Users get latest pybind11 version
  • Standard dependency management
  • Easier to update pybind11 version

Updated documentation:

  • README.md now lists pybind11 in Prerequisites section
  • CLAUDE.md updated with prerequisite information
  • Build process unchanged (CMake already used find_package)

Documentation Cleanup

Removed incomplete and unused documentation directories:

Removed /docs directory:

  • Contained skeleton Sphinx setup from 2016
  • Only had a cmake_example.rst stub
  • Not being used for actual project documentation
  • Project already has comprehensive docs in README.md and specs/

Removed /ai_docs directory:

  • Completely empty directory
  • No content or purpose

Removed /node_modules directory:

  • Orphaned Node.js packages (14MB, 140+ subdirectories)
  • Leftover from previous tool experiment (May 2025)
  • No package.json in project root
  • Not needed for C++/Python build

Documentation Strategy:

  • Primary: README.md (user guide, installation, examples)
  • Developer: CLAUDE.md (build commands, code style)
  • Specifications: specs/ directory (formal architecture docs)
  • All maintained in markdown format

Python Package Installation (Added Later)

To eliminate the need for sys.path.append() in test files and Python modules, we made the Python helpers an installable package:

Created setup.py:

  • Makes the python/ directory an installable package
  • Allows using pip install -e . for development mode
  • Automatically installs dependencies (parameterized, etc.)

Benefits:

  • No more sys.path.append() boilerplate in every file
  • Clean imports: from python.networks.standard_network import ...
  • Standard Python development workflow
  • Dependencies managed via pip

Updated workflow:

pip install -e .  # Install once
# Now all imports work without sys.path manipulation

Files cleaned up:

  • Removed sys.path.append() from all 28 test files
  • Removed sys.path.append() from all 7 Python module files
  • Updated all imports to use new package structure

Summary

These improvements modernize the project structure while maintaining backward compatibility where possible. The changes make the codebase more maintainable, easier to navigate, and aligned with industry best practices for C++/Python hybrid projects.

Key benefits:

  • Cleaner git status (better .gitignore)
  • Flexible build system (out-of-source builds)
  • Organized Python code (proper package structure)
  • Consistent test organization (unified tests/ directory)
  • Better documentation (updated CLAUDE.md)
  • Clean imports (no sys.path manipulation needed)