Skip to content

README.md

Latest commit

 

History

History
405 lines (287 loc) · 13.9 KB

README.md

File metadata and controls

405 lines (287 loc) · 13.9 KB

Science Projects Management System (SPMS) Backend

Note: This is the backend component of the Science Projects monorepo. See the root README for the complete system overview. This was merged from a separate repo into monorepo.

What is SPMS?

The Science Projects Management System (SPMS) is a comprehensive web application designed to manage scientific research projects for the Department of Biodiversity, Conservation and Attractions (DBCA) in Western Australia.

SPMS provides a centralised platform for tracking research projects, managing documentation, coordinating team members, and maintaining project records throughout their lifecycle.

Purpose

SPMS serves as the primary tool for:

Project Management: Track scientific research projects from conception through completion, including project proposals, approvals, progress updates, and final reports.

Document Management: Store and organise project-related documents, including project plans, concept plans, progress reports, student progress reports, and project closures.

Annual Report Generation: Compile data from progress reports and stakeholders to generate a comprehensive annual report for demonstrating scientific achievements and securing further funding.

Team Collaboration: Coordinate research teams across multiple business areas and locations, manage team member roles and permissions, and facilitate communication through integrated messaging.

Resource Tracking: Monitor project resources, budgets, and timelines to ensure efficient allocation and utilisation of departmental resources.

Reporting and Analytics: Generate reports on project status, resource utilisation, and research outcomes to support decision-making and strategic planning.

Who Uses SPMS?

Research Scientists: Create and manage research projects, upload findings and documentation, collaborate with team members, and track project progress.

Project Leads: Oversee the project to provide final approval before Business Area Lead review.

Business Area Leads: Oversee all projects within their Business Area to provide final approval before Directorate review.

Directorate Reviewer: Key stakeholder who manages annual report generation by providing final approval on all documents and editing the Annual Report.

Administrators / Maintainers: Assists Directorate Reviewer with project approvals, deletions, user accounts and permissions, maintaining system configuration, and providing user support.

Key Features

Project Management

  • Create and track research projects with detailed metadata
  • Categorise projects by research area, status, and priority
  • Link projects to geographic locations with map visualisation
  • Track project milestones and deliverables
  • Manage project lifecycle from proposal to completion

Document Management

  • Upload and store project documents with metadata
  • Support for multiple file types (PDFs, images, data files)
  • Version control and document history
  • Secure file storage with access controls
  • Document search and filtering

User Management

  • Role-based access control (staff, researchers, external users)
  • User profiles with employment and education history
  • Integration with DBCA IT Assets system for user data
  • Caretaker system for managing staff who depart or go on leave
  • User search and directory

Collaboration Features

  • Project team management with role assignments
  • Comments and discussions on projects
  • Notifications for project updates via email
  • Shared document access

Geographic Integration

  • Project location mapping with interactive maps
  • Geographic search and filtering
  • Location-based project visualisation
  • Integration with DBCA location database

Reporting and Analytics

  • Project status dashboards
  • Resource utilisation reports
  • Research output tracking
  • Custom report generation
  • Data export capabilities

Staff Profiles

  • Integrated sister-app
  • Provides public-facing data which is maintained per-user
  • Details a user's projects and publications
  • Details a user's expertise, about, education and employment history

System Architecture

Backend (This Repository)

Django REST Framework API providing:

  • RESTful API endpoints for all system functionality
  • PostgreSQL database for data persistence
  • User authentication and authorisation
  • File storage and media management
  • Integration with external systems (IT Assets, Library API)
  • Background task processing
  • Comprehensive test coverage (>80%)

Frontend

React-based single-page application providing:

  • Modern, responsive user interface
  • Interactive project management tools
  • Document upload and management
  • Real-time updates and notifications
  • Map-based project visualisation

See: Frontend README

Infrastructure

  • Kubernetes deployment using Kustomize
  • Docker containerisation
  • PostgreSQL database
  • File storage for media and documents
  • CI/CD pipeline with GitHub Actions

Technology Stack

Backend:

  • Python
  • Django
  • Django REST Framework
  • PostgreSQL
  • Poetry (dependency management)

Testing:

  • pytest with 80%+ coverage requirement
  • Test sharding for fast execution
  • Property-based testing for critical paths

Code Quality:

  • black (code formatting)
  • isort (import sorting)
  • flake8 (linting)
  • bandit (security scanning)
  • Pre-commit hooks for automated checks

Deployment:

  • Docker containers
  • Kubernetes with Kustomize
  • GitHub Actions CI/CD
  • Automated testing and deployment

Documentation

Complete Documentation →

Quick Links

Getting Started:

Development:

Architecture:

Deployment:

Operations:

Security:

Contributing

  1. Create a feature branch: git checkout -b feature/my-feature
  2. Make changes following our Code Style
  3. Write tests (maintain >80% coverage)
  4. Run pre-commit hooks: pre-commit run --all-files
  5. Push and create a pull request

See Feature Development for the complete workflow.

Testing

Quick Start

# Fast feedback - unit tests only (26 seconds)
poetry run pytest -m unit

# Integration tests (44 seconds)
poetry run pytest -m integration

# Full test suite (68 seconds)
poetry run pytest

# With coverage report
poetry run pytest --cov=. --cov-report=html

Test Categories

Tests are organised into three categories using pytest markers:

Unit Tests (@pytest.mark.unit):

  • Pure Python logic, no database
  • Serializer validation
  • Utility functions
  • Fast execution (< 30 seconds total)

Integration Tests (@pytest.mark.integration):

  • View tests with database
  • Service layer tests
  • API endpoint tests
  • Medium execution (< 2 minutes total)

Slow Tests (@pytest.mark.slow):

  • Complex workflows
  • Multi-model transactions
  • Admin action tests
  • Slower execution (< 3 minutes total)

Running Specific Test Categories

# Unit tests only - fastest feedback
poetry run pytest -m unit

# Integration tests only
poetry run pytest -m integration

# Slow tests only
poetry run pytest -m slow

# All except slow tests
poetry run pytest -m "not slow"

# Specific app with category
poetry run pytest users/tests/ -m unit

Development Workflow

During Development (rapid iteration):

poetry run pytest -m unit  # 26 seconds

Before Committing (thorough validation):

poetry run pytest -m integration  # 44 seconds

Before Pushing (final check):

poetry run pytest  # 68 seconds

Coverage Reports

Coverage reports are available in htmlcov/index.html after running tests with coverage:

# Generate coverage report
poetry run pytest --cov=. --cov-report=html

# Open in browser
open htmlcov/index.html  # macOS
xdg-open htmlcov/index.html  # Linux

Additional Test Options

# Run specific app tests
poetry run pytest agencies/tests/

# Run in parallel (faster)
poetry run pytest -n auto

# Verbose output
poetry run pytest -v

# Stop on first failure
poetry run pytest -x

# Show test durations
poetry run pytest --durations=10

Test Documentation

For detailed testing guidelines, see:

Maintenance Commands

Django management commands for database maintenance and troubleshooting.

Fix Database Sequences

After data migrations or bulk imports, PostgreSQL sequences can become out of sync, causing "duplicate key" errors. Use the fix_sequences command to automatically fix all sequences:

# Preview what will be fixed (dry run)
poetry run python manage.py fix_sequences --dry-run

# Fix all sequences
poetry run python manage.py fix_sequences

When to run:

  • After importing data from another database
  • After running data migrations with explicit IDs
  • When encountering "duplicate key value violates unique constraint" errors
  • After bulk creating records with custom IDs

Output example:

Checking 60 models...
✓ ProjectMember                  | Fixed: 3287 → 3867 (max_id: 3866)
✓ Project                        | Fixed: 1047 → 1292 (max_id: 1291)

======================================================================
Fixed:   37 sequences
Skipped: 20 sequences (already correct)

✓ All sequences have been fixed!

See Management Commands README for complete documentation.

Code Quality

Pre-commit hooks automatically run on every commit:

# Install hooks
poetry run pre-commit install

# Run manually
poetry run pre-commit run --all-files

Hooks include:

  • black (code formatting)
  • isort (import sorting)
  • flake8 (linting)
  • bandit (security checks)
  • django-upgrade (Django best practices)

History

SPMS was originally developed by Florian Mayer as the Science Directorate Information System (SDIS). The current version represents a completely rewritten modernisation of the system, rebuilt with current technologies and additional features, following an ETL migration step.

Original SDIS: https://github.com/dbca-wa/sdis

The system has evolved to meet the growing needs of DBCA's research programs, incorporating feedback from users and adapting to changing requirements over several years of active use.

Organisation

Department of Biodiversity, Conservation and Attractions (DBCA)

DBCA is the Western Australian government department responsible for managing the state's parks, wildlife, and biodiversity. The department conducts extensive scientific research to support conservation efforts and environmental management.

SPMS supports DBCA's research mission by providing the infrastructure needed to manage complex, long-term research projects across diverse scientific disciplines and geographic locations. The key deliverable is the Annual Report.

Support and Maintenance

SPMS is actively maintained by Ecoinformatics' development team. The system receives regular updates for:

  • Security patches and dependency updates
  • New features based on user feedback
  • Performance optimisations
  • Bug fixes and improvements
  • Integration with DBCA systems

Related Systems

IT Assets: DBCA's internal system for managing IT resources and user accounts. SPMS integrates with IT Assets to synchronise user data and authentication under SSO.

Library System: DBCA's publication management system. SPMS integrates to link research projects with published outputs.

Geographic Information Systems: SPMS integrates with DBCA's GIS infrastructure for location-based project management and visualisation. Current map setup on the frontend utilises a dump of this data.

License

Copyright (c) Department of Biodiversity, Conservation and Attractions (DBCA), Western Australia.

This software is proprietary and confidential. It is developed and maintained for internal use by DBCA. While the source code is publicly visible for transparency and collaboration with authorised partners, it is not licensed for reproduction, modification, or use by external parties without explicit written permission from DBCA.

Unauthorised copying, distribution, modification, or use of this software is strictly prohibited.

Support

For issues and questions: