# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.0.0] - 2025-10-27

### BREAKING CHANGES

#### 3D Input Requirement for Multitaper Class

**BREAKING CHANGE**: `Multitaper` now requires 3D input arrays with explicit `(n_time_samples, n_trials, n_signals)` shape. Previously ambiguous 2D arrays now raise informative `ValueError`.

- **Migration Required**: Use the new `prepare_time_series()` helper function to convert 1D/2D arrays to 3D format
- **Why**: Eliminates dangerous ambiguity where `(n_time, n)` could mean either:
  - `(n_time, 1 trial, n signals)` OR
  - `(n_time, n trials, 1 signal)`

  Silent misinterpretation produces scientifically incorrect results.

**Migration Example**:
```python
# Before (ambiguous)
mt = Multitaper(eeg_data, sampling_frequency=1000)

# After (explicit)
from spectral_connectivity.transforms import prepare_time_series
eeg_3d = prepare_time_series(eeg_data, axis='signals')  # or axis='trials'
mt = Multitaper(eeg_3d, sampling_frequency=1000)
```

#### Nyquist Frequency Now Included for Even-Length FFTs

**BREAKING CHANGE**: Corrected frequency bin indexing to include Nyquist frequency for even-length FFTs

- **Impact on results**:
  - Even-length FFTs (N=1024): Now return 513 frequencies instead of 512 (adds Nyquist bin)
  - Odd-length FFTs (N=1023): No change (still return 512 frequencies)
- **Affected functions**: All connectivity measures using `@_non_negative_frequencies` decorator
- **Scientific justification**: The Nyquist frequency (sampling_rate/2) represents valid spectral information that should not be discarded
- See CHANGELOG "Unreleased" section for complete details and migration guidance

#### Development Tooling Changes

- **BREAKING**: Migrated from `black` to `ruff format` for code formatting
- **BREAKING**: Migrated from `flake8`/`pydocstyle` to `ruff` for linting
- Development workflows now use `ruff format` and `ruff check` commands
- All formatting/linting configurations moved to `pyproject.toml` under `[tool.ruff]`
- Developers must update their tooling: `pip install ruff` (100x faster than previous tools)

### Added
- `prepare_time_series()` helper function for safe dimension handling:
  - Converts 1D/2D arrays to required 3D format
  - Explicit `axis` parameter to clarify dimension meaning
  - Prevents ambiguous dimension interpretation
  - **Required for migration to v2.0.0**
- Comprehensive notebook snapshot testing with Syrupy:
  - 27 snapshot tests covering all tutorial notebook examples
  - Tests validate output shapes, data types, and numerical properties
  - Ensures tutorial examples remain accurate across releases
  - Automatically catches breaking changes in example code
  - Uses Syrupy's snapshot testing for efficient test maintenance
- **Test infrastructure** (`tests/conftest.py`):
  - Added pytest fixture to reset numpy random state before each test
  - Ensures consistent test reproducibility with fixed seed (42)
  - Helps prevent test ordering issues from shared random state
  - Auto-runs before every test without explicit fixture declaration
- **Physical constants with scientific rationale** (`spectral_connectivity/transforms.py`):
  - `MIN_EIGENVALUE_THRESHOLD = 0.9`: Minimum eigenvalue for low-bias tapers
    - Documents that 90% of taper energy must be in main lobe to reduce spectral leakage
    - Reference: Thomson (1982), "Spectrum estimation and harmonic analysis"
  - `TAPER_MULTIPLIER = 2.0`: Multiplier for calculating number of tapers
    - Documents the theoretical basis: ~2*NW orthogonal Slepian sequences are well-concentrated
    - Reference: Slepian (1978), "Prolate spheroidal wave functions"
  - All magic numbers replaced with named constants throughout codebase
  - Docstrings updated to reference constants while preserving readability
- Comprehensive test suite for advanced connectivity measures (`tests/test_advanced_connectivity.py`):
  - 18 test methods covering `canonical_coherence()`, `global_coherence()`, and `group_delay()`
  - Tests validate output shapes, value ranges, and mathematical properties (symmetry, antisymmetry)
  - Integration tests for complete Multitaper → Connectivity → advanced measures workflow
  - Tests handle edge cases: different group sizes, single signals, non-contiguous labels
  - Graceful handling of group_delay edge case when no frequencies are significant
- Parameter helper functions for guided multitaper analysis:
  - `estimate_frequency_resolution()`: Calculate frequency resolution from parameters
  - `estimate_n_tapers()`: Calculate number of tapers from time-halfbandwidth product
  - `suggest_parameters()`: Get parameter recommendations for your data
  - `MultitaperParameters` TypedDict for type-safe parameter handling
- `summarize_parameters()` method to `Multitaper` class:
  - Human-readable summary of all analysis parameters
  - Shows computed values (n_tapers, frequency_resolution, n_windows)
  - Displays overlap percentage for windowing
  - Formatted for terminal/notebook output
- Enhanced `time_halfbandwidth_product` docstring with formulas and practical guidance:
  - Mathematical relationship to frequency resolution and n_tapers
  - Typical values (NW=2,3,4,5+) with trade-off explanations
  - Examples for achieving target resolutions (1 Hz, 5 Hz, 10 Hz)
  - Cross-references to helper functions
- Comprehensive test suite for parameter helpers (`test_parameter_helpers.py`):
  - 22 tests covering all helper functions
  - Domain-specific tests (EEG, LFP typical parameters)
  - Edge cases (conflicting parameters, impossible resolutions)
  - Consistency checks with actual `Multitaper` behavior

### Changed
- **Complete type hint coverage** (`spectral_connectivity/transforms.py`, `spectral_connectivity/connectivity.py`):
  - Added type hints to ALL 28 previously untyped functions:
    - `transforms._make_tapers()`: Added parameter and return types
    - `connectivity.py`: 27 functions including all connectivity measures and helper functions
  - Used `TYPE_CHECKING` block to avoid circular imports for Multitaper type
  - Added `Literal` types for `multiple_comparisons_method` parameter
  - Fixed `Optional`/`None` handling in `_get_independent_frequency_step()` and `_bandpass()`
  - Fixed phase_lag_index return type (extract real part from complex result)
  - Enabled `disallow_untyped_defs = true` for transforms and connectivity modules
  - All 9 modules now have strict type checking enabled
  - 100% of public API is now fully type-annotated
- **Test coverage improvements**:
  - Overall coverage increased from 85% to **88%**
  - `connectivity.py` coverage improved from 71% to **93%**
  - Fixed snapshot tests after Nyquist frequency bin correction
  - Updated test assertions to use correct frequency bin calculation: `N//2 + 1` instead of `(N+1)//2`
- Improved method discovery in `multitaper_connectivity()` wrapper:
  - Replaced `dir()` with `inspect.getmembers(predicate=inspect.isfunction)` for type-safe method filtering
  - Automatically excludes properties and classmethods (more robust)
  - Renamed `bad_methods` to `excluded_methods` with clear categorization
  - Changed from list to set for O(1) membership testing
  - Added test `test_method_discovery_with_inspect()` to verify behavior
- Documented design decision for `adjust_for_multiple_comparisons()`:
  - Replaced TODO comment with clear explanation of current behavior
  - Explained why axis parameter is not implemented (standard approach treats all p-values as single family)
  - Left open for future enhancement if needed

### Removed
- All TODO comments from codebase (2 resolved)

### Changed
- **Development tooling modernization**:
  - Migrated from `black` to `ruff format` for code formatting (100x faster)
  - Migrated from `flake8`/`pydocstyle` to `ruff check` for linting
  - Updated GitHub Actions CI to use `ruff format --check` and `ruff check`
  - Updated [CLAUDE.md](CLAUDE.md) with new ruff commands
  - All formatting and linting now unified under single fast tool
- **Code quality improvements**:
  - Applied `ruff format` to all source files (10 files reformatted)
  - Fixed 19 auto-fixable ruff linting issues
  - All source code and tests now pass `ruff check` with zero warnings
  - Type hints improved with better union handling for `detrend()` function
- Updated tutorial notebooks to use `prepare_time_series()` for 3D input
- Improved random number generation in tests for better isolation

### Fixed
- **MyPy type annotation error** in `detrend()` function (`transforms.py:1867-1876`):
  - Fixed union type handling for `bp` parameter (now `int | list[int] | NDArray[np.integer]`)
  - Added support for list input in addition to int and ndarray
  - Eliminated unreachable code warnings from MyPy
  - All 9 source files now pass strict mypy type checking

#### **Nyquist Frequency Fix (from earlier release)**

- **Critical bug fix**: Corrected frequency bin indexing to include Nyquist frequency for even-length FFTs
  - **Affected code**: Changed frequency indexing from `(N+1)//2` to `N//2 + 1` in three locations:
    - `_non_negative_frequencies` decorator (line 107)
    - `canonical_coherence` method (line 638)
    - `_estimate_spectral_granger_prediction` function (line 2108)
  - **Impact on results**:
    - Even-length FFTs (N=1024): Now return 513 frequencies instead of 512 (adds Nyquist bin)
    - Odd-length FFTs (N=1023): No change (still return 512 frequencies)
  - **Affected functions**: All connectivity measures using `@_non_negative_frequencies` decorator:
    - `coherency()`, `coherence_magnitude()`, `coherence_phase()`, `imaginary_coherence()`
    - `phase_lag_index()`, `weighted_phase_lag_index()`, `debiased_squared_phase_lag_index()`
    - `debiased_squared_weighted_phase_lag_index()`, `phase_locking_value()`, `pairwise_phase_consistency()`
    - `power()`, all Granger causality measures (DTF, PDC, etc.)
  - **Scientific justification**: The Nyquist frequency (sampling_rate/2) represents the highest frequency
    that can be unambiguously represented in sampled data. For even-length FFTs, this frequency should be
    included once (not in negative frequencies). Excluding it discards valid spectral information and
    violates standard FFT conventions (numpy.fft.rfft, scipy.fft.rfft).
  - **Migration impact**:
    - New analyses: More accurate (includes previously missing frequency information)
    - Comparing to old results: Array shapes differ by 1 in frequency dimension for even-length FFTs
    - Published results: Document which version was used in methods section
  - **Tests added**:
    - `test_nyquist_bin_even_n()`: Validates N=1024 produces 513 frequencies
    - `test_nyquist_bin_odd_n()`: Validates N=1023 produces 512 frequencies
  - **Example**: With 1000 Hz sampling and 1024-sample FFT:
    - Old (incorrect): 512 bins, missing 500 Hz (Nyquist)
    - New (correct): 513 bins, includes 500 Hz (Nyquist)
  - See PR #71 for detailed analysis and discussion

- **Tikhonov regularization for MVAR matrix inversion stability**:
  - Replaced direct matrix inverse (`xp.linalg.inv()`) with Tikhonov-regularized solve in MVAR computations
  - Prevents `LinAlgError` exceptions when computing Granger causality with near-singular matrices
  - Affected functions: `_MVAR_Fourier_coefficients` property and `_estimate_transfer_function` function
  - Uses scale-aware regularization: λ = `TIKHONOV_REGULARIZATION_FACTOR` × mean(||H||²)
  - Added module-level constant `TIKHONOV_REGULARIZATION_FACTOR = 1e-12` for consistency
  - Solves `(H + λI)x = I` instead of computing `inv(H)` for better numerical stability
  - Added stress test `test_mvar_regularized_inverse_near_singular()` validating near-singular cases
  - All Granger causality measures now handle highly correlated signals gracefully
  - See PR #72 for detailed numerical analysis

- **Numerical stability improvements for coherence calculations**:
  - Replaced zero-clamping with epsilon-clamping in `coherency()` to prevent division by zero
  - Changed from `norm[norm == 0] = xp.nan` to `norm = xp.maximum(norm, xp.finfo(norm.dtype).eps)`
  - Added bounds clipping to `coherence_magnitude()` to ensure values stay in [0, 1] range
  - Added bounds clipping to `imaginary_coherence()` with epsilon protection
  - Prevents numerical artifacts from floating-point precision issues
  - More graceful degradation for low-power signals compared to NaN propagation
  - Added comprehensive test suite (`test_coherence_bounds.py`) with 3 tests covering edge cases
  - See PR #62 for detailed analysis

- CHANGELOG.md to track version changes following Keep a Changelog format
- Ruff linter configuration for faster, more comprehensive Python linting
- Enhanced package metadata with additional project URLs (Changelog, Source Code, Issue Tracker)
- Modern unified CI/CD workflow (`release.yml`) with automated PyPI publishing
- Support for Python 3.13
- Comprehensive parameter validation to `Multitaper` class:
  - Validates `sampling_frequency > 0` with domain-specific examples (EEG, LFP, fMRI)
  - Validates `time_halfbandwidth_product >= 1` with physical meaning explanation
  - Validates `time_window_duration > 0` (when provided) with frequency resolution formula
  - Validates `time_window_step > 0` (when provided) with overlap guidance
  - Warns when `time_halfbandwidth_product > 10` (unusually large, performance impact)
  - Warns when `time_window_step > time_window_duration` (creates data gaps)
  - Warns when data appears transposed (`n_time < n_signals`)
  - Warns when input contains NaN or Inf values with recovery suggestions
- Input shape validation to `Connectivity` class:
  - Requires 5D `fourier_coefficients` with clear error messages
  - Validates minimum 2 signals for connectivity analysis
  - Warns on NaN/Inf values in Fourier coefficients
- `prepare_time_series()` helper function for safe dimension handling:
  - Converts 1D/2D arrays to required 3D format
  - Explicit `axis` parameter to clarify dimension meaning
  - Prevents ambiguous dimension interpretation
- Enhanced error messages following WHAT/WHY/HOW pattern throughout
- 3D input requirement for `Multitaper` class to eliminate dimension ambiguity
- Intelligent error suggestion for `expectation_type` parameter:
  - Detects wrong word order (e.g., "tapers_trials" instead of "trials_tapers")
  - Suggests correct ordering with helpful explanation
  - Lists all valid options with most common choice highlighted
- Improved `detrend()` function error messages:
  - Clear explanation of linear vs constant detrending
  - Examples with domain-specific terminology (DC offset, best-fit line)
  - Actionable guidance for parameter selection
- Enhanced breakpoint validation in `detrend()`:
  - Shows specific invalid breakpoint values
  - Displays valid range based on actual data dimensions
  - Includes user's original input for easy debugging
- Comprehensive test suite for error message quality (`test_error_messages.py`)
- GPU status utility function `get_compute_backend()`:
  - Returns dict with backend type, GPU availability, device name, and helpful message
  - Shows actual GPU model name (e.g., "NVIDIA Tesla V100-SXM2-16GB") instead of compute capability
- Comprehensive test suite for block-wise computation:
  - 5 new tests validating correctness, symmetry, edge cases, and memory reduction
  - Empirical memory measurement showing 73% reduction for n_signals=50, blocks=5
  - Tests cover different expectation types and block configurations
- Enhanced documentation for `blocks` parameter in Connectivity class:
  - "When to use" and "When NOT to use" guidance with specific thresholds
  - Quick decision guide based on n_signals
  - Memory-speed tradeoff quantified (70-80% memory reduction, <10% speed penalty)
  - GPU VRAM considerations documented
  - Detects CuPy availability without side effects (uses `importlib.util.find_spec`)
  - Provides 4 different message variants for different GPU configurations
  - Includes comprehensive NumPy-style docstring with 3 usage examples
  - Example return value documented in docstring
  - Located in new `spectral_connectivity.utils` module
- Enhanced GPU device logging in `transforms.py` and `connectivity.py`:
  - Now shows actual GPU model name in log messages
  - Graceful fallback to compute capability if name unavailable
- Comprehensive GPU documentation in README.md:
  - 130+ line GPU Acceleration section with setup, troubleshooting, and usage guidance
  - 3 setup methods documented (shell, Python script, Jupyter notebook)
  - Verification steps included in all setup examples
  - Simplified CuPy installation instructions (conda recommended first)
  - Troubleshooting guide with 4 common issues and solutions
  - Clear explanation of import timing requirement (why "before importing" matters)
  - Example outputs shown for all code samples
  - Kernel restart guidance for Jupyter notebook users
  - Guidance on when GPU acceleration is beneficial
- Comprehensive test suite for GPU backend (`tests/test_gpu.py`):
  - 13 test methods covering all GPU configuration scenarios
  - Tests for CPU mode (default and explicit)
  - Tests for GPU mode (with and without CuPy available)
  - Validation of return value structure and types
  - Mock-based testing to avoid CuPy dependency
  - All tests pass (11 passed, 1 skipped when CuPy unavailable)

### Changed
- **BREAKING**: Minimum Python version raised from 3.9 to 3.10
- Migrated from flake8 to ruff for linting (100x faster, replaces flake8, isort, pydocstyle)
- Updated dependency pins: numpy>=1.24, scipy>=1.10, xarray>=2023.1, matplotlib>=3.7
- Improved mypy configuration with stricter type checking and per-module overrides
- Updated development documentation (CLAUDE.md, CONTRIBUTING.md) to reflect current tooling
- Expanded test matrix to Python 3.10, 3.11, 3.12, 3.13
- Consolidated CI workflows: removed redundant PR-test.yml and linting.yml in favor of release.yml
- Simplified CI from conda-based to pip-based installation (faster builds)
- Enhanced black configuration to target Python 3.10-3.13
- Updated ReadTheDocs to use Python 3.10

### Fixed
- Outdated release instructions in CONTRIBUTING.md (removed setup.py references)
- Deprecation warning in `minimum_phase_decomposition.py`: Changed `xp.linalg.linalg.LinAlgError` to `xp.linalg.LinAlgError` for compatibility with NumPy 2.0+
- **Critical bug in block-wise computation**: Fixed missing diagonal elements in cross-spectral matrix when using `blocks` parameter (changed `k=1` to `k=0` in `triu_indices`)
- **Critical bug in block-wise computation**: Fixed incorrect Hermitian symmetry in blocked cross-spectral matrix computation (added conjugate when filling transpose positions)

## [1.1.2] - 2023-10-17

### Added
- Conda packaging support with conda-recipe directory
- CLAUDE.md with development commands and architecture documentation
- Pinned coverage reporter version in CI workflow to avoid bugs

### Changed
- Updated module docstrings for clarity and context
- Updated README with Contributing and License sections

### Fixed
- Linting issues resolved

## [1.1.1] - 2023-09-15

### Changed
- Switch build system from setuptools to Hatch
- Add py.typed marker for type hint support
- Update and reorganize dependencies in environment.yml

### Fixed
- Resolve n_time_samples_per_window property logic error in transforms module
- Resolve mypy Optional[int] vs int return type errors
- Correct _fix_taper_sign return type annotation

## [1.1.0] - 2023-08-20

### Added
- GPU request guard feature to safely handle CUDA availability
- Complete audit of connectivity metric ranges documentation
- ValueError raised when window size parameters are unset

### Changed
- Updated GitHub Actions to latest versions
- Improved type hints throughout codebase

## [1.0.4] - 2023-03-15

### Fixed
- Bug fixes in connectivity calculations
- Improved numerical stability

## [1.0.3] - 2023-02-10

### Changed
- Performance improvements
- Documentation updates

## [1.0.2] - 2023-01-20

### Fixed
- Minor bug fixes
- Test coverage improvements

## [1.0.1] - 2022-12-15

### Fixed
- Package distribution fixes
- Documentation corrections

## [1.0.0] - 2022-12-01

### Added
- First stable release
- Full implementation of 15+ connectivity measures
- GPU acceleration support via CuPy
- Comprehensive test suite
- Complete documentation on ReadTheDocs

### Changed
- API stabilized for 1.0 release
- Performance optimizations

## [0.2.7] - 2022-06-15

### Added
- Additional connectivity measures
- Improved caching strategy

### Changed
- API improvements and refinements

## [0.2.6] - 2022-03-10

### Added
- Initial GPU support
- More connectivity measures

### Changed
- Refactored core architecture
- Improved documentation

---

[Unreleased]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v2.0.0...HEAD
[2.0.0]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.1.2...v2.0.0
[1.1.2]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.1.1...v1.1.2
[1.1.1]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.1.0...v1.1.1
[1.1.0]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.0.4...v1.1.0
[1.0.4]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.0.3...v1.0.4
[1.0.3]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.0.2...v1.0.3
[1.0.2]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.0.1...v1.0.2
[1.0.1]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v1.0.0...v1.0.1
[1.0.0]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v0.2.7...v1.0.0
[0.2.7]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/compare/v0.2.6...v0.2.7
[0.2.6]: https://github.com/Eden-Kramer-Lab/spectral_connectivity/releases/tag/v0.2.6
