392 convert layers to numpower by apphp · Pull Request #393 · RubixML/ML

apphp · 2025-12-06T17:15:04Z

No description provided.

…neural network layers.

…hensive unit tests and fixed broken source file link in the documentation

…d updated documentation with fixed source file link. Added `Parametric` interface to define parameterized layers.

…on support

…ith `NumPower` utilities

…ical stability during inference, and gradient computation logic

…itional tests and updated shape handling

…interface definition for output layers.

…entation and unit tests

…d/backward passes

…ence/backward passes, unit tests, and documentation updates

…ith `NumPower` utilities

…rd/inference/backward passes, unit tests

…ference/backward passes, unit tests

…rward/inference/backward passes, unit tests

Copilot

Pull request overview

This pull request converts neural network layers and optimizers to use the NumPower library for numerical operations. The changes include implementing new optimizer classes (StepDecay, RMSProp, Momentum, Cyclical, Adam, AdaMax, AdaGrad), new layer implementations (Swish, Placeholder1D, PReLU, Noise, Multiclass, Dropout, Dense, Continuous, Binary, BatchNorm, Activation), updating tests to use PHPUnit attributes, and reorganizing code into namespaced subdirectories with corresponding documentation updates.

Key Changes

Migration from tensor library to NumPower for all neural network operations
Reorganization of optimizers and layers into subdirectory namespaces (e.g., Adam/Adam.php instead of Adam.php)
Conversion of PHPUnit test annotations to attributes (#[Test], #[TestDox], etc.)

Reviewed changes

Copilot reviewed 71 out of 71 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/Specifications/SamplesAreCompatibleWithDistanceTest.php	Updated to use PHPUnit attributes for test annotations
tests/NeuralNet/Optimizers/Stochastic/StochasticTest.php	Refactored test with data providers and PHPUnit attributes
tests/NeuralNet/Optimizers/StepDecay/StepDecayTest.php	New comprehensive test suite for StepDecay optimizer
tests/NeuralNet/Optimizers/RMSProp/RMSPropTest.php	New test suite covering RMSProp optimizer with NumPower
tests/NeuralNet/Optimizers/Momentum/MomentumTest.php	New test suite for Momentum optimizer
tests/NeuralNet/Optimizers/Cyclical/CyclicalTest.php	New test suite for Cyclical learning rate optimizer
tests/NeuralNet/Optimizers/Adam/AdamTest.php	New comprehensive Adam optimizer tests
tests/NeuralNet/Optimizers/AdaMax/AdaMaxTest.php	New AdaMax optimizer test suite
tests/NeuralNet/Optimizers/AdaGrad/AdaGradTest.php	New AdaGrad optimizer tests
tests/NeuralNet/Layers/*/Test.php (multiple)	New test suites for all layer types using NumPower
tests/NeuralNet/FeedForwardTest.php	Updated to use PHPUnit attributes
tests/Helpers/GraphvizTest.php	Updated to use PHPUnit attributes
src/Traits/AssertsShapes.php	Updated exception import and error message
src/NeuralNet/Parameters/Parameter.php	Formatting improvements and method signature updates
src/NeuralNet/Optimizers/*/Optimizers.php (multiple)	New optimizer implementations using NumPower
src/NeuralNet/Layers/*/Layers.php (multiple)	New layer implementations using NumPower
src/NeuralNet/Initializers/*/Initializers.php (multiple)	Added explicit `loc` parameter to NumPower calls
phpunit.xml	Increased memory limit to 256M
docs/neural-network/*/.md (multiple)	Updated file paths and added mathematical formulations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/NeuralNet/Optimizers/Cyclical/Cyclical.php

docs/neural-network/optimizers/cyclical.md

Copilot

Pull request overview

Copilot reviewed 72 out of 72 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

andrewdalpino

This looks great @apphp, mostly good to go I just had a few questions before I approved.

andrewdalpino · 2026-02-20T00:44:39Z

src/NeuralNet/Layers/BatchNorm/BatchNorm.php

+     *
+     * @var int
+     */
+    protected const int AXIS_SAMPLES = 0;


I would just double check that this is correct and not the other way around. From my understanding the input to each layer is shaped [num_features, num_samples] which would mean that axis 0 is the features not the samples.

Done. It was rewritten to use the convention [num_features, num_samples], meaning features-first.

andrewdalpino · 2026-02-20T00:47:04Z

src/NeuralNet/Layers/BatchNorm/BatchNorm.php

+        $m = $dOut->shape()[0];
+
+        // Compute gradient per formula: dX = (dXHat * m - dXHatSigma - xHat * xHatSigma) * (stdInv / m)
+        return NumPower::multiply(


Can we break this apart to help make it more human-readable?

andrewdalpino · 2026-02-20T00:50:15Z

src/NeuralNet/Layers/Binary/Binary.php

+    {
+        $n = $output->shape()[1];
+
+        if ($this->costFn instanceof CrossEntropy) {


I don't remember if I left a comment here or not in the original implementation, but it's worth noting to future developers that this is an optimization specific to the Binary Cross Entropy loss function. Through some clever mathematical cancellation tricks this gradient calculation is just a simple subtraction operation.

No, but this is was a good point - I added comment for both implementations.

andrewdalpino · 2026-02-20T01:04:42Z

src/NeuralNet/Layers/Dense/Dense.php

+
+        if ($this->biases) {
+            // Reshape bias vector [fanOut] to column [fanOut, 1] to match output [fanOut, n]
+            $bias = NumPower::reshape($this->biases->param(), [$this->neurons, 1]);


Was this reshaping required in the original Tensor code?

No, original Tensor code didnt need it, but NumPower version does to get correct bias broadcasting across the batch dimension.

This is because in the NumPower NumPower::add($output, $biasVector) does not reliably broadcast a 1-d vector across the second dimension

That's great to know, thanks!

andrewdalpino · 2026-02-20T01:08:47Z

src/NeuralNet/Layers/Swish/Swish.php

+        // Gradient of the loss with respect to beta
+        // dL/dbeta = sum_over_batch(dL/dy * dy/dbeta)
+        // Here we use a simplified formulation: dL/dbeta ~ sum(dOut * input)
+        $dBetaFull = NumPower::multiply($dOut, $this->input);


Interesting comment, you know I'm not entirely sure I had the original gradient calculation 100% correct in the first place.

Do you want me refactor this calculation?

Let's leave it for now as these change are more about converting rather than fixing. But if indeed the calculation is wrong then we'll fix it in a followup.

…atures, num_samples]

…tropy loss function

andrewdalpino · 2026-02-20T22:40:06Z

At some point we should resolve the static analysis warnings as well

apphp added 2 commits December 6, 2025 18:55

ML-392 Added Layer and Hidden interfaces to define contracts for …

fc14381

…neural network layers.

ML-393 Added standalone Activation layer implementation with compre…

9546c9d

…hensive unit tests and fixed broken source file link in the documentation

apphp requested review from SkibidiProduction and andrewdalpino December 6, 2025 17:15

apphp self-assigned this Dec 6, 2025

apphp changed the base branch from master to 3.0 December 6, 2025 17:15

apphp added 20 commits December 7, 2025 00:41

ML-394 Implemented BatchNorm layer with comprehensive unit tests an…

14a2b6f

…d updated documentation with fixed source file link. Added `Parametric` interface to define parameterized layers.

ML-395 Refactored BatchNorm layer to improve row/column normalizati…

46e101b

…on support

ML-396 Refactored BatchNorm layer to optimize normalization logic w…

a44e86a

…ith `NumPower` utilities

ML-397 Enhanced BatchNorm layer with improved axis constants, numer…

5a3f5a7

…ical stability during inference, and gradient computation logic

ML-397 Enhanced BatchNorm layer with improved axis constants, numer…

ec3c236

…ical stability during inference, and gradient computation logic

ML-398 Improved BatchNorm behavior for varying batch sizes with add…

00955f9

…itional tests and updated shape handling

ML-398 Improved BatchNorm behavior for varying batch sizes with add…

3fb79a9

…itional tests and updated shape handling

ML-399 Added Binary output layer and comprehensive unit tests with …

b4e507d

…interface definition for output layers.

ML-400 Added Continuous output layer with complete interface implem…

d5f7c57

…entation and unit tests

ML-392 Added Dense hidden layer implementation with complete forwar…

cca97f9

…d/backward passes

ML-401 Added Dropout hidden layer implementation with forward/infer…

9767a1f

…ence/backward passes, unit tests, and documentation updates

ML-392 Refactored Dropout layer to replace custom mask generation w…

8d7c938

…ith `NumPower` utilities

ML-392 Added Multiclass output layer with complete interface, forwa…

5ad0ed2

…rd/inference/backward passes, unit tests

ML-392 Added Noise output layer with complete interface, forward/in…

a6c634f

…ference/backward passes, unit tests

ML-392 Added Placeholder1D output layer with complete interface, fo…

7013a56

…rward/inference/backward passes, unit tests

ML-392 Fixed wrong exception for AssertsShapes and exception texts

08ad6b4

ML-392 Increased memory for tests

ce99147

ML-392 Added PReLU output layer with complete interface and unit tests

7628fec

ML-392 Added Swish output layer with complete interface and unit tests

f83fed6

ML-392 Fixed syntax issues in tests annotations

be52a09

andrewdalpino requested review from a team and Copilot and removed request for SkibidiProduction and andrewdalpino December 30, 2025 00:57

Copilot started reviewing on behalf of andrewdalpino December 30, 2025 00:58 View session

Copilot AI reviewed Dec 30, 2025

View reviewed changes

src/NeuralNet/Optimizers/Cyclical/Cyclical.php Outdated Show resolved Hide resolved

docs/neural-network/optimizers/cyclical.md Outdated Show resolved Hide resolved

apphp requested review from SkibidiProduction and andrewdalpino January 3, 2026 12:12

ML-392 Typo fixes

0b591ed

apphp requested review from SkibidiProduction and andrewdalpino and removed request for SkibidiProduction and andrewdalpino January 4, 2026 17:48

andrewdalpino requested a review from Copilot January 5, 2026 19:53

Copilot started reviewing on behalf of andrewdalpino January 5, 2026 19:53 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

Merge branch '3.0' into 392-convert-layers-to-numpower

c346296

andrewdalpino reviewed Feb 20, 2026

View reviewed changes

apphp mentioned this pull request Feb 20, 2026

394 convert misc nn classes to num power #395

Open

apphp added 4 commits February 20, 2026 23:43

ML-392 rewritten BatchNorm::gradient for better readability

7fedecd

ML-392 rewritten BatchNorm::forward for better readability

ee43ccc

ML-392 rewritten BatchNorm to use the library-wide convention [num_fe…

a8a1a54

…atures, num_samples]

ML-392 added comment for optimization specific to the Binary Cross En…

0ce683c

…tropy loss function

Uh oh!

Comments

Conversation

apphp commented Dec 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

andrewdalpino left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apphp Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apphp Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewdalpino commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

apphp Feb 20, 2026 •

edited

Loading

apphp Feb 20, 2026 •

edited

Loading