Simple Neural Network

A from-scratch implementation of a feedforward neural network in C# (.NET 8) without using any machine learning frameworks. This project demonstrates both backpropagation and genetic algorithm training methods to solve the multiplication table problem (learning to multiply two numbers from 1 to 9).

Project Overview

This educational project implements a complete neural network from the ground up, including:

• Custom neuron and synapse implementations
• Sigmoid activation function with numerical stability
• Backpropagation with momentum
• Genetic algorithm evolution with elitism
• Parameter optimization tools

Problem: Learn the multiplication table (1×1 to 9×9)

• Input: Two numbers (normalized to [0, 1])
• Output: Their product (normalized to [0.095, 0.905])

Features

Dual Training Methods

1. Backpropagation with Momentum

   • Classic gradient descent optimization
   • Configurable learning rate and momentum
   • Fast convergence with proper hyperparameters

2. Genetic Algorithm

   • Population-based evolution
   • Elitism to preserve best solutions
   • Crossover and mutation operators
   • Parallel fitness evaluation

Architecture Highlights

• Flexible Network Structure: Configurable input, hidden, and output layers
• Numerical Stability: Sigmoid clipping to prevent overflow
• Thread-Safe: Parallel processing in genetic algorithm
• Parameter Optimization: Built-in genetic parameter tuning

Architecture

Core Components

1. Neuron (Neuron.cs)

• Represents a single neuron in the network
• Properties:
  • Value: Current activation value
  • Bias: Neuron bias term
  • Gradient: Calculated gradient for backpropagation
  • InputSynapses: Connections from previous layer
  • OutputSynapses: Connections to next layer

2. Synapse (Synapse.cs)

• Represents a weighted connection between neurons
• Properties:
  • Weight: Connection weight
  • WeightDelta: Weight change for momentum
  • InputNeuron: Source neuron
  • OutputNeuron: Destination neuron

3. Brain (Brain.cs)

• Main neural network class
• Methods:
  • Train(): Trains using backpropagation
  • Think(): Evaluates performance on dataset
  • Compute(): Forward pass for prediction
  • Clone(): Deep copy for genetic algorithm

4. Sigmoid (Sigmoid.cs)

• Activation function implementation
• Output(): σ(x) = 1 / (1 + e^(-x))
• Derivative(): σ'(x) = σ(x) × (1 - σ(x))
• Includes clipping at ±10 for numerical stability

5. Simulation (Simulation.cs)

• Genetic algorithm implementation
• Features:
  • Population-based evolution
  • Tournament-style selection
  • Crossover and mutation
  • Elitism (preserves top 20% of selection)

6. GeneticOptimizer (GeneticOptimizer.cs)

• Hyperparameter optimization tool
• Tests combinations of:
  • Mutation range (weight perturbation magnitude)
  • Mutation rate (probability of mutation)
  • Domination rate (parent contribution ratio)

Usage

Running the Project

dotnet run

The program will:

1. Train a network using backpropagation
2. Evaluate and display results
3. Train a network using genetic algorithm
4. Compare the two methods

Sample Output

=== BACKPROPAGATION METHOD ===

EPOCH = 0, ERROR = 2.456789
EPOCH = 500, ERROR = 0.123456
...
EPOCH = 10000, ERROR = 0.012345

Final error=0.01234567

Backpropagation Results:
maxB=1.234
minB=-1.123
maxW=2.345
minW=-2.234

1*1 = 1 ok
1*2 = 2 ok
...
9*9 = 81 ok

=== ACCURACY: 81/81 (100.0%) ===


=== GENETIC ALGORITHM METHOD ===

Genetic Algorithm Results:
...

Customizing Network Configuration

Backpropagation Example

Brain neuralNet = new Brain(
    inputSize: 2,          // Two inputs (multiplicands)
    hiddenSize: 16,        // 16 neurons per hidden layer
    outputSize: 1,         // One output (product)
    numHiddenLayers: 2,    // 2 hidden layers
    learnRate: 0.4,        // Learning rate
    momentum: 0.95         // Momentum coefficient
);

neuralNet.Train(dataSet, epochs: 10000);

Genetic Algorithm Example

var simulation = new Simulation(
    dataSet,
    epochs: 100,           // 100 generations
    selection: 32,         // Top 32 brains as parents
    mutationRange: 0.5,    // ±0.5 mutation range
    mutationRate: 1000,    // 10% mutation rate
    dominationRate: 0.6    // 60% dominant, 40% recessive
);

simulation.Init(
    inputSize: 2,
    hiddenSize: 24,
    outputSize: 1,
    numHiddenLayers: 3
);

Brain best = simulation.Start();

Technical Details

Data Normalization

Inputs: Linear normalization to [0, 1]

normalizedInput = (value - 1.0) / 8.0  // Maps 1-9 to 0-1

Outputs: Normalized to [0.095, 0.905] range

// Step 1: Normalize product to [0, 1]
normalized = (product - 1.0) / 80.0  // Maps 1-81 to 0-1

// Step 2: Scale to optimal sigmoid range
output = 0.095 + normalized * 0.81   // Maps to [0.095, 0.905]

This range avoids sigmoid saturation regions where gradients approach zero.

Backpropagation Details

1. Forward Pass: Calculate activations layer by layer
2. Output Gradient: δ_output = (target - output) × σ'(output)
3. Hidden Gradients: δ_hidden = Σ(δ_next × weight) × σ'(activation)
4. Weight Update: w_new = w_old + η × δ × input + α × Δw_prev
   • η = learning rate
   • α = momentum coefficient

Genetic Algorithm Details

Selection: Top N performers become parents

Crossover: Weighted average of parent parameters

child_weight = dominant_weight × dominationRate + 
               recessive_weight × (1 - dominationRate)

Mutation: Random perturbation with probability

if (random < mutationRate)
    weight += random(-mutationRange, +mutationRange)

Elitism: Top 20% of selected brains preserved unchanged

Performance Comparison

Method	Typical Accuracy	Training Time	Consistency
Backpropagation	95-100%	Fast (~seconds)	High
Genetic Algorithm	85-95%	Slower (~minutes)	Variable

Backpropagation is generally more efficient for this problem due to:

• Direct gradient information
• Consistent convergence path
• Lower computational cost per iteration

Genetic Algorithm advantages:

• No gradient calculation needed
• Can escape local minima
• Works with non-differentiable functions

Parameter Optimization

To find optimal genetic algorithm parameters:

// Uncomment in Program.cs
GeneticOptimizer.FindOptimalParameters();

This will test various combinations and report the best configuration.

License

This project is open source and available for educational purposes.

Contributing

This is an educational project. Feel free to fork, experiment, and learn!

Further Reading

• Backpropagation Algorithm
• Genetic Algorithms
• Sigmoid Function
• Momentum in Neural Networks