You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+31-61Lines changed: 31 additions & 61 deletions
Original file line number
Diff line number
Diff line change
@@ -12,31 +12,26 @@ Found on:
12
12
13
13
# Velora
14
14
15
-
**Velora** is a lightweight and extensible framework built on top of powerful libraries like [Gymnasium](https://gymnasium.farama.org/) and [PyTorch](https://pytorch.org/), specializing in a unique approach to Deep Reinforcement Learning (RL) algorithms, a paradigm we call *Liquid RL*.
15
+
**Velora** is a lightweight and modular framework built on top of powerful libraries like [Gymnasium](https://gymnasium.farama.org/) and [PyTorch](https://pytorch.org/). It is home to a new type of RL agent called ***NeuroFlow*** (NF) that specializes in Autonomous Cyber Defence through a novel Deep Reinforcement Learning (RL) approach we call ***Liquid RL***.
16
16
17
-
Instead of Fully-connected Networks, Velora combines [Liquid Neural Networks](https://arxiv.org/abs/2006.04439) (LNNs) with [Neural Circuit Policies](https://arxiv.org/abs/1803.08554) (NCPs), specifically [Ordinary Neural Circuits](https://proceedings.mlr.press/v119/hasani20a.html) (ONCs).
17
+
## Benefits
18
18
19
-
These two components have interesting benefits:
20
-
21
-
- LNNs are a powerful RNN architecture that learns system dynamics, not just data patterns.
22
-
- NCPs focus on sparsely connected neurons with distinct functions, mimicking biological behaviour.
23
-
24
-
From what we've seen, these networks are powerful, small-scale architectures that excel in model explainability, making them perfect for control tasks.
25
-
26
-
Velora offers Liquidfied PyTorch-based implementations of RL algorithms, designed to be intuitive, easy to use, and customizable.
27
-
28
-
In other frameworks, we've seen a trend of heavy abstraction in favour of minimal lines of code. Our approach aims to offer a best of both worlds, abstracting code away but making the details explainable on the backend, while giving you the freedom to customize as needed.
19
+
-**Explainability**: NF agents use [Liquid Neural Networks](https://arxiv.org/abs/2006.04439) (LNNs) and [Neural Circuit Policies](https://arxiv.org/abs/1803.08554) (NCPs) to model Cyber system dynamics, not just data patterns. Also, they use sparse NCP connections to mimic biological efficiency, enabling clear, interpretable strategies via a labeled Strategy Library.
20
+
-**Adaptability**: NF agents dynamically grow their networks using a fitness score, adding more neurons to a backbone only when new Cyber strategies emerge, keeping agents compact and robust.
21
+
-**Planning**: NF agents use a Strategy Library and learned environment model to plan strategic sequences for proactive Cyber defense.
22
+
-**Always Learning**: using [EWC](https://arxiv.org/abs/1612.00796), NF agents refine existing strategies and learn new ones post-training, adapting to evolving Cyber threats like new attack patterns.
23
+
-**Customizable**: NF agents are PyTorch-based, designed to be intuitive, easy to use, and modular so you can easily build your own!
29
24
30
25
## Installation
31
26
32
-
To get started, simply install it through [pip](https://pypi.org/) using one of the options below.
27
+
To get started, simply install it through [pip](https://pypi.org/project/velora) using one of the options below.
33
28
34
29
### GPU Enabled
35
30
36
31
For [PyTorch](https://pytorch.org/get-started/locally/) with CUDA (recommended):
# Or, use the standard gym API (recommended for this env)
80
-
env = gym.make("InvertedPendulum-v5")
81
-
82
-
# Set core variables
83
-
state_dim = env.observation_space.shape[0] # in features
84
-
n_neurons =20# decision/hidden nodes
85
-
action_dim = env.action_space.shape[0] # out features
86
-
87
-
buffer_size =100_000
88
-
batch_size =128
89
-
90
-
# Train a model
91
-
model = LiquidDDPG(
92
-
state_dim,
93
-
n_neurons,
94
-
action_dim,
95
-
buffer_size=buffer_size,
56
+
# For continuous tasks
57
+
model = NeuroFlowCT(
58
+
"InvertedPendulum-v5",
59
+
20, # actor neurons
60
+
128, # critic neurons
96
61
device=device,
62
+
seed=64, # remove for automatic generation
97
63
)
98
-
model.train(env, batch_size, n_episodes=300)
64
+
65
+
# For discrete tasks
66
+
model = NeuroFlow(
67
+
"CartPole-v1",
68
+
20, # actor neurons
69
+
128, # critic neurons
70
+
device=device,
71
+
)
72
+
73
+
# Train the model using a batch size of 64
74
+
model.train(64, n_episodes=50, display_count=10)
99
75
```
100
76
101
-
Currently, the framework only supports [Gymnasium](https://gymnasium.farama.org/) environments and is planned to expand to [PettingZoo](https://pettingzoo.farama.org/index.html) for Multi-agent (MARL) tasks.
77
+
Currently, the framework only supports [Gymnasium](https://gymnasium.farama.org/) environments and is planned to expand to [PettingZoo](https://pettingzoo.farama.org/index.html) for Multi-agent (MARL) tasks, with updated adaptations of [CybORG](https://github.com/cage-challenge/CybORG/tree/main) environments.
102
78
103
79
## API Structure
104
80
@@ -122,16 +98,10 @@ from velora.gym import [method]
122
98
from velora.utils import [method]
123
99
```
124
100
125
-
## Customization
126
-
127
-
Customization is at the heart of Velora but requires a deeper understanding of the API.
128
-
129
-
You can read more about it in the [documentation tutorials](https://velora.achronus.dev/learn/customize).
130
-
131
101
## Active Development
132
102
133
103
🚧 View the [Roadmap](https://velora.achronus.dev/starting/roadmap) 🚧
134
104
135
-
**Velora** is a tool that is continuously being developed. There's still a lot to do to make it a fully functioning framework, such as detailed API documentation, and more RL algorithms.
105
+
**Velora** is a tool that is continuously being developed. There's still a lot to do to make it a great framework, such as detailed API documentation, and expanding our NeuroFlow agents.
136
106
137
107
Our goal is to provide a quality open-source product that works 'out-of-the-box' that everyone can experiment with, and then gradually fix unexpected bugs and introduce more features on the road to a `v1` release.
Copy file name to clipboardExpand all lines: docs/index.md
+9-14Lines changed: 9 additions & 14 deletions
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ hide:
11
11
12
12
<pid="slogan"align="center"markdown>
13
13
14
-
*Velora, a lightweight and modular <spanstyle="color: #38e2e2;">Liquid Reinforcement Learning (RL)</span> framework.*
14
+
*Velora, a <spanstyle="color: #38e2e2;">Liquid RL</span> framework for <spanstyle="color: #38e2e2;">NeuroFlow</span> agents, empowering <spanstyle="color: #38e2e2;">Autonomous Cyber Defence</span>.*
15
15
16
16
</p>
17
17
@@ -30,20 +30,15 @@ hide:
30
30
31
31
---
32
32
33
-
**Velora** is a lightweight and extensible framework built on top of powerful libraries like [Gymnasium [:material-arrow-right-bottom:]](https://gymnasium.farama.org/) and [PyTorch [:material-arrow-right-bottom:]](https://pytorch.org/), specializing in a unique approach to Deep Reinforcement Learning (RL) algorithms, a paradigm we call *Liquid RL*.
33
+
**Velora** is a lightweight and modular framework built on top of powerful libraries like [Gymnasium [:material-arrow-right-bottom:]](https://gymnasium.farama.org/) and [PyTorch [:material-arrow-right-bottom:]](https://pytorch.org/). It is home to a new type of RL agent called ***NeuroFlow*** (NF) that specializes in Autonomous Cyber Defence through a novel Deep Reinforcement Learning (RL) approach we call ***Liquid RL***.
- LNNs are a powerful RNN architecture that learns system dynamics, not just data patterns.
40
-
- NCPs focus on sparsely connected neurons with distinct functions, mimicking biological behaviour.
41
-
42
-
From what we've seen, these networks are powerful, small-scale architectures that excel in model explainability, making them perfect for control tasks.
43
-
44
-
Velora offers Liquidfied PyTorch-based implementations of RL algorithms, designed to be intuitive, easy to use, and customizable.
45
-
46
-
In other frameworks, we've seen a trend of heavy abstraction in favour of minimal lines of code. Our approach aims to offer a best of both worlds, abstracting code away but making the details explainable on the backend, while giving you the freedom to customize as needed.
37
+
-**Explainability**: NF agents use [Liquid Neural Networks [:material-arrow-right-bottom:]](https://arxiv.org/abs/2006.04439) (LNNs) and [Neural Circuit Policies [:material-arrow-right-bottom:]](https://arxiv.org/abs/1803.08554) (NCPs) to model Cyber system dynamics, not just data patterns. Also, they use sparse NCP connections to mimic biological efficiency, enabling clear, interpretable strategies via a labeled Strategy Library.
38
+
-**Adaptability**: NF agents dynamically grow their networks using a fitness score, adding more neurons to a backbone only when new Cyber strategies emerge, keeping agents compact and robust.
39
+
-**Planning**: NF agents use a Strategy Library and learned environment model to plan strategic sequences for proactive Cyber defense.
40
+
-**Always Learning**: using [EWC [:material-arrow-right-bottom:]](https://arxiv.org/abs/1612.00796), NF agents refine existing strategies and learn new ones post-training, adapting to evolving Cyber threats like new attack patterns.
41
+
-**Customizable**: NF agents are [PyTorch-based [:material-arrow-right-bottom:]](https://pytorch.org/), designed to be intuitive, easy to use, and modular so you can easily build your own!
47
42
48
43
<divclass="grid cards"markdown>
49
44
@@ -67,7 +62,7 @@ In other frameworks, we've seen a trend of heavy abstraction in favour of minima
67
62
68
63
## Active Development
69
64
70
-
**Velora** is a tool that is continuously being developed. There's still a lot to do to make it a fully functioning framework, such as detailed API documentation, and more RL algorithms.
65
+
**Velora** is a tool that is continuously being developed. There's still a lot to do to make it a great framework, such as detailed API documentation, and expanding our NeuroFlow agents.
71
66
72
67
Our goal is to provide a quality open-source product that works 'out-of-the-box' that everyone can experiment with, and then gradually fix unexpected bugs and introduce more features on the road to a [`v1`](#active-development) release.
The `BasicCNN` uses a static architecture from the DQN Nature paper: [Human-level control through deep reinforcement learning [:material-arrow-right-bottom:]](https://www.nature.com/articles/nature14236).
56
56
57
57
The paper used it for Atari games, but has been adopted in other libraries such as [Stable-Baselines3 [:material-arrow-right-bottom:]](https://stable-baselines3.readthedocs.io/en/master/index.html) as a go-to CNN architecture, so we thought we'd use the same one! 😊
58
58
59
59
As an added bonus, it makes things easier for comparing SB3 baselines with our algorithms 😉.
60
60
61
-
???+ note "Backbones with Velora algorithms"
61
+
???+ abstract "Backbones with Velora agents"
62
62
63
-
Currently, Velora doesn't directly use backbones in it's prebuilt algorithms, they are strictly LNN architectures. So, you need to manually apply them yourself (we'll show you how to do this shortly).
63
+
Currently, Velora doesn't directly use backbones in it's agents, they are strictly LNN or NCP architectures. So, you need to manually apply them yourself (we'll show you how to do this shortly).
64
64
65
-
We plan to change this in the future, but right now we are focusing on building a robust baseline for our algorithms.
65
+
Typically, cyber environments don't use images as inputs, so we have no intention of changing this.
66
66
67
67
To use the `BasicCNN` architecture, we pass in the number of `in_channels` and then can call the `forward()` or `out_size()` methods:
0 commit comments