Skip to content

Commit

Permalink
chore: update docs on real-time and VILLASnode interfaces
Browse files Browse the repository at this point in the history 
Signed-off-by: Niklas Eiling <[email protected]>
  • Loading branch information
@n-eiling
n-eiling committed Feb 17, 2025
1 parent 006535c commit ab0d97a
Show file tree
Hide file tree
Showing 2 changed files with 117 additions and 57 deletions.
123 changes: 86 additions & 37 deletions docs/hugo/content/en/docs/Getting started/real-time.md
View file
Original file line number Diff line number Diff line change
@@ -1,52 +1,101 @@
---
title: "Real-Time"
linkTitle: "Real-Time"
date: 2017-01-05
date: 2025-02-13
---

This page describes recommended techniques to optimize the host operating system for real-time execution of DPsim.

In principle, real-time execution is supported on all platforms.
However, we recommend to use an optimized Linux installation.
DPsim supports real-time execution, where the simulation time equals the real or wall clock time, on any system.
However, without modifying system parameters the minimum time step reliably achievable will not be in the range of microseconds.
This is due to operating system noise and other processes interfering with the simulation.
With proper tuning, we have achieved real-time time steps as low as 5 us synchronized to a FPGA using VILLASnode.
Synchronizing the time step to an external source is only necessary, when very high time step accuracy, with maximum deviations in the nanoseconds, is required.

## Operating System and Kernel

For minimum latency several kernel and driver settings can be optimized.

To get started, we recommend the [Redhat Real-time Tuning Guide](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/2/html/Realtime_Tuning_Guide/index.html).

A [PREEMPT_RT patched Linux](https://rt.wiki.kernel.org/index.php/Main_Page) kernel is recommended.
Precompiled kernels for Fedora can be found here: http://ccrma.stanford.edu/planetccrma/software/

Use the *tuned* tool for improving general real-time performance.
Please adjust the setting `isolated_cpucores` according to your hardware and enable the `realtime` profile as follows:

$ dnf install tuned-profiles-realtime
$ echo "realtime" > /etc/tuned/active_profile
$ echo "isolated_cpucores=6-7" >> /etc/tuned/realtime-variables.conf
$ systemctl enable tuned && systemctl start tuned
$ systemctl reboot

## Running a real-time simulation

As a reference, real-time simulation examples are provided in the Examples/Cxx folder of the DPsim repository.

In order to run a real-time simulation, the simulation process must be started in a special way in order to change the execution priority, scheduler and CPU affinity.
For this purpose the `chrt` and `taskset` commands are used.
In the following example, we pin the execution of the simulation to CPU cores 6-7 which have been reserved previously by the tuned real-time profile (see above).

$ taskset --all-tasks --cpu-list 6-7 \
$ chrt --fifo 99 \
$ Examples/Cxx/RT_DP_CS_R_1

More details:
## Operating System and Kernel

Using a Linux kernel with the `PREEMPT_RT` feature improves latency when issuing system calls and enables the FIFO scheduler that lets us avoid preemption during the real-time simulation.

Most distributions offer a binary package for a `PREEMPT_RT` enabled kernel. For example on Rocky Linux:
```bash
sudo dnf --enablerepo=rt install kernel-rt kernel-rt-devel
```

More aggressive tuning can involve isolating a set of cores for exclusive use by the real-time simulation.
This way, the kernel will not schedule any processes on these cores.
Add the kernel parameters `isolcpus` and `nohz_full` using, for example, `grubby`:
```bash
sudo grubby --update-kernel=ALL --args="isolcpus=9,11,13,15 nohz_full=9,11,13,15"
```

Something similar, but less invasive and non-permanent can be achieved using `tuna`:
```bash
sudo tuna isolate -c 9,11,13,15
```

To avoid real-time throttling to cause overruns disable this feature:
```bash
sudo bash -c "echo -1 > /proc/sys/kernel/sched_rt_runtime_us"
```
Note that this is not persistent when rebooting.

## Simulation Model Tuning

Real time capable models cannot issue any system calls during simulation as the context switch to the kernel introduces unacceptable latencies.
This means models cannot allocate memory, use mutexes or other interupt-driven synchronization primitives, read or write data from files.
You should turn off logging, when time steps in the low milliseconds are desired.
There is a `RealTimeDataLogger` that can be used to output simulation results in these cases.
Note however, that this logger pre-allocated the memory required for all of the logging required during simulations.
Your machine may run out of memory, when the simulation is long or you log too many signals.

You can increase the performance of your simulation by adding the `-flto` and `-march=native` compiler flags:
```
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8801cbe8d..4a2843269 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -79,7 +79,7 @@ include(CheckSymbolExists)
check_symbol_exists(timerfd_create sys/timerfd.h HAVE_TIMERFD)
check_symbol_exists(getopt_long getopt.h HAVE_GETOPT)
if(CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
- add_compile_options(-flto -march=native -Ofast)
+ add_compile_options( -Ofast)
endif()
# Get version info and buildid from Git
```

## Running a Real-Time Simulation

Before running a simulation, you can run the following commands as root:
```bash
echo "evacuating cores"
tuna isolate -c 9,11,13,15

echo "disabling RT throttling"
echo -1 > /proc/sys/kernel/sched_rt_runtime_us

echo "stopping systemd services"
systemctl stop polkit
systemctl stop containerd
systemctl stop crond
systemctl stop chronyd
```

As a reference, real-time simulation examples are provided in the `dpsim/examples/cxx` and `dpsim-villas/examples/cxx` folder of the DPsim repository.

To benefit from the `PREEMPT_RT` feature and the isolated cores, the simulation has to be started using the `chrt` command to set the scheduling policy and priority, and the `taskset` command to pin the process to the isolated cores.
- [chrt man-page](http://man7.org/linux/man-pages/man1/chrt.1.html)
- [taskset man-page](http://man7.org/linux/man-pages/man1/taskset.1.html)

## Recommended Hardware
In the following example, we set the FIFO scheduling policy with the highest priority (99) and pin the execution of the simulation to CPU cores 9,11,13,15 which have been reserved previously (see above).

Some proposals for the selection of appropriate server hardware:
```bash
# the simple RT_DP_CS_R_1 simulation
taskset -c 9,11,13,15 chrt -f 99 build/dpsim/examples/cxx/RT_DP_CS_R_1

- Server-grade CPU, e.g. Intel Xeon. A multi-core system enables true parallel execution of several decoupled systems
- Server-grade network cards, e.g. Intel PRO/1000. These allow offloading of UDP checksumming to the hardware
# Cosimulation using VILLASnode, FPGA synchronized time step, and exchanging data via Aurora interface.
# Here we need sudo, to interact with the FPGA. We disable logging (log=false) and set the time step to 50 us (-t 0.00005).
sudo taskset -c 9,11,13,15 chrt -f 99 build/dpsim-villas/examples/cxx/FpgaCosim3PhInfiniteBus -o log=false -t 0.00005 -d 10
```
51 changes: 31 additions & 20 deletions docs/hugo/content/en/docs/Overview/interfaces.md
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,41 @@
---
title: "Interfaces"
linkTitle: "Interfaces"
date: 2022-12-13
date: 2025-02-13
---

Interfaces can be used to share data between a DPsim simulation and other, external services, for example an MQTT-broker. For the purpose of this guide, all services receiving and transmitting data besides the running DPsim instance are grouped in the term **environment**. Therefore, interfaces
provide a way for a DPsim simulation to exchange data with the environment.
This data is stored in the form of [Attributes]({{< ref "./Attributes/index.md" >}}) and can be **imported** or **exported** in every simulation time step. Exporting an attribute means that on every time step, the current value of that attribute is read and written out to the environment.
Importing an attribute means that on every time step, a new value is read from the environment and the attribute in the simulation is updated to match this value.
Interfaces can be used to exchange simulation signals between a DPsim simulation and other soft- or hardware, for example an MQTT-broker or an FPGA.
Simulation signals in the form of [Attributes]({{< ref "./Attributes/index.md" >}}) can be **imported** or **exported** once per simulation time step.
Interfaces are subclasses of `Interface` and implement the methods `addExport` and `addImport`, which add dependencies to the passed attribute that forward the attribute value from or to the interface.
This way, attributes that are imported are read from the interface before they are used in any DPsim component.
Attributes that are exported are written to the interface after they are set by a DPsim component.

## Configuring an Interface
On the configuration level, an interface is an instance of the `Interface` class.
Because the base `Interface` class requires an instance of an `InterfaceWorker` to construct, it is recommended to not use this base class directly,
but instead construct a subclass derived from `Interface` which internally handles the construction of the `InterfaceWorker`. Currently, there exists
only one such subclass in DPsim which is the `InterfaceVillas`.
## Interfacing with VILLASnode

### Configuring the InterfaceVillas
> This feature requires the compilation of DPsim with the `WITH_VILLAS` feature flag. For use of the VILLASnode interface in python, the `dpsimpyvillas` target has to built in addition to the normal `dpsimpy` package.
> This feature requires the compilation of DPsim with the `WITH_VILLAS` feature flag. For use of the `InterfaceVillas` in python, the `dpsimpyvillas` target has to built in addition to the normal `dpsimpy` package.
The VILLASnode interface is designed to make use of the various node types and protocols supported by the [VILLASframework](https://github.com/VILLASframework/node).
By utilizing the nodes provided by VILLASnode, it can be configured to import and export attributes using a wide range of protocols.
There are two interface implementations for VILLASnode: `InterfaceVillasQueued` and `InterfaceVillasQueueless`.
`InterfaceVillasQueued` uses a ring buffer to store signal data between DPsim and VILLASnode to allow the protocol used in VILLASnode to operate at a different rate and non-synchronized to the DPsim time step.
`InterfaceVillasQueueless` uses direct communication with a VILLASnode node type implementing a specific protocol without using a buffer, thus enabling significantly lower latency communication.
With `InterfaceVillasQueueless`, the protocol operates at the time step of DPsim, i.e., an attribute update directly triggers a `write()` call to the connected VILLASnode node type.
`InterfaceVillasQueued` should be used when using non- or soft real-time protocols or communication mediums, such as MQTT or connections via the internet.
`InterfaceVillasQueueless` should be used when communicating using reliable, low latency, real-time protocols, e.g., with FPGAs, via dedicated fibre networks, or with local real-time applications.

The `InterfaceVillas` is an interface designed to make use of the various node types and protocols supported by the [VILLASframework](https://github.com/VILLASframework/node). By utilizing the nodes provided by VILLASnode, the `InterfaceVillas` can be configured to import and export attributes from and to a wide range of external services. To create and configure an `InterfaceVillas` instance, create a new shared pointer of type `InterfaceVillas` and supply it with a configuration string in the first constructor argument. This configuration must be a valid JSON object containing the settings for the `VILLASnode` node that should be used for data import and export.
This means that the JSON contains a `type` key describing what node type to use, as well as any additional configuration options required for this node type. The valid configuration keys can be found in the [VILLASnode documentation](https://villas.fein-aachen.org/doc/node-node-types.html).
To create and configure one of the VILLASnode interface instance, create a new shared pointer of type `InterfaceVillasQueued` or `InterfaceVillasQueueless` and supply it with a configuration string in the first constructor argument.
This configuration must be a valid JSON object containing the settings for the VILLASnode node type that should be used for data import and export.
This means that the JSON contains a `type` key describing what node type to use, as well as any additional configuration options required for this node type.
The valid configuration keys can be found in the [VILLASnode documentation](https://villas.fein-aachen.org/doc/node-node-types.html).

After the `InterfaceVillas` object is created, the `exportAttribute` and `importAttribute` methods can be used to set up the data exchange between
the DPsim simulation and the configured node. For an explanation of the various parameters, see the code documentation in `InterfaceVillas.h`. The attributes given as the first parameter to these methods are attributes belonging to components in the simulation which should be read or updated by the interface.
As an example, for exporting and importing attributes via the MQTT protocol, the `InterfaceVillas` can be configured as follows:
> Note for `InterfaceVillasQueueless`:
> The queueless interface expects the first input signal in the VILLASnode configuration to be a sequence number that is incremented every time step.
> If the value of the sequence number is not incremented by one between two consecutive time steps, an overrun is detected.
> Because logging outputs can cause large delays and overruns should not occur spuriously, the interface only reports a warning when a large number of overruns occur.
After the object is created, the `exportAttribute` and `importAttribute` methods can be used to set up the data exchange between the DPsim simulation and the configured node.
The attributes given as the first parameter to these methods are attributes belonging to components in the simulation which should be read or updated by the interface.
As an example, for exporting and importing attributes via the MQTT protocol, the VILLASnode interfaces can be configured as follows:

Using C++:
```cpp
Expand All @@ -42,7 +53,7 @@ std::string mqttConfig = R"STRING({
})STRING";

// Creating a new InterfaceVillas object
std::shared_ptr<InterfaceVillas> intf = std::make_shared<InterfaceVillas>(mqttConfig);
std::shared_ptr<InterfaceVillasQueued> intf = std::make_shared<InterfaceVillasQueued>(mqttConfig);

// Configuring the InterfaceVillas to import and export attributes
intf->importAttribute(evs->mVoltageRef, 0, true, true);
Expand Down Expand Up @@ -91,7 +102,7 @@ sim.addInterface(intf);
This method will add two new [Tasks]({{< ref "./Scheduling/index.md" >}}) to the simulation. The interface's `PreStep` task is set to modify all attributes that are imported from the environment and is therefore scheduled to execute before any other simulation tasks that depend on these attributes.
The interface's `PostStep` task is set to depend on all attributes that are exported to the environment and is therefore scheduled to execute after any other simulation tasks that might modify these attributes. To prevent the scheduler from just dropping the `PostStep` task since it does not modify any attributes and is therefore not seen as relevant to the simulation, the task is set to modify the `Scheduler::external` attribute.
Note that the execution of these tasks might not necessarily coincide with the point in time at which the values are actually written out to or read from the environment.
This is because the interface internally spawns two new threads for exchanging data with the environment and then uses a **lock-free queue** for communication between these reader and writer threads, and the simulation. Because of this, time-intensive import or export operations will not block
This is because the interface internally spawns two new threads for exchanging data with the environment and then uses a **lock-free queue** for communication between these reader and writer threads, and the simulation. Because of this, time-intensive import or export operations will not block
the main simulation thread unless this is explicitly configured in the interface's `importAttribute` and `exportAttribute` methods.
## Synchronizing the Simulation with the Environment
Expand All @@ -103,4 +114,4 @@ which have `syncOnSimulationStart` set, the `Simulation::sync` will be called be
- block until all attributes with `syncOnSimulationStart` set have been read from the environment at least once
- write out all exported attributes again
Note that this setting operates independently of the `blockOnRead` flag. This means that with both flags set, the simulation will block again after the synchronization at the start of the first time step until another value is received for the attribute in question.
Note that this setting operates independently of the `blockOnRead` flag. This means that with both flags set, the simulation will block again after the synchronization at the start of the first time step until another value is received for the attribute in question.

0 comments on commit ab0d97a

Please sign in to comment.