Beginners Questions

[poldi871]

Hello,

I am evaluating libelektra for an embedded system under BuildRoot. And as noted in the FAQ of libelektra "ask before you search too long", I'll do that now ;-)
In advance, after reading a lot (but not yet all) of the documentation and viewed the youtube videos, it seems to be a great work, even a kind of a developer's dream, if it will work for me.

Here are my problems / questions:

• My evaluation system is a Leap-15.2. Unfortunately there are only versions of 0.8.20 out there. I've searched for SRPM's of newer versions, because I would not use an old version for my system. I have some "hen egg problem" here. On the one hand my build under BuildRoot was successful at the first look, but at runtime there are still problems (warnings C01200), so the build recipe will need some more efforts. On the other hand I don't want to invest too much, in case it turns out that libelektra is not suitable after all. I've also looked through "libelektra4 aus Projekt home:bekun:devel", but there are also only old versions. Is there any RPM or SRPM for the 0.9.2 version?

• By the way BuildRoot:
  I found a recipe for OpenWrt. Does a recipe exist for BuildRoot already?

• On a first simple test, I have an entry in a mounted "ini"-file like this:
  Parameter = "[10,20]; [11,35]"
  Reading the value with "kdb get" results in: [10,20]; [11,35]
  Writing that value back to the configuration file results in: [10,20]; [11,35] (without the quotes)
  It is not possible to quote this entry with \" or "". If I do a double quoting, double quotes are written:
  kdb set /users/Parameter ""[10,20]; [11,35]""
  writes: Parameter = ""[10,20]; [11,35]"" to the file.

• On a second test I was not able to process structured "ini"-files.
  Example:
  [Services]
  [Services:html]
  port = 80
  [Services:ssh]
  port = 22
  On a "kdb ls /user/example" I get:
  ...
  /user/example/Services/[Services:html]
  but I don't get the "port" key.
  If I do a "kdb get /user/example/Services/[Services:html]" I get a "duplicated key entry in INI file"
  What's wrong?

Thanks in advance.

Poldi

[markus2330]

Thank you for writing! Always good to hear when someone starts using Elektra 💖

May I ask you where you intend to use Elektra?

"ask before you search too long"

🚀

even a kind of a developer's dream

Very nice to hear 🎆

Is there any RPM or SRPM for the 0.9.2 version?

This is more @darddan and @mpranj's domain. We have spec files (e.g. #2519) which worked across different versions and our CI system also uses Fedora. I am not sure if our CI produces (S)RPMs, though.

I found a recipe for OpenWrt.

Yes, this is very well maintained by @haraldg.

Does a recipe exist for BuildRoot already?

I do not know.

@darddan can you look into BuildRoot?

What's wrong?

The INI plugin is a bit quirky. @bauhaus93 develops a TOML plugin #3292 which is superior in many ways and will replace the INI plugin for most use-cases. @bauhaus93 can you maybe answer the question? Does the TOML plugin makes this quoting/structure shown above as wanted?

[bauhaus93]

@poldi871 @markus2330
I just checked it with the toml plugin, and it seems to produce the correct quotings and structure. Did I translate the second example correctly into toml?

I created the following script for checking. Command output is annotated with #>.
(The kdb file command is used to get the file path of a certain mount point)

#!/bin/sh

# Mount file
sudo kdb mount example.toml user/example toml type

# Write first example to file
echo 'Parameter = "[10,20]; [11,35]"' > `kdb file user/example`

kdb get 'user/example/Parameter'
#> [10,20]; [11,35]

kdb set 'user/example/Parameter' '[10,20]; [11,35]'
#> Set string to "[10,20]; [11,35]"

cat `kdb file user/example`
#> Parameter = "[10,20]; [11,35]"

# Write second example to file
echo -e '[Services.html]\nport = 80\n[Services.ssh]\nport = 22' > `kdb file user/example`

cat `kdb file user/example`
#> [Services.html]
#> port = 80
#> [Services.ssh]
#> port = 22

kdb ls user/example
#> user/tests/storage/Services/html
#> user/tests/storage/Services/html/port
#> user/tests/storage/Services/ssh
#> user/tests/storage/Services/ssh/port

# Get port values
kdb get user/example/Services/html/port
#> 80
kdb get user/example/Services/ssh/port
#> 22

# Cleanup
kdb rm -r user/example
sudo kdb umount user/example

[poldi871]

Hello,

thanks for the quick and good answers!

• "Thank you for writing! Always good to hear when someone starts using Elektra sparkling_heart":
  I'm making embedded SW for decades now, and it annoyed me all this time to have no configuration at a certain abstraction level. I had a very similar idea (build a registry of keys and values) a long time ago when I was chief developer of a little company, but never got the man power to realize it. At that time the majority of decision-makers decided to implement a proprietary webmin based system. Some times later I became a free developer and as a "one man band" didn't have even less resources for such a task ;-)
  Now and then I've looked for configuration frameworks which are usable for embedded systems, but obviously I never stumbled upon this project. The search always resulted in projects which were (in my mind) too proprietary, or too specialized, also preventing an ability to choose a practical migration strategy.
  Elektra seems to meet all of these requirements and I wonder why it hasn't become a standard tool yet.

• "May I ask you where you intend to use Elektra?":
  I'm about to implement a next generation of an embedded gateway application for a customer now using "systemd" as the init system built with BuildRoot. The base design of that application reaches back until the late 1990s (strong arm SA110), and evolved over some generations up to now running now on an x86 based system.

• RPM and SRPM:
  I took a quick look at it and it seems to be helpful for evaluation. But meanwhile my focus is already on the BuildRoot recipe.

• Recipe for BuildRoot:
  I had a closer look on the OpenWrt recipe. I think the information there is enough to generate a BuildRoot recipe from it. At the moment there exists no recipe in BuildRoot. I asked, because sometimes you can find BuildRoot recipes outside of BuildRoot itself.

• "ini" plugin:
  Thanks I will try the suggestions.

Poldi

Addendum:
the recipe for BuildRoot works.
Not very "nice" at the moment, but a first step:

################################################################################

libelektra

################################################################################

LIBELEKTRA_VERSION = 0.9.2
LIBELEKTRA_SITE_METHOD = file
LIBELEKTRA_SITE = $(BR2_EXTERNAL)/package/TestBed/libelektra
LIBELEKTRA_INSTALL_STAGING = YES
LIBELEKTRA_CONF_OPTS =
-DTARGET_PLUGIN_FOLDER=""
-DBUILD_FULL=OFF
-DBUILD_STATIC=OFF
-DBUILD_DOCUMENTATION=OFF
-DFORCE_IN_SOURCE_BUILD=ON
-DBUILD_TESTING=OFF
-DKDB_DEFAULT_RESOLVER=resolver_fm_pb_b
-DKDB_DEFAULT_STORAGE=ini
-DENABLE_OPTIMIZATIONS=OFF
-DPLUGINS="ALL;-multifile"
-DICONV_FIND_REQUIRED=ON

LIBELEKTRA_SUPPORTS_IN_SOURCE_BUILD = NO

$(eval $(cmake-package))

[mpranj]

Thank you so much for your interest and for sharing your BuildRoot recipe with us.

Is there any RPM or SRPM for the 0.9.2 version?

Unfortunately we don't have any official ones now, but they are planned. We do offer current DEBs and packages for macOS (via homebrew).

The INI plugin is a bit quirky.

I also agree. If your scenario allows it, wait for us to publish the TOML plugin, which is superior and almost ready. If your project timeline does not allow it, then use what is available.

If you have any further questions, feedback and bugreports (or even success stories) we are happy to hear from you.

[poldi871]

@poldi871 @markus2330
I just checked it with the toml plugin, and it seems to produce the correct quotings and structure. Did I translate the second example correctly into toml?

I created the following script for checking. Command output is annotated with #>.
(The kdb file command is used to get the file path of a certain mount point)

#!/bin/sh

# Mount file
sudo kdb mount example.toml user/example toml type

# Write first example to file
echo 'Parameter = "[10,20]; [11,35]"' > `kdb file user/example`

kdb get 'user/example/Parameter'
#> [10,20]; [11,35]

kdb set 'user/example/Parameter' '[10,20]; [11,35]'
#> Set string to "[10,20]; [11,35]"

cat `kdb file user/example`
#> Parameter = "[10,20]; [11,35]"

# Write second example to file
echo -e '[Services.html]\nport = 80\n[Services.ssh]\nport = 22' > `kdb file user/example`

cat `kdb file user/example`
#> [Services.html]
#> port = 80
#> [Services.ssh]
#> port = 22

kdb ls user/example
#> user/tests/storage/Services/html
#> user/tests/storage/Services/html/port
#> user/tests/storage/Services/ssh
#> user/tests/storage/Services/ssh/port

# Get port values
kdb get user/example/Services/html/port
#> 80
kdb get user/example/Services/ssh/port
#> 22

# Cleanup
kdb rm -r user/example
sudo kdb umount user/example

• sorry, another foolish question:
  How to get the TOML plugin working?
  With 0.9.2 i get:
  kdb mount OtaModemOperatorDaemon.conf user/OtaModemOperatorDaemon toml type The command kdb mount terminated unsuccessfully with the info: Was not able to load such a plugin!

I downloaded the git master branch of today. The build shows the plugins in- and excluded. "toml" is mentioned nowhere.
A file search 'toml' through the source tree only shows some references within the rust binding.

"On stupid" I included "toml" in "-DPLUGINS" -> no chance ;-)

... beg your pardon, I still did not rtfm completely up to now -> should do it, but at the moment it is too exciting to experiment with libelekta ;-))

[mpranj]

You can not test it yet comfortably, because it is not released and not even merged into master.

The development branch can be seen in #3292, but we do not recommend basing any other work on top of development branches.

[poldi871]

I checked out "bauhaus93, libelektra-66cbf47" and found the toml plugin there ;-)

• Build: successful

• Tested 'Parameter = "[10,20]; [11,35]"':
  successful.

• Tested "structured ini file":
  the second example was not exactly reproduced by marcus2330, but it worked nevertheless.

Example-File:

[Services] 
    [Services.html] 
            port = 80 
    [Services.ssh] 
            port = 22

root@Vd:~/.config# kdb ls user/example/Services
user/example/Services
user/example/Services/html
user/example/Services/html/port
user/example/Services/ssh
user/example/Services/ssh/port

• Comments:
  Comments in the file remain intact, that's fine!

• But there was a little "inelegance"
  Setting a value destroys the formatting of the file (indents get lost).
  Command:
  "kdb set user/example/Services/html/port 81"

` And now the file looks like:

[Services]
[Services.html]
port = 81
[Services.ssh]
port = 22

`

Next beginner' problem - export to xml:
I used:

[Services]
[Services.html]
# This is a test comment on Services.html
port = 81
[Services.ssh]
# This is a test comment on Services.ssh
port = 22

The command was: "kdb export user xmltool >example.xml"

The resulting file was:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated by Elektra API. Total of 5 keys. -->
<keyset xmlns="https://www.libelektra.org"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="https://www.libelektra.org elektra.xsd"
>
<key name="user/example/Services">

     <comment><![CDATA[]]></comment>
</key>

<key name="user/example/Services/html">

     <comment><![CDATA[]]></comment>
</key>

<key name="user/example/Services/html/port"
	value="81">

     <comment><![CDATA[]]></comment>
</key>

<key name="user/example/Services/ssh">

     <comment><![CDATA[]]></comment>
</key>

<key name="user/example/Services/ssh/port"
	value="22">

     <comment><![CDATA[]]></comment>
</key>

</keyset>

--> CDATA's are empty.

Background:
At the moment the plan is to create a xml-file at the center and file it to the gateway. On the gateway side there may be some additional processing and the generation of the original config files for those apps, which were not "electrified" from the beginning or are not "electrifiable" at that time. Therefore it would be great to have the comments intact.

By the way:
The manual on "https://doc.libelektra.org/api/current/html/doc_help_kdb-export_md.html" says in the "examples" section:

To view a keyset stored in user/keyset in the XML format:
kdb export user/keyset xmltool

But within the example itself the following is used:

kdb set user/tests/kdb-export/one one
kdb set user/tests/kdb-export/two two
 
kdb export user/tests/kdb-export/ xml
                                  ^^^

-> my be a typo, because trying that form leads to an error message (no such plugin).

• once again concerning the toml plugin:
  is there a plan when it will be merged? It's no problem for me to start working on the "hidden" bauhaus93-release, but it should be available when the other side in the computing center will start testing. I think this will be in a few weeks.

Thanks for your patience!

Poldi
ps: what a damn editor. Writing this text took more time than the tests ;-)

[markus2330]

Setting a value destroys the formatting of the file (indents get lost).

I agree that this would be a nice-to-have for TOML as the files otherwise easily get hard-to-read.

says in the "examples" section

Thank you for pointing this out. This is not a good example, as xmltool is only legacy (to import Elektra 0.7 configs). Tip: With kdb plugin-info xmltool (shows what https://www.libelektra.org/plugins/xmltool contains) you would have seen that the plugin is discouraged.

my be a typo, because trying that form leads to an error message (no such plugin).

It should use the xerces plugin (see also below in that man page), maybe you did not include it, as it needs the xerces XML parser. See https://www.libelektra.org/plugins/xerces

is there a plan when it will be merged? It's no problem for me to start working on the "hidden" bauhaus93-release, but it should be available when the other side in the computing center will start testing. I think this will be in a few weeks.

This should be easily doable. Can you sponsor development to speed things up?

[poldi871]

Thank you for pointing this out. This is not a good example, as xmltool is only legacy (to import Elektra 0.7 configs). Tip: With kdb plugin-info xmltool (shows what https://www.libelektra.org/plugins/xmltool contains) you would have seen that the plugin is discouraged.

my be a typo, because trying that form leads to an error message (no such plugin).

It should use the xerces plugin (see also below in that man page), maybe you did not include it, as it needs the xerces XML parser. See https://www.libelektra.org/plugins/xerces

Thanks, will try it tomorrow. Hope that xerces is buildable in BuildRoot.

is there a plan when it will be merged? It's no problem for me to start working on the "hidden" bauhaus93-release, but it should be available when the other side in the computing center will start testing. I think this will be in a few weeks.

This should be easily doable. Can you sponsor development to speed things up?

Unfortunately not. As I already mentioned, I'm a "one man band". I'm spending my time on my own risk at the moment and this will continue until the result of my evaluation will end in a positive decision. And the risk for me is not only spending my time, but also get an unjustifiable delay in my own project. Even in the positive case I have to prepare a realizable proposal to convince the customer. From the view of the customer, this will be only a very little part of the project and I think he will not even pay for (yes, I'm a technician, not a sales man). I see my part in helping to make the project a little better known by trying to integrate libelektra into my application because i am convinced of this approach. And hand on heart, after all the resources that are already in this project, there are still problems with ini files in the officially available release? I think the reality is, that most evaluators would have stopped evaluation at that point. And I'm still a long way from the end of my tests...

[markus2330]

I'm spending my time on my own risk

💖

Even in the positive case I have to prepare a realizable proposal to convince the customer.

Can you also write here what your goals and requirements are? Maybe we can tell you if and how much implementation effort will be needed and what we might be able to do.

I see my part in helping to make the project a little better known

🚀

And hand on heart, after all the resources that are already in this project, there are still problems with ini files in the officially available release?

Yes, this was also very surprising for me that this is such a hard problem. In the beginning we were hoping that there would already be some library that would simply read&write configuration files and to integrate this library to Elektra. But we did not find such a library, they always fail in either:

• round-tripping (not any KeySet can be represented)
• preserving what the user wrote (comments, spaces, order)
• giving a data structure describing the configuration and not the syntax of the file

The INI plugin was the first attempt to solve all three requirements but it became so complex that it is unmaintainable now. That is why @bauhaus93 wrote the TOML plugin using better parser technology.

I think the reality is, that most evaluators would have stopped evaluation at that point.

We have quite a lot of customers in the embedded systems world (e.g. Broadcom, Toshiba, Kapsch, ...), as in embedded systems human-editable configuration files are usually not a big topic. There are plenty of plugins in Elektra that reliably serialize and deserialize any KeySet with whatsoever weird configurations, e.g. dump, quickdump or mmapstorage. Alternatively, the ni plugin writes anything reliably to and from INI files but does not care about preserving what the user did. But I agree with you, what the admin usually wants is what the toml plugin does: provide all three properties at the same time in a best-effort. But afaik Elektra is the only software that does this at the moment at all, others do not even try properly, e.g. hyperrealm/libconfig#35 or jbeder/yaml-cpp#154.

[poldi871]

Can you also write here what your goals and requirements are? Maybe we can tell you if and how much implementation effort will be needed and what we might be able to do.

Background:
At the moment a gateway is configured by a proprietary system, which was adapted very often in the history of this long running project. And all adaptions had to be downward compatible. You may imagine the number of exceptions the system has to deal with meanwhile.
Driven by the eol of some tools and packets in the current version the decision was made to build a "next generation" based on "systemd" and "DBUS". Because the last and current versions are/were running nearly "bug free" for a long period there is enough time (even for a "one man band") to generate that next generation. The introduction of "systemd" and "DBUS" leads to a quite different system construct. As a consequence the idea was born to take the chance and also clean up the configuration handling and introduce the base for further migration. At least this task alone would have been worth to generate a project by itself. But that's life ;-)

Back to your question:
At the moment I can't give you a serious answer. But thanks for your offer. There are a bunch of standard configuration files (host, hosts, ipsec,...) which are generated at the center and sent to a special directory on the device. The device knows that files and has links established from the standard location to the file in the special directory. There are also a bunch of configuration files necessary for the customer apps. These apps are configured by simple ini files at the moment and would be the candidates for "electrifying" them immediately or during a migration step. The possibility of migration is necessary because not only the device is affected by a change but also at least the whole organization on the center side. In the past we have made too good experience with that strategy, even at next generation steps, and although the current generation is absolutely different from former generations from an implementation point of view.
At the moment my plan is to generate a xml file on the center side and send it to the device for further processing. At first glance an unnecessary intermediate processing, because the base structure of the configuration is quite the same for all devices, but there are exceptions caused by network infrastructure and other reasons that even cause the need to generate quite different files. At the moment this processing is done in the center with a big effort, although the device could handle those exceptions much more transparent and therefore much more effective. For the non exception cases the device would simply load the xml file into the virtual configuration filesystem of libelektra and write that parts to the configuration files of the "non electrified" applications.

The INI plugin was the first attempt to solve all three requirements but it became so complex that it is unmaintainable now. That is why @bauhaus93 wrote the TOML plugin using better parser technology.

I agree concerning the processing of ini files. Last week I studied a lot of competitors, even whole distributions and all have their problems.

We have quite a lot of customers in the embedded systems world (e.g. Broadcom, Toshiba, Kapsch, ...), as in embedded systems human-editable configuration files are usually not a big topic.

Nice to hear that (customers)
human-readable configuration files: in our case this is a big topic.