ip collection does not seem to work properly with latest ModSecurity

[ne20002]

I am using @rbl xbl.spamhaus.org. rule for protecting my Fediverse server and I face the issue that with latest version of either ModSecurity or the docker.io/owasp/modsecurity-crs:nginx container the collection handling for the IP collection seems to not longer work properly.

In my pi-hole I see a huge number of queries to .xbl.spamhaus.org for all the ip addresses checked and with queries to the same ip based query many times where I would expect that this is not the case due to how I set up the ModSecurity rules.

• The issue is happening with the latest version of the docker.io/owasp/modsecurity-crs:nginx container.
• It is far less a occuring with docker.io/owasp/modsecurity-crs:4.10-nginx-202501050801 even when mounting ruleset 4.14 to this container.
• I don't believe it is a problem with the CRS.

Based on what I see I would see it as if

• the expirevar for the IP collection is not working properly
• maybe the persistent store is causing the problem (as I don't see any problems with the TX collection which is also a collection but not persistet). This might be an docker image problem.

According to modsecurity.conf the SecTmpDir is set to /tmp/modsecurity/tmp (as data and upload directory are also located in /tmp/modsecurity).
This directory is empty. Wouldn't this be the directory for persistent storage and shouldn't there be a file for the IP collection database?

This is the rules I use:

# xbl.spamhaus.org to block malicious/infected ips
SecAction \
    "phase:1,id:1100,\
    t:none,pass,nolog,\
    tag:'COLLECTIONS',\
    initcol:ip=%{remote_addr},\
    setvar:'tx.real_ip=%{remote_addr}'"

SecRule IP:SPAMMER "@eq 1" \
    "phase:1,id:1101,\
    t:none,deny,log,auditlog,\
    msg:'Request from Known SPAM Source (Previous RBL Match)',\
    tag:'AUTOMATION/MALICIOUS',\
    severity:'CRITICAL',\
    setvar:'tx.msg=%{rule.msg}'"

SecRule IP:PREVIOUS_RBL_CHECK "@eq 1" "phase:1,id:1102,t:none,pass,nolog,skipAfter:END_RBL_LOOKUP"

SecAction "phase:1,id:1103,\
    t:none,pass,nolog,\
    setvar:ip.previous_rbl_check=1,\
    expirevar:ip.previous_rbl_check=3600"

SecRule REMOTE_ADDR "@rbl xbl.spamhaus.org." \
   "phase:1,id:1104,\
    t:none,deny,log,auditlog,\
    msg:'RBL Match for SPAM Source',\
    tag:'AUTOMATION/MALICIOUS',\
    severity:'CRITICAL',\
    setvar:'tx.msg=%{rule.msg}',\
    setvar:ip.spammer=1,\
    expirevar:ip.spammer=3600"

SecMarker END_RBL_LOOKUP

This is part of my setup.conf before including the CRS rules. As said, it worked much better (but also not perfect) with the older docker image mentioned above.

[ne20002]

I have tried to capture some log output and this is what I found:

Occurence of 205.166.94.38:

[174852413370.405117] [/inbox] [4] (Rule: 1100) Executing unconditional rule...
[174852413370.405117] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852413370.405117] [/inbox] [4] Running (non-disruptive) action: tag
[174852413370.405117] [/inbox] [5] Collection `ip' initialized with value: 205.166.94.38
[174852413370.405117] [/inbox] [4] Running (disruptive)     action: pass.
[174852413370.405117] [/inbox] [4] (Rule: 1101) Executing operator "Eq" with param "1" against IP:SPAMMER.
[174852413370.405117] [/inbox] [4] Rule returned 0.
[174852413370.405117] [/inbox] [4] (Rule: 1102) Executing operator "Eq" with param "1" against IP:PREVIOUS_RBL_CHECK.
[174852413370.405117] [/inbox] [4] Rule returned 0.
[174852413370.405117] [/inbox] [4] (Rule: 1103) Executing unconditional rule...
[174852413370.405117] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852413370.405117] [/inbox] [4] Running (disruptive)     action: pass.
[174852413370.405117] [/inbox] [4] (Rule: 1104) Executing operator "Rbl" with param "xbl.spamhaus.org." against REMOTE_ADDR.
[174852413370.405117] [/inbox] [5] RBL lookup of 205.166.94.38 failed.
[174852413370.405117] [/inbox] [4] Rule returned 0.

[174852421441.145581] [/inbox] [4] (Rule: 1100) Executing unconditional rule...
[174852421441.145581] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852421441.145581] [/inbox] [4] Running (non-disruptive) action: tag
[174852421441.145581] [/inbox] [5] Collection `ip' initialized with value: 205.166.94.38
[174852421441.145581] [/inbox] [4] Running (disruptive)     action: pass.
[174852421441.145581] [/inbox] [4] (Rule: 1101) Executing operator "Eq" with param "1" against IP:SPAMMER.
[174852421441.145581] [/inbox] [4] Rule returned 0.
[174852421441.145581] [/inbox] [4] (Rule: 1102) Executing operator "Eq" with param "1" against IP:PREVIOUS_RBL_CHECK.
[174852421441.145581] [/inbox] [4] Rule returned 0.
[174852421441.145581] [/inbox] [4] (Rule: 1103) Executing unconditional rule...
[174852421441.145581] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852421441.145581] [/inbox] [4] Running (disruptive)     action: pass.
[174852421441.145581] [/inbox] [4] (Rule: 1104) Executing operator "Rbl" with param "xbl.spamhaus.org." against REMOTE_ADDR.
[174852421441.145581] [/inbox] [5] RBL lookup of 205.166.94.38 failed.

[174852413370.405117] [/inbox] [4] (Rule: 1100) Executing unconditional rule...
[174852413370.405117] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852413370.405117] [/inbox] [4] Running (non-disruptive) action: tag
[174852413370.405117] [/inbox] [5] Collection `ip' initialized with value: 205.166.94.38
[174852413370.405117] [/inbox] [4] Running (disruptive)     action: pass.
[174852413370.405117] [/inbox] [4] (Rule: 1101) Executing operator "Eq" with param "1" against IP:SPAMMER.
[174852413370.405117] [/inbox] [4] Rule returned 0.
[174852413370.405117] [/inbox] [4] (Rule: 1102) Executing operator "Eq" with param "1" against IP:PREVIOUS_RBL_CHECK.
[174852413370.405117] [/inbox] [4] Rule returned 0.
[174852413370.405117] [/inbox] [4] (Rule: 1103) Executing unconditional rule...
[174852413370.405117] [/inbox] [4] Running [independent] (non-disruptive) action: setvar
[174852413370.405117] [/inbox] [4] Running (disruptive)     action: pass.
[174852413370.405117] [/inbox] [4] (Rule: 1104) Executing operator "Rbl" with param "xbl.spamhaus.org." against REMOTE_ADDR.
[174852413370.405117] [/inbox] [5] RBL lookup of 205.166.94.38 failed.

So setting the ip address to the collection seems to have no effect?

[airween]

HI @ne20002,

thanks for reporting this issue, and for the detailed explanation.

Let me check this soon and give a feedback.

[ne20002]

I have done a logging session with debug level 9. It creates a rather big log file. From what I see there is no other log statement related to collection. I would have expected at least something according to storing the collection. But .. nothing like that.

@airween, if should do some special tests or can help in any other way ...

[airween]

@ne20002

sorry for the late reply, finally I was able to take a look at this issue.

I tried your setup and got the same result - especially this line in debug.log:

RBL lookup of 205.166.94.38 failed.

I think this line is generated by this part of @rbl operator's code.

    rc = getaddrinfo(host.c_str(), NULL, NULL, &info);

    if (rc != 0) {
        if (info != NULL) {
            freeaddrinfo(info);
        }
        ms_dbg_a(t, 5, "RBL lookup of " + ipStr + " failed.");
        return false;
    }

The used host variable is set up by mapIpToAddress function, this produces the expected format by Spamhaus - in your example above, from the IP 205.166.94.38 this function creates the host name 38.94.166.205.xbl.spamhaus.org and does a DNS query. But because that IP is not on the list, therefore the getaddrinfo() returns with a non-zero value.

I suggest you to first try to use the IP collection with some other way, eg. use a static list with @ipMatch operator.

[ne20002]

Thank you @airween for the answer. Actually the lookup of 38.94.166.205.xbl.spamhaus.org is not the problem.

The inverse number order is how this is handled by the rbl systems (they all use it) and a query to 38.94.166.205.xbl.spamhaus.org results in NXDOMAIN as expected as 205.166.94.38 is not listed as a spam source. So returning false here is correct and 205.94.166.205 should not be added to the ip:spammers collection.

But the 205.166.94.38 seems also not to be added to ip:previous_rbl_check and this is a problem. The ip:previous_rbl_check in this setup is used to bypass further checks for requests from 205.166.94.38.

This is what I wanted to achieve:

• Rule 1101: Check if ip is a known spammer by looking up ip:spammer. If the ip is known as spammer deny the request.

If the ip is not known as spammer...

• Rule 1102: Check is the ip has been checked already within the last hour (timeout 3600s on the ip:previous_rbl_check). If the ip is known as already checked then skip the following test.

If the ip has not been tested before...

• Rule 1103: mark the ip as tested in the ip collection with ip.previous_rbl_check set to 1

• Rule 1104: now test the ip with the rbl check. If the ip is a spammer, add it to ip.collection setting ip:spammer=1

So I expect for the second and following requests from an ip address (within the 3600s timeframe) that it is

• either known as spammer and the request is denied or

• the rbl check is skipped as it has already been done.

The last part is the problem. From what I see is, that the checks on the ip collection (for ip:spammer or ip:previous_rbl_check) always return 0. And this then causes a rbl check for each request as I see it on my pi-hole.

The interessting part is, that it has worked before. So I believe the problem is not the rbl check but the collection.

[airween]

hi @ne20002,

ah, thanks for the clarification, now it's clear. I see the problem.

The interessting part is, that it has worked before. So I believe the problem is not the rbl check but the collection.

Do you remember in which version was that this worked as well?

[ne20002]

I'm using the docker.io/owasp/modsecurity-crs:nginx container, but with my own configuration which includes the setup as described above. I don't use the container's startup scripts, just the nginx with the modsecurity module.

With a previous version (docker.io/owasp/modsecurity-crs:4.10-nginx-202501050801) this worked not fully, but mostly well.

I also read the documentation, and based on that:

• docs say I can't have more than one expirevar statement per rule, that's the reason for having rule 1103 (and it also worked with the previous image).
• the trx collection doesn't have problems.

So I don't think it is the collection code except for (based on what I see happening here):

• it might be a problem with the persistance (one difference between trx and ip collection) or/and
• synchronization (ip collection should be shared between requests, trx not) or/and
• maybe a size problem with the collection or problem with the expirevar code (not used in trx)

This is also why I looked for the persistant storage file. And I haven't found one (so this e.g. might be a container problem).
I don't have the ability to setup a plain nginx with modsecurity so that is why I'm limited to the container.

So I'm limited in options to dig deeper but still I'm willing to try or test whatever you need.

[airween]

CRS's Docker uses LMDB (based on this configuration part), which means you have to have a file inside the container with name modsec-shared-collections (may be with some extension).

Here you can find a Gist, with that you can read the content of the LMDB file (which is a "database": keys and values).

Could you try to read that file (and share the relevant content here)?

[ne20002]

I think I don't have that file.

pi@dmz1:~$ podman exec -it friendica-web /bin/bash
nginx@friendica:/$ id
uid=101(nginx) gid=101(nginx) groups=101(nginx)
nginx@friendica:/$ cd /tmp
nginx@friendica:/tmp$ ls -la 
total 8
drwxrwxrwt 8 root  root   180 Jun  6 00:12 .
dr-xr-xr-x 1 root  root  4096 Jun  6 00:12 ..
drwx------ 2 nginx nginx   40 Jun  6 00:12 client_temp
drwx------ 2 nginx nginx   40 Jun  6 00:12 fastcgi_temp
drwxr-xr-x 5 nginx nginx  100 Jun  6 00:12 modsecurity
-rw-r--r-- 1 nginx nginx    2 Jun  6 00:12 nginx.pid
drwx------ 2 nginx nginx   40 Jun  6 00:12 proxy_temp_path
drwx------ 2 nginx nginx   40 Jun  6 00:12 scgi_temp
drwx------ 2 nginx nginx   40 Jun  6 00:12 uwsgi_temp
nginx@friendica:/tmp$ cd modsecurity/
nginx@friendica:/tmp/modsecurity$ ls -la
total 0
drwxr-xr-x 5 nginx nginx 100 Jun  6 00:12 .
drwxrwxrwt 8 root  root  180 Jun  6 00:12 ..
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 data
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 tmp
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 upload
nginx@friendica:/tmp/modsecurity$ cd data
nginx@friendica:/tmp/modsecurity/data$ ls -la
total 0
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 .
drwxr-xr-x 5 nginx nginx 100 Jun  6 00:12 ..
nginx@friendica:/tmp/modsecurity/data$ cd ../tmp
nginx@friendica:/tmp/modsecurity/tmp$ ls -la
total 0
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 .
drwxr-xr-x 5 nginx nginx 100 Jun  6 00:12 ..
nginx@friendica:/tmp/modsecurity/tmp$ cd ../upload/
nginx@friendica:/tmp/modsecurity/upload$ ls -la
total 0
drwxr-xr-x 2 nginx nginx  40 Jun  6 00:12 .
drwxr-xr-x 5 nginx nginx 100 Jun  6 00:12 ..
nginx@friendica:/tmp/modsecurity/upload$ 

nginx@friendica:/$ find / -type f -name "modsec-shared-collection*"
find: '/root': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/lib/apt/lists/partial': Permission denied
find: '/proc/tty/driver': Permission denied

[ne20002]

@airween

Can you help me with defining (a few) rules that should result in writing a lmdb file? Maybe by setting dummy values?
I will then check if it works and also do a debug log on debug level 9.

This way we may get info if writing a lmdb is working at all or not with the container.

[airween]

@ne20002,

Today I investigated this issue and found a few problems. I will write a summary later.

[airween]

Here is what I found:

• in case of LMDB, collections' data (except TX) stored in a file
• the path of this file is hardcoded
• this path is the / (root) directory; engine needs to create, open and write files /modsec-shared-collections and /modsec-shared-collections-lock in name of running user (www-data? or I don't know what's that in Docker...)
• unfortunately if the engine cannot open the file, this fact remains hidden - see this part of code; there is no else branch (rc == 0 means the file was opened successfully)
• the other problem is that the collections are case sensitive in case of using LMDB; if you want to store and retrieve a variable in a collection (like you do in your example), then you should put and refer the key with the same format. In your rules you use mixed capital and lower case.

I changed your rules and set the keys to upper case format, then I get this:

[174993034342.235412] [/] [4] (Rule: 1102) Executing operator "Eq" with param "1" against IP:PREVIOUS_RBL_CHECK.
[174993034342.235412] [/] [4] Rule returned 0.
[174993034342.235412] [/] [9] Matched vars cleaned.
[174993034342.235412] [/] [4] (Rule: 1103) Executing unconditional rule...
[174993034342.235412] [/] [4] Running [independent] (non-disruptive) action: setvar
--
[174993035551.202951] [/?q=1] [4] (Rule: 1102) Executing operator "Eq" with param "1" against IP:PREVIOUS_RBL_CHECK.
[174993035551.202951] [/?q=1] [9] Target value: "1" (Variable: IP:172.35.40.50::::PREVIOUS_RBL_CHECK)
[174993035551.202951] [/?q=1] [9] Matched vars updated.
[174993035551.202951] [/?q=1] [4] Rule returned 1.
[174993035551.202951] [/?q=1] [9] Saving msg: Found IP in collection.

As you can see, in the second request the rule 1102 matched.

Other sad fact, that this does not work in case of in-memory backend. In that case the collection handling does not work to me at all - this needs more investigations.

Workaround

Even though I changed the code and put the database to under /tmp, I tried these steps:

sudo systemctl nginx stop
sudo rm /tmp/modsec-shared-collections*
sudo touch /tmp/modsec-shared-collections /tmp/modsec-shared-collections-lock
sudo chown www-data:www-data /tmp/modsec-shared-collections /tmp/modsec-shared-collections-lock
sudo systemctl nginx start

After the engine started and received the first request, the files were filled with data, so this could work in your case too.

I don't know how to solve it permanently in Docker. @theseion do you have some idea?

Note that if the engine runs and uses those files, you can open and look its content with this version of lmdbread, this can open the file for read in a different process.

[ne20002]

Thank you very much @airween

I updated my config to fix the lower-/uppercase issue (all lowercase now).

According to the documentation the storage path for the persistant collections should be SecDataDir (but is not supported in v3). This is by default set to /tmp/modsecurity/data/ in the docker image. This path is created and it is owned by user nginx but is empty. So I assume you're right with the / being the path used.

I tried to create the files as u did with touch by mounting a shell script into the docker-entrypoint.d directory (mounting the script file worked).
I could touch the files in /tmp/modsecurity/data/ but it is not used (still always 0 bytes size). So again, SecDataDir seems not be used as path.
I wasn't able to create the files in the root directory or in /var/www. In both cases I got a 'Permission denied'. Seems as if the scripts in docker-entrypoint.d are runned with the nginx user.

I have now created those two files on my host and mounted them rw to ogw to the root directory inside the container. The files seems to be used as the size increased. According to the debug log it now works. :)

I believe the best option to solve the issue would be to fix the creation of the persistent storage to use SecDataDir as according to the documentation. If so, no change to the docker image would be required.

And also an error log message should be written if the attempt to create or write to the persistent store fails.

[airween]

I updated my config to fix the lower-/uppercase issue (all lowercase now).

I think that has no effect, because as you wrote you don't have the database files...

According to the documentation the storage path for the persistant collections should be SecDataDir (but is not supported in v3). This is by default set to /tmp/modsecurity/data/ in the docker image. This path is created and it is owned by user nginx but is empty. So I assume you're right with the / being the path used.

Vainly, because - as your wrote too - SecDataDir is not supported in v3. Unfortunately, in default modsecurity.conf there is no any warning... :(

I tried to create the files as u did with touch by mounting a shell script into the docker-entrypoint.d directory (mounting the script file worked). I could touch the files in /tmp/modsecurity/data/ but it is not used (still always 0 bytes size). So again, SecDataDir seems not be used as path. I wasn't able to create the files in the root directory or in /var/www. In both cases I got a 'Permission denied'. Seems as if the scripts in docker-entrypoint.d are runned with the nginx user.

May be @theseion can help with this.

And also an error log message should be written if the attempt to create or write to the persistent store fails.

I think Nginx shouldn't start if a necessary file does not exists...

Anyway, I think this is another important part of the engine that we have to care about and fix the issues.

Moreover, if we deal with it, we should do some other missing things. Eg. it always bothered that a user have to choose in build time which backend want they use for collections (in-memory or LMDB). There is no option to build LMDB and choose in-memory if it's better for the requirements. I was thinking to build two separated packages from the library (for Debian and Ubuntu systems): once without LMDB and another one with it. Debian developers helped me, ahd suggested to solve this issue inside the library, and don't split the package into more parts. There we can add some code which controls the database path.

[ne20002]

I agree with the idea to fix a few more things then.

For me, with mounting the files into the docker container in its root directory, it is working now. Thank you very much for the investigation, I wouldn't have expected this as the reason for the issue.

By documentation SecDataDir should be the place for the persitent storage. So this should be implemented anyway. An option to choose for memory or lmdb would still be a good idea (but this is also not available in v2, am I right?)

For the docker image it would be an option to create/provide the two files writeable in the root directory by default. So persistent collection would work and other user won't run into this problem. But it would break the read-only feature in case someone uses the collections. If I remember correctly, collections are used in PL2 so this can currently be considered broken for the docker image.

[airween]

For me, with mounting the files into the docker container in its root directory, it is working now. Thank you very much for the investigation, I wouldn't have expected this as the reason for the issue.

you're welcome!

By documentation SecDataDir should be the place for the persitent storage.

I have to investigate this in case of v2, but I have some concerns regarding to use this as the path. For eg. what if a user has several vhost contexts, and set this value as a different locations...? Now I can't tell yo what could we expect.

So this should be implemented anyway. An option to choose for memory or lmdb would still be a good idea (but this is also not available in v2, am I right?)

V2 does not support in-memory collections (only the TX, but there is no other option). This "feature" was introduced in v3.

For the docker image it would be an option to create/provide the two files writeable in the root directory by default. So persistent collection would work and other user won't run into this problem. But it would break the read-only feature in case someone uses the collections. If I remember correctly, collections are used in PL2 so this can currently be considered broken for the docker image.

Yes, but as I wrote, if a user chooses in-memory collection then it does not work at all... we have to inspect that too.

[theseion]

I wasn't able to create the files in the root directory or in /var/www. In both cases I got a 'Permission denied'. Seems as if the scripts in docker-entrypoint.d are runned with the nginx user.

That's on purpose :) We could probably create the files as symlinks to a writable location. I'm not sure it's worth the effort though. @ne20002 you're the first with this issue that I've seen and we now have a workaround. I'd rather wait for a proper working solution in ModSecurity than having to document how this is broken and how the workaround works for the container image.