[bug]: Blocked IP address (security.ip-blocked) of the http port behind apache

[samweisgamdschie]

What happened?

After installing stalwart via docker, leaving common parameters at default values i configured an apache web server in front of the http port. It works for a while, but now I get

2025-01-06T21:16:35Z INFO Blocked IP address (security.ip-blocked) listenerId = "http", localPort = 8080, remoteIp = 172.18.0.1, remotePort = 60454

When I try reaching the management interface.

At first, there is no information available in documentation from which feature this message comes from.

I tried the following config settings with a restart afterwards with no success/change in getting rid of the block above.

server.allowed-ip = [ "172.18.0.1" ]
server.allowed-ip.0 = "172.18.0.1" #  <= not even working, I got an error.
server.allowed-ip = "172.18.0.1"
server.allowed-ip = { "172.18.0.1" }

server.http.allowed-endpoint = [ { if = "starts_with( remote_ip, '172.1.' )", then = "200" },{ else = "400" } ]


server.auto-ban.auth.rate = "100/1m"
server.auto-ban.abuse.rate = "100/1m"
server.auto-ban.loiter.rate = "100/1m"
server.auto-ban.scan.rate = "100/1m"
server.http.hsts = false
server.http.permissive-cors = false

Because of the fact, that I cant configure via the UI, I did it via
# vi /var/lib/docker/volumes/stalwart_data/_data/etc/config.toml

Am I missing something here?

Here is my obviously straight forward config:

authentication.fallback-admin.secret = "shurelysecure"
authentication.fallback-admin.user = "admin"
directory.internal.store = "rocksdb"
directory.internal.type = "internal"
lookup.default.hostname = "mail.mydomain.com"
server.listener.http.bind = "[::]:8080"
server.listener.http.protocol = "http"
server.listener.https.bind = "[::]:443"
server.listener.https.protocol = "http"
server.listener.https.tls.implicit = true
server.listener.imap.bind = "[::]:143"
server.listener.imap.protocol = "imap"
server.listener.imaptls.bind = "[::]:993"
server.listener.imaptls.protocol = "imap"
server.listener.imaptls.tls.implicit = true
server.listener.pop3.bind = "[::]:110"
server.listener.pop3.protocol = "pop3"
server.listener.pop3s.bind = "[::]:995"
server.listener.pop3s.protocol = "pop3"
server.listener.pop3s.tls.implicit = true
server.listener.sieve.bind = "[::]:4190"
server.listener.sieve.protocol = "managesieve"
server.listener.smtp.bind = "[::]:25"
server.listener.smtp.protocol = "smtp"
server.listener.submission.bind = "[::]:587"
server.listener.submission.protocol = "smtp"
server.listener.submissions.bind = "[::]:465"
server.listener.submissions.protocol = "smtp"
server.listener.submissions.tls.implicit = true
server.auto-ban.auth.rate = "100/1m"
server.auto-ban.abuse.rate = "100/1m"
server.auto-ban.loiter.rate = "100/1m"
server.auto-ban.scan.rate = "100/1m"
storage.blob = "rocksdb"
storage.data = "rocksdb"
storage.directory = "internal"
storage.fts = "rocksdb"
storage.lookup = "rocksdb"
store.rocksdb.compression = "lz4"
store.rocksdb.path = "/opt/stalwart-mail/data"
store.rocksdb.type = "rocksdb"
tracer.log.ansi = false
tracer.log.enable = true
tracer.log.level = "info"
tracer.log.path = "/opt/stalwart-mail/logs"
tracer.log.prefix = "stalwart.log"
tracer.log.rotate = "daily"
tracer.log.type = "log"

docker container version is v0.10.7

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

• connecting to http port either via ProxyPass from apache or with curl via cli directly:

$ curl http://localhost:8003
curl: (52) Empty reply from server

where I get

2025-01-06T21:32:25Z INFO Blocked IP address (security.ip-blocked) listenerId = "http", localPort = 8080, remoteIp = 172.18.0.1, remotePort = 39380

Version

v0.10.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

None

What operating system are you using?

Docker

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[andreymal]

leaving common parameters at default values

But you should have configured server.http.use-x-forwarded, I guess?

[samweisgamdschie]

I haven't configured this flag. I use apache config like this, where i tried both, with and without ProxyPreserveHost (where stalwart should not need to have server.http.use-x-forwarded = true set, if I am right).

        ProxyPass / http://127.0.0.1:8003/
        ProxyPassReverse / http://127.0.0.1:8003/
        # ProxyPreserveHost On

But with no success :/, see

2025-01-07T20:00:02Z INFO Blocked IP address (security.ip-blocked) listenerId = "smtp", localPort = 25, remoteIp = 172.18.0.1, remotePort = 43352
2025-01-07T20:18:32Z INFO Blocked IP address (security.ip-blocked) listenerId = "http", localPort = 8080, remoteIp = 172.18.0.1, remotePort = 52632

==> /var/log/apache2/mail_error.log <==
[Tue Jan 07 19:58:18.895736 2025] [proxy_http:error] [pid 194209:tid 194209] (20014)Internal error (specific information not available): [client 1.2.3.4:41398] AH01102: error reading status line from remote server 127.0.0.1:8003
[Tue Jan 07 19:58:18.895837 2025] [proxy:error] [pid 194209:tid 194209] [client 1.2.3.4:41398] AH00898: Error reading from remote server returned by /

==> /var/log/apache2/mail_access.log <==
1.2.3.4 - - [07/Jan/2025:19:58:18 +0000] "GET / HTTP/1.1" 502 889 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0"

[andreymal]

Stalwart always needs this flag because it needs to know that it can actually trust the X-Forwarded-For header.

Anyway, this doesn't matter now because Apache is already blocked and can't even send X-Forwarded-For to Stalwart. You have to find another way to reach Stalwart without using the blocked IP.

If you have (or can install) curl inside the Stalwart container, you can try clearing blocked IPs by sending an API request to 127.0.0.1, something like:

docker exec -it stalwart-container-name bash
apt-get update && apt-get -y install curl
curl -u 'admin:PASSWORD' \
  --header 'Content-Type: application/json' \
  --data '[{"type": "Clear", "prefix": "server.blocked-ip."}]' \
  http://127.0.0.1:8080/api/settings

It should print {"data":null}, then restart the Stalwart container

[wrenix]

type needs to written small

-  --data '[{"type": "Clear", "prefix": "server.blocked-ip."}]' \
+  --data '[{"type": "clear", "prefix": "server.blocked-ip."}]' \

[andreymal]

This depends on the Stalwart version - "clear" in 0.11 and "Clear" in 0.10

[mdecimus]

Enable the proxy protocol in your reverse proxy so Stalwart knows where the connections are coming from. If you are just forwarding HTTP traffic then enable the setting to obtain the remote IP from the X-Forwarded headers. More details at https://stalw.art/docs

[samweisgamdschie]

A quick test with apache config like

        ProxyPass /asdf http://127.0.0.1:8005/
        ProxyPassReverse /asdf http://127.0.0.1:8005/

and netcat listening on Port 8005 with nc -l -p 8005, I get the following headers set:

# nc -l -p 8005

GET / HTTP/1.1
Host: 127.0.0.1:8005
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
DNT: 1
Cookie: PHPSESSID=fjdu1d95bqjqpsn7t9fb1sr3jf; K2P=desktop
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Priority: u=0, i
X-Forwarded-For: 91.114.142.77
X-Forwarded-Host: mail.mydomain
X-Forwarded-Server: mail.mydomain
Connection: Keep-Alive

where you can see, that apache sends my client IP correctly in the header X-Forwarded-For.

But I still got banned (see logs above).

Do I need to set X-Forwarded, Forwarded or X-Forwarded-For?

[mdecimus]

You need to enable parsing of the X-Forwarded header as others mentioned.
If your IP is already blocked you can unblock it from local host using the REST API or CLI.

[wrenix]

What is the Problem here?

that the log message contains the wrong ip address?

2025-01-06T21:16:35Z INFO Blocked IP address (security.ip-blocked) listenerId = "http", localPort = 8080, remoteIp = 172.18.0.1, remotePort = 60454

or that the server.allowed-ip has no effect? (i have that problem for my health-check)

or that the proxy setup do not forward the correct ip-address?