Let's Encrypt TLS renew

[Loughty]

Hi, this is not actually a problem with GeoIP, but with Let's Encrypt, they are very unflexible about users that use any kind of geoblocker, 'cause for renewing their certificated they use a domain that change of IP often, and they don't want to say what is the pool or list of public IPs they use to add them to the whitelist, they only say to users that don't use geoblocker directly and allow all traffic. The only exception to this was a more moderate solution that is allow all traffic but only to: /.well-known/acme-challenge/

Is this posible todo using geoip-shell? If not, geoip-shell allows you to whitelist a domain instead of an IP? (I haven't tried it yet) If any of that is not posible, there's not problem, I still can temporally disable the geoblocking so the service can renew the certs, something that fortunately is not so often, but I will prefer to leave it full automatic, so any solution on or outside of geoip-shell is welcome :)

[friendly-bits]

Hi, I'll copy-paste relevant bits of a similar discussion in #35 :

geoip-shell deals with the system firewall, and that firewall doesn't understand the notion of domain names. It works with IP addresses. Technically, it is possible to periodically resolve certain domain names to IP addresses and add those addresses to trusted IP addresses list in geoip-shell. This has a few problems: DNS resolution is relatively slow, which limits the number of domains it is feasible to resolve; if IP addresses change after the last periodic DNS resolution then you get incorrect IP addresses, so this won't work; if you only need to access the given service (certbot) once in a long while then there is no point to keep an updated ipset with its IP addresses at all times.

So while technically it is possible to implement, I think at least for the case of certbot, a better solution is to use its built-in feature of pre-hook and post-hook scripts. German-speaking users of geoip-shell came up with this solution here. If you don't speak German, Google Translate seems to translate the text to English pretty well.

---

Another solution would be to use the 'dns-01' challenge rather than the 'http-01' challenge:

https://security.stackexchange.com/questions/256920/what-does-it-mean-to-create-a-lets-encrypt-certificate-automatically-rather-t/256924#256924

This seems more difficult but should still be possible to automate.

[Loughty]

Hi, thanks for the response and the help!

I'm not using Certbot, but Caddy, which have an automatic HTTPS function, but they don't have nothing to execute a script during the operation, as far as I know; but I think is still posible, they leave logs when is about to do that task, and Fail2ban does allow you to run scripts when it detects a pattern in the logs, so, this going to be...

1. filter.d/pattern.conf: where im gonna put the regex that detects the specific log entry when a domain needs to update his certs
2. jail.local: where im gonna put the condition needed (how many times the pattern is detected) to activated an action during an specific amount of minutes...
3. action.d/script.conf: and the action itself, where instead of blocking, they gonna execute two scripts, one to deactivate geoip, and one to activate it again one the amount of minutes passes

I don't know if this is going to work, but if that the case, I'm gonna leave examples in another comment ;D

Also, this is offtopic, but could be relevant to someone: using Geoip-shell in combination with UFW and Fail2ban, it doesn't seem to cause any problems for now (all of them use iptables), although I have not tested it rigorously, and I have no idea what could happen when there are conflicting rules between them, I know that Fail2ban rules have a higher priority than UFW rules, but i'm not sure about geoip-shell, what you know about it?

[friendly-bits]

I'm not using Certbot, but Caddy

I looked at the documentation as well and yeah, looks like Caddy has no option to run a script before and after renewing certificates. Perhaps if someone filed a feature request, the team behind Caddy would implement this?

so, this going to be...

Sounds like a reasonable plan. Please do update here how it goes.

using Geoip-shell in combination with UFW and Fail2ban, it doesn't seem to cause any problems for now (all of them use iptables), although I have not tested it rigorously, and I have no idea what could happen when there are conflicting rules between them

I can not really imagine how geoip-shell rules could conflict with any other rules. geoip-shell simply blocks or allows packets from user-specified certain IP ranges. Packets which are allowed by geoip-shell are still processed by all your other firewall rules and may get dropped by them. UFW (AFAIK) is simply a front-end for iptables. geoip-shell doesn't mind that. The only potential issue might happen if some other utility decides to remove geoip-shell rules, which would disable geoblocking.

I know that Fail2ban rules have a higher priority than UFW rules, but i'm not sure about geoip-shell, what you know about it?

I'm running a very old version of fail2ban on my server and at least that version with these settings creates its rules in table filter. geoip-shell creates its rules in table mangle which is processed earlier. In terms of netfilter hooks, fail2ban uses the input hook, geoip-shell uses the prerouting hook (for inbound geoblocking). It is possible that newer versions of fail2ban act differently but I doubt this.

[blandureauj-hash]

Hello !

Have you managed to implement what you described above?
I'm interested by your fail2ban script to deactivate geoip when attempt to renew the certificate with Caddy

[Loughty]

Hello !

Have you managed to implement what you described above? I'm interested by your fail2ban script to deactivate geoip when attempt to renew the certificate with Caddy

Hi @blandureauj-hash , sorry for the late reply

In my logs Caddy show the next output when try to renew with geoblock:

systemd[1]: Started Caddy.
caddy[91294]: {"level":"info","ts":1759953624.368078,"logger":"tls","msg":"certificate needs renewal based on ARI window","subjects":(...)
caddy[91294]: {"level":"info","ts":1759953624.3800461,"logger":"tls.renew","msg":"acquiring lock","identifier":"ur.domain.com"}
(...)
caddy[91294]: {"level":"info","ts":1759953631.3407302,"msg":"trying to solve challenge","identifier":"ur.domain.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy[91294]: {"level":"error","ts":1759953643.77937,"msg":"challenge failed","identifier":"ur.domain.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection",(...)
caddy[91294]: {"level":"error","ts":1759953643.7797263,"msg":"validating authorization","identifier":"ur.domain.com","problem":{"type":"urn:ietf:params:acme:error:connection",(...) Timeout during connect (likely firewall problem)",(...)
(this repeats several times until give up)

And this when do the same without geoblock:

systemd[1]: Started Caddy.
(...)
caddy[91407]: {"level":"info","ts":1759955256.3931909,"logger":"tls.renew","msg":"acquiring lock","identifier":"ur.domain.com"}
caddy[91407]: {"level":"info","ts":1759955256.3985553,"logger":"tls.renew","msg":"lock acquired","identifier":"ur.domain.com"}
caddy[91407]: {"level":"info","ts":1759955256.399505,"logger":"tls.renew","msg":"renewing certificate","identifier":"ur.domain.com","remaining":1110808.600501692}
(...)
caddy[91407]: {"level":"info","ts":1759955273.577867,"msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/(...)
(...)
caddy[91407]: {"level":"info","ts":1759955275.184867,"msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/(...)
caddy[91407]: {"level":"info","ts":1759955275.199106,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"ur.domain.com","issuer":"acme-v02.api.letsencrypt.org-directory"}
(...)
caddy[91407]: {"level":"info","ts":1759955275.199502,"logger":"tls","msg":"reloading managed certificate","identifiers":["ur.domain.com"]}
caddy[91407]: {"level":"info","ts":1759955275.2005901,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["ur.domain.com"],"new_expiration":1767727763}
(...)
caddy[91407]: {"level":"info","ts":1759977456.699333,"logger":"tls.cache.maintenance","msg":"updated and stored ACME renewal information","identifiers":["ur.domain.com"],"cert_hash":(...)
(here the renew process ends)

(I removed some parts for privacy or not being relevant.)

Then I create these files:

cat /etc/fail2ban/filter.d/caddy-renewal.conf

[Definition]

failregex = caddy\[\d+\]: .*level\":\"error\".*msg\":\"validating authorization\".*likely firewall problem\"

ignoreregex = caddy\[\d+\]: .*level\":\"info\".*msg\":\"certificate renewed successfully\"

cat /etc/fail2ban/action.d/geoip-on-off.conf 

[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
actionstart = 

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
actionstop = 

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
actioncheck = 

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights. 
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionban = /bin/bash -c "/etc/fail2ban/scripts/desactivar_geoblock.sh"

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionunban = /bin/bash -c "/etc/fail2ban/scripts/activar_geoblock.sh"

cat /etc/fail2ban/jail.d/caddy-cert.conf

[caddy-renewal-error]
enabled = true
# NECESARIO PARA LEER LOS LOGS DEL JOURNALCTL
backend = systemd
journalmatch = _SYSTEMD_UNIT=caddy.service

# ESTE ES EL NOMBRE DEL FILTRO REGEX QUE USAREMOS
filter = caddy-renewal
# SE ACTIVARA LUEGO DE DETECTAR ESTE NUMERO DE ERRORES DE RENOVACION
maxretry = 1
# MANTIENE LA ACCION DURANTE EL TIEMPO DEFINIDO
# PASADO ESE TIEMPO DEBERIA ACTIVARSE EL ACTION STOP
bantime = 15m

# NECESARIO PARA EJECUTAR LOS SCRIPTS EN LUGAR DE LA ACCION POR DEFECTO
action = geoip-on-off

And of course, the scripts:

cat /etc/fail2ban/scripts/activar_geoblock.sh

#!/bin/bash
# Script para reactivar el geoblock después de la renovación exitosa del certificado
echo "$(date): Activando geoblock..." | logger -t caddy-renewal
/usr/bin/geoip-shell on
echo "$(date): Geoblock activado." | logger -t caddy-renewal
exit 0

cat /etc/fail2ban/scripts/desactivar_geoblock.sh

#!/bin/bash
# Script para desactivar temporalmente el geoblock para permitir la renovación de Let's Encrypt
echo "$(date): Desactivando geoblock..." | logger -t caddy-renewal
/usr/bin/geoip-shell off
echo "$(date): Geoblock desactivado." | logger -t caddy-renewal
exit 0

(Don't forget to give it execution permissions, I think fail2ban executes as root, so nothing more is needed apart from that)

Take into account that I haven't tested any of this get, so something will probably (and surely) go wrong. I'm bad at scripting and regex, so any corrections are welcome. In the meantime, I'm waiting for the website certificate to expire to see how it goes. There are about two months left until then.

[oraculix]

a better solution is to use its built-in feature of pre-hook and post-hook scripts.

FWIW, this is kind of the solution I'm running, although my setup differs from Loughty's:
I run geoip-shell in my homelab directly on the Caddy machine, so I don't have to waste space and CPU cycles on running it on every target behind Caddy.

Then I resorted to a simple cron job that

• runs once a week,
• switches geoblocking off,
• lets Caddy renew the Letsencrypt certs and
• switches geoblocking on again.

It's a fairly simple and pragmatic solution for the time being, but it leaves my shields down for maybe half a minute. I'd wish for a nicer solution as well, preferrably with a yet-to-be-implemented functionality in Caddy.

Another solution would be to use the 'dns-01' challenge rather than the 'http-01' challenge:

This works fine if you have control over your DNS entries for your domain(s). When you run your environment at home using a typical DynDNS service to cope with always-changing IPs, this often isn't an option.

Cheers, Uwe

[Loughty]

Yep @oraculix you're right, months have passed since then, and as expected, my solution didn't work. When I tried to fix it, I realized that fail2ban wasn't necessary for what I wanted to do, since I can read the expiration date directly from the certificate without having to check the Caddy's logs. So my solution ended up being quite similar to yours, here an edited copy of my script:

#!/bin/bash

DOMAIN="your.domain.com"
CERT_PATH="/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/$DOMAIN/$DOMAIN.crt"

# TIME REMAINING IN SECONDS
UMBRAL_SEGUNDOS=2592000 
# 2592000 ITS 30 DAYS
# CADDY IN THEORY BEGINS RENEWING THE CERTS STARTING FROM THAT

echo "DATE: $(date '+%Y-%m-%d %H:%M:%S')"
echo "DOMAIN: $DOMAIN"

# CHECK IF THE CERT EXISTS
if [ ! -f "$CERT_PATH" ]; then
    echo "ERROR: NO CERT IN $CERT_PATH"
    exit 1
fi

# CERT INFO
EMISOR=$(openssl x509 -noout -issuer -in "$CERT_PATH" | sed 's/issuer= //')
FIN_VALIDEZ=$(openssl x509 -noout -enddate -in "$CERT_PATH" | sed 's/notAfter=//')
FECHA_FIN_S=$(date -d "$FIN_VALIDEZ" +%s)
AHORA_S=$(date +%s)
DIAS_RESTANTES=$(( (FECHA_FIN_S - AHORA_S) / 86400 ))

echo "ISSUER: $EMISOR"
echo "EXPIRE IN: $FIN_VALIDEZ ($DIAS_RESTANTES DAYS LEFT)"

# CHECK IF CERTS EXPIRATION DATE ITS UNDER THE TIME SPECIFIED
if ! openssl x509 -checkend $UMBRAL_SEGUNDOS -noout -in "$CERT_PATH"; then
    echo "THE CERT ITS ABOUT TO EXPIRE."

# IF THATS TRUE, THEN STOP GEOBLOCK, RELOAD CADDY, AND WAIT
    echo "DEACTIVATING GEOIP-SHELL..."
    /usr/bin/geoip-shell off
    echo "RELOADING CADDY TO FORCE RENOVATION..."
    systemctl reload caddy
    echo "PLEASE WAIT 5 MINUTES..."
    sleep 300

# AFTER THAT TIME, START GEOBLOCK AGAIN
    /usr/bin/geoip-shell on
    echo "REACTIVATING GEOIP-SHELL"

# CHECKING THE CERT DATE AGAIN
    NUEVA_FECHA=$(openssl x509 -noout -enddate -in "$CERT_PATH" | sed 's/notAfter=//')
    echo "NEW CERT DATE: $NUEVA_FECHA"

else
    echo "THE CERT ITS STILL VALID."
fi

And to make it execute automatically, just add it to the cronjob, with an ">> /var/log/caddy_renewal_check.log 2>&1" at the end, so u can have a record of the script doing his job. My cronjob was running everyday at midnight (0 */12 * * *), and only stopped geoip if 24 hours left to cert to expire, but reading ur commend I changed it to execute only on sunday at midnight, and stop geoip if 30 days left, that will undoubtedly reduce the number of entries in the log. So my current cronjob now is:

0 0 * * 0 /usr/local/bin/verificar-certificado.sh >> /var/log/caddy_certificado.log 2>&1

I suspect that the same thing could be done with Certbot, not only with Caddy, just change the cert route, and replace the command to reload caddy, for the one that uses Certbot to force is renewing process 🤔

[blandureauj-hash]

Finally, I also did it differently: my homelab restarts every 15 days, so I delayed the startup of geoip-shell so that it starts 5 minutes after the reboot, giving caddy time to do its little things...

[friendly-bits]

Thank you all for sharing your workarounds. I filed a feature request with Caddy, see above.

[friendly-bits]

To update here as well, Caddy maintainer replied

That's already a thing: https://caddyserver.com/docs/caddyfile/options#event-options

I looked at the link briefly and it seems that indeed, functionality similar to Certbot pre- and post-hooks is implemented in Caddy via "events".

I'm not a Caddy user, so I'm not sure whether following is correct but just from the documentation, it seems that you would need to add something like

{
	events {
		on cert_obtaining exec geoip-shell off
		on cert_obtained exec geoip-shell on
		on cert_failed exec geoip-shell on
	}
}

to (your Caddy config file?).

The doc page mentions that in order to be able to run the commands, a third-party plugin is required:
https://github.com/mholt/caddy-events-exec

Again, I do not know whether this works. If it doesn't, please report this in the Caddy issue I opened (and update here if possible).

[oraculix]

Thanks for the update!
I implemented this in Caddy as described, and at least it was able to start up again without errors. 😆
But when I forced a TLS renewal, I noticed an error message in Caddy's journal:

geoip-shell must be run as root

My minimal container setup didn't include sudo, so I had to install it first and add an entry for the user "caddy" to the sudoers directory:

caddy   ALL=(ALL) NOPASSWD:/usr/bin/geoip-shell

At the beginning of the caddyfile, the entry for the event handler now looks like this:

# Switch GeoIP fencing OFF during certificate renewal
# Needs module "caddy-events-exec"
{
        events {
                on cert_obtaining exec sudo geoip-shell off
                on cert_obtained exec sudo geoip-shell on
                on cert_failed exec sudo geoip-shell on
        }
}

Now, after a systemctl restart caddy, I could see a successful outcome in the journal:

...
  24   │ Mar 17 16:53:10 caddy sudo[49790]:    caddy : PWD=/ ; USER=root ; COMMAND=/usr/bin/geoip-shell off
  25   │ Mar 17 16:53:10 caddy sudo[49790]: pam_limits(sudo:session): Could not set limit for 'core' to soft=0, hard=-1: Operation 
       │ not permitted; uid=999,euid=0
  26   │ Mar 17 16:53:10 caddy sudo[49790]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
  27   │ Mar 17 16:53:10 caddy caddy[49791]: Updating the config file... /usr/lib/geoip-shell/geoip-shell-lib-common.sh: line 579: 
       │ /etc/geoip-shell/geoip-shell.conf: Read-only file system
  28   │ Mar 17 16:53:10 caddy caddy[49791]: geoip-shell: Error: setconfig: Failed to write to '/etc/geoip-shell/geoip-shell.conf'
  29   │ Mar 17 16:53:10 caddy caddy[49808]: Removing the geoblocking enable rule for direction 'inbound'... Ok.
  30   │ Mar 17 16:53:10 caddy caddy[49808]: outbound geoblocking is already disabled.
  31   │ Mar 17 16:53:10 caddy sudo[49790]: pam_unix(sudo:session): session closed for user root
  32   │ Mar 17 16:53:12 caddy caddy[49570]: {"level":"info","ts":1773766392.3339403,"msg":"trying to solve challenge","identifier"
...
  40   │ Mar 17 16:53:35 caddy sudo[49857]:    caddy : PWD=/ ; USER=root ; COMMAND=/usr/bin/geoip-shell on
...

HTH, Uwe

[friendly-bits]

Thanks for the update. You do have this in the above log:

27 │ Mar 17 16:53:10 caddy caddy[49791]: Updating the config file... /usr/lib/geoip-shell/geoip-shell-lib-common.sh: line 579:
│ /etc/geoip-shell/geoip-shell.conf: Read-only file system
28 │ Mar 17 16:53:10 caddy caddy[49791]: geoip-shell: Error: setconfig: Failed to write to '/etc/geoip-shell/geoip-shell.conf'
29 │ Mar 17 16:53:10 caddy caddy[49808]: Removing the geoblocking enable rule for direction 'inbound'... Ok.

So apparently, it can control the firewall but can not write to the config file.

[oraculix]

So apparently, it can control the firewall but can not write to the config file.

Yes, I noticed that. Unfortunately, I ran into a rate limit at letsencrypt, so I'll have to wait until I can check this again, later.
OTOH, I already renewed another subdomain with these settings in place, so I'm not sure if the write failure really is an issue in this context.
Seems like the blocking status isn't changed due to the write issue, because I get a

Timeout during connect (likely firewall problem)

while the ACME challenge is running.
Allowing the caddy user to run every command without a password for sudo did not help. I'll go search for more troubleshooting ideas — any help is welcome!

[oraculix]

Found it: My Caddy runs in an unprivileged LXC, which has all kinds of security-related restrictions. I tried a few workarounds with "sudo" and "su -c", but then other restrictions hit, e. g. with iptables.

With Caddy being the primary entrance from the public Internet, I definitely won't consider making this LXC privileged. So I'll fall back to my pragmatic cron-job solution.

My above description should work for for other setups, though.