Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

client: cert renewal errors in a loop until restart #28

Closed
lidel opened this issue Dec 29, 2024 · 2 comments · Fixed by #42
Closed

client: cert renewal errors in a loop until restart #28

lidel opened this issue Dec 29, 2024 · 2 comments · Fixed by #42
Labels
bug Something isn't working

Comments

@lidel
Copy link
Contributor

lidel commented Dec 29, 2024 •
edited
Loading

p2p-forge/client seems to fail to renew cert.
restarting kubo node seems to fix it eventually, but we should look into this, as people running long-lived stable servers may have a window of time where old cert expired and new one is not set up yet.

Problem description

I have test node which had cert long enough to trigger reneval.

Kubo 0.33.0-rc1 seems to detect need to refresh correctly:

  • [email protected]/certificates.go:159 certificate is in configured renewal window based on expiration date

but then it fails to get new one

  • ERROR autotls.renew [email protected]/config.go:919 could not get certificate from issuer {"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/2028917947/338498640095) (ca=https://acme-v02.api.letsencrypt.org/directory)"}

It then fails the same way (no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) with backoff.

Restarting fixed the issue, and the node was able to get new cert.

ℹ️ For reference, full log (unfortunately was at INFO level and not DEBUG) 
2024-12-29T02:53:22.464Z	INFO	autotls.maintenance	[email protected]/maintain.go:584	updated ACME renewal information	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "cert_hash": "4dda083d4f05993691a37adc58ad2153923aa0d8bb645f57fe3f2f255d3e0cec", "ari_unique_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "cert_expiry": "2025-01-28T12:43:11.000Z", "selected_time": "2024-12-29T23:10:37.000Z", "next_update": "2024-12-29T08:53:22.348Z", "explanation_url": ""}
2024-12-29T09:03:22.172Z	INFO	autotls.default_acme_client_fixme.acme_client	acme/ari.go:215	got renewal info	{"names": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "selected_time": "2024-12-29T06:56:02.000Z", "recheck_after": "2024-12-29T15:03:22.172Z", "explanation_url": ""}
2024-12-29T09:03:22.218Z	INFO	autotls.maintenance	[email protected]/maintain.go:584	updated ACME renewal information	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "cert_hash": "4dda083d4f05993691a37adc58ad2153923aa0d8bb645f57fe3f2f255d3e0cec", "ari_unique_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "cert_expiry": "2025-01-28T12:43:11.000Z", "selected_time": "2024-12-29T23:10:37.000Z", "next_update": "2024-12-29T15:03:22.172Z", "explanation_url": ""}
2024-12-29T12:43:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2591990.420553425}
2024-12-29T12:43:21.604Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2591990.395778225}
2024-12-29T12:43:21.604Z	INFO	autotls.maintenance	[email protected]/maintain.go:251	attempting certificate renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2591990.395640887}
2024-12-29T12:43:21.654Z	INFO	autotls.renew	[email protected]/config.go:787	acquiring lock	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"}
2024-12-29T12:43:21.690Z	INFO	autotls.renew	[email protected]/config.go:805	lock acquired	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"}
2024-12-29T12:43:21.691Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2591990.308839202}

2024-12-29T12:43:21.736Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeclient.go:328	waiting on internal rate limiter	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024-12-29T12:43:21.736Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeclient.go:337	done waiting on internal rate limiter	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024-12-29T12:43:21.736Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/2028917947", "account_contact": []}
2024-12-29T12:43:22.767Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/2028917947/338498640095) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2024-12-29T12:43:22.768Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/2028917947/338498640095) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.077152872, "max_duration": 2592000}
2024-12-29T12:44:22.769Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2591929.23017441}
2024-12-29T12:44:22.806Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T12:44:24.060Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650086454) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T12:44:24.060Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650086454) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 2, "retrying_in": 120, "elapsed": 62.369935843, "max_duration": 2592000}
2024-12-29T12:46:24.062Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2591807.937340641}
2024-12-29T12:46:24.065Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T12:46:24.788Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650112194) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T12:46:24.788Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650112194) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 3, "retrying_in": 120, "elapsed": 183.097650058, "max_duration": 2592000}
2024-12-29T12:48:24.789Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2591687.21013008}
2024-12-29T12:48:24.793Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T12:48:25.527Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650140654) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T12:48:25.527Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650140654) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 4, "retrying_in": 300, "elapsed": 303.836685208, "max_duration": 2592000}
2024-12-29T12:53:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2591390.420673534}
2024-12-29T12:53:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2591390.419330619}
2024-12-29T12:53:25.529Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2591386.470713419}
2024-12-29T12:53:25.532Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T12:53:26.626Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650215144) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T12:53:26.626Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650215144) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 5, "retrying_in": 600, "elapsed": 604.9353128, "max_duration": 2592000}
2024-12-29T13:03:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2590790.420398779}
2024-12-29T13:03:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2590790.419263556}
2024-12-29T13:03:26.627Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2590785.372387967}
2024-12-29T13:03:26.629Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T13:03:27.719Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650360824) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T13:03:27.719Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650360824) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 6, "retrying_in": 600, "elapsed": 1206.02860991, "max_duration": 2592000}
2024-12-29T13:13:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2590190.421073503}
2024-12-29T13:13:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2590190.420728749}
2024-12-29T13:13:27.720Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2590184.27937022}
2024-12-29T13:13:27.722Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T13:13:28.849Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650509624) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T13:13:28.849Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650509624) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 7, "retrying_in": 600, "elapsed": 1807.158436529, "max_duration": 2592000}
2024-12-29T13:23:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2589590.421281737}
2024-12-29T13:23:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2589590.420212301}
2024-12-29T13:23:28.850Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2589583.149500662}
2024-12-29T13:23:28.853Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T13:23:29.939Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650648924) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T13:23:29.939Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650648924) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 8, "retrying_in": 1200, "elapsed": 2408.248323, "max_duration": 2592000}
2024-12-29T13:33:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2588990.420629257}
2024-12-29T13:33:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2588990.419444005}
2024-12-29T13:43:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2588390.421302571}
2024-12-29T13:43:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2588390.420199631}
2024-12-29T13:43:29.940Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2588382.059669962}
2024-12-29T13:43:29.944Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T13:43:31.025Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650925814) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T13:43:31.025Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21650925814) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 9, "retrying_in": 1200, "elapsed": 3609.334406574, "max_duration": 2592000}
2024-12-29T13:53:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2587790.420877975}
2024-12-29T13:53:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2587790.419555141}
2024-12-29T14:03:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2587190.421224961}
2024-12-29T14:03:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2587190.420265738}
2024-12-29T14:03:31.026Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2587180.973655376}
2024-12-29T14:03:31.029Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T14:03:32.073Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651195464) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T14:03:32.073Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651195464) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 10, "retrying_in": 1200, "elapsed": 4810.382305482, "max_duration": 2592000}
2024-12-29T14:13:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2586590.421232684}
2024-12-29T14:13:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2586590.420144747}
2024-12-29T14:23:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2585990.42117584}
2024-12-29T14:23:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2585990.419853625}
2024-12-29T14:23:32.074Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2585979.925924952}
2024-12-29T14:23:32.077Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T14:23:33.124Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651466504) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T14:23:33.125Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651466504) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 11, "retrying_in": 1200, "elapsed": 6011.434136077, "max_duration": 2592000}
2024-12-29T14:33:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2585390.421045374}
2024-12-29T14:33:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2585390.419827154}
2024-12-29T14:43:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2584790.42119307}
2024-12-29T14:43:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2584790.420711232}
2024-12-29T14:43:33.127Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2584778.872994137}
2024-12-29T14:43:33.129Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T14:43:34.208Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651756394) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T14:43:34.209Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21651756394) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 12, "retrying_in": 1800, "elapsed": 7212.51813141, "max_duration": 2592000}
2024-12-29T14:53:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2584190.420890342}
2024-12-29T14:53:21.579Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2584190.420187069}
2024-12-29T15:03:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2583590.421134567}
2024-12-29T15:03:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2583590.419854252}
2024-12-29T15:13:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T15:03:22.172Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2582990.420244154}
2024-12-29T15:13:22.268Z	INFO	autotls.default_acme_client_fixme.acme_client	acme/ari.go:215	got renewal info	{"names": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "selected_time": "2024-12-29T11:32:38.000Z", "recheck_after": "2024-12-29T21:13:22.268Z", "explanation_url": ""}
2024-12-29T15:13:22.316Z	INFO	autotls.maintenance	[email protected]/maintain.go:584	updated ACME renewal information	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "cert_hash": "4dda083d4f05993691a37adc58ad2153923aa0d8bb645f57fe3f2f255d3e0cec", "ari_unique_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "cert_expiry": "2025-01-28T12:43:11.000Z", "selected_time": "2024-12-29T23:10:37.000Z", "next_update": "2024-12-29T21:13:22.268Z", "explanation_url": ""}
2024-12-29T15:13:22.316Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2582989.683371124}
2024-12-29T15:13:34.210Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2582977.789995201}
2024-12-29T15:13:34.212Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T15:13:35.299Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21652200814) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T15:13:35.300Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21652200814) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 13, "retrying_in": 1800, "elapsed": 9013.609198204, "max_duration": 2592000}
2024-12-29T15:23:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2582390.420495145}
2024-12-29T15:23:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2582390.419210943}
2024-12-29T15:33:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2581790.42050137}
2024-12-29T15:33:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2581790.419727822}
2024-12-29T15:43:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2581190.421286873}
2024-12-29T15:43:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2581190.419971187}
2024-12-29T15:43:35.301Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2581176.698055262}
2024-12-29T15:43:35.305Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T15:43:36.373Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21652636574) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T15:43:36.373Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21652636574) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 14, "retrying_in": 1800, "elapsed": 10814.682649924, "max_duration": 2592000}
2024-12-29T15:53:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2580590.420278021}
2024-12-29T15:53:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2580590.419074364}
2024-12-29T16:03:21.578Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2579990.421048168}
2024-12-29T16:03:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2579990.419819377}
2024-12-29T16:13:21.579Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2579390.420610068}
2024-12-29T16:13:21.580Z	INFO	autotls.maintenance	[email protected]/maintain.go:239	certificate expires soon; queuing for renewal	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "remaining": 2579390.419412874}
2024-12-29T16:13:36.373Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2579375.626105528}
2024-12-29T16:13:36.374Z	INFO	autotls.default_acme_client_fixme	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/169298433", "account_contact": []}
2024-12-29T16:13:37.430Z	ERROR	autotls.renew	[email protected]/config.go:919	could not get certificate from issuer	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21653055054) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2024-12-29T16:13:37.430Z	ERROR	autotls.renew	[email protected]/async.go:117	will retry	{"error": "[*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] Renew: [*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct] solving challenges: *.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/169298433/21653055054) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 15, "retrying_in": 1800, "elapsed": 12615.739628141, "max_duration": 2592000}

Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
ipfs version 0.33.0-rc1
Found IPFS fs-repo at /data/ipfs
Initializing daemon...
Kubo version: 0.33.0-rc1-1b5aa0b
Repo version: 16
System version: amd64/linux
Golang version: go1.23.4
PeerID: 12D3KooWBdmLJjhpgJ9KZgLM3f894ff9xyBfPvPjFNn7MKJpyrC2
2024-12-29T16:17:45.165Z	INFO	autotls	node/groups.go:178	appended AutoWSS listener: /ip6/::/tcp/4001/tls/sni/*.libp2p.direct/ws
2024-12-29T16:17:45.165Z	INFO	autotls	node/groups.go:178	appended AutoWSS listener: /ip4/0.0.0.0/tcp/4001/tls/sni/*.libp2p.direct/ws
2024-12-29T16:17:45.188Z	INFO	autotls.maintenance	[email protected]/maintain.go:63	started background certificate maintenance	{"cache": "0xc0001c3b80"}
2024-12-29T16:17:45.324Z	INFO	autotls.start	client/acme.go:356	found preexisting cert for "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct" in local storage
2024-12-29T16:17:45.696Z	INFO	autotls	[email protected]/certificates.go:159	certificate is in configured renewal window based on expiration date	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "expiration": "2025-01-28T12:43:12.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.BPwMMBt6MN1aGaCzgVPdjm2A", "next_ari_update": "2024-12-29T21:13:22.268Z", "renew_check_interval": 600, "window_start": "2024-12-28T13:02:41.000Z", "window_end": "2024-12-30T13:02:41.000Z", "remaining": 2579126.303230176}
2024-12-29T16:17:45.744Z	INFO	autotls.renew	[email protected]/config.go:787	acquiring lock	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"}
Swarm listening on 127.0.0.1:4001 (TCP+UDP)
Swarm listening on 172.17.0.1:4001 (TCP+UDP)
Swarm listening on 172.18.0.1:4001 (TCP+UDP)
Swarm listening on 192.168.50.102:4001 (TCP+UDP)
Swarm listening on [::1]:4001 (TCP+UDP)
Run 'ipfs id' to inspect announced and discovered multiaddrs of this node.
RPC API server listening on /ip4/127.0.0.1/tcp/5001
WebUI: http://127.0.0.1:5001/webui
Gateway server listening on /ip4/127.0.0.1/tcp/8080
Gateway server listening on /ip4/127.0.0.1/tcp/8089
Gateway server listening on /ip4/0.0.0.0/tcp/58080
Gateway server listening on /ip6/::/tcp/58180
Gateway server listening on /ip6/::1/tcp/8080
Routing V1 API exposed at http://127.0.0.1:8080/routing/v1
Routing V1 API exposed at http://127.0.0.1:8089/routing/v1
Routing V1 API exposed at http://0.0.0.0:58080/routing/v1
Routing V1 API exposed at http://[::]:58180/routing/v1
Routing V1 API exposed at http://[::1]:8080/routing/v1
Daemon is ready
2024/12/29 16:17:51 [INFO][FileStorage:/data/ipfs/p2p-forge-certs] Lock for 'issue_cert_*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct' is stale (created: 2024-12-29 12:43:21.655065032 +0000 UTC, last update: 2024-12-29 16:17:40.776814936 +0000 UTC); removing then retrying: /data/ipfs/p2p-forge-certs/locks/issue_cert_wildcard_.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct.lock
2024-12-29T16:17:51.817Z	INFO	autotls.renew	[email protected]/config.go:805	lock acquired	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"}
2024-12-29T16:17:51.817Z	INFO	autotls.renew	[email protected]/config.go:829	renewing certificate	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "remaining": 2579120.182599468}
2024-12-29T16:17:51.818Z	INFO	autotls	[email protected]/acmeclient.go:328	waiting on internal rate limiter	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024-12-29T16:17:51.818Z	INFO	autotls	[email protected]/acmeclient.go:337	done waiting on internal rate limiter	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024-12-29T16:17:51.818Z	INFO	autotls	[email protected]/acmeissuer.go:473	using ACME account	{"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/2028917947", "account_contact": []}
2024-12-29T16:17:53.154Z	INFO	autotls.acme_client	[email protected]/client.go:404	trying to solve challenge	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2024-12-29T16:18:01.128Z	INFO	autotls.acme_client	[email protected]/client.go:581	authorization finalized	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "authz_status": "valid"}
2024-12-29T16:18:01.128Z	INFO	autotls.acme_client	[email protected]/client.go:173	validations succeeded; finalizing order	{"order": "https://acme-v02.api.letsencrypt.org/acme/order/2028917947/338551397685"}
2024-12-29T16:18:02.611Z	INFO	autotls.acme_client	acme/ari.go:215	got renewal info	{"names": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "window_start": "2025-02-26T15:39:00.000Z", "window_end": "2025-02-28T15:39:00.000Z", "selected_time": "2025-02-27T01:10:50.000Z", "recheck_after": "2024-12-29T22:18:02.611Z", "explanation_url": ""}
2024-12-29T16:18:02.969Z	INFO	autotls.acme_client	acme/ari.go:215	got renewal info	{"names": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "window_start": "2025-02-26T15:39:00.000Z", "window_end": "2025-02-28T15:39:00.000Z", "selected_time": "2025-02-26T21:41:57.000Z", "recheck_after": "2024-12-29T22:18:02.969Z", "explanation_url": ""}
2024-12-29T16:18:02.969Z	INFO	autotls.acme_client	[email protected]/client.go:207	successfully downloaded available certificate chains	{"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/04579d82fb4596c1b27579530ba52d7dfb6a"}
2024-12-29T16:18:03.146Z	INFO	autotls.renew	[email protected]/config.go:955	certificate renewed successfully	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory"}
2024-12-29T16:18:03.146Z	INFO	autotls.renew	[email protected]/config.go:796	releasing lock	{"identifier": "*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"}
2024-12-29T16:18:03.146Z	INFO	autotls	[email protected]/certificates.go:490	reloading managed certificate	{"identifiers": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"]}
2024-12-29T16:18:03.422Z	INFO	autotls	[email protected]/cache.go:327	replaced certificate in cache	{"subjects": ["*.k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r.libp2p.direct"], "new_expiration": "2025-03-29T15:19:31.000Z"}

Initial thoughts

  • We should check if there is an implicit configuration of challenges, and confirm we only have dns-01 enabled (instead of http-01 and tls-alpn-01). If not, make dns-01 the default and explicitly disable other ones.
  • Would be good to have confirmation of this behavior from another node.
    • Out collab cluster peers will likely want to renew cert soon too, and looking for no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[dns-01] remaining=[dns-01] in logs would be good way to confirm.
  • 👉 this could be another race condition caused by the way we initialize ACME in p2p-forge/client
    • we may have some implicit default that still runs the background job (note default_acme_client_fixme we set in Kubo, it is a smell present only when issue happens – default manager may still run and may be getting lock on the cert store, and the right one is unable to do the job)
@lidel lidel added the bug Something isn't working label Dec 29, 2024
@lidel lidel changed the title Cert renewal errors in a loop until restart client: cert renewal errors in a loop until restart Dec 29, 2024
@lidel
Copy link
Contributor Author

lidel commented Dec 29, 2024 •
edited
Loading

I won't be able to look into this until Jan 6th+, but if useful, default_acme_client_fixme is something i've set in Kubo to catch if/when default certmagic is used, helping debugging issues like this:

certmagic.Default.Logger = rawLogger.Named("default_fixme")
certmagic.DefaultACME.Logger = rawLogger.Named("default_acme_client_fixme")

default_acme_client_fixme should not be present at all, so:

  • it is a strong signal we have wrong or two cert managers running (ours + the implicit default).
  • newCertmagicConfig or related code may be still initializing the default manager

@lidel
Copy link
Contributor Author

lidel commented Jan 16, 2025 •
edited
Loading

Confirmed same issue at collab-cluster-dc13-2. It seems the redundant cert manager obtained lock, and bricked TCP:

Jan 16 17:22:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:22:24.215Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 17:22:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:22:24.216Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 583258.783317273}
Jan 16 17:32:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:32:24.216Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 17:32:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:32:24.216Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 582658.783203719}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.689Z        INFO        autotls.renew        [email protected]/config.go:829        renewing certificate        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "remaining": 582609.310627755}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.689Z        DEBUG        autotls        [email protected]/config.go:1023        created CSR        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "san_dns_names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "san_emails": [], "common_name": "", "extra_extensions": 0}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.690Z        DEBUG        autotls.default_acme_client_fixme        [email protected]/account.go:50        using existing ACME account because key found in storage associated with email        {"email": "default", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.691Z        DEBUG        autotls.default_acme_client_fixme        [email protected]/account.go:50        using existing ACME account because key found in storage associated with email        {"email": "", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.691Z        INFO        autotls.default_acme_client_fixme        [email protected]/acmeissuer.go:473        using ACME account        {"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/177442124", "account_contact": []}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.861Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/http.go:275        http request        {"method": "GET", "url": "https://acme-staging-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["974"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 17:33:13 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.861Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/order.go:130        creating order        {"account": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/177442124", "identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"]}
Jan 16 17:33:13 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:13.915Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/http.go:275        http request        {"method": "HEAD", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 16 Jan 2025 17:33:13 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IIrnThrgC_tFWhyD360VSC0cVFvgcv4tnYUvD33xo8M9xLi2sjs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.021Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["177442124"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["424"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 17:33:13 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/177442124/22036120244"],"Replay-Nonce":["IIrnThrgpnF_FSOHp2QQ7JdVriAvB_FDKJEIbdeD-Zy7gVnDRcc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.085Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz/177442124/15729093094", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["177442124"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["464"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 17:33:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["IIrnThrgWdOBJ4WGahISVO68hQJrwu04TBP13mhvaldB4URC2l8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.085Z        DEBUG        autotls.default_acme_client_fixme.acme_client        [email protected]/client.go:506        no solver configured        {"challenge_type": "dns-01"}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.156Z        DEBUG        autotls.default_acme_client_fixme.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/authz/177442124/15729093094", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["177442124"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 17:33:14 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["m146KzL7kxIgxoxLk2MAu5Uqsa4gCKEB5p9gVT0hHS2SY83Ic0Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.156Z        ERROR        autotls.renew        [email protected]/config.go:919        could not get certificate from issuer        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct] solving challenges: *.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct: no solvers available for remaining challenges (configured=[tls-alpn-01 http-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/177442124/22036120244) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Jan 16 17:33:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:33:14.156Z        ERROR        autotls.renew        [email protected]/async.go:117        will retry        {"error": "[*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct] Renew: [*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct] solving challenges: *.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct: no solvers available for remaining challenges (configured=[tls-alpn-01 http-01] offered=[dns-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/177442124/22036120244) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 115, "retrying_in": 21600, "elapsed": 2008849.869753334, "max_duration": 2592000}
Jan 16 17:34:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:34:14.521Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 17:34:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:34:14.595Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 17:42:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:42:24.216Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 17:42:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:42:24.216Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 582058.783313245}
Jan 16 17:42:37 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:42:37.111Z        ERROR        fullrtdht        fullrt/dht.go:328        Accelerated DHT client was unable to fully refresh its routing table due to Resource Manager limits, which may degrade content routing. Consider increasing resource limits. See debug logs for the "dht-crawler" subsystem for details.
Jan 16 17:46:14 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:46:14.984Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 17:47:37 collab-cluster-dc13-2 ipfs[4085403]: 2025/01/16 17:47:37 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:65)
Jan 16 17:47:37 collab-cluster-dc13-2 ipfs[4085403]: 2025/01/16 17:47:37 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.(*responseWriterDelegator).WriteHeader (delegator.go:65)
Jan 16 17:52:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:52:24.216Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 17:52:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:52:24.217Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 581458.782816359}
Jan 16 17:58:15 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:58:15.241Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 17:58:15 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T17:58:15.298Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 18:02:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T18:02:24.215Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 18:02:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T18:02:24.216Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 580858.783386523}
Jan 16 18:10:15 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T18:10:15.629Z        ERROR        core/commands/cmdenv        pin/pin.go:155        context canceled
Jan 16 18:12:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T18:12:24.215Z        INFO        autotls        [email protected]/certificates.go:135        certificate needs renewal based on ARI window        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "ari_cert_id": "kydGmAOpUWiOmNbEQkjbI79YlNI.A6y5lwCt3i28YyBMDW3hAEWu", "next_ari_update": "2025-01-16T18:22:24.446Z", "renew_check_interval": 600, "window_start": "2024-12-23T11:42:52.000Z", "window_end": "2024-12-25T11:42:52.000Z", "selected_time": "2024-12-24T23:42:42.000Z", "renewal_cutoff": "2024-12-24T23:32:42.000Z"}
Jan 16 18:12:24 collab-cluster-dc13-2 ipfs[4085403]: 2025-01-16T18:12:24.216Z        INFO        autotls.maintenance        [email protected]/maintain.go:239        certificate expires soon; queuing for renewal        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "remaining": 580258.783429764}

Restarting daemon fixed issue, new cert got renewed correctly:

Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: Daemon is ready
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.987Z        DEBUG        autotls.handshake        [email protected]/handshake.go:191        no matching certificates and no custom selection logic        {"identifier": "2604-1380-45f1-d800--3.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.987Z        DEBUG        autotls.handshake        [email protected]/handshake.go:198        choosing certificate        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "num_choices": 1}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.987Z        DEBUG        autotls.handshake        [email protected]/handshake.go:204        default certificate selection results        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.987Z        DEBUG        autotls.handshake        [email protected]/handshake.go:270        matched certificate in cache        {"remote_ip": "64.176.223.253", "remote_port": "1616", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "expiration": "2025-01-23T11:23:23.000Z", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.988Z        DEBUG        autotls.handshake        [email protected]/handshake.go:191        no matching certificates and no custom selection logic        {"identifier": "147-75-63-129.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.988Z        DEBUG        autotls.handshake        [email protected]/handshake.go:198        choosing certificate        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "num_choices": 1}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.988Z        DEBUG        autotls.handshake        [email protected]/handshake.go:204        default certificate selection results        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:05 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:05.988Z        DEBUG        autotls.handshake        [email protected]/handshake.go:270        matched certificate in cache        {"remote_ip": "64.176.223.253", "remote_port": "39577", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "expiration": "2025-01-23T11:23:23.000Z", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025/01/16 22:56:07 [INFO][FileStorage:/home/ipfs/.ipfs/p2p-forge-certs] Lock for 'issue_cert_*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct' is stale (created: 2024-12-24 11:32:24.275467082 +0000 UTC, last update: 2025-01-16 22:55:56.21875182 +0000 UTC); removing then retrying: /home/ipfs/.ipfs/p2p-forge-certs/locks/issue_cert_wildcard_.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct.lock
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.198Z        INFO        autotls.renew        [email protected]/config.go:805        lock acquired        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.198Z        INFO        autotls.renew        [email protected]/config.go:829        renewing certificate        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "remaining": 563235.80104424}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.199Z        DEBUG        autotls        [email protected]/config.go:1023        created CSR        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "san_dns_names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "san_emails": [], "common_name": "", "extra_extensions": 0}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.200Z        DEBUG        autotls        [email protected]/account.go:50        using existing ACME account because key found in storage associated with email        {"email": "default", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.200Z        DEBUG        autotls        [email protected]/account.go:50        using existing ACME account because key found in storage associated with email        {"email": "", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.200Z        INFO        autotls        [email protected]/acmeclient.go:328        waiting on internal rate limiter        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.200Z        INFO        autotls        [email protected]/acmeclient.go:337        done waiting on internal rate limiter        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.200Z        INFO        autotls        [email protected]/acmeissuer.go:473        using ACME account        {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/2019807727", "account_contact": []}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.395Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["828"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:07 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.395Z        DEBUG        autotls.acme_client        acme/order.go:130        creating order        {"account": "https://acme-v02.api.letsencrypt.org/acme/acct/2019807727", "identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"]}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.449Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 16 Jan 2025 22:56:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["695xoWTDqSZMeaeacsHFILCIZBUG1TV89rzF1EIBWpR-Itso9_Q"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.565Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["412"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/2019807727/345087934645"],"Replay-Nonce":["TuPv3Ouuxm28NFUPRZPioLsIdE5d5S3U4LAJwTG731tevwUO84U"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.621Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz/2019807727/461721768355", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["458"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:07 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["TuPv3Ouu1pcCEJPIDYOc1TyNrL9O75WBILIfbEQawU5V0TzFJJU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.622Z        INFO        autotls.acme_client        [email protected]/client.go:404        trying to solve challenge        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.622Z        DEBUG        autotls.dns01solver        client/acme.go:487        getting DNS-01 challenge value from CA        {"acme_challenge": {"type":"dns-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/2019807727/461721768355/0Ph28Q","status":"pending","token":"AttphU1qmbhSUWh8wgqiMcnCHG1qHPrcD2DjPQDqSy0","keyAuthorization":"AttphU1qmbhSUWh8wgqiMcnCHG1qHPrcD2DjPQDqSy0.1In3uI_XVkIwpw7LF8HUVDxjknYQeV1hAufrHm58qFw","identifier":{"type":"dns","value":"k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}}}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.623Z        DEBUG        autotls.dns01solver        client/acme.go:509        advertised libp2p addrs for p2p-forge broker to try        {"addrs": ["/ip4/147.75.63.129/tcp/4001","/ip4/147.75.63.129/tcp/4002/tls/sni/147-75-63-129.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct/ws","/ip4/147.75.63.129/udp/4001/webrtc-direct/certhash/uEiC8-nHTUvOVnmjDbbvvICFm2Gz18GjIg-O3abp6XUVhHA","/ip4/147.75.63.129/udp/4001/webrtc-direct/certhash/uEiC8-nHTUvOVnmjDbbvvICFm2Gz18GjIg-O3abp6XUVhHA","/ip4/147.75.63.129/udp/4001/quic-v1","/ip4/147.75.63.129/udp/4001/quic-v1/webtransport/certhash/uEiChmnTa0SZIhCwuplClPrb_LW5w9v1P3g4NmmLeU_6HDQ/certhash/uEiAXssBXU6cPoOEHSd5AoO5oIKxWlP3DC82boua9Z3IUqQ","/ip4/147.75.63.129/udp/4001/quic-v1/webtransport/certhash/uEiChmnTa0SZIhCwuplClPrb_LW5w9v1P3g4NmmLeU_6HDQ/certhash/uEiAXssBXU6cPoOEHSd5AoO5oIKxWlP3DC82boua9Z3IUqQ","/ip6/2604:1380:45f1:d800::3/tcp/4001","/ip6/2604:1380:45f1:d800::3/tcp/4002/tls/sni/2604-1380-45f1-d800--3.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct/ws","/ip6/2604:1380:45f1:d800::3/udp/4001/webrtc-direct/certhash/uEiC8-nHTUvOVnmjDbbvvICFm2Gz18GjIg-O3abp6XUVhHA","/ip6/2604:1380:45f1:d800::3/udp/4001/webrtc-direct/certhash/uEiC8-nHTUvOVnmjDbbvvICFm2Gz18GjIg-O3abp6XUVhHA","/ip6/2604:1380:45f1:d800::3/udp/4001/quic-v1","/ip6/2604:1380:45f1:d800::3/udp/4001/quic-v1/webtransport/certhash/uEiChmnTa0SZIhCwuplClPrb_LW5w9v1P3g4NmmLeU_6HDQ/certhash/uEiAXssBXU6cPoOEHSd5AoO5oIKxWlP3DC82boua9Z3IUqQ","/ip6/2604:1380:45f1:d800::3/udp/4001/quic-v1/webtransport/certhash/uEiChmnTa0SZIhCwuplClPrb_LW5w9v1P3g4NmmLeU_6HDQ/certhash/uEiAXssBXU6cPoOEHSd5AoO5oIKxWlP3DC82boua9Z3IUqQ"]}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.623Z        DEBUG        autotls.dns01solver        client/acme.go:511        asking p2p-forge broker to set DNS-01 TXT record        {"url": "https://registration.libp2p.direct", "dns01_value": "YtdLA8qhu_FYvDESNamXwaqWU88yjh0rRea1BZGkh9o"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.906Z        DEBUG        autotls.acme_client        [email protected]/client.go:435        waiting for solver before continuing        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "challenge_type": "dns-01"}
Jan 16 22:56:07 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:07.906Z        DEBUG        autotls.dns01solver        client/acme.go:480        waiting for DNS-01 TXT record to be set
Jan 16 22:56:10 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:10.983Z        DEBUG        autotls.handshake        [email protected]/handshake.go:191        no matching certificates and no custom selection logic        {"identifier": "147-75-63-129.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}
Jan 16 22:56:10 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:10.983Z        DEBUG        autotls.handshake        [email protected]/handshake.go:198        choosing certificate        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "num_choices": 1}
Jan 16 22:56:10 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:10.983Z        DEBUG        autotls.handshake        [email protected]/handshake.go:204        default certificate selection results        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:10 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:10.983Z        DEBUG        autotls.handshake        [email protected]/handshake.go:270        matched certificate in cache        {"remote_ip": "45.120.177.231", "remote_port": "59082", "subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "managed": true, "expiration": "2025-01-23T11:23:23.000Z", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf"}
Jan 16 22:56:12 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:12.906Z        DEBUG        autotls.acme_client        [email protected]/client.go:441        done waiting for solver        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "challenge_type": "dns-01"}
Jan 16 22:56:12 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:12.978Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/chall/2019807727/461721768355/0Ph28Q", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["194"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:12 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz/2019807727/461721768355>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall/2019807727/461721768355/0Ph28Q"],"Replay-Nonce":["TuPv3OuuBQL3HUSIuN3aYArFtTEyI1nd0MCBNJWGf5M0xrVo-Uc"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:12 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:12.978Z        DEBUG        autotls.acme_client        [email protected]/client.go:478        challenge accepted        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "challenge_type": "dns-01"}
Jan 16 22:56:13 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:13.286Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz/2019807727/461721768355", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["458"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:13 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["695xoWTDGZ3aN2oanWDra1rmd_hCPLuWNTNQNVR4pg0qFfTU3So"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:13 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:13.593Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz/2019807727/461721768355", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["458"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:13 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["TuPv3OuuQ-JKFpUHCNzVTJU0Xt64N2KkSt4jCztPEKcHecksFA0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:13 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:13.900Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz/2019807727/461721768355", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["655"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:13 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["695xoWTDxgsgfztPXQmU2M3YnAKzbGx15WJK5C8xHxUirbs2whU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:13 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:13.900Z        INFO        autotls.acme_client        [email protected]/client.go:581        authorization finalized        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "authz_status": "valid"}
Jan 16 22:56:13 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:13.900Z        INFO        autotls.acme_client        [email protected]/client.go:173        validations succeeded; finalizing order        {"order": "https://acme-v02.api.letsencrypt.org/acme/order/2019807727/345087934645"}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.225Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/finalize/2019807727/345087934645", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2019807727"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["514"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/2019807727/345087934645"],"Replay-Nonce":["695xoWTDNBA8HsCQ0aFrQpdqhfX0WOppIffMZqHE7OJ4Yq4K8ag"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.283Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["2893"],"Content-Type":["application/pem-certificate-chain"],"Date":["Thu, 16 Jan 2025 22:56:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88/1>;rel=\"alternate\""],"Replay-Nonce":["TuPv3OuueWd8IfCDJqZYA47v1idR_2l8kCdYssePkdxKocFmW74"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.284Z        DEBUG        autotls.acme_client        acme/ari.go:134        getting renewal info        {"names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"]}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.354Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo/nytfzzwhT50Et-0rLMTGcIvS1w0.AxklyZ1Z6HAG61i8AsuOli2I", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["101"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Retry-After":["21600"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.355Z        INFO        autotls.acme_client        acme/ari.go:215        got renewal info        {"names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "window_start": "2025-03-16T22:17:12.000Z", "window_end": "2025-03-18T22:17:12.000Z", "selected_time": "2025-03-18T01:26:14.000Z", "recheck_after": "2025-01-17T04:56:14.355Z", "explanation_url": ""}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.414Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88/1", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["2324"],"Content-Type":["application/pem-certificate-chain"],"Date":["Thu, 16 Jan 2025 22:56:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88/0>;rel=\"alternate\""],"Replay-Nonce":["TuPv3OuuMUvxxdEbPHnLlPvr_0-UEjlApc-OVRwYhcCP1wCjoKw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.414Z        DEBUG        autotls.acme_client        acme/ari.go:134        getting renewal info        {"names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"]}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.473Z        DEBUG        autotls.acme_client        acme/http.go:275        http request        {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo/nytfzzwhT50Et-0rLMTGcIvS1w0.AxklyZ1Z6HAG61i8AsuOli2I", "headers": {"User-Agent":["CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["101"],"Content-Type":["application/json"],"Date":["Thu, 16 Jan 2025 22:56:14 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Retry-After":["21600"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.473Z        INFO        autotls.acme_client        acme/ari.go:215        got renewal info        {"names": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "window_start": "2025-03-16T22:17:12.000Z", "window_end": "2025-03-18T22:17:12.000Z", "selected_time": "2025-03-17T08:43:01.000Z", "recheck_after": "2025-01-17T04:56:14.473Z", "explanation_url": ""}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.473Z        INFO        autotls.acme_client        [email protected]/client.go:207        successfully downloaded available certificate chains        {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88"}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.473Z        DEBUG        autotls        [email protected]/acmeissuer.go:518        selected certificate chain        {"url": "https://acme-v02.api.letsencrypt.org/acme/cert/031925c99d59e87006eb58bc02cb8e962d88"}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.544Z        INFO        autotls.renew        [email protected]/config.go:955        certificate renewed successfully        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct", "issuer": "acme-v02.api.letsencrypt.org-directory"}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.545Z        INFO        autotls.renew        [email protected]/config.go:796        releasing lock        {"identifier": "*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.545Z        INFO        autotls        [email protected]/certificates.go:490        reloading managed certificate        {"identifiers": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"]}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.678Z        DEBUG        autotls        [email protected]/cache.go:307        removed certificate from cache        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-01-23T11:23:23.000Z", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "c3bba9959c55d58c543e48f1826bc94b37d111bafa631b4357e0a515a95e3abf", "cache_size": 0, "cache_capacity": 0}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.678Z        DEBUG        autotls        [email protected]/cache.go:271        added certificate to cache        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "expiration": "2025-04-16T21:57:43.000Z", "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "4c0dc2075deb606f7304b456fdd60593041c39e94d714d850c38ad875b114f79", "cache_size": 1, "cache_capacity": 0}
Jan 16 22:56:14 collab-cluster-dc13-2 ipfs[1480079]: 2025-01-16T22:56:14.678Z        INFO        autotls        [email protected]/cache.go:327        replaced certificate in cache        {"subjects": ["*.k51qzi5uqu5dht5qyglpp8q4qldzx6d094lqdffp5n80zj5u6vfxk7n4pmutoo.libp2p.direct"], "new_expiration": "2025-04-16T21:57:43.000Z"}

@lidel lidel mentioned this issue Jan 17, 2025
Merged
4 tasks
@lidel lidel closed this as completed in #42 Jan 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(client): race in cert renewal ipshipyard/p2p-forge
1 participant
@lidel