minio failed to mirror bucket; failing with "transport connection broken"

[larsks]

Expected behavior

I expect mc mirror to successfully mirror a bucket from a source to a destination.

Actual behavior

The mirror operation fails with:

mc: Failed to copy https://s3.openshift-storage.svc/openshift-logging-objectbu-aceba6c1-ad0b-48e1-8bd6-53b816aeec9c/application/4b634d8c87a01139:18932aab87f:1893318bd1b:e4c9fb25. Put "https://log-backup.s3.dualstack.us-east-1.amazonaws.com/application/4b634d8c87a01139%3A18932aab87f%3A1893318bd1b%3Ae4c9fb25": net/http: HTTP/1.x transport connection broken: http: ContentLength=1882613 with Body length 0

Steps to reproduce the behavior

The failing script looks like this:

mc -C /tmp/.mc alias set openshift $SRC_ENDPOINT $SRC_ACCESS_ID $SRC_SECRET_KEY
mc -C /tmp/.mc alias set backup $DEST_ENDPOINT $DEST_ACCESS_ID $DEST_SECRET_KEY
mc -C /tmp/.mc mirror --newer-than 14d --overwrite openshift/$SRC_BUCKET backup/$DEST_BUCKET

Where SRC_* points at a Noobaa bucket, and DEST_* points at either a Minio bucket or an AWS S3 bucket (we've seen the same problem with multiple destinations).

mc --version

mc version RELEASE.2023-07-11T23-30-44Z (commit-id=c1193fc29d61d8aa69e3725a519abd95ee9da910)
Runtime: go1.19.11 linux/amd64
Copyright (c) 2015-2023 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

System information

Red Hat CoreOS (OpenShift 4.10)

[larsks]

Astute readers will note that the symptoms here are a duplicate of #4231. That issue was closed without much of an explanation; @harshavardhana says:

mc gave up because this is an unrecoverable error - it cannot be retried since the connection was broken mid-flight during the socket read @dingsongjie

Why is this an unrecoverable error? We're transferring distinct objects over a stateless protocol. If the connection breaks, you establish a new connection and re-send the object.

Is the solution here to wrap mc mirror in a while loop?

while ! mc -C /tmp/.mc mirror --newer-than 14d --overwrite openshift/$SRC_BUCKET backup/$DEST_BUCKET; do
  sleep 1
done

That seems like logic that should be wrapped around the transfers of each object, rather than the entire mirror operation.

[klauspost]

If the connection breaks, you establish a new connection and re-send the object.

There is already retry attempts made AFAIK, but we need to fail at some point.

[zveinn]

@larsks I recently made a PR that should implement another layer of retries on a higher level.
#4715

This should already be released, give it a try.