Corrupted files on whosonfirst.mapzen.com?

[ong]

Running the download script results in the following corrupted file error on multiple bundles:

curl https://whosonfirst.mapzen.com/bundles/wof-region-latest-bundle.tar.bz2 | tar -xj --strip-components=1 --exclude=README.txt -C ...

curl: (56) GnuTLS recv error (-54): Error in the pull function.

bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
	Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

[ong]

Error disappears upon rerunning download script.

[trescube]

Sorry for the late reply, but if you encounter further issues with the data or data bundles, please report them to https://github.com/whosonfirst-data. This repository is just the Pelias importer of Who's on First data.

[orangejulius]

@thisisaaronland might be interested in seeing this even though it's now fixed.

[thisisaaronland]

This appears to be a TLS issue rather than a data issue?

[ong]

The only difference I could tell between the unsuccessful run and the successful one is that the successful one had a faster network connection and thus finished downloading faster. Could this be a timeout issue followed by the script attempting to uncompress a partially downloaded file?

[jeremy-rutman]

I've hit this on the same file twice now:

error downloading whosonfirst-data-locality-latest.tar.bz2 bundle: Error: Command failed: curl -s https://dist.whosonfirst.org/bundles/whosonfir
st-data-locality-latest.tar.bz2 | tar -xj --strip-components=1 --exclude=README.txt -C /mnt/storage-proc2/users/jeremy/pelias_metal/data/whosonf
irst && mv /mnt/storage-proc2/users/jeremy/pelias_metal/data/whosonfirst/whosonfirst-data-locality-latest.csv /mnt/storage-proc2/users/jeremy/pe
lias_metal/data/whosonfirst/meta

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

[missinglink]

We don't have any control over the WhosOnFirst servers, please report the issue with the data provider here: https://github.com/whosonfirst-data/whosonfirst-data

[jeremy-rutman]

possibly a long untar leads to this error? I added a bit at 405