Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload 300GB radar/satellite/lightning files to S3 - compress first #20

Open
obriensystems opened this issue Jul 30, 2021 · 0 comments
Open

Upload 300GB radar/satellite/lightning files to S3 - compress first #20

obriensystems opened this issue Jul 30, 2021 · 0 comments
Assignees
@obriensystems

Comments

@obriensystems
Copy link
Member

obriensystems commented Jul 30, 2021 •
edited
Loading

8tb/../tars.sh
start 1730 on mbp5 65% of 1600 cpu

HDD via TB is about 5x the speed as internet upload for rogers
write .28/50 .42/57 = .14GB/7sec = 20MB/sec write
read 68/30 79/34 = .11GB/4 sec = 27MB/sec
55 64 17-21 sec = .09GB/4sec = 22MB/sec (HDD not SSD)
lightning half at 16GB took 12 min = (predicted 13.5 min) - so 2h to compress 160GB
300GB will take 85h to compress

compression ration of 2:2.5 = 80%
use SSD raid on my PC's
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@obriensystems obriensystems

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@obriensystems