Aci multi part upload pool

[rescuerunaways]

No description provided.

[zsolt-marta-bitrise]

The end of the chunk is at chunkStart + chunkSize or chunkStart + lastChunkSize. I see you're actually passing the size of it, so let's just call it currentChunkSize or similar

[zsolt-marta-bitrise]

You don't need a chunkCount capacity channel for this, just have one channel and at the end try to select from it to chek if there's been any errors.

Also, cancel the context at the first error so that other workers stop downloading other chunks if their work will be discarded anyways

[zsolt-marta-bitrise]

Use defer() instead of adding this to all lines

[zsolt-marta-bitrise]

Even better yet - use the WaitGroup only for synchronizing the goroutines, since we're waiting on the goroutines, they can handle their own shutdown when there are no more items in the channel. So like

var wg sync.WaitGroup

ctx, cancelFunc := context.WithCancel(context.Background()) // Or if we already have a parent, use that
ch := make(chan int, chunkCount)
errCh := make(chan error)
for i := 0; i < chunkCount; i++ {
	ch <- job
}
close(ch)
for i := 0; i < 10; i++ {
	wg.Add(1)
	go func() {
		defer wg.Done()
		for {
			select {
			case job, ok := <-ch:
				if !ok {
					return
				}
				if err := uploadChunk(job); err != nil {
					cancelFunc()
					return
				}
			case <-ctx.Done():
			default:
				return
			}
		}
	}()
}
wg.Wait() // Wait on workers

[zsolt-marta-bitrise]

Create a cancelable context and check if it's cancelled to propagate cancellation (and to support error handling - see above).

For this, you'll need to change the range to a select: the context cancellation or the closure of the channel should end the goroutine.