[WIP] Tus dataprovider

[butonic]

revival of #286

I also added tus.io create extension support for dataprovider so a /bif.txt with content bar can be created with:

$ echo -n /bif.txt | base64
L2JpZi50eHQ=
$ curl -X POST -H "Tus-Resumable: 1.0.0" -H "Content-Length: 3" -H "Upload-Length: 3" http://localhost:9156/data -v -u einstein:relativity -H "Content-Type: application/offset+octet-stream" -d "bar" -H "Upload-Metadata: filename L2JpZi50eHQ="

• implement upload for eos fs
• implement upload for local fs
• implement upload for s3 fs
• refactor owncloud fs NewUpload and InitiateUpload. The share 99% of the code
• check if chunking v2 now works? AFAICT the ocdav endpoint should support chunking v2 and use tus to upload the file to the dataprovider
• pass on tus.io requests in the ocdav handler to allow clients to use tus directly? The endpoint then becomes /remote.php/dav/uploads but how do we determine which storage to use? For the dataprovider the path must be relative to its root. we may need to use the datagateway to lookup the correct dataprovider based on the path in the tus Upload-Metadata header.

[PVince81]

frontend bits, might help for testing: owncloud/web#3345

[butonic]

I made every collection in the webdav endpoint accept tus.io POST requests that can be used to start an upload. It currently returns a Location header with the port of the dataprovider. I tested it with the go-tus client https://github.com/eventials/go-tus

This is my test code to upload a file:

package main

import (
	"crypto/tls"
	"encoding/base64"
	"net/http"
	"os"

	"github.com/eventials/go-tus"
	"github.com/eventials/go-tus/memorystore"
)

func basicAuth(username, password string) string {
	auth := username + ":" + password
	return base64.StdEncoding.EncodeToString([]byte(auth))
}

func main() {
	f, err := os.Open("my-file.bin")
	if err != nil {
		panic(err)
	}

	defer f.Close()

	http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
	c := tus.DefaultConfig()
	c.Resume = true
	c.Store, err = memorystore.NewMemoryStore()
	if err != nil {
		panic(err)
	}

	c.Header.Add("Authorization", "Basic "+basicAuth("einstein", "relativity"))

	// create the tus client.
	client, err := tus.NewClient("https://localhost:9200/remote.php/dav/files/einstein/", c)
	if err != nil {
		panic(err)
	}

	// create an upload from a file.
	upload, err := tus.NewUploadFromFile(f)
	if err != nil {
		panic(err)
	}

	// create the uploader.
	uploader, err := client.CreateUpload(upload)
	if err != nil {
		panic(err)
	}

	// start the uploading process.
	err = uploader.Upload()
	if err != nil {
		panic(err)
	}
}

https://localhost:9200 is ocis, but it should work against an ocdav endpoint with plain reva as well.

That the Location header points to a different port might be problematic for deployments that only allow a single port to the outside world. In those cases we can either register different routes in the proxy or use the datagateway to route requests ... something to track ... until we tested tus in general

I'll test this tomorrow with @PVince81 and @LukasHirt

Another bash script to see what needs to be done to create an upload:

#!/bin/bash -x
UNAME=einstein
PASS=relativity
SOURCE=bin/ocis
BASE="https://localhost:9200/remote.php/dav"
DESTINATION_DIR="$BASE/files/$UNAME/" 
ID=`uuidgen`
DESTINATION_FILE="testchunk-$ID.bin"
METADATA="filename `echo -n $DESTINATION_FILE | base64`"
UPLOAD_LENGTH="11534336"

# check tus support using HEAD, result MUST have the "Tus-Resumable: 1.0.0" header
curl --head $DESTINATION_DIR -u $UNAME:$PASS -k 

# POST to the dir
curl -X POST $DESTINATION_DIR -u $UNAME:$PASS -k -v -H "Tus-Resumable: 1.0.0" -H "Upload-Length: $UPLOAD_LENGTH" -H "Upload-Metadata: $METADATA"

I stopped here, because I would have had to parse the headers to follow the location ... and my bash-fu is not up to the task, so I used the above go code.

[labkode]

@butonic nice progress. So the Tus-Resumable header is the one that says: Hi, this is TUS request? Is very important to be explicit on the ocdav part of what is accepted.

[PVince81]

@butonic I think we're missing a "Tus-Resumable: 1.0.0" response header for the POST.
Seems the JS lib is not strict about this though.

[hound]

exported function LogEvent should have comment or be unexported

[hound]

exported var ErrUnsupportedVersion should have comment or be unexported

[hound]

exported var ErrUnsupportedVersion should have comment or be unexported

[butonic]

this branch also contains an initial oc chunking v2 implementation, but it needs to be restarted on #674

and better yet, after having sorted out how to stream directly to eos