PUT returns bad/illegal format or missing URL on ARM 64-bit

[swanlm]

Expected behavior

PUT should run with no errors.

Actual behavior

Code run per the example given to PUT ftp file returns "URL using bad/illegal format or missing URL" when build architecture set to ARM 64-bit

Steps to reproduce the behavior

Run the demo app with the latest version with build architecture set to X86 64-bit. PUT a file to a server. File transfer is successful.
Close the demo app, change the architecture to ARM 64-bit. PUT the same file to a server and response is "CURLE_URL_MALFORMAT(3): URL using bad/illegal format or missing URL"

[swanlm]

Tested as not compatible with ARM 64-bit.

[charonn0]

I don't have an ARM64 device around to test on, so I'm not sure what the issue might be. But at least we're getting far enough along in the process to get a "bad URL" error, which is promising.

What do the debug messages say about the URL? Does it seem like the URL in the messages is empty or corrupted?

[charonn0]

Also, do other operations besides PUT fail on ARM64?

[swanlm]

I don't have an ARM64 device around to test on, so I'm not sure what the issue might be. But at least we're getting far enough along in the process to get a "bad URL" error, which is promising.

What do the debug messages say about the URL? Does it seem like the URL in the messages is empty or corrupted?

In the Stack; libcURL.ResponseHeaderEngine.RequestCount shows Exception FunctionNotFoundException with Text "Could not load curl_easy_nextheader from libcurl.4.dylib."
I have seen somewhere on the libcURL docs that the CURLE_URL_MALFORMAT error can be a mismatch in version; not necessarily a problem with the header - but that may be a red herring.

[swanlm]

Also, do other operations besides PUT fail on ARM64?

I tried GET and HEAD and got same response.

[charonn0]

In the Stack; libcURL.ResponseHeaderEngine.RequestCount shows Exception FunctionNotFoundException with Text "Could not load curl_easy_nextheader from libcurl.4.dylib."

This is a separate bug (now fixed) that your problem exposed. The curl_easy_nextheader function is part of a new API in libcurl for handling response headers, so if you're not using the very latest release then it won't be available. I failed to account for the case when the API is not available and also no headers were received.

---

If you'd be willing to conduct a little experiment, paste this in the App.Opening event of the demo project and run it on your ARM64 device:

Dim c As New cURLClient

Dim url As MemoryBlock = "http://www.example.com/" + Chr(0)
If Not c.EasyHandle.SetOptionPtr(libcURL.Opts.URL, url) Then
  Dim err As New libcURL.cURLException(c.EasyHandle)
  MessageBox("In SetOptionPtr: " + err.Message)
End If

If c.Get("") Then
  MessageBox(c.GetInfo(libcURL.Info.EFFECTIVE_URL))
Else
  Dim err As New libcURL.cURLException(c.EasyHandle)
  MessageBox("In request: " + err.Message)
End If

This snippet makes a GET request but bypasses the normal way of setting the URL. If the request succeeds then the actual URL that was used is shown. If not the error details are shown. Either case will narrow down the problem a bit.

[swanlm]

Hi,

I added the code. As soon as the App opens I get: URL using bad/illegal format or missing URL (CURLE_URL_MALFORMAT). I tried during debug to access the cURLClient properties but as soon as I click on them the App crashes.

[charonn0]

That suggests that the problem is at the interface between libcurl and Xojo. But without being able to reproduce the problem myself it's going to be hard to figure out.

[swanlm]

That suggests that the problem is at the interface between libcurl and Xojo. But without being able to reproduce the problem myself it's going to be hard to figure out.

If I can help somehow let me know.

[charonn0]

So I've been thinking about this, and I'd like to ask you to do a couple more experiments. You'll need to be using at least libcurl 7.62.0 since that's when libcurl exposed it's internal URL parser.

First, let's see if the URL parser will kindly tell us what it thinks the problem is:

  Dim url As New libcURL.URLParser("http://www.example.com/")
  If url.LastError = 0 Then
    MsgBox(url.StringValue) ' successfully parsed
  Else
    Raise New libcURL.cURLException(url) ' error
  End If

If the URL parser reports no parsing error, then let's see what happens if we use the URL parser to set the URL for a transfer:

Dim url As New libcURL.URLParser("http://www.example.com/")
Dim c As New cURLClient
Call c.SetOption(libcURL.Opts.URL, url)
If c.Get("") Then
  MsgBox("Success!")
Else
  Raise New libcURL.cURLException(c.EasyHandle)
End If

On the other hand, if the parser reported an error, let's see what happens if we build the URL rather than parse it:

Dim url As New libcURL.URLParser
url.Scheme = "http"
url.Hostname = "www.example.com"
url.Path = "/"
Dim c As New cURLClient
Call c.SetOption(libcURL.Opts.URL, url)
If c.Get("") Then
  MsgBox("Success!")
Else
  Raise New libcURL.cURLException(c.EasyHandle)
End If

If the transfer succeeds then we have a workaround. If not then hopefully the URL parser error will give a hint on what to look at next.

[swanlm]

The first bit of code runs successfully and returns a message box with "http://www.example.com/".

[swanlm]

The second bit of code generates cURLException with the Reason "URL using bad/illegal format or missing URL (CURLE_URL_MALFORMAT)". Error number 3.

[swanlm]

Same result with the third piece of code.

[swanlm]

Output from curl -V in terminal gives curl 7.79.1 libcurl 7.79.1.

[swanlm]

Finally; in debug mode; here are the values for libCURL.URLparser.

[swanlm]

Hi,

I'm not are whether this is relevant but I was stepping through the code in debug mode and noted the following in libcURL.cURLException.Constructor:

Case ErrantItem IsA libcURL.URLParser Me.Message = libcURL.FormatURLError(Me.ErrorNumber)

Else
Me.Message = libcURL.FormatError(Me.ErrorNumber) + " (" + libcURL.Errors.Name(Me.ErrorNumber) + ")"

End Select

The error is being generated at the "Else" statement not "Case ErrantItem IsA libcURL.URLParser". It is definitely Error number 3, but perhaps the error is not actually coming from the URL Parser?

[charonn0]

Yeah, at this point it seems pretty clear that the malformatted URL error is a red herring. Something else is going on, though I'm not sure what to look at next.

[swanlm]

In case this helps; any time I try to inspect the contents on EasyHandle in Debug mode, the app crashes. I'm attaching the crash report in case you can make head or tails of it.

As soon as I click on libcURL.EasyHandle, app crashes.

crashreport.txt

[SchroedersKater]

Any news on this? I have the same or a very similar problem, but it seems hard to catch.

[charonn0]

I don't have the hardware to reproduce it with, which means there's not a whole lot I can do about fixing it for now.

Here's what I do know

The Problem

The symptoms are a classic example of trying to read a value from an invalid memory address. If something by chance already is stored at that address we get garbage data, like a malformed URL.

If nothing is stored there the OS kills our process for an access violation, which is what is happening when you examine the EasyHandle in the debugger. The debugger will trigger the EasyHandle's computed properties which in turn call into libcurl which triggers an access violation and--poof.

Can it be fixed?

Yes. Exactly the same sort of problems cropped up in the transition to 64-bit. It's simply a matter of accounting for the differences between different CPU architectures.

If you search the code for #If Target you'll see how many times the subtle binary-level differences between 32-bit and 64-bit Intel architectures had to be accounted for. To fix this we need #If Target statements that account or ARM64's differences.

I guesstimate approximately the same amount of #If Target statements will have to be written for ARM64, probably in most of the same places. The hard part is knowing what's different. That's where trial-and-error, and hence reproducibility, come in.

[SchroedersKater]

Thank you very much for your effort! I wish I were able to get into this deeper, but I'm afraid, I'm not that deep in this business any more. Sorry. Anyway, let me say that I appreciate the effort that you made for programming this beautiful piece of software. Thank you very much again!