Partial function application: Keywords and placeholders

[ChristianGruen]

The test suite contains test cases – FunctionCall-414 … FunctionCall-417 – for partially applied functions with keywords and placeholders:

<test-case name="FunctionCall-414" covers-40="keywords">
  <description>Use of keyword arguments with placeholders on user-defined function</description>
  <created by="Michael Kay" on="2023-03-13"/>
  <modified by="Michael Kay" on="2023-12-13" change="do what the description says"/>
  <dependency type="spec" value="XQ40+"/>
  <test><![CDATA[
     declare function local:diff ($s as xs:integer, $t as xs:integer) as xs:integer { 
        $s - $t
     };  
     local:diff(s := 12, t := ?)(8)
  ]]></test>
  <result>
    <assert-eq>4</assert-eq>
  </result>
</test-case>
...

I didn’t find information on this feature combination in the spec; is it already covered? If yes, is it also possible to partially apply function items with keywords?…

declare function local:f($s, $t) { $s - $t };
local:f#2(s := 12, t := ?)(8),
local:f(?, ?)(s := 12, t := ?)(8)
...

If 2x yes, I can try to add some more test cases (for example, I assume that $f(t := 12, ?) is illegal, as arguments without keywords probably need to be placed first).

[michaelhkay]

I didn’t find information on this feature combination in the spec; is it already covered?

Yes. §4.6.1.1 says:

[Definition: An argument to a function call is either an argument expression or an [ArgumentPlaceholder] (?); in both cases it may either be supplied positionally, or identified by a name (called a keyword).]

(Note that this allows reordering of the parameters, which caused me some trouble, hence the tests...)

If yes, is it also possible to partially apply function items with keywords?…

No, the syntax of DynamicFunctionCall prevents it.

[ChristianGruen]

I see. And I notice that this affects our implementation quite fundamentally (which creates function items first before checking the placeholders). Indeed I wouldn’t be unhappy if we could look at this feature separately.

I assume it also applies to the arrow expression, 1 => (sum(zero := ?))(1) and similar?

I would be interested in the experiences of other implementors (provided there are any).

[michaelhkay]

Note the earlier discussion on this topic in issue #47.

[ChristianGruen]

Note the earlier discussion on this topic in issue #47.

I feel similar.

[michaelhkay]

I think it would be a rather artificial constraint to say that placeholders have to be positional, or that they have to retain order, and there's no good reason for the constraint other than saving implementors a bit of legwork.

It's a tradeoff here between complexity in the spec and complexity in the implementation, and I think it's always better in such cases to put the complexity in the implementation.

[ChristianGruen]

I think it would be a rather artificial constraint to say that placeholders have to be positional, or that they have to retain order, and there's no good reason for the constraint other than saving implementors a bit of legwork.

I'd still claim this contrasts with the previous comment that I quoted. Regarding implementation work, I’ve often observed that something that appeared trivial to us didn’t seem that trivial to other implementors. I think it’s a matter of perspective, which is why I would (ideally) be happy to learn about the assessment of other potential implementors.

[michaelhkay]

In Saxon, the implementation was a bit disruptive because it changed some internal APIs, but it wasn't in any way difficult. Certainly easier than introducing keyword parameters generally.

The question is, if you want to change the spec, how would you like to change it?

[rhdunn]

My XQuery plugin has support for an older keyword syntax. It does not currently handle placeholders well (with or without keywords). -- I'm doing this for things like providing inlay parameter hints for the parameter names when I can determine the function statically.

What I'm currently doing is:

1. resolving to the statically known function that the function call references by name, filtered by the supported arity range of the function;
2. binding the first parameter to the source expression if it is an arrow function call;
3. binding the positional arguments to the corresponding positional parameters;
4. binding the keyword arguments to the corresponding parameter name;
5. marking any unassigned parameters as missing.

What I would need to do to properly support argument placeholders (including in keyword arguments) is to convert the function call into an inline function declaration with a parameter list corresponding to the argument placeholders and use the variable bindings to map between the constructed inline function and the target function call.

I've not made any progress on this, nor catching up with XPath/XQuery 4.0 features as I'm working out how to make a functional processor with LSP (Language Server Protocol) support for IDEs. And I've been distracted by other projects.

[rhdunn]

For the bound variables, I'm constructing them as follows:

1. variable name -- the parameter's variable name;
2. variable type -- the parameter's variable type;
3. variable expression -- the argument expression if present, or the parameter's default expression.

[ChristianGruen]

In Saxon, the implementation was a bit disruptive because it changed some internal APIs, but it wasn't in any way difficult. Certainly easier than introducing keyword parameters generally.

As so often, it depends on the way how it’s implemented. In our case, supporting keyword parameters was not a big deal (and it will certainly used by much more people than placeholder with keywords).

The question is, if you want to change the spec, how would you like to change it?

I believe the current solution is half-baked. I would propose to…

1. either exclude the combination of placeholder and keywords, or
2. be serious, go a step further and support keywords dynamically (i.e., for function items). It feels not very convincing to be able to write sum(values := ?, zero := 0), but to disallow sum(values := ?, zero := 0)(values := 1 to 5).

[rhdunn]

It looks like it should be straightforward to implement that logic as opposed to fully dynamic parameter resolution. That is, given a partial function application, any argument placeholders assigned via a keyword parameter would propagate that parameter name to the dynamic function.

The reason I think this would be straightforward to support is that all the information needed to define the named parameter is present statically. Thus, the processor would need to track the parameter name (from the keyword argument name) as well as the parameter position (from the number of argument placeholder encountered thus far) when constructing the inline function.

[michaelhkay]

We've rehearsed the reasons for not allowing keywords on dynamic function calls many times. In general with a dynamic function call you don't know what function you are calling and you don't know statically what its parameter names are. fn:for-each-pair() might be called with a function having parameter names (x, y) or with one having parameter names (arg1, arg2) or with one having parameter names (y, x). To make keywords on dynamic calls viable we would need some way of making the parameter names part of the required function type.

The example

sum(values := ?, zero := 0)(values := 1 to 5)

is completely unrealistic because it's a rare case where it's a dynamic function call, but it is statically known what function is being called.

[ChristianGruen]

To make keywords on dynamic calls viable we would need some way of making the parameter names part of the required function type.

…exactly. – With XQuery, functions can be declared with function and variable declarations…

declare function compute($a := 1, $b := 2) { ... };
declare variable $compute := function($a := 1, $b :=2) { ... };

…and it will not be unrealistic if people expect to be able to use the following function calls interchangeably (with and without placeholders):

compute(b := 3)
$compute(b := 3)

I think my point is that it’s hard to convey/understand (for non-implementors) why compute(a := ?, ?) is allowed, whereas $compute(a := ?, ?) is not.

[rhdunn]

Regarding function types, I've always wondered why you cannot assign parameter names e.g. to document the parameters. With that, the parameter names could come from the earliest/nearest/closest function signature. I.e.:

1. the parameter type declaration for function parameters -- e.g. in fn:for-each-pair;
2. the inline function bound to a variable or step in a path expression -- e.g. the $compute example above, or implicitly created inline functions per the sum(values := ?, zero := 0) example;
3. the named static function declaration, including in named function references -- e.g. compute#2.

This way, the logic would be deterministic and resolvable statically without e.g. having to resolve the function passed to the parameter of a function. It would also be the most logical from a user perspective -- if implementing fn:for-each-pair or similar, I'd want to use the names I specified in the parameter declaration.

[michaelhkay]

Regarding function types, I've always wondered why you cannot assign parameter names e.g. to document the parameters.

It creates a lot of opportunities for complex rules and/or user confusion for example when a function with declared parameters (x, y) is passed as an argument to something that expects a function with parameters (y, x).

What exactly is the impact on instance matching, subtyping, and function coercion?

This seems to be a proposal to add something pretty complex with very little user benefit.

[rhdunn]

Why do the parameter names need to be considered for instance matching, subtyping, and function coercion? -- I would advocate that they don't affect those. Thus, they are a purely static context feature and is backward compatible with signatures that don't specify names.

If I'm implementing a function like fn:for-each-pair I don't care what the names are of the passed in function, I only care about the names I declare in the parameter signature. -- Specifically, the names allow the following:

1. documenting what the parameters mean for someone calling the function by looking at the signature only (e.g. in an IDE);
2. being able to refer to the names in the spec to make it clearer and easier to read;
3. being able to specify those parameters by name to make the implementation more readable.

[ChristianGruen]

I think we talk about different things:

1. My proposal would be to introduce (dynamic runtime) support for placeholders and keywords in function items, such that we can do things as follows:

for $op in (fn($a, $b) { $a - $b }, fn($a, $b) { $a div $b })
let $inverted := $op(b := ?, a := ?)
return for-each-pair($seq1, $seq2, $inverted)

If we believe that’s one step too far, I would like to question whether/why we need the placeholder/keyword support for static functions.

2. If we had (optional) named parameters in function types, …

fn:for-each-pair(
  $input1  as item()*,	
  $input2  as item()*,	
  $action  as function(
    item1  as item(),
    item2  as item(),
    pos    xs:integer
  ) as item()*	
) as item()*

…it could be very useful to document our increasing number of higher-order functions, and to help IDEs and processors to generate more helpful error messages, but it would (in my point of view) not affect the way how placeholders and keywords work. Function calls like for-each-pair(1, 2, compare#2) would still be possible (and need to be possible), even though the first parameter names of fn:compare are value1 and value2 (instead of item1 and item2).

@rhdunn Is 2. in line with the way you see it, or am I missing something here?

[rhdunn]

Yes, that's in line with what I'm thinking.

There are two aspects to this:

1. a static context only feature that does not affect the runtime;
2. a dynamic context feature that supports a more generally applicable use of keywords and argument placeholders.

I'm thinking of this statically -- i.e. when it is possible to determine that the target function is a single known item (function declaration, inline function, or parameter/variable with a declared type) then the keywords should be resolved against that.

If it is possible to do dynamically, we shoud aim to do that. However, I think getting the static cases would be easier to specify/define and implement. -- That would also be what an IDE, compiler, or other static analysis tool would check.

[ChristianGruen]

@rhdunn Thanks.

I'm thinking of this statically -- i.e. when it is possible to determine that the target function is a single known item (function declaration, inline function, or parameter/variable with a declared type) then the keywords should be resolved against that.

Just to be sure, this is not about 2. anymore (the named parameters in function types), right? Or would you like to treat named parameters as keywords?

When we look at the “static” analysis of a query, we need to be aware that this depends on the facilities of an implementation). For example, a processor compiler might (or might not) statically rewrite the query…

for $op in (fn($a, $b) { $a - $b }, fn($a, $b) { $a div $b })
let $inverted := $op(b := ?, a := ?)
return for-each-pair($seq1, $seq2, $inverted)

…to…

for-each-pair($seq1, $seq2, fn($b, $a) { $a * $b }),
for-each-pair($seq1, $seq2, fn($b, $a) { $a div $b })

[rhdunn]

I would like the named parameters in function types to work as keywords. -- I suggested that to help with the static analysis of names for the case when implementing e.g. for-each-pair. (I also think that it is a good idea for the other reasons mentioned, so I've created a separate proposal for this.)

With the static analysis approach, I wouldn't expect the for example (specifically the $inverted part) as $op does not either:

1. have a static typed function test that defines the parameters (e.g. via Defining names for parameters on typed function tests #1136);
2. refer to a single inline function with defined named parameters;
3. refer to an implicit inline function with named parameters referencing argument placeholders -- this is the static nature I'm referring to for this issue;
4. refers to a single function by named reference or complete argument placeholder binding;
5. calling a function directly.

I would not expect any more complex cases to support static named keywords. We currently only support case 5. Other more complex examples like you mention would be dynamic as you are proposing.

In the static case, I would expect $inverted to have an error:

let $inverted := $op(b := ?, a := ?)
                     ^^^^
Error: cannot determine the keyword name in this context. $op is bound to a dynamic expression without a defined static type.

[ChristianGruen]

I can’t help, but I still find the combination of placeholders and keywords confusing (no matter how it’s to be implemented, after all). I bet that hardly anyone reading the next function calls…

a) sort(?),
b) sort(input := ?, keys := ?)
c) sort(keys := ?, input := ?)
d) sort(?, keys := ?),
e) sort(keys := ?, ?)
f) sort(keys := ?),

…would currently be able to guess which variants are legal (ɐ/q/ɔ/p, I assume) and which are not.

Do we lose so much if we stick with function($a, $b) { f($b, $a) } ?

[michaelhkay]

Do we lose so much if we stick with function($a, $b) { f($b, $a) } ?

I'm not sure precisely what restriction you want to impose (no keywords with placeholders? Placeholders must be in the right order?) but either way you add a extra rule to the spec that articifially restricts what users can do in the interests of making life easier for implementors, and that's usually a bad trade-off in my view.

[ChristianGruen]

either way you add a extra rule to the spec that articifially restricts what users can do in the interests of making life easier for implementors, and that's usually a bad trade-off in my view.

In my last comment, I tried to focus on user expectations. Even if we allow keywords and placeholders, it’s not obvious (and questionable, in my point of view) that arguments are swapped.

I'm not sure precisely what restriction you want to impose (no keywords with placeholders? Placeholders must be in the right order?)

My observation/claim is that the current behavior, which I believe implied from rule §4.6.1.1…

(Note that this allows reordering of the parameters, which caused me some trouble, hence the tests...)

…is not defined/described at all in the current spec:

• The function calls sort(keys := $k, input := $i) and sort(input := $i, keys := $k) are equivalent: It makes no difference in which order the parameters are supplied.
• In contrast, sort(keys := ?, input := ?) and id(input := ?, keys := ?) are not equivalent. How is one supposed to know that without trying the latest Saxon implementation, or interpreting the referenced test cases?

[michaelhkay]

In contrast, sort(keys := ?, input := ?) and id(input := ?, keys := ?) are not equivalent. How is one supposed to know that without trying the latest Saxon implementation, or interpreting the referenced test cases?

It's explicitly stated in the rules for partial function application:

parameter names: The names of the parameters of FD that have been identified as placeholder parameters, retaining the order in which the placeholders appear in the function call.
Note:
A partial function application can be used to change the order of parameters, for example fn:contains(substring := ?, value := ?) returns a function item that is equivalent to fn:contains#2, but with the order of arguments reversed.
signature: The parameters in the returned function are the parameters of FD that have been identified as placeholder parameters, retaining the order in which the placeholders appear in the function call. The result type of the returned function is the same as the result type of FD.

[ndw]

From the Prague F2F, consensus drifting towards requiring more compelling reasons to change the status quo.

[ndw]

At meeting 095, the CG agreed to close this issue with no further action.