Identify optional arguments in callback functions

[michaelhkay]

It was pointed out today that is not obvious, looking at a function signature like

fn:filter(
  $input as item()*, |  
  $predicate as function(item(), xs:integer) as xs:boolean |  
) as item()*

that the second argument of the $predicate function is optional.

At least in the documentation, it would be useful to capture this in some way. Being "optional" here means that it makes sense, semantically, to supply an arity-1 function, in which case the caller will not supply the second argument.

Perhaps it would also be useful to go beyond documentation, and attach some syntax and semantics to it. Specifically, if the signature of the callback function indicates that the first N arguments are required, then supplying a function item of arity less than N will result in a type error.

[michaelhkay]

I propose as follows:

1. The syntax of a TypedFunctionTest is extended to allow, for example, function(item(), %optional xs:integer) as xs:boolean. This is essentially the union of function(item()) and function(item(), xs:integer): it matches both arity-1 functions with signature function(item()) as xs:boolean and arity-2 functions with signature function(item(), xs:integer) as xs:boolean. If a parameter is designated %optional then all subsequent parameters must also be designated %optional.
2. A specific FunctionItem still has a fixed arity. A type may allow functions with different arities, but each instance has a fixed arity, and a dynamic function call must supply the correct number of arguments.
3. In the function coercion rules, we drop the arity-bending coercion. If the signature of a higher-order function declares a callback parameter of type function(item(), %optional xs:integer) as xs:boolean then (subject to the old function coercion rules) the supplied argument must match this type, which means it must match one of the two allowed possibilities.
4. To make it easier for user-defined functions to invoke a callback that allows functions of different arities, we extend fn:apply() so that excess arguments in the argument array are dropped. So if fn:apply($f, [$arg1, $arg2]) supplies an array of two arguments to a function item that only expects one, the second argument is silently ignored.
5. In those F&O functions where we have added an optional $position argument, this will be declared as %optional in the type signature. In typical cases such as fn:filter the first argument will revert to being mandatory, which means fn:filter will no longer accept true#0 and false#0 as arguments. (Discuss.)

[michaelhkay]

Unfortunately the syntax %optional does have complications because a SequenceType can already start with the annotation %optional. We could get around that if we want to by reserving the annotation name. However, I chose the syntax with the thought that we might eventially allow general annotations on function parameters, and that appears to be tricky. Perhaps we should use %%optional.

[ndw]

Point 4, silently dropping arguments in fn:apply(), makes me a little nervous if this applies in the general case. If I think I'm applying a function with three arguments, but I've identified a function that only takes two, that's probably an error on my part.

Did you intend, or could it be written, to apply only in cases where the extra arguments are %optional (or, ugh %%optional)?

[michaelhkay]

Well we could generalise fn:apply() so that instead of taking an array as the second argument, it takes a function which, given an integer N, returns the value to be used for the Nth argument - explicit lazy evaluation. Would that make you more comfortable? (That's backwards compatible because we had the foresight to ensure that an array can be used where a function-taking-an-integer-argument is expected).

[ChristianGruen]

I’m not keen on adding even more syntax for this. It would need to be correctly understood again, and it would be yet another approach to tackle optional arguments. In static functions, arguments are optional if default arguments exist…

declare function predicate(
  $item      as item()*,
  $position  as xs:integer := ()
) as xs:boolean

…so (if we think we need) we should rather choose a similar solution for function items.

Next, if we make the first argument(s) mandatory again, we would need to revive some discussions that have been resolved in the past, e.g. on fn:identity (#858).

But I definitely agree that the documentation of functions with higher-order function arguments gets more and more arcane, no matter if arguments may be more or less optional. I think we can win a lot by improving the presentation of the arguments of higher-order functions in general.

We could start by adding comments for function item arguments.

fn:filter(
  $input      as item()*,
  $predicate  as function(
    item(),    (: current item :)
    xs:integer (: current position :)
  ) as xs:boolean
) as item()*

This is how the filter function is presented in JavaScript; maybe we can learn from them?

[dnovatchev]

I propose as follows:

1. The syntax of a TypedFunctionTest is extended to allow, for example, function(item(), %optional xs:integer) as xs:boolean. This is essentially the union of function(item()) and function(item(), xs:integer): it matches both arity-1 functions with signature function(item()) as xs:boolean and arity-2 functions with signature function(item(), xs:integer) as xs:boolean. If a parameter is designated %optional then all subsequent parameters must also be designated %optional.
2. A specific FunctionItem still has a fixed arity. A type may allow functions with different arities, but each instance has a fixed arity, and a dynamic function call must supply the correct number of arguments.
3. In the function coercion rules, we drop the arity-bending coercion. If the signature of a higher-order function declares a callback parameter of type function(item(), %optional xs:integer) as xs:boolean then (subject to the old function coercion rules) the supplied argument must match this type, which means it must match one of the two allowed possibilities.
4. To make it easier for user-defined functions to invoke a callback that allows functions of different arities, we extend fn:apply() so that excess arguments in the argument array are dropped. So if fn:apply($f, [$arg1, $arg2]) supplies an array of two arguments to a function item that only expects one, the second argument is silently ignored.
5. In those F&O functions where we have added an optional $position argument, this will be declared as %optional in the type signature. In typical cases such as fn:filter the first argument will revert to being mandatory, which means fn:filter will no longer accept true#0 and false#0 as arguments. (Discuss.)

It seems a lot more simple to just split this into two separate overloads - as this is done in C# and probably in other languages.
Thus we will avoid having to create a complicated machinery and we will be able to keep the signatures for each of the overloads more readable and easily understandable.

What is wrong with having two overloads?

From Microsoft's documentation:

Overloads

Where(IEnumerable, Func<TSource,Boolean>)
Filters a sequence of values based on a predicate.

Where(IEnumerable, Func<TSource,Int32,Boolean>)
Filters a sequence of values based on a predicate. Each element's index is used in the logic of the predicate function.

[michaelhkay]

What is wrong with having two overloads?

We would need a completely different type system. We don't have that kind of polymorphism in the language and it would be an immense effort to introduce it, especially if we were to retain any kind of backwards compatibility.

What we could potentially do would be to introduce a general union type so the signature could be

filter($input as item()*, $predicate as any-of(function(item()) as boolean, function(item(), integer) as boolean)

That's essentially equivalent to what I'm proposing.

There's a world of difference between having one function that accepts different kinds of arguments, and having two functions where the function despatch algorithm depends on the type of the supplied arguments.

You're constantly comparing with C#, which is a strongly typed language. For better or worse, that's not where we are coming from. We are much closer to weakly typed languages like Javascript.

[dnovatchev]

We would need a completely different type system. We don't have that kind of polymorphism in the language and it would be an immense effort to introduce it, especially if we were to retain any kind of backwards compatibility.

If we are facing this kind of difficulties, then I'd rather prefer the $action function type to be specified as function(*), and still say in the Spec that this is equivalent to having two separate overloads and list each of these overloads in full.

Thus readability will not suffer, we will continue to be "dynamic" - seems like a win-win situation, doesn't it?

[Arithmeticus]

I think this thread was catalyzed by discussion on fn:fold/scan-left/right. Can we get a list of all XPath functions that might be relevant/affected?

[michaelhkay]

Can we get a list of all XPath functions that might be relevant/affected?

A quick (and possibly incomplete) search of the function catalog gives

for-each filter fold-left fold-right while-do do-until for-each-pair index-where for-each filter fold-left fold-right for-each-pair every index-where some partition

[ChristianGruen]

A quick (and possibly incomplete) search of the function catalog gives […]

…and basically every other function that has function arguments. For example, the action of fn:replace has two parameters, both of which could be regarded as optional.

I'm interested what some of you think about my suggestion to focus on our documentation? JavaScript users (and there are lots of them) seem to have no problem with the solution that we're about to take.

[dnovatchev]

Instead of polluting the definition of well-known and understood functions by adding a new argument that nobody seems to use (I tried hard to find examples of using the index argument!) let us create a separate function that has this extended capabilities - for example we could use a name like: fold-left-extended or fold-left-position-aware.

The mere fact that it is so difficult finding examples of actually using this index-argument clearly tells us that this is a separate, much limited functionality.

Let us not pollute some of the most-popular and well-known functions in functional programming, making them unrecognizable and difficult to understand.

[michaelhkay]

As I said at the start of the thread, the main issue is making it clear to the reader what the options are, and we don't necessarily need to change the language to do that: we could go with a documentation solution. Building on Christian's suggestion, something like:

fn:filter(
  $input      as item()*,
  $predicate  as function(
    item(),    (: current item :)
    xs:integer (: current position - optional :)
  ) as xs:boolean
) as item()*

[Arithmeticus]

Should line 5 be...
xs:integer? := () (: current position - optional :)
?

[dnovatchev]

As I said at the start of the thread, the main issue is making it clear to the reader what the options are, and we don't necessarily need to change the language to do that: we could go with a documentation solution. Building on Christian's suggestion, something like:

fn:filter(
  $input      as item()*,
  $predicate  as function(
    item(),    (: current item :)
    xs:integer (: current position - optional :)
  ) as xs:boolean
) as item()*

This is still quite unreadable - but we were talking more specifically about the need for having such additional/optional argument for the fold - functions. And there there is absolutely insignificant amount of evidence (concrete code examples of usage) that adding such additional, new argument is really needed in the case of the fold functions.

If the new argument would be specified in less that 0.1% of all cases, this clearly means that if (for whatever reason) somebody (whoever they may be) desperately needs it, then they can have a separate new function, say fold-left-extended

If 80% of the world already knows well the fold functions from languages such as Haskell and C#, why would we want to disguise our fold functions and make them completely unrecognizable to this majority of people?

Maybe we intentionally want to repel them from using our languages?

[ChristianGruen]

Should line 5 be... xs:integer? := () (: current position - optional :) ?

It could look like that if we decide to add default arguments to dynamically called function items in the future (but there are various implications that would need to be solved).

With the current solution in the spec, both arguments are optional, i.e. you can also supply a function like fn() { true() } as predicate (see function coercion for details).

[ChristianGruen]

For those who want to learn about use cases for folds with index access, two examples in JavaScript:

https://stackoverflow.com/questions/35034006/javascript-array-reduce-start-from-index
https://blog.alexdevero.com/javascript-reduce-method/

[michaelhkay]

I would be happy to go with something that's a bit more type-safe than Javascript, i.e. not allowing arguments to be omitted unless there's something explicit to say they're optional. But I certainly don't accept the argument that something we've adopted from Javascript is going to be unacceptable to our user community.

[michaelhkay]

but we were talking more specifically about the need for having such additional/optional argument for the fold - functions

Not on this thread we weren't. This thread is about how to indicate in function signatures that some arguments of callback functions are optional. The question of whether to have such optional arguments on the fold and scan functions is another issue entirely.

[ChristianGruen]

I would be happy to go with something that's a bit more type-safe than Javascript, […].

True. At the moment, it’s convenient to pass something as simple as true#0 as a dynamic argument to a function…

let $select := function($test) {
  fn:filter($some-data, $test)
}
return $select(true#0)

…but if the first parameter of the predicate function of fn: filter was mandatory, passing fn { true() } or fn($i) { true() } would certainly be ok, and possibly clearer.

Regarding the syntax, @Arithmeticus’ observation sticks in my mind: Wouldn't it be best and consistent if we introduced default values for function items (as @rhdunn had suggested a long time ago)? Which would certainly be one more hard nut to crack.

[dnovatchev]

but we were talking more specifically about the need for having such additional/optional argument for the fold - functions

Not on this thread we weren't. This thread is about how to indicate in function signatures that some arguments of callback functions are optional. The question of whether to have such optional arguments on the fold and scan functions is another issue entirely.

OK, I will be happy just not to add this new argument to the $action function of the folds/scans.

For all other functions, still we need to design their signatures having in mind the importance of readability and understandability.

[dnovatchev]

For those who want to learn about use cases for folds with index access, two examples in JavaScript:

https://stackoverflow.com/questions/35034006/javascript-array-reduce-start-from-index https://blog.alexdevero.com/javascript-reduce-method/

These only prove the point that the use of the artificially added new index argument is extremely rare . The blog article proposes a solution for computing the average that uses completely unnecessarily the index argument. This, of course is not needed to find the average. And there is no other, example in all the provided 5 examples that uses the index-argument.

As for the StackOverflow question - it was asked 8 years ago, and of the 3 solutions that use the reduce function , only one uses the index argument. Again, this shows that the index argument is not necessary in this case.

Also, in the last 8 years approximately 2136 Javascript questions were asked on StackOverflow. Only one solution to only one of these tried to use (unnecessarily) the index argument This is 0.05% of all questions, and 0.01% of all solutions. These numbers speak for themselves.

To summarize: The provided "evidence" actually tells us that calling the $action function with the index argument is extremely rare, and even when this has been done, it was completely unnecessary.

[ChristianGruen]

The provided "evidence" […] it was completely unnecessary.

Examples and evidence are different things.

Conversations are pointless if the mind is fixed. Let’s focus on the actual topic of this issue.

[dnovatchev]

The provided "evidence" […] it was completely unnecessary.

Examples and evidence are different things.

Conversations are pointless if the mind is fixed. Let’s focus on the actual topic of this issue.

Still the fact remains that it is difficult to find usage of this parameter and even when found this usage is artificial and unnecessary.

As @michaelhkay pointed out, this belongs to the other issue - hope we can keep the folds clean.

[dnovatchev]

I propose as follows:

1. The syntax of a TypedFunctionTest is extended to allow, for example, function(item(), %optional xs:integer) as xs:boolean. This is essentially the union of function(item()) and function(item(), xs:integer): it matches both arity-1 functions with signature function(item()) as xs:boolean and arity-2 functions with signature function(item(), xs:integer) as xs:boolean. If a parameter is designated %optional then all subsequent parameters must also be designated %optional.
2. A specific FunctionItem still has a fixed arity. A type may allow functions with different arities, but each instance has a fixed arity, and a dynamic function call must supply the correct number of arguments.
3. In the function coercion rules, we drop the arity-bending coercion. If the signature of a higher-order function declares a callback parameter of type function(item(), %optional xs:integer) as xs:boolean then (subject to the old function coercion rules) the supplied argument must match this type, which means it must match one of the two allowed possibilities.
4. To make it easier for user-defined functions to invoke a callback that allows functions of different arities, we extend fn:apply() so that excess arguments in the argument array are dropped. So if fn:apply($f, [$arg1, $arg2]) supplies an array of two arguments to a function item that only expects one, the second argument is silently ignored.
5. In those F&O functions where we have added an optional $position argument, this will be declared as %optional in the type signature. In typical cases such as fn:filter the first argument will revert to being mandatory, which means fn:filter will no longer accept true#0 and false#0 as arguments. (Discuss.)

I think this is a good and clear proposal and it is a positive step in solving the issue.

I would prefer to have not:

   function(item(), %optional xs:integer) as xs:boolean

but the easier to read (no '%' character):

   function(item(), optional xs:integer) as xs:boolean

Or even:

   function(item() [, xs:integer]) as xs:boolean

[rhdunn]

I like the idea of using the := syntax for this, instead of providing yet another syntax:

function(item(), xs:integer := 0) as xs:boolean

That would resolve the issue of what value should be supplied if the function is called with the lower arity.

From an API perspective, should we also make xs:integer an optional sequence? I.e.:

function(item(), xs:integer? := ()) as xs:boolean

That's clearer to the user that the second parameter is optional, even if it will never be called with an empty value.

[dnovatchev]

I like the idea of using the := syntax for this, instead of providing yet another syntax:

function(item(), xs:integer := 0) as xs:boolean

That would resolve the issue of what value should be supplied if the function is called with the lower arity.

This doesn't resolve the issue of providing a function with arity 2, when a function with arity 3 is expected. "Forgiving" this results in a flood of unintentional user errors going unnoticed and resulting in difficult to debug runtime errors.

From an API perspective, should we also make xs:integer an optional sequence? I.e.:

function(item(), xs:integer? := ()) as xs:boolean

That's clearer to the user that the second parameter is optional, even if it will never be called with an empty value.

Except that this still requires the call to be:

funName(3, ())

and not

funName(3)

[ChristianGruen]

Except that this still requires the call to be:
funName(3, ())

The signature of fn:sum is:

fn:sum(
  $values  as xs:anyAtomicType*,	
  $zero    as xs:anyAtomicType?	:= 0
) as xs:anyAtomicType?

…and you can omit the second argument and write fn:sum(1). We could apply a similar principle to function items.

[dnovatchev]

Except that this still requires the call to be:
funName(3, ())

The signature of fn:sum is:

fn:sum(
  $values  as xs:anyAtomicType*,	
  $zero    as xs:anyAtomicType?	:= 0
) as xs:anyAtomicType?

…and you can omit the second argument and write fn:sum(1). We could apply the same principle to function items.

And what sensible default would there be for the $predicate argument in the case of:

array:filter(
$predicate as function(item()*, xs:integer) as xs:boolean   
) as array(*)

Wouldn't it be nice if we could provide as the default a 1-argument function? No, because there are so many such 1-argument predicates and each could be handy. Here the real question is not about the default value for a function, or for a function argument. What we really want somehow to do, is specify the default arity of a function, when this arity can vary from one call to another.

Even if we specify that one argument is optional, the function still must be of of arity 2, and be able to handle the 2nd argument's (default) value.

In many cases when a function of N - 1 arguments is passed N arguments this could be an user error. By masking this error we are doing a bad service to the user, as they will be allowed to execute and will observe unexpected, strange and difficult to explain runtime behavior of their code.

[ChristianGruen]

And what sensible default would there be for the $predicate argument in the case of:

array:filter(
  $predicate as function(item()*, xs:integer) as xs:boolean   
) as array(*)

I would propose xs:integer := (), as the value will not be used.

Even if we specify that one argument is optional, the function still must be of of arity 2, and be able to handle the 2nd argument's (default) value.

Currently, due/thanks to the function coercion rules, fn:filter($value, true#0) is valid.

Personally, I believe our major challenge is the improvement of the existing documentation. Think e.g. of:

fn:replace(
  $value         as xs:string?,	
  $pattern       as xs:string,	
  $replacement   as xs:string?	:= (),
  $flags         as xs:string?	:= '',
  $action        as (function(xs:untypedAtomic, xs:untypedAtomic*) as item()?)?	:= ()
) as xs:string

By just reading the signature, it’s impossible to guess what the parameters of $action are supposed to do, no matter what the default arity would be. And things won’t get easier when more high-order functions will be added in the future.

[rhdunn]

I've wondered why function types don't allow named parameters. That's a separate issue, but allowing (optional) named parameters on type signatures would help with documenting the behaviour of the function.

[ChristianGruen]

I've wondered why function types don't allow named parameters. That's a separate issue, but allowing (optional) named parameters on type signatures would help with documenting the behaviour of the function.

That’s true.

Your issue on supporting optional parameters on dynamic functions is still open as well (#158). I believe to remember that we observed it’s more challenging to realize/formalize this for dynamic function items.

[dnovatchev]

And what sensible default would there be for the $predicate argument in the case of:

array:filter(
  $predicate as function(item()*, xs:integer) as xs:boolean   
) as array(*)

I would propose xs:integer := (), as the value will not be used.

An integer value cannot be the empty sequence.

[dnovatchev]

Even if we specify that one argument is optional, the function still must be of of arity 2, and be able to handle the 2nd argument's (default) value.

Currently, due/thanks to the function coercion rules, fn:filter($value, true#0) is valid.

This is even worse!

Passing a 0-argument function where a 1-argument function (which is the majority of use cases) or a 2-argument function is expected.

This is meaningless and misleading when the main goal of fn:filter is considered.

[ChristianGruen]

This is even worse!

Passing a 0-argument function where a 1-argument function (which is the majority of use cases) or a 2-argument function is expected.

All this has already been discussed in the past.

[dnovatchev]

You're constantly comparing with C#, which is a strongly typed language. For better or worse, that's not where we are coming from. We are much closer to weakly typed languages like Javascript.

@michaelhkay ,

Whether or not we should closely copy Javascript is a separate topic (which probably we need to discuss separately and reach some conclusion/agreement on), but even if this is so imperative to do, why not copy from the "bigger brother" Typescript, which allows function overloads?

Function Overloads in Typescript.

Why did they do this? Clearly not for the convenience of the implementors (who, in our case, (said in comments) don't care) but for the Typescript programmers, who need to both easily read the specification of someone else's function and also be able to specify their own-created functions in the most clear and understandable way to future users.

In Typescript it is easy to raise a compile-time error when a function is called with 2 arguments, but no overload expects 2 arguments, there are just 2 overloads, expecting either 1 or 3 arguments.

I deeply suspect that we are not able to have such level of validation/diagnostics at the moment.

[rhdunn]

@dnovatchev The point is that XPath, XQuery, and XSLT function overloading is only based on the arity of the function. Likewise, for types it coerces (matches) values not types.

A such, it is not possible to define two functions with the same arity but different parameter types. So they cannot support one taking a function(item()) as xs:boolean and another taking a function(item(), xs:integer) as xs:boolean.

Doing so now would break a lot of code and existing implementations, so is not technically feasible in XPath, XQuery, and XSLT.

Incidentally, that's the same roadblock that the creators of TypeScript had, hence them creating a separate language to JavaScript!

[ChristianGruen]

In Typescript it is easy to raise a compile-time error when a function is called with 2 arguments, but no overload expects 2 arguments, there are just 2 overloads, expecting either 1 or 3 arguments.

Note that in Typescript it’s even discouraged to make parameters of callback (i.e., higher-order) functions optional, as it’s completely valid as well to provide callbacks that accept fewer arguments:

https://www.typescriptlang.org/docs/handbook/declaration-files/do-s-and-don-ts.html#optional-parameters-in-callbacks

[dnovatchev]

Perhaps it would also be useful to go beyond documentation, and attach some syntax and semantics to it. Specifically, if the signature of the callback function indicates that the first N arguments are required, then supplying a function item of arity less than N will result in a type error.

Well. any function could potentially be used as a callback - then following the above would mean not to specify at all required and optional arguments - that is: all arguments become optional.

Down the rabbit hole ...

I think the opening comment by @michaelhkay is absolutely correct (bold hi-lighting is mine):

Perhaps it would also be useful to go beyond documentation, and attach some syntax and semantics to it. Specifically, if the signature of the callback function indicates that the first N arguments are required, then supplying a function item of arity less than N will result in a type error.

I strongly support the idea that we need to make the language type-safe where this is possible.

[dnovatchev]

@dnovatchev The point is that XPath, XQuery, and XSLT function overloading is only based on the arity of the function. Likewise, for types it coerces (matches) values not types.

A such, it is not possible to define two functions with the same arity but different parameter types. So they cannot support one taking a function(item()) as xs:boolean and another taking a function(item(), xs:integer) as xs:boolean.

Doing so now would break a lot of code and existing implementations, so is not technically feasible in XPath, XQuery, and XSLT.

Doing so for ver. 4.0 and later will not break any existing pre-ver. 4.0 code.

It is true that having a function reference such as myFun#3 cannot distinguish between two overloads that have the same arity of 3.

But we could augment the current syntax for a named function reference so that the exact overload becomes part of it, for example something like:

myFun#3^1 meaning: the 1st overload that has arity 3

This is the first that comes to mind, maybe there could be even better ways to distinguish between different same-arity overloads.

We simply do need to do our job on this, and I believe we can achieve a good and useful result.

[ChristianGruen]

I strongly support the idea that we need to make the language type-safe where this is possible.

I agree in principle (although I encountered use cases in practice where a 0-arity function like true#0 can be useful argument; see #981 (comment)).

A pragmatic solution would be to simplify the signatures to…

fn:xyz(
  $input   as item()*,
  $action  as function(*)
)

…and define the supported types and supported arities for $action in the detailed rules of the corresponding function. A type error could be raised if the argument doesn’t match these rules (but it will be difficult for IDEs to raise such errors statically).

If we generalized local union types, the following syntax would become legal…

fn:xyz(
  $input   as item()*,
  $action  as union(
    function(item()) as xs:boolean,
    function(item(), xs:integer) as xs:boolean
  )
)

…but pretty verbose.

[line-o]

Now that we have ChoiceTypes we can resolve this by

fn:xyz(
  $input as item()*,
  $action as function(item()) as xs:boolean | function(item(), xs:integer) as xs:boolean
)

[ChristianGruen]

I think we can close this issue:

• With 1136 Optional names in typed function types #1696, names are now possible in typed function types
• Editorial: optional position argument in function signature for for-each and other HOF  #1514 presents ideas on how to make optional HOF arguments mandatory.