Signature::Coercible with user defined implicit casting #14440

jayzhan211 · 2025-02-03T13:37:38Z

Which issue does this PR close?

Part of Epic: Simplify functions signature with LogicalType #13301.
It would be helpful for Fix Type Coercion for UDF Arguments #14268

Rationale for this change

Inspired from Fix Type Coercion for UDF Arguments #14268 from @shehabgamin we used to define some implicit coercion inside datafusion which turns out it is not flexible and doesn't work for all the use case. We turn it explicitly and customizable by default.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jayzhan211 · 2025-02-03T13:40:04Z

datafusion/expr-common/src/signature.rs

+
+#[derive(Debug, Clone, Eq, PartialOrd, Hash)]
+pub struct ParameterType {
+    pub param_type: LogicalTypeRef,


param_type: target type of function signature
allowed_casts: implicit coercion allowed to cast to target type.

For example,
param_type: string
allowed_casts: binary, int

Valid: All are casted to string func(string) func(binary) func(int) Invalid: func(float or other types)

I am worried that Vec<Vec<ParameterType>> might become challenging to reason about. It would also be confusing about when one use this new signature rather than Signature::Coercable 🤔

Given this seems very similar to Signature::Coercable, and Signature::Coercable mirrors what we want pretty well, could add some new information there on the allowed coercions rather than an entire new type signature. Something like extending Coercable with rules could be used to coerce to the target type:

pub enum TypeSignature { ... Coercible(Vec<Coercion>), ... }

Where Coercion looks like

struct Coercion { desired_type: TypeSignatureClass, allowed_casts: ... // also includes an option for User Defined? }

🤔

I would like to have a breaking change to Signature::Coercible too, the only concern is whether this cause regression or large impact to downstream

@shehabgamin If we replace CoercibleV2 with Signature::Coercible would it be a large change we should be concerned to?

If not, I'll replace Signature::Coercible with Signature::CoercibleV2.

Signed-off-by: Jay Zhan <[email protected]>

jayzhan211 · 2025-02-07T01:55:22Z

datafusion/functions/src/string/repeat.rs

                vec![
-                    TypeSignatureClass::Native(logical_string()),
-                    TypeSignatureClass::Native(logical_int64()),


In v1 version, any integer is casted to the defined NativeType's default casted type which is i64 in this case.

I do like how much more explicit the new formualtion is

jayzhan211 · 2025-02-07T01:55:39Z

datafusion/functions/src/string/repeat.rs

+                    // Accept all integer types but cast them to i64
+                    Coercion {
+                        desired_type: TypeSignatureClass::Native(logical_int64()),
+                        allowed_casts: vec![TypeSignatureClass::Integer],


Without this, i32 is rejected.

jayzhan211 · 2025-02-07T02:00:08Z

datafusion/functions/src/string/ascii.rs

+            signature: Signature::coercible_v2(
+                vec![Coercion {
+                    desired_type: TypeSignatureClass::Native(logical_string()),
+                    allowed_casts: vec![TypeSignatureClass::Native(logical_binary())],


Coercing binary to string is now easily customizable

Signed-off-by: Jay Zhan <[email protected]>

alamb · 2025-02-07T19:17:17Z

I will try and review this carefully over the weekend

Maybe @shehabgamin has some time to take a look too

shehabgamin · 2025-02-08T08:25:54Z

I will try and review this carefully over the weekend

Maybe @shehabgamin has some time to take a look too

I will review over the weekend as well!

jayzhan211 · 2025-02-12T05:27:48Z

@jayzhan211 Should we port any relevant tests from the old PR? #14268

or close this first then revisit #14268

BTW, most of the binary-to-string conversions mentioned in #14268 might not be ideal for DataFusion. We should reconsider them.

alamb · 2025-02-13T10:49:04Z

@jayzhan211 is this PR ready for another review?

jayzhan211 · 2025-02-13T11:06:24Z

@jayzhan211 is this PR ready for another review?

yes, please

alamb

In general I think this PR moves the coercsion story forward significantly. Thank you @jayzhan211 and @shehabgamin

I think the Coercion struct(maybe enum) is really close to the general purpose API we have been desperately lacking these many years in DataFusion to describe coercions. Let me try some things and report back

alamb · 2025-02-13T15:14:42Z

datafusion/common/src/types/native.rs

@@ -198,6 +198,11 @@ impl LogicalType for NativeType {
        TypeSignature::Native(self)
    }

+    /// Returns the default casted type for the given arrow type


alamb · 2025-02-13T15:16:31Z

datafusion/expr-common/src/signature.rs

+    ///
+    /// Get all possible types for `information_schema` from the given `TypeSignature`
+    //
+    // TODO: Make this function private


we could potentially mark it as deprecated and make the new (non pub) version with a different name

Maybe something like this (can do in a follow on PR)

pub fn get_example_types(&self) -> Vec<Vec<DataType>> {

I think we would need NativeType for get_example_types

alamb · 2025-02-13T15:18:07Z

datafusion/expr-common/src/signature.rs

+    match signature_classes {
+        TypeSignatureClass::Native(l) => get_data_types(l.native()),
+        TypeSignatureClass::Timestamp => {
+            vec![


Is listing all possible DataType combination helpful?

I don't think this is helpful to be honest -- just the combination of classes

alamb · 2025-02-13T15:19:23Z

datafusion/expr-common/src/signature.rs

@@ -431,6 +463,35 @@ impl TypeSignature {
    }
 }

+fn get_possible_types_from_signature_classes(


I agree with @shehabgamin on both points

alamb · 2025-02-13T15:29:09Z

datafusion/expr-common/src/signature.rs

@@ -431,6 +463,35 @@ impl TypeSignature {
    }
 }

+fn get_possible_types_from_signature_classes(


Suggested name:

Suggested change

fn get_possible_types_from_signature_classes(

fn get_example_types(

And we can put it on TypeSignatureClass

alamb · 2025-02-13T15:29:46Z

datafusion/expr-common/src/signature.rs

+            vec![DataType::Date64]
+        }
+        TypeSignatureClass::Time => {
+            vec![DataType::Time64(TimeUnit::Nanosecond)]


I think it is ok to have one example type as this is only used for inforamtin schema -- the naming is super confusing though -- I think renaming the functions would make it much better

alamb · 2025-02-13T15:31:37Z

datafusion/expr/src/type_coercion/functions.rs

+            for (current_type, param) in current_types.iter().zip(param_types.iter()) {
+                let current_logical_type: NativeType = current_type.into();
+
+                fn is_matched_type(


this function also seems like it would make more sense as a method on TypeSignatureClass

alamb · 2025-02-13T15:32:43Z

datafusion/functions/src/string/repeat.rs

                vec![
-                    TypeSignatureClass::Native(logical_string()),
-                    TypeSignatureClass::Native(logical_int64()),


I do like how much more explicit the new formualtion is

alamb · 2025-02-13T15:37:32Z

datafusion/expr-common/src/signature.rs

@@ -460,6 +521,44 @@ fn get_data_types(native_type: &NativeType) -> Vec<DataType> {
    }
 }

+#[derive(Debug, Clone, Eq, PartialOrd)]
+pub struct Coercion {


@jayzhan211 , I believe @alamb 's question (please correct me if I'm wrong) is about creating functionality for a downstream user to override the default signature of a UDF in order to provide their own coercion rules.

This is what I was getting at

What do you think about making Coercion an enum like this (I can make a follow on PR / propose changes to this PR):

#[derive(Debug, Clone, Eq, PartialOrd)] pub enum Coercion { /// the argument type must match desired_type Exact { desired_type: TypeSignatureClass, } /// The argument type must be coercable to the desired type using the implicit coercion Implict { desired_type: TypeSignatureClass, implicit_coercion: ImplicitCoercion, } }

Then we can eventually maybe add a "user defined coercion" variant

Let me make a PR to see what this would look like.

alamb

Here is one software engineering PR:

Encapsulates logic from TypeSignatureClass, rename methods jayzhan211/datafusion#4
(to encapsulate this code more)

jayzhan211 · 2025-02-14T01:03:58Z

Enum looks good to me but "adding user defined coercion variant" is not clear to me since I think the Coercible signature is user-defined builtin, they can change the type they need and set it into their own function 🤔

alamb · 2025-02-14T11:54:02Z

Enum looks good to me but "adding user defined coercion variant" is not clear to me since I think the Coercible signature is user-defined builtin, they can change the type they need and set it into their own function 🤔

That is true that TypeSignature::Coercable allows users to define whatever they want.

I was thinking for @shehabgamin 's usecase where he wanted to implement coercsion that had a slightly different rules about when to cast Integer --> String. He could implement that with an entirely custom coercion, but it might be nice to just adjust the rules somehow (and reuse most of the rest of the logic ) 🤔

jayzhan211 · 2025-02-14T12:23:25Z

Enum looks good to me but "adding user defined coercion variant" is not clear to me since I think the Coercible signature is user-defined builtin, they can change the type they need and set it into their own function 🤔

That is true that TypeSignature::Coercable allows users to define whatever they want.

I was thinking for @shehabgamin 's usecase where he wanted to implement coercsion that had a slightly different rules about when to cast Integer --> String. He could implement that with an entirely custom coercion, but it might be nice to just adjust the rules somehow (and reuse most of the rest of the logic ) 🤔

Modifying Signature::Coercible alone won't achieve what @shehabgamin wants because, regardless of the approach, they would still need to:

Fork the entire function
Modify its signature
Register it in the function map

Even if we introduce another variant, we would still maintain a version for DataFusion, while others would have to go through the same process to customize it for their needs.

A better solution would be to separate the function signature from the function itself, as shown below:

pub struct ScalarFunctionExpr {
    fun: Arc<ScalarUDF>,
    name: String,
    args: Vec<Arc<dyn PhysicalExpr>>,
    return_type: DataType,
    nullable: bool,

   // new
   signature: TypeSignature
}

function_expr.with_signature(customize signature)

This change fundamentally impacts how scalar functions are structured...

Encapsulates logic from TypeSignatureClass, rename methods

jayzhan211 · 2025-02-14T12:56:30Z

To support int->string in coercion,

This change is all you need

                    Coercion::new_exact(TypeSignatureClass::Native(logical_string())),

                    Coercion::new_implicit(TypeSignatureClass::Native(logical_string()), vec![TypeSignatureClass::Integer], NativeType::String),

alamb · 2025-02-14T16:54:34Z

I took the liberty of merging up from main to get the CI to pass again.

I hope to spend a bit longer today messing around with potential changes to Coercion for your consideration

alamb · 2025-02-14T16:56:19Z

Modifying Signature::Coercible alone won't achieve what @shehabgamin wants because, regardless of the approach, they would still need to:

Fork the entire function

Modify its signature

Register it in the function map

I was hoping it might be possible to make a 'wrapper' function that only overrides the signature but passes everything else in

Though now that I type this, perhaps what is really needed is user defined coercion for operators (like = or other comparisons 🤔 )

alamb · 2025-02-14T16:57:09Z

To support int->string in coercion,

This change is all you need

                    Coercion::new_exact(TypeSignatureClass::Native(logical_string())),

                    Coercion::new_implicit(TypeSignatureClass::Native(logical_string()), vec![TypeSignatureClass::Integer], NativeType::String),

Maybe we could make this a builder style API too -- I hope to mess around with it later todya

shehabgamin · 2025-02-15T02:33:30Z

@jayzhan211 I will re-review by tomorrow EOD, exciting progress!

github-actions bot added logical-expr Logical plan and expressions functions labels Feb 3, 2025

jayzhan211 commented Feb 3, 2025

View reviewed changes

jayzhan211 mentioned this pull request Feb 3, 2025

Fix Type Coercion for UDF Arguments #14268

Open

jayzhan211 force-pushed the signature-v2 branch from 23953dd to 806c6a6 Compare February 6, 2025 13:46

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Feb 6, 2025

jayzhan211 marked this pull request as draft February 7, 2025 00:38

jayzhan211 added 7 commits February 7, 2025 08:38

coerciblev2

386c7ed

Signed-off-by: Jay Zhan <[email protected]>

repeat

579ffef

Signed-off-by: Jay Zhan <[email protected]>

fix possible types

104da43

replace all coerciblev1

aae48ff

cleanup

f585136

remove specialize logic

bad7348

comment

c99e986

jayzhan211 force-pushed the signature-v2 branch from 806c6a6 to c99e986 Compare February 7, 2025 01:49

github-actions bot added the common Related to common crate label Feb 7, 2025

jayzhan211 changed the title ~~Draft: coercible signature~~ Signature::Coercible with user defined implicit casting Feb 7, 2025

jayzhan211 commented Feb 7, 2025

View reviewed changes

jayzhan211 added 2 commits February 7, 2025 10:15

err msg

07f97d0

ci escape

da84394

jayzhan211 marked this pull request as ready for review February 7, 2025 03:57

jayzhan211 added 3 commits February 7, 2025 20:23

rm coerciblev1

e0a889e

Signed-off-by: Jay Zhan <[email protected]>

fmt

7a78a6d

rename

2f8c2ad

jayzhan211 added the api change Changes the API exposed to users of the crate label Feb 7, 2025

jayzhan211 requested a review from alamb February 7, 2025 13:04

jayzhan211 added 3 commits February 12, 2025 13:29

Merge branch 'main' of github.com:apache/datafusion into signature-v2

cb30d91

conflit

2cd8f5f

fmt

6ce159f

alamb reviewed Feb 13, 2025

View reviewed changes

alamb added 2 commits February 13, 2025 11:33

Rename get_example_types, make method on TypeSignatureClass

a3120f7

Move more logic into TypeSignatureClass

bde60f6

alamb mentioned this pull request Feb 13, 2025

Encapsulates logic from TypeSignatureClass, rename methods jayzhan211/datafusion#4

Merged

alamb reviewed Feb 13, 2025

View reviewed changes

fix docs

5d47a62

Merge pull request #4 from alamb/alamb/signature_suggestions

59d06d8

Encapsulates logic from TypeSignatureClass, rename methods

github-actions bot added the catalog Related to the catalog crate label Feb 14, 2025

jayzhan211 added 3 commits February 14, 2025 20:35

46

ff5d063

enum

77fe4b8

fmt

7dc7365

jayzhan211 requested review from alamb and shehabgamin February 14, 2025 13:18

jayzhan211 and others added 3 commits February 14, 2025 21:23

Merge branch 'main' of github.com:apache/datafusion into signature-v2

6ce2c8c

fmt

56838b7

Merge remote-tracking branch 'apache/main' into signature-v2

6ceb9a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Signature::Coercible with user defined implicit casting #14440

Signature::Coercible with user defined implicit casting #14440

jayzhan211 commented Feb 3, 2025 •

edited

Loading

jayzhan211 Feb 3, 2025

alamb Feb 3, 2025

jayzhan211 Feb 6, 2025

jayzhan211 Feb 6, 2025

jayzhan211 Feb 7, 2025

jayzhan211 Feb 7, 2025

alamb Feb 13, 2025

jayzhan211 Feb 7, 2025

jayzhan211 Feb 7, 2025

alamb commented Feb 7, 2025

shehabgamin commented Feb 8, 2025

jayzhan211 commented Feb 12, 2025 •

edited

Loading

alamb commented Feb 13, 2025

jayzhan211 commented Feb 13, 2025

alamb left a comment

alamb Feb 13, 2025

alamb Feb 13, 2025

jayzhan211 Feb 14, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb Feb 13, 2025

alamb left a comment •

edited

Loading

jayzhan211 commented Feb 14, 2025

alamb commented Feb 14, 2025 •

edited

Loading

jayzhan211 commented Feb 14, 2025 •

edited

Loading

jayzhan211 commented Feb 14, 2025

alamb commented Feb 14, 2025

alamb commented Feb 14, 2025

alamb commented Feb 14, 2025

shehabgamin commented Feb 15, 2025

	fn get_possible_types_from_signature_classes(
	fn get_example_types(

Signature::Coercible with user defined implicit casting #14440

Are you sure you want to change the base?

Signature::Coercible with user defined implicit casting #14440

Conversation

jayzhan211 commented Feb 3, 2025 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Feb 7, 2025

shehabgamin commented Feb 8, 2025

jayzhan211 commented Feb 12, 2025 • edited Loading

alamb commented Feb 13, 2025

jayzhan211 commented Feb 13, 2025

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment • edited Loading

Choose a reason for hiding this comment

jayzhan211 commented Feb 14, 2025

alamb commented Feb 14, 2025 • edited Loading

jayzhan211 commented Feb 14, 2025 • edited Loading

jayzhan211 commented Feb 14, 2025

alamb commented Feb 14, 2025

alamb commented Feb 14, 2025

alamb commented Feb 14, 2025

shehabgamin commented Feb 15, 2025

jayzhan211 commented Feb 3, 2025 •

edited

Loading

jayzhan211 commented Feb 12, 2025 •

edited

Loading

alamb left a comment •

edited

Loading

alamb commented Feb 14, 2025 •

edited

Loading

jayzhan211 commented Feb 14, 2025 •

edited

Loading