My understanding of what you have written is that there will need to be a unique name (key?) for each possible variation in function behavior. I'm not sure this is practical as there is a surprising amount of choices for even simple operations. So by this argument I would lean towards add_error being add and add has an option controlling error behavior. I've got some additional thoughts on "options" I'll add as separate discussion points.

My thinking of key is that it would be "implicit". See my comment below where key might be add[0][error:OVERFLOW_SILENTLY]

Is the ENUM approach too generic at the moment? Are there other examples of things where we'd need that approach, or can we special case cast for now?

From my exercise with Arrow I found that I needed to know (I'm not 100% sure I'm using the term constant argument correctly here. In Arrow terms this would be "function options"):

• The other types
• The value of potential constant arguments
• The arity of the function

Some examples:

• strptime has a constant argument "units" which describes the units the output timestamps should have (e.g. 'us', 'ms') [in some ways strptime is a cast though]
• assume_timezone is Arrow specific (as it has a parameterized time zone) but it is the timestamp -> timestamptz function and requires knowing the destination time zone to compute the output type. This would not be a concern for a substrait system
• make_struct is an Arrow function which takes in multiple arrays and a constant argument (list of field names) and returns a single struct array. Inspection of both the input types and the constant argument is needed in this case.
• A truncate / round kernel for decimal types that truncates the precision to a target decimal precision
• A make_fixed_list function which takes in X arguments of the same type and returns a fixed_size_list needs to know the arity of the function to create the fixed size list

These are great examples to work against, thanks. Here is some categorization. I'm ignoring the fact that a couple of these aren't actually datatypes in substrait.

Is it fair that those are three subcategories? Do we have other things that don't fall into one of those?

Honestly, as I think about it more, make_struct actually falls into the category of what I was expressing as the projection equivalent of the field references "Masked Complex Expression". I'm inclined to think that we should not try to deal with that in the context of function definitions. It feels too first class.

I introduced the concept of "constant" arguments in the argument type definitions which I think fills half the need. The big questions are:

• Can output derivation get constant values during derivation determination?
• Is exposing as an argument like this make it harder for a tool to expose signatures directly to end users? Maybe we should introduce "options" as things engines define, not users and they are passed separately?

It seems like it would require additional work to hide constant values from output determination. Wouldn't the constants be available immediately?

I wouldn definitely say that at least in some implementations, the data is entirely separate from the types and it is NOT easier to expose that information :D

Making all constant data available to a type expression makes the type expression language much more complex and starts turning it into a new programming language.

As I've thought more about this, I'm inclined to a structure something like this (as opposed to what is currently sketched in spec/extension files):

add:

• args: [i32,i32]
  options:
  • error: [OVERFLOW_SILENTLY, FAIL, NULL_OUT]
    output: i32
• args: [i64,i64]
  output: i64
  • variation: [OVERFLOW_SILENTLY, FAIL, NULL_OUT]

so a plan would bind add[0][error:OVERFLOW_SILENTLY] according to the appropriate expectations.

My expectation is that the plan should always be fully bound (e.g. casts are explicitly included as necessary). Substrait would have declarations for cast functions but it would be up to the plan producer to insert where they want and use the appropriate functions as expected.

+1 for keeping specific choices about when to cast out of Substrait. Those semantics are different across systems, and I think more people will adopt the spec if the spec supports whatever casting behavior they want.

I've also come around to +1 that casts should be explicitly included as necessary. Are the two of you saying the same thing though?

Substrait would have declarations for cast functions

Those semantics are different across systems, and I think more people will adopt the spec if the spec supports whatever casting behavior they want.

I think my understanding is: Substrait will define a minimum subset of valid cast operations but it will not define when adding a cast to the plan is the right thing to do. Implementations will need to figure this out and are free to add their own internal casting options on top of Substrait's defined set of casts.

Hopefully this is in agreement with both of your points.

Rather than think of that as core to substrait specification itself, my expectation has been that the community would build a library of common plan transformations and so people could use the fallback behavior they wanted (substrait polyfills). If you don't have X, then substrait will do something with a rewrite to Y. You pick the polyfills that you're okay with.

Some polyfills would be lossy and thus a user/builder would need to decide and declare what polyfills they are applying to "roughly" consume the intention. For example, imagine a polyfill for a system that only supports fp64 but not Decimal. It could rewrite a plan to work using fp64 as opposed to Decimal. It's lossy but may be fine in certain circumstances. Another polyfill might be converting count distinct to approximate count distinct for systems that have a restricted capacity for count distinct. I can also imagine a polyfill for window frame operations when a system doesn't support it directly (I think).

It comes down to extensibility. If we want systems to be able to work with functions that they didn't know at compile time, we need a way to communicate the behavior of those functions. It could be in a single system or communication between two systems. For sure an example is an engine has a udf and wants to inform a producer of that udf. In that case, the producer wants to expose the function to an end user but validate the plan before submitting it. To do so, they need to know the function input and derivation rules.

Thanks for sharing. I'd say the main observation I have about your initial implementation is that it seems like your scheme is quite verbose. For example, you have: <T:string>(T)=>T. Why not just make that string => string?

Ease of parsing but I see the point

CC @westonpace , @cpcloud @rdblue

Gentle ping @jacques-n

Thanks, forgot about this one.

When we had the call a week or so ago, my interpretation of the discussion was that we wouldn't have anything in the signatures themselves to support this pattern. Our conversation, though, was mostly focused on the type derivation stuff so I can see how this got lost.

I'm somewhat torn on the situation. Options I see:

• Option A:, stay as currently stated and people can use constant string arguments if they'd like. This at least makes things clearer than a i8. At the same time, it means that substrait really can't do much in the way of validation (and neither can a frontend system that gets a list of function signatures).
• Option B: Propose actually support enum arguments for and then FE systems can expose/validate/etc (as well as substrait intermediaries). This would not be a datatype in substrait, but a third type of argument. Basically add another option to the current set of value and type arguments here.

Note, in the latest yaml function list, I proposed something which is slightly different: behavioral options which could be specify as part of the function selection process but are generally envisioned as internal concerns of the system as opposed to external user behavior properties.

As I look at the things I did in the latest yaml, I wonder if we'd be better off just formalizing as option B. It really feels like this is a real thing and treating it as a special kind of "enum constant" or similar would be best.

There was some discussion on this discussion topic on Slack (https://substrait.slack.com/archives/C02D7CTQXHD/p1633566594104600). In summary I was a little behind on what "options" were in the current functions yaml.

Neither of us was in favor of "Option A" in the above post (using string arguments). One place where this is used that could be improved was:

- name: extract
  description: Extract portion of a date/time value.
  parameters:
    - D: [timestamp, timestamp_tz, date, time]
  arguments:
    - type: D
      name: Date/time value to extract information from.
    - type: STRING
      name: The part of the value to extract.
      constant: TRUE
  return: i64

Having type: STRING for the second argument is undesirable because it makes validation difficult. For that reason it seems that enum arguments do make sense.

Once we have enum arguments then the current "options" feature becomes a little redundant. For example:

  - name: '+'
  description: "Add two numeric values."
  options:
    overflow: [SILENT, SATURATE, ERROR]
  variants:
    - variant: scalar
      parameters:
        - K: [i8,i16,i32,i64,fp32,fp64]
      arguments:
        - type: K
        - type: K
      return: K

The options: overflow could easily be a third argument to add which has a type of "overflow enum".

This avoids the need for extension types.

I've posted #49 to try to add this functionality. We should also add the explicit resolution example you gave here.

From the discussion on slack:

• The argument in the yaml would have an enum type
• The argument would have a possible value of UNSPECIFIED (this may also be the default value)
• If the producer chooses "UNSPECIFIED" then the consumer should consider the enum definition to be an ordered list of choices and should choose the first choice that the consumer supports.

For example, given an add function with an argument overflow which is an enum option with the choices UNSPECIFIED, SILENT, SATURATE, ERROR and given a consumer that only supports SATURATE and ERROR:

• If the producer picks SILENT then the consumer should return an error
• If the producer picks ERROR then the consumer should use ERROR
• If the producer picks UNSPECIFIED then the consumer should use SATURATE

If an enum option does not have UNSPECIFIED as a choice then it is always an error if the consumer does not support it (for example, the enum used for time extraction given in the other discussion).