What are "operations" in the vision statement #7
Replies: 2 comments 7 replies
-
|
Thanks for bringing this up! My intention was definitely option 2+. The plus is because function is pretty generic and some kinds of things (say batch level columnar transformations) are included within what I'd like to cover but I don't think most people will think of them when they initially say "function". A good example is when working with geo, a contains operation may actually take a set, build an S2 geometry version of the data and do contains operations using that. It's kind of a function except that it has some kind of intermediate set-level state. I think that's a type of operation that should also be covered. Do you have some proposed language to help clarify things? |
Beta Was this translation helpful? Give feedback.
-
No, and it probably doesn't have to be solved in the vision statement itself as long as it is expanded somewhere else on the site. Perhaps it could be covered by another use case. Something like one of...
|
Beta Was this translation helpful? Give feedback.
-
Is this referring to a common semantic interpretation of specific functions, or one of the structure of functions, operators and transformations themselves? |
Beta Was this translation helpful? Give feedback.
-
|
I think we should primarily be focused on the first but I believe that naturally leads to us at least defining a common vocabulary for the second (although not necessarily a formal specification). For example we may define a scalar function as |
Beta Was this translation helpful? Give feedback.
-
|
I was thinking about facts like...
As for general concepts like "scalar function" I'm not sure. It isn't what I was thinking. It could help to define the logical concept of a "scalar function" to group semantic properties (for example, rather than writing "if f([x_n]) => f([y_n]) then f([x_1, x_2]) => f([y_1, y_2])" for add, multiply, divide, etc. we could define the logical concept "scalar function" and then say add, multiply, divide, etc. are "scalar functions". Then if a consumer never had anything called "scalar function" in its implementation that would be perfectly fine (and "scalar function" need not be a discoverable property in a function registry, if such a thing exists). |
Beta Was this translation helpful? Give feedback.
-
I believe you're saying these would be things that substrait would specify. If so, I agree. Not that a user couldn't specify alternatives but there would be clear semantics for the things defined in substrait. That being said, there may be alternatives that are closely related. It is very likely, for example that you might have two definitions of add(i64,i64), one which overflows and one which will error. A producer would decide which behavior they want and then pick the specific one that has that behavior. As of yet, I haven't sketched a formal relationship to achieve this in the spec. We should consider whether it makes sense to build this concept in as first class in the context of formalizing the functions section once we get there.
While I don't really care about the specific name, there are classes of functions that allowed or disallowed in specific contexts. For example, I would expect that we would define rules that state one can use a scalar functions in a join condition but cannot use aggregate functions in a join condition. The class or category of functions are important for the contexts where their semantics make sense. |
Beta Was this translation helpful? Give feedback.
-
|
Forgot to reply but yes, I agree with what you have said here. I did not mean to imply there is going to be a single simple definition. There may be many variations. Also, I see what you mean by "scalar function" now. That makes sense to me. |
Beta Was this translation helpful? Give feedback.
-
The vision statement currently reads...
What is an "operation"? For example:
Option 1: operation => relational algebra operator
In this definition an operation would be something like "select", "project", "union", "order by", "udf", etc. Given such a definition, you could even envision a specification that doesn't have any type system at all. A type is just a tuple of (unique identifier, is_sortable, is_hashable). Cast would simply be another udf. The unique identifier could include physical representation, if needed by an implementation. This would be a smaller and simpler vision.
Option 2: operation => relational algebra operator OR function
In this definition operations would include things like "add", "upper", "ucase", "lcase", "round". In such a definition you would need a type system because you would need to be able to formally define the semantics of an operation. What behavior happens if integer addition overflows? What expectations are there are the precision of floating point operations? Does F & null => null or F? Can an interval be multiplied by an integer of floating point number? Option 2 has a much broader vision.
Options 3+: ???
My interpretation is that this spec is aimed at option 2 but I wanted to clarify because it also might be somewhere in between options 1 and 2 and I think a very clear idea of the scope is going to be important.
Beta Was this translation helpful? Give feedback.
All reactions