[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-spec compliant behavior of dynamic dispatch on Object methods with unusual arguments #3895

Closed
stereotype441 opened this issue Jun 11, 2024 · 19 comments
Labels
breaking-change inference specification technical-debt Dealing with a part of the language which needs clarification or adjustments type-inference Type inference, issues or improvements

Comments

@stereotype441
Copy link
Member

While writing up a detailed specification for type inference, I noticed a difference between the behavior of the analyzer and CFE and the current text in specification/dartLangSpec.tex.

In the "Method Invocation -> Ordinary Invocation" section of the spec, in the case explaining how to interpret a method invocation i of the form e.m<...>(...) (where <...> is understood to be optional), there is this text:

Let T be the static type of e. It is a compile-time error if T does not have an accessible (6.2) instance member named m, unless either:

  • T is dynamic bounded (17.15.3); in this case no further static checks are performed on i (apart from separate static checks on subterms like arguments) and the static type of i is dynamic.

This seems to pretty clearly imply that the following method invocations should all be accepted, and have static type dynamic (assuming that the type of d is dynamic bounded):

  • d.toString()
  • d.toString(0)
  • d.toString(x: 0)

(Note that all of these invocations could be semantically meaningful, since it is legal for an override of toString to accept optional arguments.)

It also seems to imply that the following method invocations should all be accepted at compile time and have static type dynamic (again assuming that the type of d is dynamic bounded), even though they are guaranteed to fail at runtime regardless of the value bound to d (since a non-generic function can't be overridden by a generic function):

  • d.toString<int>()
  • d.noSuchMethod<int>(d)
  • d.toString<int>(0)
  • d.toString<int>(x: 0)
  • d.noSuchMethod<int>()
  • d.noSuchMethod<int>(d, d)
  • d.noSuchMethod<int>(d, x: d)

But the actual behavior of the analyzer and the CFE is that:

  • d.toString() has static type String,
  • d.toString(0) and d.toString(x: 0) have static type dynamic,
  • d.toString<int>() and d.noSuchMethod<int>(d) are rejected with a compile-time error, and
  • The remaining forms (d.toString<int>(0), d.toString<int>(x: 0), d.noSuchMethod<int>(), d.noSuchMethod<int>(d, d), and d.noSuchMethod(d, x: d)) have no compile-time error, and have static type dynamic.

Although we have specified a few rare circumstances in which the static type of an invocation depends on the type of invocation arguments (see number-operation-typing.md), this is the only situation I know of where the static type of an invocation depends on the presence or absence of invocation arguments. And it's the only situation I know of where the presence or absence of invocation arguments affects whether type arguments are accepted.

This all feels like an unnecessary source of complexity in the type inference logic. My preference would be to change the implementations to comply with the existing spec here, and accept all of these forms without complaint at compile time (and make them all have static type dynamic).

However, I could see two possible counter-arguments to that:

  1. It could arguably create a worse user experience if d.toString() has static type dynamic rather than String, since it might lead to more dynamism and less static error checking in code that follows.
  2. It's conceivable that making this change might lead to a performance regression (especially if it's done in a naive way), by causing dynamic dispatch logic or dynamic downcasts to be invoked in circumstances where they aren't necessary.

It's possible that I could be persuaded to accept, as an alternative, that:

  • d.toString(...) always has static type String
  • d.toString<...>(...) is always a compile-time error
  • d.noSuchMethod<...>(...) is always a compile-time error

Note that either of these possibilities would technically be a breaking change, so they would have to be done using the breaking change process or using language versioning (I would prefer to use the breaking change process, since I believe code that is affected is extremely rare).

@dart-lang/language-team what do you think?

@stereotype441 stereotype441 added specification breaking-change inference technical-debt Dealing with a part of the language which needs clarification or adjustments type-inference Type inference, issues or improvements labels Jun 11, 2024
@lrhn
Copy link
Member
lrhn commented Jun 11, 2024

I believe we decided at some point, possibly around Dart 2, that dynamic invocations of "Object members" with structurally valid argument lists were considered as having the return type of the member in the Object interface, and maybe also a tear-of could have the function type, but I'm not sure that was considered.

You could still call with seemingly invalid arguments, because they might be supported, or they might hit noSuchMethod, in which case we don't know the returned value's type.

That would prevent some possible operations, fx dyn.runtimeType("boo") could be valid if the returned Type object has a call method, but it was considered worth it to break such crazy code.

There should be issues discussing this.

@stereotype441
Copy link
Member Author

Thanks, @lrhn. After a bunch of digging I found dart-lang/sdk#32414, which I assume is the issue you're referring to.

@lrhn
Copy link
Member
lrhn commented Jun 11, 2024

That does sound like the discussion I remember. Well found!

@leafpetersen
Copy link
Member

I'm quite opposed to changing this back to the "just use dynamic always" behavior - I think doing so would be a real regression in user facing behavior. I'm ok keeping the "just use dynamic if the argument list is weird" behavior, though I continue to find it pointless. :).

This section is also relevant to this discussion.

@eernstg
Copy link
Member
eernstg commented Jun 12, 2024

We do have rules that correspond to the implemented behavior, in the section 'Type dynamic' in the language specification.

So we could just keep the currently implemented behavior and section 'Type dynamic' in the specification unchanged, and all is good...

... except that the language specification needs to be modified slightly: In section 17.15.3 'Binding Actuals to Formals' at the location where member accesses on a receiver whose type is dynamic bounded is specified, we should add a reference to the section 'Type dynamic' saying that the general rule doesn't apply when the function invocation is an ordinary member invocation whose name is one of the Object member names.

@eernstg
Copy link
Member
eernstg commented Jun 12, 2024

@stereotype441, I was just about to write a PR to add those missing references in the language specification. If you want to do it then just give me a heads up.

@stereotype441
Copy link
Member Author

@eernstg:

@stereotype441, I was just about to write a PR to add those missing references in the language specification. If you want to do it then just give me a heads up.

No, go right ahead. I'll be happy to review it.

@eernstg
Copy link
Member
eernstg commented Jun 12, 2024

Very good! I'll do it tomorrow, CEST.

@lrhn
Copy link
Member
lrhn commented Jun 12, 2024

I'm ok keeping the "just use dynamic if the argument list is weird" behavior, though I continue to find it pointless. :).

It's not there because it's awesome, but because it's sound.

The way dynamic invocations work, in the presence of noSuchMerhod forwarders, is that any invocation that we know hits a forwarder can be trusted to satisfy the declared return type. Either the invocation throws, or it returns a value that is type checked before being returned by the forwarder (and if it fails, write back to throwing).

That means we can soundly use the declared return type from Object as the static type.

If the argument list shape matches the parameter list from the Object interface, then we know it will try to invoke a method with a function type that is a subtype of that interface signature, whether a forwarder or not. If the argument types don't match, the call will throw.

If the argument list shape doesn't match, then the call may invoke noSuchMethod directly, and can return any value. We would have to insert a runtime type check on the returned value to keep that sound. We could do that, but so far we've just assumed a dynamic return.

@leafpetersen
Copy link
Member

@lrhn I think you've reverse the interpretation of what I said in the quoted bit? You just gave the justification for why we use the declared Object method return types (because it is sound to do so). I agree that it is sound to do so, and it is useful. What the quoted bit is saying is that I don't understand why we choose to statically accept (3 as dynamic).toString("Fizzlepoof").arglebargle. It is true that it is sound to accept it, but is it really useful? To me it seems pointless.

@lrhn
Copy link
Member
lrhn commented Jun 12, 2024

The alternatives would be:

  • not accepting it, which means leaving users with no workaround for doing an untyped call. A dynamic call is the workaround that we provide. (Say having an instance of a class C<T> with a toString([T? optarg]) for some T, and a value of that type T, but without knowing which T it is. You can still do (o as dynamic).toString(value) today, but this option would disable the workaround. But you could also upcast to C<Object?> and rely on unsafe covariance, or tear off the method and cast that to dynamic.)

  • allow the call with return type String, and add a runtime type check on the returned value

  • allow the call with return type dynamic, as we do today.

Each has trade offs. I don't think any of the choices are inherently superior, or more pointless than the other.

@leafpetersen
Copy link
Member
leafpetersen commented Jun 12, 2024

not accepting it,

Bingo! :). Yes, I understand that if we choose not to accept calls to Object methods with weird argument lists then people can't override the Object methods with weird but type compatible signatures. My hypothesis is that the number of classes in real actual code in the wild which override the Object methods with weird but type compatible signatures is not meaningfully different from 0. Hence "pointless". Am I wrong?

@natebosch
Copy link
Member

or they might hit noSuchMethod, in which case we don't know the returned value's type.

Do we allow this because a runtime type check is too expensive?

@tatumizer
Copy link

Showing an error for
(3 as dynamic).toString("Fizzlepoof").arglebargle
will be inconsistent with NOT showing an error for
("x" as dynamic).toInt().arglebargle (where toInt - some user-defined method not inherited from Object)
From the user's viewpoint, toInt might be self-explanatory (it must return int!), but this is not obvious to the compiler.
It might be better to keep everything uniform.

@stereotype441
Copy link
Member Author

not accepting it,

Bingo! :). Yes, I understand that if we choose not to accept calls to Object methods with weird argument lists then people can't override the Object methods with weird but type compatible signatures. My hypothesis is that the number of classes in real actual code in the wild which override the Object methods with weird but type compatible signatures is not meaningfully different from 0. Hence "pointless". Am I wrong?

For what it's worth, I wrote this code in the SDK long before we started this discussion:

  /// Returns a string representation of this type.
  ///
  /// If [parenthesizeIfComplex] is `true`, then the result will be surrounded
  /// by parenthesis if it takes any of the following forms:
  /// - A type with a trailing `?` or `*`
  /// - A function type (e.g. `void Function()`)
  /// - A promoted type variable type (e.g. `T&int`)
  @override
  String toString({bool parenthesizeIfComplex = false}) =>
      switch (nullabilitySuffix) {
        NullabilitySuffix.question => _parenthesizeIf(
            parenthesizeIfComplex,
            '${_toStringWithoutSuffix(parenthesizeIfComplex: true)}'
            '?'),
        NullabilitySuffix.star => _parenthesizeIf(
            parenthesizeIfComplex,
            '${_toStringWithoutSuffix(parenthesizeIfComplex: true)}'
            '*'),
        NullabilitySuffix.none =>
          _toStringWithoutSuffix(parenthesizeIfComplex: parenthesizeIfComplex),
      };

(context: https://github.com/dart-lang/sdk/blob/e23e032406e2ab41641a5ea16a5f7020e06ff83b/pkg/_fe_analyzer_shared/test/mini_types.dart#L432-L452)

Not sure if you count the brainchild of someone like me to be "meaningfully different from 0" though 😃

Also relevant: I don't believe I ever invoke this method dynamically.

@Wdestroier
Copy link

I found 6.8k files where people use optional named parameters in toString and 968 files where people use optional positional parameters in toString on GitHub.

toString with optional named parameters: https://github.com/search?q=toString%28%7B+language%3ADart+&type=code
toString with optional positional parameters: https://github.com/search?q=toString%28%5B+language%3ADart+&type=code

From the DX point of view, the last alternative can be great imo. Always show the error at compile time if it's known to always be an error at runtime (if the code might work, then don't show an error). And infer the known type if any (toString can only return a String effectively) or else dynamic.

@lrhn
Copy link
Member
lrhn commented Jun 13, 2024

... people use optional ... parameter in toString ...

But do they call those methods dynamically? (Statistically ... probably some of them.)

Do we allow this because a runtime type check is too expensive?

We allow it beacuse we have never disallowed it.

There is no difference (in the language semantics) between calling (o as dynamic).foo(42); on an object with no foo method and with a foo method which cannot be called with one argument. They both go to noSuchMethod.

We could change that, and say that if o has no foo member, then it goes to noSuchMethod, and if foo has a foo member, then it tries to call that. It cannot, that throws a NoSuchMethodError directly, instead of going to noSuchMethod. (And then noSuchMethod would again be appropriately named!)

You can still try to call the method with more arguments than the interface allows, but if that fails, it'll always throw.

Since any object implementing an interface with a foo member will have a foo member, even if it's just an nSM-forwarder, it means you can start trusting the return type of known interface methods, since it's either called or it throws. (If we introduce dynamic<T> then we'll have more interface methods.)

Or, if we are willing to change behavior of noSuchMethod invocations, we could make dynamic invocations throw immediately if the object has no foo member, only ever going to noSuchMethod through nSM-forwarders. Then noSuchMethod is no longer a way to implement "anything", but only to have a dynamic implementation of an existing interface.
(I'd like that!)

That is: A dynamic invocation of (o as dynamic).foo(args) would check the object o for a foo member. If it's a getter, it's read, at type dynamic, and then invoked dynamically with (args) as arguments. If it's a method, then it's attempted invoked with (args) as arguments if possible, and otherwise an error is thrown. Otherwise there is no instance foo member, and an error is thrown. The only platform code calling noSuchMethod is inside the nSM-forwarders.
And then we can always trust the return type of Object members, and we can use their function signatures for tear-offs, even in dynamic invocations.

@leafpetersen
Copy link
Member

But do they call those methods dynamically? (Statistically ... probably some of them.)

Yeah, I should have qualified my statement with this. It's not terribly surprising that someone would add an optional parameter to toString. It would be pretty surprising to me if there's significant deliberate reliance on dynamic calls to such a method. I strongly suspect that essentially all such dynamic calls are accidental.

In any case, I'm not proposing that we remove support for this, just opposing removing the existing support for statically checked dynamic calls to the object methods .

At a meta-level, I think that a big factor in this is the fact that even today it's very easy to unexpectedly fall into to dynamic code in Dart. If it was essentially impossible to accidentally end up with unexpected dynamic calls, I'd be more open to not special casing dynamic calls to the object methods.

@stereotype441
Copy link
Member Author

Thanks for the discussion, everyone! I now understand that:

  • the behavior I observed is spec compliant after all (and @eernstg further clarified this in 0a25c3a, thanks Erik!), and
  • the change I was most in favor of (giving a static type of String to a dynamic invocation of e.g. toString(x: 0)) is actually unsound, since such an invocation might be handled by noSuchMethod, which can return any type.

Although there's some discussion here about possibly changing the behavior, it reads to me as largely speculative, and I think if anyone wants to pursue it, it would be better addressed as a separate proposal anyhow. (For example, @lrhn's comment that (o as dynamic).foo(args) could check the object o for a foo member, and then invoke that member, supplying args). So I'm going to go ahead and close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change inference specification technical-debt Dealing with a part of the language which needs clarification or adjustments type-inference Type inference, issues or improvements
Projects
None yet
Development

No branches or pull requests

7 participants