Scribe Android handwriting text input #148784

justinmc · 2024-05-21T18:17:04Z

Enables the Scribe feature, or Android stylus handwriting text input.

This PR only implements basic handwriting input. Other features will be done in subsequent PRs:

I created and fixed issue about stylus hovering while working on this: #148810

Original PR for iOS Scribble, the iOS version of this feature: #75472
FYI @fbcouch

~~Depends on flutter/engine#52943~~ (merged).

Example code I'm using to test this feature (but any TextField works)

import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';
import 'package:flutter/services.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final FocusNode _focusNode1 = FocusNode();
  final FocusNode _focusNode2 = FocusNode();
  final FocusNode _focusNode3 = FocusNode();
  final TextEditingController _controller3 = TextEditingController(
    text: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.',
  );

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Scribe demo'),
      ),
      body: Center(
        child: Padding(
          padding: const EdgeInsets.symmetric(horizontal: 74.0),
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: <Widget>[
              TextField(
                focusNode: _focusNode1,
                autofocus: false,
              ),
              TextField(
                focusNode: _focusNode2,
              ),
              TextField(
                focusNode: _focusNode3,
                minLines: 4,
                maxLines: 4,
                controller: _controller3,
              ),
              TextButton(
                onPressed: () {
                  _focusNode1.unfocus();
                  _focusNode2.unfocus();
                  _focusNode3.unfocus();
                },
                child: const Text('Unfocus'),
              ),
              TextButton(
                onPressed: () {
                  _focusNode1.requestFocus();
                  SchedulerBinding.instance.addPostFrameCallback((Duration _) {
                    SystemChannels.textInput.invokeMethod('TextInput.hide');
                  });
                },
                child: const Text('Focus 1'),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

vistaar-mlopes · 2024-05-29T09:00:58Z

hi. @justinmc I have Samsung galaxy s9 ultra tab, I wanted to implement the above feature. I went through above PR and till it gets approved, can you please suggest a temporary solution that I can implement using native android?
I found your comment on using 2 native methods :
"Android's InputMethodManager class already has support for stylus handwriting. We'll need to tie these into the framework similarly to how it's done for iOS.

Some relevant methods:

1.InputMethodManager.startStylusHandwriting
2.View.setAutoHandwritingEnabled"
can I implement method channnel and access this native features?

Thanks.

…l text though)

justinmc · 2024-05-31T20:30:05Z

@vistaar-mlopes I'm not sure whether you could get this working by writing a plugin. The bare minimum thing you'd need to do is to call startStylusHandwriting on the InputMethodManager. If you couldn't do that with a plugin, you would need to build a custom engine and framework.

By the way, if you end up trying this feature out using your Samsung device, could you let me know if it works? I'm assuming/hoping that other Android devices will work the same as what I'm using (Pixel Tablet).

Currently I have a hacked version of OverflowBox that does the same thing. Need to modify it to allow the margin to overflow.

I can draw outside of the bounds nicely, but receiving gestures is another story. The parent will stop hit testing when it sees that an event is outside of its boundaries, so I can never receive it. Instead, maybe I should first look into the hover icon. How will I do that?

justinmc · 2024-06-06T21:29:28Z

packages/flutter/lib/src/widgets/editable_text.dart

+
+  @override
+  Widget build(BuildContext context) {
+    return Listener(


Note to self: I can listen directly to the global pointer binding to avoid the problem with clipping out events that I want.

vistaar-mlopes · 2024-07-23T10:28:06Z

@justinmc Recently I got a Lenovo tab. I tried to scribble on the text field and it worked. I typed hello and it was there in the text field. whereas I tried the same on the Samsung tab with S-pen it didn't work. Now I'm not sure where the issue is. Is it a flutter issue or is it device dependent?

justinmc · 2024-09-24T23:42:26Z

@vistaar-mlopes Interesting! You tried this on master, not on this branch, right? I'm not sure how those other devices do handwriting. Maybe they have their own implementations since I believe Android hasn't supported this officially until recently.

* Removed ScribeClient! Seems unnecessary for now, though I'll need it for gestures. * Used the variable for default value of stylusHandwritingEnabled, and fixed one case where it was incorrect!

justinmc · 2024-10-23T17:55:25Z

@vistaar-mlopes If you could try this branch out with your Samsung and Lenovo tablets I'd be interested to see if it solves your problem! Now no engine change is needed, just this framework branch.

LongCatIsLooong

(looked at widgets and services)

LongCatIsLooong · 2024-10-23T23:01:53Z

packages/flutter/lib/src/services/scribe.dart

+///    built-in text fields support handwriting input.
+///  * [SystemChannels.scribe], which is the [MethodChannel] used by this
+///    class, and which has a list of the methods that this class handles.
+class Scribe {


final class since everything seems to be static.

LongCatIsLooong · 2024-10-23T23:02:09Z

packages/flutter/lib/src/services/scribe.dart

+///  * [SystemChannels.scribe], which is the [MethodChannel] used by this
+///    class, and which has a list of the methods that this class handles.
+class Scribe {
+  Scribe._() {


Who calls this private constructor?

Also how's linter not complaining if this is not used.

Ah this is leftover from when I was handling gesture calls, before I moved it to a separate branch. I'll just remove all this. Not sure about the linter! Maybe the linter can't handle constructors?

LongCatIsLooong · 2024-10-23T23:10:02Z

packages/flutter/lib/src/services/scribe.dart

+  /// A convenience method to check if the device currently supports Scribe
+  /// stylus handwriting input.
+  ///
+  /// Call this before calling [startStylusHandwriting] to make sure it's


How often does the caller have to check if the feature is available? Can I assume the return value won't change while the app is running?

It must be called each time before calling startStylusHandwriting. I'll update the docs.

LongCatIsLooong · 2024-10-23T23:11:23Z

packages/flutter/lib/src/services/scribe.dart

+  }
+
+  /// Returns true if the InputMethodManager supports Scribe stylus handwriting
+  /// input.


Do false and null mean the same thing?

The nullability is a limitation of invokeMethod (docs don't explain it super well either but maybe it's just that the codec can't enforce non-nullability?). I will explain in the docs here.

LongCatIsLooong · 2024-10-23T23:22:29Z

packages/flutter/lib/src/services/scribe.dart

+
+  /// Tell Android to begin receiving stylus handwriting input.
+  ///
+  /// This is typically called after detecting a [PointerDownEvent] indicating


From the Android documentation this can't be called when there is no active input connection it seems (maybe I'm reading the doc wrong, it says "This will always be preceded by onStartInput(android.view.inputmethod.EditorInfo, boolean) for the EditorInfo and InputConnection for which stylus handwriting is being requested")?

Good call, I tried calling it with no active TextInputConnection and it was ignored. I'll update the docs to explain this.

LongCatIsLooong · 2024-10-23T23:31:19Z

packages/flutter/lib/src/widgets/editable_text.dart

+
+  /// Returns a new Rect whose size has changed by the given padding while
+  /// remaining centered.
+  static Rect _pad(Rect rect, EdgeInsets padding) {


https://main-api.flutter.dev/flutter/painting/EdgeInsets/deflateRect.html ?

LongCatIsLooong · 2024-10-23T23:34:13Z

packages/flutter/lib/src/widgets/editable_text.dart

+  /// [Rect] in global coordinates.
+  static Rect _localToGlobalRect(Rect rect, RenderBox renderBox) {
+    return Rect.fromPoints(
+      renderBox.localToGlobal(rect.topLeft),


https://main-api.flutter.dev/flutter/painting/MatrixUtils/transformRect.html is probably more idiomatic.

LongCatIsLooong · 2024-10-23T23:43:43Z

packages/flutter/lib/src/widgets/editable_text.dart

+  void initState() {
+    super.initState();
+    if (widget.enabled) {
+      GestureBinding.instance.pointerRouter.addGlobalRoute(_handlePointerEvent);


What prevents 2 overlapping text fields from competing for stylus point down events? Also, the stylus hitting an invisible text field (in an inactive route, for example) would also trigger scribble it seems?

Also why can't this be a regular gesture recognizer / detector? Is it because of the padding?

I will follow up on this one on Monday.

What prevents 2 overlapping text fields from competing for stylus point down events?

Good catch, I just tried this out and it looks like the behavior to the user is that the field lowest in the tree will win if I write directly in between the two fields, where their paddings are overlapping. If I get really accurate with the stylus I can write just in the bottom of the upper field but the lower field will receive the text, but for the most part the behavior is probably fine in this ambiguous situation.

However, both fields do receive the event. I'll need to follow up and do something to prevent that.

3 TextFields right next to each other

child: Column( mainAxisAlignment: MainAxisAlignment.center, children: <Widget>[ TextField( controller: controller, stylusHandwritingEnabled: true, decoration: InputDecoration( border: OutlineInputBorder(), ), ), TextField( stylusHandwritingEnabled: true, decoration: InputDecoration( border: OutlineInputBorder(), ), ), TextField( stylusHandwritingEnabled: true, decoration: InputDecoration( border: OutlineInputBorder(), ), ), ], ),

Also, the stylus hitting an invisible text field (in an inactive route, for example) would also trigger scribble it seems?

Should I check ModalRoute.of(context) or something like that? Anything else I should check?

Also why can't this be a regular gesture recognizer / detector? Is it because of the padding?

Yes, this was my solution to needing to listen to an area outside of the size of the field.

LongCatIsLooong · 2024-10-23T23:50:11Z

packages/flutter/lib/src/widgets/editable_text.dart

+    return _ScribbleFocusable(
+      focusNode: focusNode,
+      editableKey: editableKey,
+      enabled: enabled,


enabled is always true here right?

Yes, and it is more readable if I pass true here 👍

LongCatIsLooong · 2024-10-24T00:05:45Z

packages/flutter/lib/src/widgets/editable_text.dart

+    }
+
+    // A stylus event that starts on a selection handle does not start
+    // handwriting, it moves the handle.


I wish this logic lives in the TextSelectionHandle (or whatever it is called) since we have different handle classes for android and iOS (I think)?

…ack later

LongCatIsLooong

Regarding hit-testing, how important is it to support _handwritingPadding for how?

Can this be implemented using gesture recognizers, by either:

move the logic to TextField like other gesture recognizers
ignore the padding for now, and implement Opt a subtree out of hitTestSelf culling #75747 to allow the RenderBox to accept out-of-bounds events for the subtree

So we don't have to replicate the hit testing logic and z-index hittest behavior logic?

LongCatIsLooong · 2024-10-28T21:10:35Z

packages/flutter/lib/src/services/scribe.dart

+///    built-in text fields support handwriting input.
+///  * [SystemChannels.scribe], which is the [MethodChannel] used by this
+///    class, and which has a list of the methods that this class handles.
+final class Scribe {


nit: abstract final class to indicate the class is just a namespace?

LongCatIsLooong · 2024-10-28T21:13:19Z

packages/flutter/lib/src/services/scribe.dart

+  ///
+  /// The return type is nullable due to the way
+  /// [MethodChannel.invokeMethod](https://main-api.flutter.dev/flutter/services/MethodChannel/invokeMethod.html)
+  /// works, but a successful call will never resolve to `null`.


Would it be possible to do .then<bool>((bool? value) => value ?? false)

or .then<bool>((bool? value) => value ?? throw SomeError) then?

Good call, I should do this checking and not the user.

LongCatIsLooong · 2024-10-28T21:20:20Z

packages/flutter/lib/src/services/scribe.dart

+  /// Returns true if the InputMethodManager supports Scribe stylus handwriting
+  /// input, false otherwise.
+  ///
+  /// Call this each time before calling [startStylusHandwriting] to make sure


If this is the case shouldn't this API be incorporated into startStylusHandwriting then? Since they are always paired?

It sounds like it should, but that's not the way that Android designed their API for whatever reason. I want to provide an accurate reproduction of the Android API in case our users need it.

LongCatIsLooong · 2024-10-28T21:25:13Z

packages/flutter/lib/src/services/scribe.dart

+  ///   which is the corresponding API on Android.
+  ///  * [EditableText.stylusHandwritingEnabled], which controls whether
+  ///    Flutter's built-in text fields support handwriting input.
+  static Future<void> startStylusHandwriting() {


so the IME is responsible for determining when to stop?

Yes. It seems like it's a timeout from the last screen touch.

LongCatIsLooong · 2024-10-28T21:32:10Z

packages/flutter/lib/src/services/system_channels.dart

+  ///
+  /// The following methods are defined for this channel:
+  ///
+  ///  * `Scribe.startStylusHandwriting`: Indicates that stylus input has been


nit: [] instead of ``?

Not sure why I didn't do that in the first place...

LongCatIsLooong · 2024-10-28T21:33:30Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -1975,6 +2001,9 @@ class EditableText extends StatefulWidget {
  /// {@macro flutter.widgets.magnifier.intro}
  final TextMagnifierConfiguration magnifierConfiguration;

+  /// The default value for [stylusHandwritingEnabled].
+  static const bool defaultStylusHandwritingEnabled = true;


Why is the default exposed?

In order to reference it from other files that also need the same default, instead of duplicating it.

LongCatIsLooong · 2024-10-28T21:45:32Z

packages/flutter/lib/src/widgets/editable_text.dart

+
+class _ScribeState extends State<_Scribe> {
+  // The handwriting bounds padding of EditText in Android API 34.
+  static const EdgeInsets _handwritingPadding = EdgeInsets.symmetric(


Is this a fixed size or just the size of the input decorator?

This is the fixed amount of padding that native Android uses to detect Scribe events outside of the field.

But, I also am seeing the magnifier appear for swipes that go from down to up. That seems to be a bug on master that shouldn't happen, and fixing it might improve the feel of scribe.

Proof of concept for scribe handwriting input

afe5fc7

justinmc self-assigned this May 21, 2024

github-actions bot added a: text input Entering text in a text field or keyboard related problems framework flutter/packages/flutter repository. See also f: labels. labels May 21, 2024

justinmc mentioned this pull request May 21, 2024

Scribe (Android stylus handwriting text input) flutter/engine#52943

Merged

justinmc added 5 commits May 21, 2024 11:47

Support writing on a field that is not yet focused.

76b957c

Properly return future, maybe consider error handling

b70b25a

Note about padding

60f09e6

TODO about the keyboard

59cb9f1

Merge branch 'master' into scribe

54adc2b

justinmc added 2 commits May 29, 2024 14:52

Only count stylus events

35ddd87

Receive selection gesture from engine (still need to select the actua…

187c494

…l text though)

justinmc mentioned this pull request May 30, 2024

Native ios context menu #143002

Merged

Select works, but not accurate due to selectPositionAt

458e26c

justinmc added 2 commits June 3, 2024 15:19

stylusHandwritingEnabled param

a36e165

stylusHandwritingEnabled in other classes

d97cb44

github-actions bot added f: material design flutter/packages/flutter/material repository. f: cupertino flutter/packages/flutter/cupertino repository labels Jun 3, 2024

justinmc added 3 commits June 4, 2024 17:25

WIP handwriting padding

bd38fe5

Currently I have a hacked version of OverflowBox that does the same thing. Need to modify it to allow the margin to overflow.

TODO for handwriting cursor

0937d8f

justinmc force-pushed the scribe branch from 1377452 to 0937d8f Compare June 5, 2024 22:59

justinmc commented Jun 6, 2024

View reviewed changes

justinmc mentioned this pull request Jun 17, 2024

Support iOS 18 formatting menu #150068

Open

Merge branch 'master' into scribe

9c04551

justinmc mentioned this pull request Sep 30, 2024

[Scribe] Hover icon #155948

Open

justinmc added 9 commits October 22, 2024 12:54

Hit test the field before the handles for efficiency

7a0933b

scribbleEnabled default value and deprecation version

283bc22

Merge branch 'master' into scribe

c88ac89

Analyzer

44c53d0

Merge branch 'master' into scribe

80fab0c

Docs link fix

26eb96b

Whoops, I wanted the default for the new parameter

86d91f7

Clean up

8493a64

* Removed ScribeClient! Seems unnecessary for now, though I'll need it for gestures. * Used the variable for default value of stylusHandwritingEnabled, and fixed one case where it was incorrect!

Docs cleanup

78f79d0

justinmc requested a review from LongCatIsLooong October 23, 2024 17:56

justinmc marked this pull request as ready for review October 23, 2024 17:56

justinmc changed the title ~~(WIP) Scribe Android handwriting text input~~ Scribe Android handwriting text input Oct 23, 2024

LongCatIsLooong reviewed Oct 24, 2024

View reviewed changes

justinmc added 5 commits October 25, 2024 11:18

Scribe class should be final

a4d68cc

Not necessary to listen to the method channel in this PR, will come b…

6a03f17

…ack later

Docs improvements

1b70d2d

inflateRect

969f982

MatrixUtils.transformRect

62c9a3f

LongCatIsLooong reviewed Oct 28, 2024

View reviewed changes

justinmc added 3 commits October 29, 2024 11:38

Can pass true directly

eb6ebcb

Cleanup unused method

a6acb3f

Nonnullable method return values

138c855

justinmc force-pushed the scribe branch from e01d2a3 to 138c855 Compare October 29, 2024 23:26

justinmc added 5 commits October 30, 2024 10:40

Link methods in docs

718075b

Fix docs analyzer

00172f1

Circular dependency

8c716b3

WIP Investigating using a real GestureDetector

86a5d39

WIP trying out a custom gesture recognizer.

ea3da65

But, I also am seeing the magnifier appear for swipes that go from down to up. That seems to be a bug on master that shouldn't happen, and fixing it might improve the feel of scribe.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scribe Android handwriting text input #148784

Scribe Android handwriting text input #148784

Scribe Android handwriting text input #148784

Are you sure you want to change the base?

Scribe Android handwriting text input #148784

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment