Release 2.6.0

Bug Fixes and Other Changes

• Update __init__.py: Added a __version__ variable
• Fixes the benchmark suite for graph mode. While using tf.function prevented caching, it was also causing the graph being tested to rebuild each time. Using placeholder instead fixes this.
• Pin nightly version.
• Remove TF patch as it is not needed anymore. The code is in core TF.
• Typos
• Format and lint NBs, add images
• Add a couple notes to the BertTokenizer docs.
• Narrative docs migration: TF Core -> TF Text
• Update nmt_with_attention
• Moved examples of a few API docs above the args sections to better match other formats.
• Fix NBs
• Update Installation from source instruction.
• Add SplitterWithOffsets as an exported symbol.
• Fix a note to the BertTokenizer docs.
• Remove unused index.md
• Convert tensorflow_text to use public TF if possible.
• Fix failing notebooks.
• Create user_ops BUILD file.
• Remove unnecessary METADATA.
• Replace tf.compat.v2.xxx with tf.xxx, since tf_text is using tf2 only.
• Fix load_data function in nmt tutorial
• Update tf.data.AUTOTUNE in Fine-tuning a BERT model
• Switch TF to OSS keras (1/N).
• added subspaces
• Disable TSAN for tutorial tests that may run for >900sec when TSAN is enabled.
• Adds a short description to the main landing page of our GitHub repo to point users to the tf.org subsite.
• Phrasing fix to TF Transformer tutorial.
• Disable RTTI when building Tf.Text kernels for mobile
• Migrate the references in third_party/toolchains directory as it is going to be deleted soon.
• Fix bug in RoundRobinTrimmer. Previously the stopping condition was merging and combining from across different batches. Instead now the stopping condition is first determined in each batch, then aggregated.
• Set mask_token='' to make it work with TF 2.6.0
• Builds TF Text with C++14 by default. This is already done by TensorFlow, and the TF Lite shim has C++14 features used within; thus, this is needed to build kernels against it.
• This is a general clean up to the build files. The previous tf_deps paradigm was confusing. By encapsulating everything into a single call lib, I'm hoping this makes it easier to understand and follow.
• Update the WORKSPACE to not use the same "workspace" name when initializing TensorFlow.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

8bitmp3, akiprasad, bongbonglemon, Jules Gagnon-Marchand, Stonepia

Release 2.6.0-rc0

Bug Fixes and Other Changes

• Update __init__.py: Added a __version__ variable
• Fixes the benchmark suite for graph mode. While using tf.function prevented caching, it was also causing the graph being tested to rebuild each time. Using placeholder instead fixes this.
• Pin nightly version.
• Remove TF patch as it is not needed anymore. The code is in core TF.
• Typos
• Format and lint NBs, add images
• Add a couple notes to the BertTokenizer docs.
• Narrative docs migration: TF Core -> TF Text
• Update nmt_with_attention
• Moved examples of a few API docs above the args sections to better match other formats.
• Fix NBs
• Update Installation from source instruction.
• Add SplitterWithOffsets as an exported symbol.
• Fix a note to the BertTokenizer docs.
• Remove unused index.md
• Convert tensorflow_text to use public TF if possible.
• Fix failing notebooks.
• Create user_ops BUILD file.
• Remove unnecessary METADATA.
• Replace tf.compat.v2.xxx with tf.xxx, since tf_text is using tf2 only.
• Fix load_data function in nmt tutorial
• Update tf.data.AUTOTUNE in Fine-tuning a BERT model
• Switch TF to OSS keras (1/N).
• added subspaces
• Disable TSAN for tutorial tests that may run for >900sec when TSAN is enabled.
• Adds a short description to the main landing page of our GitHub repo to point users to the tf.org subsite.
• Phrasing fix to TF Transformer tutorial.
• Disable RTTI when building Tf.Text kernels for mobile

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

8bitmp3, akiprasad, bongbonglemon, Jules Gagnon-Marchand, Stonepia

Release 2.5

We want to particularly point out that guides, tutorials, and API docs are currently being published to http://tensorflow.org/text ! This should make it easier for users to find our documentation. We worked hard on improving docs across the board, so feel free to let us know if further clarification is needed.

Major Features and Improvements

• API docs, guides, & tutorial are now available on http://tensorflow.org/text
• New guides & tutorials including: tokenizers, subwords tokenizer, and BERT text preprocessing guide.
• Add RoundRobinTrimmer
• Add a function to generate a BERT vocab from a tf.data.Dataset.
• Add detokenize methods for BertTokenizer and WordpieceTokenizer.
• Enable NFD and NFKD in NormalizeWithOffset op

Bug Fixes and Other Changes

• Many API updates (eg. adding descriptions & examples) to various ops.
• Let SentencePieceTokenizer optionally return the nbest tokenizations instead of sampling from them.
• Fix a bug in split mode tokenizers that caused tests to fail on Windows.
• Fix broadcasting bugs in RoundRobinTrimmer
• Add WordpieceTokenizeWithOffsets with ALLOW_STATEFUL_OP_FOR_DATASET_FUNCTIONS for tf.data
• Remove PersistentTensor from sentencepiece_kernels.cc
• Document examples are now tested.
• Fix benchmarking of graph mode ops through use of tf.function.
• Set the default for mask_token for StringLookup and IntegerLookup to None
• Update the sentence_breaking_ops docstring to indicate that it's deprecated.
• Adding an i18n-friendly BasicTokenizer that can preserve accents
• For Windows, always include ICU data files since they need to be built in statically.
• Rename documentation file WordShape.md to WordShape_cls.md. Fix #361.
• Convert input to tensor to allow for numpy inputs to state based sentence breaker.
• Add classifiers to py packages and fix header image.
• Fix for the model server test.
• Update regression test for break_sentences_with_offsets.
• Add a shape attribute to the ToDense Keras layer.
• Add support for [batch, 1] shaped inputs in StateBasedSentenceBreaker
• Fix for the model server test.
• Refactor saved_model.py to make it easier to comment out blocks of related code to identify problems.
• Add regression test for Find Source Offsets
• Fix unselectable_ids shape check in ItemSelector.
• Switch out architecture image in tf.Text documentation.
• Fix regression test for state_based_sentence_breaker_v2
• Update run_build with enable_runfiles flag.
• Update the version of bazel_skylib to match TF's and fix a possible visibility issue.
• Simplify tf-text WORKSPACE, by relying on tf_workspace().
• Update transformer.ipynb to use a saved text.BertTokenizer
• Update mobile targets to use :mobile rather than separate :android & :ios targets.
• Make tools part of the tensorflow_text pip package.
• Import tools from the tf-text package, instead of cloning the git repo.
• Minor cleanups to make some code compile on the android build system.
• Fix pip install command in readme
• Fix tools pip package inclusion.
• A tensorfow.org compatible docs generator for tf-text.
• Sample random tokens correctly during MLM.
• Treat Sentencepiece ops as stateful in tf.data pipelines.
• Replacing use of TFT's deprecated dataset_schema.from_feature_spec with its replacement schema_utils.schema_from_feature_spec.

Release 2.5.0-rc0

Major Features and Improvements

• Add a subwords tokenizer tutorial to text/examples.
• Add a function to generate a BERT vocab from a tf.data.Dataset.
• Add detokenize methods for BertTokenizer and WordpieceTokenizer.
• Let SentencePieceTokenizer optionally return the nbest tokenizations instead of sampling from them.
• Enable NFD and NFKD in NormalizeWithOffset op
• Adding an i18n-friendly BasicTokenizer that can preserve accents
• Create guide for tokenizers.

Breaking Changes

Bug Fixes and Other Changes

• Other:
  • For Windows, always include ICU data files since they need to be built in statically.
  • Patches TF to fix windows builds to not look for a python3 executable.
  • Rename documentation file WordShape.md to WordShape_cls.md. The problem is on MacOS (and maybe Windows) this filename collides with wordshape.md, because the filesystem does not differentiate cases for the files. This is purely a QOL change for anybody checking out the library on a non-Linux platform. Fix #361.
  • Convert input to tensor to allow for numpy inputs to state based sentence breaker.
  • Add classifiers to py packages and fix header image.
  • fix bad rendering for add_eos add_bos description in SentencepieceTokenizer.md
  • Fix for the model server test. Make sure our test tensors have the expected
  • Update regression test for break_sentences_with_offsets.
  • Add a shape attribute to the ToDense Keras layer.
  • Add support for [batch, 1] shaped inputs in StateBasedSentenceBreaker
  • Fix for the model server test. The result of the tokenize() method of
  • Refactor saved_model.py to make it easier to comment out blocks of related code to identify problems. Also moved out the vocab for Wordpiece due to a tf bug.
  • Update documentation for SplitMergeFromLogitsTokenizer
  • Add regression test for Find Source Offsets
  • Fix unselectable_ids shape check in ItemSelector.
  • changing two tests, to debug failure on Kokoro Windows build.
  • Switch out architecture image in tf.Text documentation.
  • Fix regression test for state_based_sentence_breaker_v2
  • Update run_build with enable_runfiles flag.
  • Update the version of bazel_skylib to match TF's and fix a possible visibility issue.
  • Simplify tf-text WORKSPACE, by relying on tf_workspace().
  • Update transformer.ipynb to use a saved text.BertTokenizer
  • typos
  • Update mobile targets to use :mobile rather than separate :android & :ios targets.
  • Make tools part of the tensorflow_text pip package.
  • Import tools from the tf-text package, instead of cloning the git repo.
  • Minor cleanups to make some code compile on the android build system.
  • Fix pip install command in readme
  • Fix tools pip package inclusion.
  • Clear outputs
  • A tensorfow.org compatible docs generator for tf-text.
  • Formatting fixes for tensorflow.org
  • Sample random tokens correctly during MLM.
  • Internal repo change
  • Treat Sentencepiece ops as stateful in tf.data pipelines.
  • Reduce the critical section range. Because the options are
  • Replacing use of TFT's deprecated dataset_schema.from_feature_spec with its replacement schema_utils.schema_from_feature_spec.
  • Updating guide with new template

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Rens, Samuel Marks, thuang513

Release 2.4.3

Bug Fixes and Other Changes

• Fix export as saved model of hub_module_splitter
• Fix bug in regex_split_with_offsets when input.ragged_rank > 1
• Convert input to tensor to allow for numpy inputs in state based sentence breaker.
• Add more classifiers to py packages.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

fsx950223

Release 2.4.2

Major Features and Improvements

• We are now building a nightly package - tensorflow-text-nightly. This is available for Linux immediately, with other platforms to be added soon.

Bug Fixes and Other Changes

• Fixes a bug which prevented the sentence_fragmenter from being able to process tensors with a rank > 1.
• Update documentation filenames to prevent collisions when checking out the code on filesystems that do not have case sensitivity.

Release 2.4.1

Major Features and Improvements

• New APIs proposed in RFC: End-to-end text preprocessing with TF.Text #283 have been added, including:

• Splitter
  • RegexSplitter
  • StateBasedSentenceBreaker
• Trimmer
  • WaterfallTrimmer
  • RoundRobinTrimmer
• ItemSelector
  • RandomItemSelector
  • FirstNItemSelector
• MaskValuesChooser
• mask_language_model()
• combine_segments()
• pad_model_inputs()

• Windows support!
• Released our first TF Hub module for Chinese segmentation! Please visit the hub module page here for more info including instructions on how to use the model.
• Added Spliter / SplitterWithOffsets abstract base classes. These are meant to replace the current Tokenizer / TokenizerWithOffsets base classes. The Tokenizer base classes will continue to work and will implement these new Splitter base classes. The reasoning behind the change is to prevent confusion when future splitting operations that also use this interface do not tokenize into words (sentences, subwords, etc).
• With this cleanup of terminology, we've also updated the documentation and internal variable names for token offsets to use "end" instead of "limit". This is purely a documentation change and doesn't affect any current APIs, but we feel it more clearly expresses that offset_end is a positional value rather than a length.
• Added new HubModuleSplitter that helps handle ragged tensor input and outputs for hub modules which implement the Splitter class.
• Added new SplitMergeFromLogitsTokenizer which is a narrowly focused tokenizer that splits text based on logits from a model. This is used with the newly released Chinese segmentation model.
• Added normalize_utf8_with_offsets and find_source_offsets ops.
• Added benchmarking for tokenizers and other ops. Allows for comparisons of dense vs ragged and TF1 vs TF2.
• Added string_to_id to SentencepieceTokenizer.
• Support Android build.
• RegexSplit op now caches regular expressions between calls.

Bug Fixes and Other Changes

• Add a minimal count_words function to wordpiece_vocabulary_learner.
• Test cleanup - use assertAllEqual(expected, actual), instead of (actual, expected), for better error messages.
• Add dep on tensorflow_hub in pip_package/setup.py
• Add filegroup BUILD target for test_data segmentation Hub module.
• Extend documentation for class HubModuleSplitter.
• Read SP model file in bytes mode in tests.
• Update intro.ipynb colab.
• Track the Sentencepiece model resource via a TrackableResource so it can be saved within Keras layers.
• Update StateBasedSentenceBreaker handling of text input tensors.
• Reduce over-broad dependencies in regex_split library.
• Fix broken builds.
• Fix comparison between signed and unsigned int in FindNextFragmentBoundary.
• Update README regarding versions.
• Fixed bug in WordpieceTokenizer so end offset is preserved when an unknown token of long size is found.
• Convert non-tensor inputs in pad along dimension op.
• Add the necessity to install coreutils to the build instructions if building on MacOS.
• Add filegroup BUILD target for test_data segmentation Hub module.
• Add long and long long overloads for RegexSplit so as to be TF agnostic c++ api.
• Add Spliter / SplitterWithOffsets abstract base classes.
• Update setup.py. TensorFlow has switched to the default package being GPU, and having users explicitly call out when wanting just CPU.
• Change variable names for token offsets: "limit" -> "end".
• Fix presubmit failed for MacOS.
• Allow dense tensor inputs for RegexSplit.
• Fix imports in tools/.
• BertTokenizer: Error out if the user passes a normalization_form that will be ignored.
• Update documentation for Sentencepiece.tokenize_with_offsets.
• Let WordpieceTokenizer read vocabulary files.
• Numerous build improvements / adjustments (mostly to support Windows):
  • Patch out googletest & glog dependencies from Sentencepiece.
  • Switch to using Bazel's internal patching.
  • ICU data is built statically for Windows.
  • Remove reliance on tf_kernel_library.
  • Patch TF to fix problematic Python executable searching.
  • Various other updates to .bazelrc, build_pip_package, and configuration to support Windows.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Pranay Joshi, Siddharths8212376, Vincent Bodin

Release 2.4.0-rc1

Major Features and Improvements

• Windows support!
• Released our first TF Hub module for Chinese segmentation! Please visit the hub module page here for more info including instructions on how to use the model.
• Added Spliter / SplitterWithOffsets abstract base classes. These are meant to replace the current Tokenizer / TokenizerWithOffsets base classes. The Tokenizer base classes will continue to work and will implement these new Splitter base classes. The reasoning behind the change is to prevent confusion when future splitting operations that also use this interface do not tokenize into words (sentences, subwords, etc).
• With this cleanup of terminology, we've also updated the documentation and internal variable names for token offsets to use "end" instead of "limit". This is purely a documentation change and doesn't affect any current APIs, but we feel it more clearly expresses that offset_end is a positional value rather than a length.
• Added new HubModuleSplitter that helps handle ragged tensor input and outputs for hub modules which implement the Splitter class.
• Added new SplitMergeFromLogitsTokenizer which is a narrowly focused tokenizer that splits text based on logits from a model. This is used with the newly released Chinese segmentation model.
• Added normalize_utf8_with_offsets and find_source_offsets ops.
• Added benchmarking for tokenizers and other ops. Allows for comparisons of dense vs ragged and TF1 vs TF2.
• Added string_to_id to SentencepieceTokenizer.
• Support Android build.
• RegexSplit op now caches regular expressions between calls.

Bug Fixes and Other Changes

• Add a minimal count_words function to wordpiece_vocabulary_learner.
• Test cleanup - use assertAllEqual(expected, actual), instead of (actual, expected), for better error messages.
• Add dep on tensorflow_hub in pip_package/setup.py
• Add filegroup BUILD target for test_data segmentation Hub module.
• Extend documentation for class HubModuleSplitter.
• Read SP model file in bytes mode in tests.
• Update intro.ipynb colab.
• Track the Sentencepiece model resource via a TrackableResource so it can be saved within Keras layers.
• Update StateBasedSentenceBreaker handling of text input tensors.
• Reduce over-broad dependencies in regex_split library.
• Fix broken builds.
• Fix comparison between signed and unsigned int in FindNextFragmentBoundary.
• Update README regarding versions.
• Fixed bug in WordpieceTokenizer so end offset is preserved when an unknown token of long size is found.
• Convert non-tensor inputs in pad along dimension op.
• Add the necessity to install coreutils to the build instructions if building on MacOS.
• Add filegroup BUILD target for test_data segmentation Hub module.
• Add long and long long overloads for RegexSplit so as to be TF agnostic c++ api.
• Add Spliter / SplitterWithOffsets abstract base classes.
• Update setup.py. TensorFlow has switched to the default package being GPU, and having users explicitly call out when wanting just CPU.
• Change variable names for token offsets: "limit" -> "end".
• Fix presubmit failed for MacOS.
• Allow dense tensor inputs for RegexSplit.
• Fix imports in tools/.
• BertTokenizer: Error out if the user passes a normalization_form that will be ignored.
• Update documentation for Sentencepiece.tokenize_with_offsets.
• Let WordpieceTokenizer read vocabulary files.
• Numerous build improvements / adjustments (mostly to support Windows):
  • Patch out googletest & glog dependencies from Sentencepiece.
  • Switch to using Bazel's internal patching.
  • ICU data is built statically for Windows.
  • Remove reliance on tf_kernel_library.
  • Patch TF to fix problematic Python executable searching.
  • Various other updates to .bazelrc, build_pip_package, and configuration to support Windows.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Pranay Joshi, Siddharths8212376, Vincent Bodin

Release 2.4.0-rc0

Major Features and Improvements

• Released our first TF Hub module for Chinese segmentation! Please visit the hub module page here for more info including instructions on how to use the model.
• Added Spliter / SplitterWithOffsets abstract base classes. These are meant to replace the current Tokenizer / TokenizerWithOffsets base classes. The Tokenizer base classes will continue to work and will implement these new Splitter base classes. The reasoning behind the change is to prevent confusion when future splitting operations that also use this interface do not tokenize into words (sentences, subwords, etc).
• With this cleanup of terminology, we've also updated the documentation and internal variable names for token offsets to use "end" instead of "limit". This is purely a documentation change and doesn't affect any current APIs, but we feel it more clearly expresses that offset_end is a positional value rather than a length.
• Added new HubModuleSplitter that helps handle ragged tensor input and outputs for hub modules which implement the Splitter class.
• Added new SplitMergeFromLogitsTokenizer which is a narrowly focused tokenizer that splits text based on logits from a model. This is used with the newly released Chinese segmentation model.
• Added normalize_utf8_with_offsets and find_source_offsets ops.
• Added benchmarking for tokenizers and other ops. Allows for comparisons of dense vs ragged and TF1 vs TF2.
• Added string_to_id to SentencepieceTokenizer.
• Support Android build.
• Support Windows build (Py3.6 & Py3.7 this release).
• RegexSplit op now caches regular expressions between calls.

Bug Fixes and Other Changes

• Test cleanup - use assertAllEqual(expected, actual), instead of (actual, expected), for better error messages.
• Add dep on tensorflow_hub in pip_package/setup.py
• Add filegroup BUILD target for test_data segmentation Hub module.
• Extend documentation for class HubModuleSplitter.
• Read SP model file in bytes mode in tests.
• Update intro.ipynb colab.
• Track the Sentencepiece model resource via a TrackableResource so it can be saved within Keras layers.
• Update StateBasedSentenceBreaker handling of text input tensors.
• Reduce over-broad dependencies in regex_split library.
• Fix broken builds.
• Fix comparison between signed and unsigned int in FindNextFragmentBoundary.
• Update README regarding versions.
• Fixed bug in WordpieceTokenizer so end offset is preserved when an unknown token of long size is found.
• Convert non-tensor inputs in pad along dimension op.
• Add the necessity to install coreutils to the build instructions if building on MacOS.
• Add filegroup BUILD target for test_data segmentation Hub module.
• Add long and long long overloads for RegexSplit so as to be TF agnostic c++ api.
• Add Spliter / SplitterWithOffsets abstract base classes.
• Update setup.py. TensorFlow has switched to the default package being GPU, and having users explicitly call out when wanting just CPU.
• Change variable names for token offsets: "limit" -> "end".
• Fix presubmit failed for MacOS.
• Allow dense tensor inputs for RegexSplit.
• Fix imports in tools/.
• BertTokenizer: Error out if the user passes a normalization_form that will be ignored.
• Update documentation for Sentencepiece.tokenize_with_offsets.
• Let WordpieceTokenizer read vocabulary files.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Pranay Joshi, Siddharths8212376, Vincent Bodin

Release 2.4.0-b0

Please note that this is a pre-release and meant to run with TF v2.3.x. We wanted to give access to some of the features we were adding to 2.4.x, but did not want to wait for the TF release.

Major Features and Improvements

• Released our first TF Hub module for Chinese segmentation! Please visit the hub module page here for more info including instructions on how to use the model.
• Added Spliter / SplitterWithOffsets abstract base classes. These are meant to replace the current Tokenizer / TokenizerWithOffsets base classes. The Tokenizer base classes will continue to work and will implement these new Splitter base classes. The reasoning behind the change is to prevent confusion when future splitting operations that also use this interface do not tokenize into words (sentences, subwords, etc).
• With this cleanup of terminology, we've also updated the documentation and internal variable names for token offsets to use "end" instead of "limit". This is purely a documentation change and doesn't affect any current APIs, but we feel it more clearly expresses that offset_end is a positional value rather than a length.
• Added new HubModuleSplitter that helps handle ragged tensor input and outputs for hub modules which implement the Splitter class.
• Added new SplitMergeFromLogitsTokenizer which is a narrowly focused tokenizer that splits text based on logits from a model. This is used with the newly released Chinese segmentation model.

Bug Fixes and Other Changes

• Test cleanup - use assertAllEqual(expected, actual), instead of (actual, expected), for better error messages.
• Add dep on tensorflow_hub in pip_package/setup.py
• Add filegroup BUILD target for test_data segmentation Hub module.
• Extend documentation for class HubModuleSplitter.
• Read SP model file in bytes mode in tests.

Thanks to our Contributors