description: An abstract base class for splitting text.

View source

An abstract base class for splitting text.

text.Splitter(
    name=None
)

A Splitter is a module that splits strings into pieces. Generally, the pieces returned by a splitter correspond to substrings of the original string, and can be encoded using either strings or integer ids (where integer ids could be created by hashing strings or by looking them up in a fixed vocabulary table that maps strings to ids).

Each Splitter subclass must implement a split method, which subdivides each string in an input Tensor into pieces. E.g.:

>>> class SimpleSplitter(tf_text.Splitter):
...   def split(self, input):
...     return tf.strings.split(input)
>>> print(SimpleSplitter().split(["hello world", "this is a test"]))
<tf.RaggedTensor [[b'hello', b'world'], [b'this', b'is', b'a', b'test']]>

View source

@abc.abstractmethod
split(
    input
)

Splits the input tensor into pieces.

Generally, the pieces returned by a splitter correspond to substrings of the original string, and can be encoded using either strings or integer ids.

>>> print(tf_text.WhitespaceTokenizer().split("small medium large"))
tf.Tensor([b'small' b'medium' b'large'], shape=(3,), dtype=string)