object

tokenizer(Representation,Language)

• Representation - Text representation. Valid values are atom, codes, and chars.

• Language - Object implementing the tokenizer_language_protocol protocol.

Tokenizer and sentence splitter parameterized by text representation and language provider.

Availability:

logtalk_load(tokenization(loader))

Author: Paulo Moura

Version: 1:0:0

Date: 2026-09-08

Compilation flags:

static, context_switching_calls

Implements:

public tokenizer_protocol

public sentence_splitter_protocol

Imports:

public options

Uses:

list

type

Remarks:

(none)

Inherited public predicates:

 check_option/1  check_options/1  default_option/1  default_options/1  option/2  option/3  split_sentences/2  split_sentences/3  tokenize/2  tokenize/3  tokenize_sentences/2  tokenize_sentences/3  valid_option/1  valid_options/1

Public predicates

(no local declarations; see entity ancestors if any)

Protected predicates

(no local declarations; see entity ancestors if any)

Private predicates

(no local declarations; see entity ancestors if any)

Operators

(none)

See also

tokenizer_protocol, sentence_splitter_protocol, tokenizer_language_protocol