Cheshire3 Objects: Tokenizer

Description

A Tokenizer takes a string of language and processes it to produce an ordered list of tokens.

Implementations

The following implementations are pre-configured and ready to use.
They may be used out of the box in configurations for Indexes, Workflows etc.

• SimpleTokenizer
  A basic Tokenizer that splits the given data at whitespace.
• RegexpSubTokenizer
  A Tokenizer that replaces regular expression matches in the data with a configurable character (defaults to whitespace), then splits the result at whitespace.
• RegexpSplitTokenizer
  A Tokenizer that splits the data at matches of a regular expression. Each match (e.g. punctuation) is preserved in the result as a separate token.
• RegexpFindTokenizer
  A Tokenizer that uses a regular expression to finds tokens. Matches a large range commonly occurring tokens (e.g. simple words, email@addresses, £monetary $amounts, ti:me:s, percentages%, a.c.r.o.n.y.m.s., hyphen-ated and abbrev'd words etc.)
• RegexpFindOffsetTokenizer
  A Tokenizer that uses a regular expression to finds tokens in the same way as RegexpFindTokenizer, but also maintains the character offset where token occurred within the data (useful for proximity searching within N characters and search term highlighting.)
• SentenceTokenizer
  A Tokenizer to split data into sentences (i.e. splits at full-stops.)
• PythonTokenizer
  A Tokenizer to split and annotate tokens in the Python programming language.
• DateTokenizer
  A Tokenizer that finds date time tokens within the given data.
• PreserveMaskingTokenizer
  A keyword Tokenizer based on RegexpSubTokenizer but which doesn't split at CQL masking characters (^*?). Especially useful when tokenizing incoming queries.

API

Module: cheshire3.tokenizer
Classes:

• SimpleTokenizer
  Splits the given data at all instances of a configurable string. Defaults to splitting on whitespace.
• OffsetTokenizer
  Abstract base class for Tokenizers that maintain information about the location of each token as character offsets.
• RegexpSubTokenizer
  Replaces regular expression matches in the data with a configurable character (defaults to whitespace), then splits the result at whitespace.
• RegexpSplitTokenizer
  Splits the data at matches of a regular expression. By default, each match (e.g. punctuation) is preserved in the result as a separate token.
• RegexpFindTokenizer
  Returns all words that match the configured regex.
• RegexpFindOffsetTokenizer
  Uses a regular expression to finds tokens in the same way as RegexpFindTokenizer, but also maintains the character offset where token occurred within the data.
• RegexpFindPunctuationOffsetTokenizer As RegexpFindOffsetTokenizer but does not include punctuation when calculating offsets.
• LineTokenizer
  Splits the data at new line characters
• SentenceTokenizer
  Splits data into sentences (i.e. splits at full-stops.)
• PythonTokenizer
  Splits and annotates tokens in the Python programming language.
• DateTokenizer
  Finds date time tokens within the given data.

Methods:

Function	Parameters	Returns	Description
__init__	config, parent
process_string	session, data	list	Process a raw string to produce an ordered list of tokens
process_hash	session, data	list	Process the text of each item in the hash to produce an ordered list of tokens for that hash entry.

Sub-Packages