TokenizerByText

java.lang.Object

com.topologi.diffx.load.text.TokenizerByText

All Implemented Interfaces:: TextTokenizer

public final class TokenizerByText extends Object implements TextTokenizer

The tokeniser for characters events.

This class is not synchronized.

Version:: 11 May 2010

Field Summary

Fields

Modifier and Type

Field

Description

private final WhiteSpaceProcessing

whitespace

Define the whitespace processing.
Constructor Summary

Constructors

Constructor

Description

TokenizerByText(WhiteSpaceProcessing whitespace)

Creates a new tokenizer.
Method Summary

Modifier and Type

Method

Description

TextGranularity

granurality()

Always TextGranularity.CHARACTER.

List<TextEvent>

tokenize(CharSequence seq)

Returns the list of TextEvent corresponding to the specified character sequence.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details
- whitespace
  
  private final WhiteSpaceProcessing whitespace
  
  Define the whitespace processing.
Constructor Details
- TokenizerByText
  
  public TokenizerByText(WhiteSpaceProcessing whitespace)
  
  Creates a new tokenizer.
  
  Parameters:
  
  whitespace - the whitespace processing for this tokenizer.
  
  Throws:
  
  NullPointerException - if the white space processing is not specified.
Method Details
- tokenize
  
  public List<TextEvent> tokenize(CharSequence seq)
  
  Returns the list of TextEvent corresponding to the specified character sequence.
  
  Specified by:
  
  tokenize in interface TextTokenizer
  
  Parameters:
  
  seq - the character sequence to tokenize.
  
  Returns:
  
  the corresponding list.
- granurality
  
  public TextGranularity granurality()
  
  Always TextGranularity.CHARACTER. Returns the text granularity of this tokenizer.
  
  Specified by:
  
  granurality in interface TextTokenizer
  
  Returns:
  
  the text granularity of this tokenizer.