Home Reference Source

import {Tokenizer} from '@hiherto-elements/natural/src/tokenizers/tokenizer.js'

public class | source

Tokenizer

Direct Subclass:

AggressiveTokenizer, CaseTokenizer, RegexpTokenizer, SentenceTokenizer, TreebankWordTokenizer

Method Summary

Public Methods
public	attach()
public	tokenize()
public	trim(array: ):

Public Methods

public attach() source

public tokenize() source

public trim(array: ): source

Params:

Name	Type	Attribute	Description
array	*

Return:

*