cmnTokenizer Class Reference

#include <cmnTokenizer.h>

Public Types
typedef std::vector< const char * >	TokensContainer
typedef TokensContainer::size_type	size_type

Public Member Functions
	cmnTokenizer ()
	~cmnTokenizer ()
void	SetDelimiters (const char *delimiters)
void	SetQuoteMarkers (const char *markers)
void	SetEscapeMarkers (const char *markers)
const char *	GetDelimiters (void) const
const char *	GetQuoteMarkers (void) const
const char *	GetEscapeMarkers (void) const
size_type	GetNumTokens (void) const
const char *	GetToken (size_type num) const
const char *const *	GetTokensArray (void) const
void	GetArgvTokens (std::vector< const char * > &argvTokens) const

Static Public Member Functions
static const char *	GetDefaultDelimiters (void)
static const char *	GetDefaultQuoteMarkers (void)
static const char *	GetDefaultEscapeMarkers (void)

Public Attributes
void Parse(const char *string) CISST_THROW(std void	Parse (const std::string &string) CISST_THROW(std

Detailed Description

cmnTokenizer provides a convenient interface for parsing a string into a set of tokens. The parsing uses several sets of control characters:

Delimiters: Separate tokens. A sequence of delimiters which is not quoted or escaped is ignored, and a new token begins after it.

Quote markers: Enclose parts of, or complete, tokens. Anything between a pair of identical quote markers is included in a token.

Escape markers: Quote the character immediately following them.

The default values for delimiters is whitespace (space, tab, CR); for quote markers is the set of double and single quotation marks (",'); for escape markers is backslash (). But the user can override them by calling the appropriate method.

A cmnTokenizer object maintains an internal copy of the tokenized text, and can return a pointer to an array of pointer, after the fashion of argv.

Note

It is important to note that the stored tokens have the life span of the tokenizer. If the tokenizer is destroyed, the user cannot access any of the tokens.

Member Typedef Documentation

size_type

typedef TokensContainer::size_type cmnTokenizer::size_type

TokensContainer

typedef std::vector<const char *> cmnTokenizer::TokensContainer

Constructor & Destructor Documentation

cmnTokenizer()

cmnTokenizer::cmnTokenizer

(

)

~cmnTokenizer()

cmnTokenizer::~cmnTokenizer

(

)

Member Function Documentation

GetArgvTokens()

void cmnTokenizer::GetArgvTokens

(

std::vector< const char * > &

argvTokens

)

const

This method will fill the input vector with the tokens, but first set the 0-index element to NULL, to follow the argv convention, where argv[0] contains the "name of the program".

GetDefaultDelimiters()

const char * cmnTokenizer::GetDefaultDelimiters ( void )

inlinestatic

GetDefaultEscapeMarkers()

const char * cmnTokenizer::GetDefaultEscapeMarkers ( void )

inlinestatic

GetDefaultQuoteMarkers()

const char * cmnTokenizer::GetDefaultQuoteMarkers ( void )

inlinestatic

GetDelimiters()

const char * cmnTokenizer::GetDelimiters ( void ) const

inline

GetEscapeMarkers()

const char * cmnTokenizer::GetEscapeMarkers ( void ) const

inline

GetNumTokens()

size_type cmnTokenizer::GetNumTokens ( void ) const

inline

Return the number of tokens stored. Note that the last token is a NULL.

GetQuoteMarkers()

const char * cmnTokenizer::GetQuoteMarkers ( void ) const

inline

GetToken()

const char * cmnTokenizer::GetToken

(

size_type

num

)

const

Return the specified token.

Parameters

num	the token number (0..GetNumTokens())

Returns

pointer to token string (0 if not valid)

GetTokensArray()

const char *const * cmnTokenizer::GetTokensArray

(

void

)

const

Return the array of tokens in an argv fashion.

Note

This parsing returns exactly the tokens in the input string. For an argv-style set of argument, one needs to have the "name of the program" argument in index 0, and the arguments starting at index 1. Use the method GetArgvTokens() for that.

SetDelimiters()

void cmnTokenizer::SetDelimiters ( const char * delimiters )

inline

SetEscapeMarkers()

void cmnTokenizer::SetEscapeMarkers ( const char * markers )

inline

SetQuoteMarkers()

void cmnTokenizer::SetQuoteMarkers ( const char * markers )

inline

Member Data Documentation

Parse

void Parse(const char *string) CISST_THROW(std void cmnTokenizer::Parse(const std::string &string) CISST_THROW(std

inline

Parse the input string and store the tokens internally. If there is a syntax error (e.g., unclosed quotes) throw an exception.

Parameters

string

the text to be parsed.

---

The documentation for this class was generated from the following file:

• tmp/cisst-saw/cisst/cisstCommon/cmnTokenizer.h