Class BaseTokeniser

Abstract Base Class for Tokeniser which handles the Position tracking.

Inheritance

object

BaseTokeniser

CsvTokeniser

NTriplesTokeniser

Notation3Tokeniser

SparqlTokeniser

TriGTokeniser

TsvTokeniser

TurtleTokeniser

Implements

ITokeniser

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Namespace: VDS.RDF.Parsing.Tokens

Assembly: dotNetRdf.dll

Syntax

public abstract class BaseTokeniser : ITokeniser

Constructors

BaseTokeniser(TextReader)

Constructor for the BaseTokeniser which takes in a TextReader that the Tokeniser will generate Tokens from.

Declaration

protected BaseTokeniser(TextReader reader)

Parameters

Type	Name	Description
TextReader	reader	TextReader to generator Tokens from.

Properties

CurrentLine

Gets the Current Line in the Input Stream.

Declaration

protected int CurrentLine { get; }

Property Value

Type	Description
int

CurrentPosition

Gets the Current Position in the Input Stream.

Declaration

protected int CurrentPosition { get; }

Property Value

Type	Description
int

EndLine

Gets the End Line in the Input Stream of the current Token.

Declaration

protected int EndLine { get; }

Property Value

Type	Description
int

EndPosition

Gets the End Position in the Input Stream of the current Token.

Declaration

protected int EndPosition { get; }

Property Value

Type	Description
int

Format

Gets/Sets the Format that this Tokeniser is used for.

Declaration

protected string Format { get; set; }

Property Value

Type	Description
string

Remarks

The value set here will replace any instances of {0} specified in inputs to the Error() function allowing messages regarding certain syntaxes not being valid in a given format to be provided.

HasBacktracked

Gets whether the Tokeniser has backtracked.

Declaration

protected bool HasBacktracked { get; }

Property Value

Type	Description
bool

LastTokenType

Gets/Sets the Last Token Type.

Declaration

protected int LastTokenType { get; set; }

Property Value

Type	Description
int

Length

Gets the current length of the Output Buffer.

Declaration

protected int Length { get; }

Property Value

Type	Description
int

StartLine

Gets the Start Line in the Input Stream of the current Token.

Declaration

protected int StartLine { get; }

Property Value

Type	Description
int

StartPosition

Gets the Start Position in the Input Stream of the current Token.

Declaration

protected int StartPosition { get; }

Property Value

Type	Description
int

Value

Gets the value of the Output Buffer.

Declaration

protected string Value { get; }

Property Value

Type	Description
string

Methods

Backtrack()

Allows you to Backtrack one character (and no more).

Declaration

protected void Backtrack()

ConsumeCharacter()

Consumes a single Character into the Output Buffer and increments the Position Counters.

Declaration

protected void ConsumeCharacter()

Exceptions

Type	Condition
RdfParseException	Thrown if the caller tries to read beyond the end of the Stream.

ConsumeCharacter(bool)

Consumes a single Character into the Output Buffer and increments the Position Counters.

Declaration

protected bool ConsumeCharacter(bool allowEOF)

Parameters

Type	Name	Description
bool	allowEOF	Whether EOF is allowed.

Returns

Type	Description
bool	True if the EOF is reached.

Remarks

If allowEOF is set to false then the normal behaviour is used and an error will be thrown on end of file.

ConsumeNewLine(bool)

Consumes a New Line (which may be a single \n or \r or the two characters following each other).

Declaration

protected void ConsumeNewLine(bool asOutput)

Parameters

Type	Name	Description
bool	asOutput	Whether the New Line should be added to the Output Buffer.

ConsumeNewLine(bool, bool)

Consumes a New Line (which may be a single \n or \r or the two characters following each other).

Declaration

protected void ConsumeNewLine(bool asOutput, bool allowEOF)

Parameters

Type	Name	Description
bool	asOutput	Whether the New Line should be added to the Output Buffer.
bool	allowEOF	Whether EOF is permitted instead of a New Line.

DiscardWhiteSpace()

Helper function which discards White Space which the Tokeniser doesn't care about and increments position counters correctly.

Declaration

protected void DiscardWhiteSpace()

Error(string)

Helper Function for generating Standardised Parser Errors.

Declaration

protected RdfParseException Error(string detail)

Parameters

Type	Name	Description
string	detail	The Error Message.

Returns

Type	Description
RdfParseException

GetNextToken()

Gets the Next available Token from the Input.

Declaration

public abstract IToken GetNextToken()

Returns

Type	Description
IToken

Exceptions

Type	Condition
RdfParseException	Parser Exception if a valid Token cannot be retrieved.

HandleComplexLocalNameEscapes()

Handles the complex escapes that can occur in a local name.

Declaration

protected void HandleComplexLocalNameEscapes()

Remarks

Unlike HandleEscapes() this only unescapes unicode escapes, other escapes are simply validated and passed through for later unescaping.

HandleEscapes(TokeniserEscapeMode)

Handles the standard escapes supported in all the UTF-8 based RDF serializations.

Declaration

protected void HandleEscapes(TokeniserEscapeMode mode)

Parameters

Type	Name	Description
TokeniserEscapeMode	mode

IsHexDigit(char)

Determines whether a given Character can be valid as a Hex Digit.

Declaration

protected bool IsHexDigit(char c)

Parameters

Type	Name	Description
char	c	Character to test.

Returns

Type	Description
bool

Peek()

Peeks at the next Character.

Declaration

protected char Peek()

Returns

Type	Description
char

SkipCharacter()

Skips a single Character and increments the Position Counters.

Declaration

protected char SkipCharacter()

Returns

Type	Description
char

Remarks

Use when you are reading characters into some local buffer and not the global token buffer, used in String escaping.

Exceptions

Type	Condition
RdfParseException	Thrown if the caller tries to read beyond the end of the Stream.

StartNewToken()

Informs the Helper that you wish to start reading a new Token.

Declaration

protected void StartNewToken()

UnexpectedCharacter(char, string)

Helper Function for generating Standardised Parser Errors about unexpected characters.

Declaration

protected RdfParseException UnexpectedCharacter(char c, string expected)

Parameters

Type	Name	Description
char	c	Unexpected Character.
string	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

UnexpectedEndOfInput(string)

Helper Function for generating Standardised Parser Errors about unexpected end of input.

Declaration

protected RdfParseException UnexpectedEndOfInput(string expected)

Parameters

Type	Name	Description
string	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

UnexpectedNewLine(string)

Helper Function for generating Standardised Parser Errors about unexpected new lines.

Declaration

protected RdfParseException UnexpectedNewLine(string expected)

Parameters

Type	Name	Description
string	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

UnexpectedToken(string, IToken)

Helper Function for generating Standardised Parser Errors about unexpected tokens.

Declaration

protected RdfParseException UnexpectedToken(string expected, IToken t)

Parameters

Type	Name	Description
string	expected	Message detailing what was expected (may be empty if no explicity expectation).
IToken	t	Token that was parsed.

Returns

Type	Description
RdfParseException

Implements

ITokeniser

Extension Methods

Extensions.ToSafeString(object)

Extensions.AsEnumerable<T>(T)