Class BaseTokeniser

Abstract Base Class for Tokeniser which handles the Position tracking.

Inheritance

System.Object

BaseTokeniser

Implements

ITokeniser

Inherited Members

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Namespace: VDS.RDF.Parsing.Tokens

Assembly: dotNetRDF.dll

Syntax

public abstract class BaseTokeniser : ITokeniser

Constructors

| Improve this Doc View Source

BaseTokeniser(TextReader)

Constructor for the BaseTokeniser which takes in a TextReader that the Tokeniser will generate Tokens from.

Declaration

protected BaseTokeniser(TextReader reader)

Parameters

Type	Name	Description
System.IO.TextReader	reader	TextReader to generator Tokens from.

Properties

| Improve this Doc View Source

CurrentLine

Gets the Current Line in the Input Stream.

Declaration

protected int CurrentLine { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

CurrentPosition

Gets the Current Position in the Input Stream.

Declaration

protected int CurrentPosition { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

EndLine

Gets the End Line in the Input Stream of the current Token.

Declaration

protected int EndLine { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

EndPosition

Gets the End Position in the Input Stream of the current Token.

Declaration

protected int EndPosition { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

Format

Gets/Sets the Format that this Tokeniser is used for.

Declaration

protected string Format { get; set; }

Property Value

Type	Description
System.String

Remarks

The value set here will replace any instances of {0} specified in inputs to the Error(String) function allowing messages regarding certain syntaxes not being valid in a given format to be provided.

| Improve this Doc View Source

HasBacktracked

Gets whether the Tokeniser has backtracked.

Declaration

protected bool HasBacktracked { get; }

Property Value

Type	Description
System.Boolean

| Improve this Doc View Source

LastTokenType

Gets/Sets the Last Token Type.

Declaration

protected int LastTokenType { get; set; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

Length

Gets the current length of the Output Buffer.

Declaration

protected int Length { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

StartLine

Gets the Start Line in the Input Stream of the current Token.

Declaration

protected int StartLine { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

StartPosition

Gets the Start Position in the Input Stream of the current Token.

Declaration

protected int StartPosition { get; }

Property Value

Type	Description
System.Int32

| Improve this Doc View Source

Value

Gets the value of the Output Buffer.

Declaration

protected string Value { get; }

Property Value

Type	Description
System.String

Methods

| Improve this Doc View Source

Backtrack()

Allows you to Backtrack one character (and no more).

Declaration

protected void Backtrack()

| Improve this Doc View Source

ConsumeCharacter()

Consumes a single Character into the Output Buffer and increments the Position Counters.

Declaration

protected void ConsumeCharacter()

Exceptions

Type	Condition
RdfParseException	Thrown if the caller tries to read beyond the end of the Stream.

| Improve this Doc View Source

ConsumeCharacter(Boolean)

Consumes a single Character into the Output Buffer and increments the Position Counters.

Declaration

protected bool ConsumeCharacter(bool allowEOF)

Parameters

Type	Name	Description
System.Boolean	allowEOF	Whether EOF is allowed.

Returns

Type	Description
System.Boolean	True if the EOF is reached.

Remarks

If allowEOF is set to false then the normal behaviour is used and an error will be thrown on end of file.

| Improve this Doc View Source

ConsumeNewLine(Boolean)

Consumes a New Line (which may be a single \n or \r or the two characters following each other).

Declaration

protected void ConsumeNewLine(bool asOutput)

Parameters

Type	Name	Description
System.Boolean	asOutput	Whether the New Line should be added to the Output Buffer.

| Improve this Doc View Source

ConsumeNewLine(Boolean, Boolean)

Consumes a New Line (which may be a single \n or \r or the two characters following each other).

Declaration

protected void ConsumeNewLine(bool asOutput, bool allowEOF)

Parameters

Type	Name	Description
System.Boolean	asOutput	Whether the New Line should be added to the Output Buffer.
System.Boolean	allowEOF	Whether EOF is permitted instead of a New Line.

| Improve this Doc View Source

DiscardWhiteSpace()

Helper function which discards White Space which the Tokeniser doesn't care about and increments position counters correctly.

Declaration

protected void DiscardWhiteSpace()

| Improve this Doc View Source

Error(String)

Helper Function for generating Standardised Parser Errors.

Declaration

protected RdfParseException Error(string detail)

Parameters

Type	Name	Description
System.String	detail	The Error Message.

Returns

Type	Description
RdfParseException

| Improve this Doc View Source

GetNextToken()

Gets the Next available Token from the Input.

Declaration

public abstract IToken GetNextToken()

Returns

Type	Description
IToken

Exceptions

Type	Condition
RdfParseException	Parser Exception if a valid Token cannot be retrieved.

| Improve this Doc View Source

HandleComplexLocalNameEscapes()

Handles the complex escapes that can occur in a local name.

Declaration

protected void HandleComplexLocalNameEscapes()

Remarks

Unlike HandleEscapes(TokeniserEscapeMode) this only unescapes unicode escapes, other escapes are simply validated and passed through for later unescaping.

| Improve this Doc View Source

HandleEscapes(TokeniserEscapeMode)

Handles the standard escapes supported in all the UTF-8 based RDF serializations.

Declaration

protected void HandleEscapes(TokeniserEscapeMode mode)

Parameters

Type	Name	Description
TokeniserEscapeMode	mode

| Improve this Doc View Source

IsHexDigit(Char)

Determines whether a given Character can be valid as a Hex Digit.

Declaration

protected bool IsHexDigit(char c)

Parameters

Type	Name	Description
System.Char	c	Character to test.

Returns

Type	Description
System.Boolean

| Improve this Doc View Source

Peek()

Peeks at the next Character.

Declaration

protected char Peek()

Returns

Type	Description
System.Char

| Improve this Doc View Source

SkipCharacter()

Skips a single Character and increments the Position Counters.

Declaration

protected char SkipCharacter()

Returns

Type	Description
System.Char

Remarks

Use when you are reading characters into some local buffer and not the global token buffer, used in String escaping.

Exceptions

Type	Condition
RdfParseException	Thrown if the caller tries to read beyond the end of the Stream.

| Improve this Doc View Source

StartNewToken()

Informs the Helper that you wish to start reading a new Token.

Declaration

protected void StartNewToken()

| Improve this Doc View Source

UnexpectedCharacter(Char, String)

Helper Function for generating Standardised Parser Errors about unexpected characters.

Declaration

protected RdfParseException UnexpectedCharacter(char c, string expected)

Parameters

Type	Name	Description
System.Char	c	Unexpected Character.
System.String	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

| Improve this Doc View Source

UnexpectedEndOfInput(String)

Helper Function for generating Standardised Parser Errors about unexpected end of input.

Declaration

protected RdfParseException UnexpectedEndOfInput(string expected)

Parameters

Type	Name	Description
System.String	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

| Improve this Doc View Source

UnexpectedNewLine(String)

Helper Function for generating Standardised Parser Errors about unexpected new lines.

Declaration

protected RdfParseException UnexpectedNewLine(string expected)

Parameters

Type	Name	Description
System.String	expected	Message detailing what was expected (may be empty if no explicit expectation).

Returns

Type	Description
RdfParseException

| Improve this Doc View Source

UnexpectedToken(String, IToken)

Helper Function for generating Standardised Parser Errors about unexpected tokens.

Declaration

protected RdfParseException UnexpectedToken(string expected, IToken t)

Parameters

Type	Name	Description
System.String	expected	Message detailing what was expected (may be empty if no explicity expectation).
IToken	t	Token that was parsed.

Returns

Type	Description
RdfParseException

Implements

ITokeniser

Extension Methods

Extensions.AsEnumerable<T>(T)