Package nz.org.riskscape.dsl

Class Lexer<T extends TokenType>

java.lang.Object

nz.org.riskscape.dsl.Lexer<T>

public class Lexer<T extends TokenType>
extends Object

Converts a String in to a stream of Tokens for parsing in to an AST by the Parser.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Lexer.Tokens<T extends TokenType>	Meta-class object for token type enumeration.

Constructor Summary

Constructors

Constructor	Description
Lexer(Lexer.Tokens<T> tokens, String source)	Builds a Lexer from a String

Method Summary

Modifier and Type	Method	Description
Optional<Token>	consumeIf(Set<? extends TokenType> oneOf)
Optional<Token>	consumeIf(TokenType conditionalToken)
final Token	expect(Set<? extends TokenType> expected)
final Token	expect(TokenType... expected)
boolean	isEOF()
Token	next()
Token	peek()
T	peekType()
String	remaining()
void	rewind(Token backTo)	Reposition the lexer so that next() returns this Token again.
final <U> U	tryThese(ConditionalParse<U>... attemptsArray)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Lexer

public Lexer(Lexer.Tokens<T> tokens,
 String source)

Builds a Lexer from a String

Method Details

next

public Token next()
           throws LexerException

Returns:

the next Token or Lexer.Tokens.eofToken if parsing as reached the end.

Throws:

LexerException

remaining

public String remaining()

Returns:

the remaining source String that still needs to be parsed. This can be useful for error reporting, i.e. show the user the unadulterated string that we were trying to parse next. Note that you may want to truncate this string in error messages, when parsing a large file.

peek

public Token peek()
           throws LexerException

Returns:

the Token that will be returned from the Lexer when you call next(), without consuming it.

Throws:

LexerException

isEOF

public boolean isEOF()
              throws LexerException

Returns:

true if the Lexer is at the end of the source

Throws:

LexerException

peekType

public T peekType()

Returns:

shortcut for peek().getTokenType()

rewind

public void rewind(Token backTo)

Reposition the lexer so that next() returns this Token again.

Parameters:

backTo -

expect

@SafeVarargs
public final Token expect(TokenType... expected)

expect

public final Token expect(Set<? extends TokenType> expected)

tryThese

@SafeVarargs
public final <U> U tryThese(ConditionalParse<U>... attemptsArray)

consumeIf

public Optional<Token> consumeIf(TokenType conditionalToken)

consumeIf

public Optional<Token> consumeIf(Set<? extends TokenType> oneOf)