TextFormat.Tokenizer

java.lang.Object
- org.fusesource.hawtbuf.proto.compiler.TextFormat.Tokenizer

Enclosing class:

TextFormat
```
private static final class TextFormat.Tokenizer
extends java.lang.Object
```
Represents a stream of tokens parsed from a String.
The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:
- java.io.StreamTokenizer: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').
- java.util.Scanner: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly, Scanner requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"), Scanner would recognize it only as a single token. Furthermore, Scanner provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.
Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need Matcher.usePattern(), which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.

Field Summary

Fields
Modifier and Type	Field and Description
`private int`	`column`
`private java.lang.String`	`currentToken`
`private static java.util.regex.Pattern`	`DOUBLE_INFINITY`
`private static java.util.regex.Pattern`	`FLOAT_INFINITY`
`private static java.util.regex.Pattern`	`FLOAT_NAN`
`private int`	`line`
`private java.util.regex.Matcher`	`matcher`
`private int`	`pos`
`private int`	`previousColumn`
`private int`	`previousLine`
`private java.lang.CharSequence`	`text`
`private static java.util.regex.Pattern`	`TOKEN`
`private static java.util.regex.Pattern`	`WHITESPACE`

Constructor Summary

Constructors
Constructor and Description

Tokenizer(java.lang.CharSequence text)
Construct a tokenizer that parses tokens from the given text.

Constructors
Constructor and Description
`Tokenizer(java.lang.CharSequence text)` Construct a tokenizer that parses tokens from the given text.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`atEnd()` Are we at the end of the input?
`void`	`consume(java.lang.String token)` If the next token exactly matches `token`, consume it.
`boolean`	`consumeBoolean()` If the next token is a boolean, consume it and return its value.
`Buffer`	`consumeBuffer()` If the next token is a string, consume it, unescape it as a `Buffer`, and return it.
`double`	`consumeDouble()` If the next token is a double, consume it and return its value.
`float`	`consumeFloat()` If the next token is a float, consume it and return its value.
`java.lang.String`	`consumeIdentifier()` If the next token is an identifier, consume it and return its value.
`int`	`consumeInt32()` If the next token is a 32-bit signed integer, consume it and return its value.
`long`	`consumeInt64()` If the next token is a 64-bit signed integer, consume it and return its value.
`java.lang.String`	`consumeString()` If the next token is a string, consume it and return its (unescaped) value.
`int`	`consumeUInt32()` If the next token is a 32-bit unsigned integer, consume it and return its value.
`long`	`consumeUInt64()` If the next token is a 64-bit unsigned integer, consume it and return its value.
`private TextFormat.ParseException`	`floatParseException(java.lang.NumberFormatException e)` Constructs an appropriate `TextFormat.ParseException` for the given `NumberFormatException` when trying to parse a float or double.
`private TextFormat.ParseException`	`integerParseException(java.lang.NumberFormatException e)` Constructs an appropriate `TextFormat.ParseException` for the given `NumberFormatException` when trying to parse an integer.
`boolean`	`lookingAtInteger()` Returns `true` if the next token is an integer, but does not consume it.
`void`	`nextToken()` Advance to the next token.
`TextFormat.ParseException`	`parseException(java.lang.String description)` Returns a `TextFormat.ParseException` with the current line and column numbers in the description, suitable for throwing.
`TextFormat.ParseException`	`parseExceptionPreviousToken(java.lang.String description)` Returns a `TextFormat.ParseException` with the line and column numbers of the previous token in the description, suitable for throwing.
`private void`	`skipWhitespace()` Skip over any whitespace so that the matcher region starts at the next token.
`boolean`	`tryConsume(java.lang.String token)` If the next token exactly matches `token`, consume it and return `true`.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - text
```
private final java.lang.CharSequence text
```
  - matcher
```
private final java.util.regex.Matcher matcher
```
  - currentToken
```
private java.lang.String currentToken
```
  - pos
```
private int pos
```
  - line
```
private int line
```
  - column
```
private int column
```
  - previousLine
```
private int previousLine
```
  - previousColumn
```
private int previousColumn
```
  - WHITESPACE
```
private static java.util.regex.Pattern WHITESPACE
```
  - TOKEN
```
private static java.util.regex.Pattern TOKEN
```
  - DOUBLE_INFINITY
```
private static java.util.regex.Pattern DOUBLE_INFINITY
```
  - FLOAT_INFINITY
```
private static java.util.regex.Pattern FLOAT_INFINITY
```
  - FLOAT_NAN
```
private static java.util.regex.Pattern FLOAT_NAN
```
- Constructor Detail
  - Tokenizer
```
public Tokenizer(java.lang.CharSequence text)
```
    Construct a tokenizer that parses tokens from the given text.
- Method Detail
  - atEnd
```
public boolean atEnd()
```
    Are we at the end of the input?
  - nextToken
```
public void nextToken()
```
    Advance to the next token.
  - skipWhitespace
```
private void skipWhitespace()
```
    Skip over any whitespace so that the matcher region starts at the next token.
  - tryConsume
```
public boolean tryConsume(java.lang.String token)
```
    If the next token exactly matches token, consume it and return true. Otherwise, return false without doing anything.
  - consume
```
public void consume(java.lang.String token)
             throws TextFormat.ParseException
```
    If the next token exactly matches token, consume it. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - lookingAtInteger
```
public boolean lookingAtInteger()
```
    Returns true if the next token is an integer, but does not consume it.
  - consumeIdentifier
```
public java.lang.String consumeIdentifier()
                                   throws TextFormat.ParseException
```
    If the next token is an identifier, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeInt32
```
public int consumeInt32()
                 throws TextFormat.ParseException
```
    If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeUInt32
```
public int consumeUInt32()
                  throws TextFormat.ParseException
```
    If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeInt64
```
public long consumeInt64()
                  throws TextFormat.ParseException
```
    If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeUInt64
```
public long consumeUInt64()
                   throws TextFormat.ParseException
```
    If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeDouble
```
public double consumeDouble()
                     throws TextFormat.ParseException
```
    If the next token is a double, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeFloat
```
public float consumeFloat()
                   throws TextFormat.ParseException
```
    If the next token is a float, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeBoolean
```
public boolean consumeBoolean()
                       throws TextFormat.ParseException
```
    If the next token is a boolean, consume it and return its value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeString
```
public java.lang.String consumeString()
                               throws TextFormat.ParseException
```
    If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - consumeBuffer
```
public Buffer consumeBuffer()
                     throws TextFormat.ParseException
```
    If the next token is a string, consume it, unescape it as a Buffer, and return it. Otherwise, throw a TextFormat.ParseException.
    
    Throws:
    
    TextFormat.ParseException
  - parseException
```
public TextFormat.ParseException parseException(java.lang.String description)
```
    Returns a TextFormat.ParseException with the current line and column numbers in the description, suitable for throwing.
  - parseExceptionPreviousToken
```
public TextFormat.ParseException parseExceptionPreviousToken(java.lang.String description)
```
    Returns a TextFormat.ParseException with the line and column numbers of the previous token in the description, suitable for throwing.
  - integerParseException
```
private TextFormat.ParseException integerParseException(java.lang.NumberFormatException e)
```
    Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse an integer.
  - floatParseException
```
private TextFormat.ParseException floatParseException(java.lang.NumberFormatException e)
```
    Constructs an appropriate TextFormat.ParseException for the given NumberFormatException when trying to parse a float or double.

Class TextFormat.Tokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

text

matcher

currentToken

pos

line

column

previousLine

previousColumn

WHITESPACE

TOKEN

DOUBLE_INFINITY

FLOAT_INFINITY

FLOAT_NAN

Constructor Detail

Tokenizer

Method Detail

atEnd

nextToken

skipWhitespace

tryConsume

consume

lookingAtInteger

consumeIdentifier

consumeInt32

consumeUInt32

consumeInt64

consumeUInt64

consumeDouble

consumeFloat

consumeBoolean

consumeString

consumeBuffer

parseException

parseExceptionPreviousToken

integerParseException

floatParseException