com.fasterxml.jackson.databind.util

## Class TokenBuffer

- java.lang.Object
- - com.fasterxml.jackson.core.JsonGenerator
  - - com.fasterxml.jackson.databind.util.TokenBuffer

All Implemented Interfaces:
  Versioned, Closeable, Flushable, AutoCloseable

    public class TokenBuffer
    extends JsonGenerator

Utility class used for efficient storage of `JsonToken` sequences, needed for temporary buffering. Space efficient for different sequence lengths (especially so for smaller ones; but not significantly less efficient for larger), highly efficient for linear iteration and appending. Implemented as segmented/chunked linked list of tokens; only modifications are via appends.

Note that before version 2.0, this class was located in the "core" bundle, not data-binding; but since it was only used by data binding, was moved here to reduce size of core package

### Nested Class Summary

Nested Classes
`protected static class` `TokenBuffer.Parser`
`protected static class` `TokenBuffer.Segment` Individual segment of TokenBuffer that can store up to 16 tokens (limited by 4 bits per token type marker requirement).

### Nested classes/interfaces inherited from class com.fasterxml.jackson.core.JsonGenerator

`JsonGenerator.Feature`

### Field Summary

Fields
`protected int` `_appendAt` Offset within last segment,
`protected boolean` `_closed`
`protected TokenBuffer.Segment` `_first` First segment, for contents this buffer has
`protected boolean` `_forceBigDecimal` Flag set during construction, if use of `BigDecimal` is to be forced on all floating-point values.
`protected int` `_generatorFeatures` Bit flag composed of bits that indicate which `JsonGenerator.Feature`s are enabled.
`protected boolean` `_hasNativeId` Do we currently have a native type or object id buffered?
`protected boolean` `_hasNativeObjectIds`
`protected boolean` `_hasNativeTypeIds`
`protected TokenBuffer.Segment` `_last` Last segment of this buffer, one that is used for appending more tokens
`protected boolean` `_mayHaveNativeIds`
`protected ObjectCodec` `_objectCodec` Object codec to use for stream-based object conversion through parser/generator interfaces.
`protected Object` `_objectId` If native object ids supported, this is the id for following value (or first token of one) to be written.
`protected Object` `_typeId` If native type ids supported, this is the id for following value (or first token of one) to be written.
`protected JsonWriteContext` `_writeContext`
`protected static int` `DEFAULT_GENERATOR_FEATURES`

### Fields inherited from class com.fasterxml.jackson.core.JsonGenerator

`_cfgPrettyPrinter`

### Constructor Summary

Constructors
Constructor and Description
`TokenBuffer(JsonParser p)`
`TokenBuffer(JsonParser p, DeserializationContext ctxt)`
`TokenBuffer(ObjectCodec codec)` Deprecated. since 2.3 preferred variant is one that takes `JsonParser` or additional boolean parameter.
`TokenBuffer(ObjectCodec codec, boolean hasNativeIds)`

### Method Summary

Methods
`protected void` `_append(JsonToken type)`
`protected void` `_append(JsonToken type, Object value)`
`protected void` `_appendRaw(int rawType, Object value)`
`protected void` `_appendValue(JsonToken type)` Similar to `_append(JsonToken)` but also updates context with knowledge that a scalar value was written
`protected void` `_appendValue(JsonToken type, Object value)` Similar to `_append(JsonToken,Object)` but also updates context with knowledge that a scalar value was written
`protected void` `_reportUnsupportedOperation()`
`TokenBuffer` `append(TokenBuffer other)` Helper method that will append contents of given buffer into this buffer.
`JsonParser` `asParser()` Method used to create a `JsonParser` that can read contents stored in this buffer.
`JsonParser` `asParser(JsonParser src)`
`JsonParser` `asParser(ObjectCodec codec)` Method used to create a `JsonParser` that can read contents stored in this buffer.
`boolean` `canWriteBinaryNatively()` Since we can efficiently store `byte[]`, yes.
`boolean` `canWriteObjectId()`
`boolean` `canWriteTypeId()`
`void` `close()`
`void` `copyCurrentEvent(JsonParser p)`
`void` `copyCurrentStructure(JsonParser jp)`
`TokenBuffer` `deserialize(JsonParser p, DeserializationContext ctxt)` Helper method used by standard deserializer.
`JsonGenerator` `disable(JsonGenerator.Feature f)`
`JsonGenerator` `enable(JsonGenerator.Feature f)`
`JsonToken` `firstToken()`
`void` `flush()`
`TokenBuffer` `forceUseOfBigDecimal(boolean b)`
`ObjectCodec` `getCodec()`
`int` `getFeatureMask()`
`JsonWriteContext` `getOutputContext()`
`boolean` `isClosed()`
`boolean` `isEnabled(JsonGenerator.Feature f)`
`JsonGenerator` `overrideStdFeatures(int values, int mask)`
`void` `serialize(JsonGenerator gen)` Helper method that will write all contents of this buffer using given `JsonGenerator`.
`JsonGenerator` `setCodec(ObjectCodec oc)`
`JsonGenerator` `setFeatureMask(int mask)` Deprecated.
`String` `toString()`
`JsonGenerator` `useDefaultPrettyPrinter()`
`Version` `version()`
`void` `writeBinary(Base64Variant b64variant, byte[] data, int offset, int len)`
`int` `writeBinary(Base64Variant b64variant, InputStream data, int dataLength)` Although we could support this method, it does not necessarily make sense: we can not make good use of streaming because buffer must hold all the data.
`void` `writeBoolean(boolean state)`
`void` `writeEndArray()`
`void` `writeEndObject()`
`void` `writeFieldName(SerializableString name)`
`void` `writeFieldName(String name)`
`void` `writeNull()`
`void` `writeNumber(BigDecimal dec)`
`void` `writeNumber(BigInteger v)`
`void` `writeNumber(double d)`
`void` `writeNumber(float f)`
`void` `writeNumber(int i)`
`void` `writeNumber(long l)`
`void` `writeNumber(short i)`
`void` `writeNumber(String encodedValue)`
`void` `writeObject(Object value)`
`void` `writeObjectId(Object id)`
`void` `writeRaw(char c)`
`void` `writeRaw(char[] text, int offset, int len)`
`void` `writeRaw(SerializableString text)`
`void` `writeRaw(String text)`
`void` `writeRaw(String text, int offset, int len)`
`void` `writeRawUTF8String(byte[] text, int offset, int length)`
`void` `writeRawValue(char[] text, int offset, int len)`
`void` `writeRawValue(String text)`
`void` `writeRawValue(String text, int offset, int len)`
`void` `writeStartArray()`
`void` `writeStartObject()`
`void` `writeString(char[] text, int offset, int len)`
`void` `writeString(SerializableString text)`
`void` `writeString(String text)`
`void` `writeTree(TreeNode node)`
`void` `writeTypeId(Object id)`
`void` `writeUTF8String(byte[] text, int offset, int length)`

### Methods inherited from class com.fasterxml.jackson.core.JsonGenerator

`_reportError, _throwInternal, _writeSimpleObject, canOmitFields, canUseSchema, canWriteFormattedNumbers, configure, getCharacterEscapes, getCurrentValue, getFormatFeatures, getHighestEscapedChar, getOutputBuffered, getOutputTarget, getPrettyPrinter, getSchema, overrideFormatFeatures, setCharacterEscapes, setCurrentValue, setHighestNonEscapedChar, setPrettyPrinter, setRootValueSeparator, setSchema, writeArray, writeArray, writeArray, writeArrayFieldStart, writeBinary, writeBinary, writeBinary, writeBinaryField, writeBooleanField, writeNullField, writeNumberField, writeNumberField, writeNumberField, writeNumberField, writeNumberField, writeObjectField, writeObjectFieldStart, writeObjectRef, writeOmittedField, writeRawValue, writeStartArray, writeStartObject, writeStringField`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

### Field Detail

#### DEFAULT_GENERATOR_FEATURES

    protected static final int DEFAULT_GENERATOR_FEATURES

#### _objectCodec

    protected ObjectCodec _objectCodec

Object codec to use for stream-based object conversion through parser/generator interfaces. If null, such methods can not be used.

#### _generatorFeatures

    protected int _generatorFeatures

Bit flag composed of bits that indicate which `JsonGenerator.Feature`s are enabled.

NOTE: most features have no effect on this class

#### _closed

    protected boolean _closed

#### _hasNativeTypeIds

    protected boolean _hasNativeTypeIds

Since:
  2.3

#### _hasNativeObjectIds

    protected boolean _hasNativeObjectIds

Since:
  2.3

#### _mayHaveNativeIds

    protected boolean _mayHaveNativeIds

Since:
  2.3

#### _forceBigDecimal

    protected boolean _forceBigDecimal

Flag set during construction, if use of `BigDecimal` is to be forced on all floating-point values.

Since:
  2.7

#### _first

    protected TokenBuffer.Segment _first

First segment, for contents this buffer has

#### _last

    protected TokenBuffer.Segment _last

Last segment of this buffer, one that is used for appending more tokens

#### _appendAt

    protected int _appendAt

Offset within last segment,

#### _typeId

    protected Object _typeId

If native type ids supported, this is the id for following value (or first token of one) to be written.

#### _objectId

    protected Object _objectId

If native object ids supported, this is the id for following value (or first token of one) to be written.

#### _hasNativeId

    protected boolean _hasNativeId

Do we currently have a native type or object id buffered?

#### _writeContext

    protected JsonWriteContext _writeContext

### Constructor Detail

#### TokenBuffer

    @Deprecated
    public TokenBuffer(ObjectCodec codec)

Deprecated. since 2.3 preferred variant is one that takes `JsonParser` or additional boolean parameter.

Parameters:
  `codec` - Object codec to use for stream-based object conversion through parser/generator interfaces. If null, such methods can not be used.

#### TokenBuffer

    public TokenBuffer(ObjectCodec codec,
               boolean hasNativeIds)

Parameters:
  `codec` - Object codec to use for stream-based object conversion through parser/generator interfaces. If null, such methods can not be used.
  `hasNativeIds` - Whether resulting `JsonParser` (if created) is considered to support native type and object ids

#### TokenBuffer

    public TokenBuffer(JsonParser p)

Since:
  2.3

#### TokenBuffer

    public TokenBuffer(JsonParser p,
               DeserializationContext ctxt)

Since:
  2.7

### Method Detail

#### forceUseOfBigDecimal

    public TokenBuffer forceUseOfBigDecimal(boolean b)

Since:
  2.7

#### version

    public Version version()

Specified by:
  `version` in interface `Versioned`
Specified by:
  `version` in class `JsonGenerator`

#### asParser

    public JsonParser asParser()

Method used to create a `JsonParser` that can read contents stored in this buffer. Will use default `_objectCodec` for object conversions.

Note: instances are not synchronized, that is, they are not thread-safe if there are concurrent appends to the underlying buffer.

Returns:
  Parser that can be used for reading contents stored in this buffer

#### asParser

    public JsonParser asParser(ObjectCodec codec)

Method used to create a `JsonParser` that can read contents stored in this buffer.

Note: instances are not synchronized, that is, they are not thread-safe if there are concurrent appends to the underlying buffer.

Parameters:
  `codec` - Object codec to use for stream-based object conversion through parser/generator interfaces. If null, such methods can not be used.
Returns:
  Parser that can be used for reading contents stored in this buffer

#### asParser

    public JsonParser asParser(JsonParser src)

Parameters:
  `src` - Parser to use for accessing source information like location, configured codec

#### firstToken

    public JsonToken firstToken()

#### append

    public TokenBuffer append(TokenBuffer other)
                       throws IOException

Helper method that will append contents of given buffer into this buffer. Not particularly optimized; can be made faster if there is need.

Returns:
  This buffer
Throws:
  `IOException`

#### serialize

    public void serialize(JsonGenerator gen)
                   throws IOException

Helper method that will write all contents of this buffer using given `JsonGenerator`.

Note: this method would be enough to implement `JsonSerializer` for `TokenBuffer` type; but we can not have upwards references (from core to mapper package); and as such we also can not take second argument.

Throws:
  `IOException`

#### deserialize

    public TokenBuffer deserialize(JsonParser p,
                          DeserializationContext ctxt)
                            throws IOException

Helper method used by standard deserializer.

Throws:
  `IOException`
Since:
  2.3

#### toString

    public String toString()

Overrides:
  `toString` in class `Object`

#### enable

    public JsonGenerator enable(JsonGenerator.Feature f)

Specified by:
  `enable` in class `JsonGenerator`

#### disable

    public JsonGenerator disable(JsonGenerator.Feature f)

Specified by:
  `disable` in class `JsonGenerator`

#### isEnabled

    public boolean isEnabled(JsonGenerator.Feature f)

Specified by:
  `isEnabled` in class `JsonGenerator`

#### getFeatureMask

    public int getFeatureMask()

Specified by:
  `getFeatureMask` in class `JsonGenerator`

#### setFeatureMask

    @Deprecated
    public JsonGenerator setFeatureMask(int mask)

Deprecated.

Specified by:
  `setFeatureMask` in class `JsonGenerator`

#### overrideStdFeatures

    public JsonGenerator overrideStdFeatures(int values,
                                    int mask)

Overrides:
  `overrideStdFeatures` in class `JsonGenerator`

#### useDefaultPrettyPrinter

    public JsonGenerator useDefaultPrettyPrinter()

Specified by:
  `useDefaultPrettyPrinter` in class `JsonGenerator`

#### setCodec

    public JsonGenerator setCodec(ObjectCodec oc)

Specified by:
  `setCodec` in class `JsonGenerator`

#### getCodec

    public ObjectCodec getCodec()

Specified by:
  `getCodec` in class `JsonGenerator`

#### getOutputContext

    public final JsonWriteContext getOutputContext()

Specified by:
  `getOutputContext` in class `JsonGenerator`

#### canWriteBinaryNatively

    public boolean canWriteBinaryNatively()

Since we can efficiently store `byte[]`, yes.

Overrides:
  `canWriteBinaryNatively` in class `JsonGenerator`

#### flush

    public void flush()
               throws IOException

Specified by:
  `flush` in interface `Flushable`
Specified by:
  `flush` in class `JsonGenerator`
Throws:
  `IOException`

#### close

    public void close()
               throws IOException

Specified by:
  `close` in interface `Closeable`
Specified by:
  `close` in interface `AutoCloseable`
Specified by:
  `close` in class `JsonGenerator`
Throws:
  `IOException`

#### isClosed

    public boolean isClosed()

Specified by:
  `isClosed` in class `JsonGenerator`

#### writeStartArray

    public final void writeStartArray()
                               throws IOException

Specified by:
  `writeStartArray` in class `JsonGenerator`
Throws:
  `IOException`

#### writeEndArray

    public final void writeEndArray()
                             throws IOException

Specified by:
  `writeEndArray` in class `JsonGenerator`
Throws:
  `IOException`

#### writeStartObject

    public final void writeStartObject()
                                throws IOException

Specified by:
  `writeStartObject` in class `JsonGenerator`
Throws:
  `IOException`

#### writeEndObject

    public final void writeEndObject()
                              throws IOException

Specified by:
  `writeEndObject` in class `JsonGenerator`
Throws:
  `IOException`

#### writeFieldName

    public final void writeFieldName(String name)
                              throws IOException

Specified by:
  `writeFieldName` in class `JsonGenerator`
Throws:
  `IOException`

#### writeFieldName

    public void writeFieldName(SerializableString name)
                        throws IOException

Specified by:
  `writeFieldName` in class `JsonGenerator`
Throws:
  `IOException`

#### writeString

    public void writeString(String text)
                     throws IOException

Specified by:
  `writeString` in class `JsonGenerator`
Throws:
  `IOException`

#### writeString

    public void writeString(char[] text,
                   int offset,
                   int len)
                     throws IOException

Specified by:
  `writeString` in class `JsonGenerator`
Throws:
  `IOException`

#### writeString

    public void writeString(SerializableString text)
                     throws IOException

Specified by:
  `writeString` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRawUTF8String

    public void writeRawUTF8String(byte[] text,
                          int offset,
                          int length)
                            throws IOException

Specified by:
  `writeRawUTF8String` in class `JsonGenerator`
Throws:
  `IOException`

#### writeUTF8String

    public void writeUTF8String(byte[] text,
                       int offset,
                       int length)
                         throws IOException

Specified by:
  `writeUTF8String` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRaw

    public void writeRaw(String text)
                  throws IOException

Specified by:
  `writeRaw` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRaw

    public void writeRaw(String text,
                int offset,
                int len)
                  throws IOException

Specified by:
  `writeRaw` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRaw

    public void writeRaw(SerializableString text)
                  throws IOException

Overrides:
  `writeRaw` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRaw

    public void writeRaw(char[] text,
                int offset,
                int len)
                  throws IOException

Specified by:
  `writeRaw` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRaw

    public void writeRaw(char c)
                  throws IOException

Specified by:
  `writeRaw` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRawValue

    public void writeRawValue(String text)
                       throws IOException

Specified by:
  `writeRawValue` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRawValue

    public void writeRawValue(String text,
                     int offset,
                     int len)
                       throws IOException

Specified by:
  `writeRawValue` in class `JsonGenerator`
Throws:
  `IOException`

#### writeRawValue

    public void writeRawValue(char[] text,
                     int offset,
                     int len)
                       throws IOException

Specified by:
  `writeRawValue` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(short i)
                     throws IOException

Overrides:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(int i)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(long l)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(double d)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(float f)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(BigDecimal dec)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(BigInteger v)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNumber

    public void writeNumber(String encodedValue)
                     throws IOException

Specified by:
  `writeNumber` in class `JsonGenerator`
Throws:
  `IOException`

#### writeBoolean

    public void writeBoolean(boolean state)
                      throws IOException

Specified by:
  `writeBoolean` in class `JsonGenerator`
Throws:
  `IOException`

#### writeNull

    public void writeNull()
                   throws IOException

Specified by:
  `writeNull` in class `JsonGenerator`
Throws:
  `IOException`

#### writeObject

    public void writeObject(Object value)
                     throws IOException

Specified by:
  `writeObject` in class `JsonGenerator`
Throws:
  `IOException`

#### writeTree

    public void writeTree(TreeNode node)
                   throws IOException

Specified by:
  `writeTree` in class `JsonGenerator`
Throws:
  `IOException`

#### writeBinary

    public void writeBinary(Base64Variant b64variant,
                   byte[] data,
                   int offset,
                   int len)
                     throws IOException

Specified by:
  `writeBinary` in class `JsonGenerator`
Throws:
  `IOException`

#### writeBinary

    public int writeBinary(Base64Variant b64variant,
                  InputStream data,
                  int dataLength)

Although we could support this method, it does not necessarily make sense: we can not make good use of streaming because buffer must hold all the data. Because of this, currently this will simply throw `UnsupportedOperationException`

Specified by:
  `writeBinary` in class `JsonGenerator`

#### canWriteTypeId

    public boolean canWriteTypeId()

Overrides:
  `canWriteTypeId` in class `JsonGenerator`

#### canWriteObjectId

    public boolean canWriteObjectId()

Overrides:
  `canWriteObjectId` in class `JsonGenerator`

#### writeTypeId

    public void writeTypeId(Object id)

Overrides:
  `writeTypeId` in class `JsonGenerator`

#### writeObjectId

    public void writeObjectId(Object id)

Overrides:
  `writeObjectId` in class `JsonGenerator`

#### copyCurrentEvent

    public void copyCurrentEvent(JsonParser p)
                          throws IOException

Overrides:
  `copyCurrentEvent` in class `JsonGenerator`
Throws:
  `IOException`

#### copyCurrentStructure

    public void copyCurrentStructure(JsonParser jp)
                              throws IOException

Overrides:
  `copyCurrentStructure` in class `JsonGenerator`
Throws:
  `IOException`

#### _append

    protected final void _append(JsonToken type)

#### _append

    protected final void _append(JsonToken type,
               Object value)

#### _appendValue

    protected final void _appendValue(JsonToken type)

Similar to `_append(JsonToken)` but also updates context with knowledge that a scalar value was written

Since:
  2.6.4

#### _appendValue

    protected final void _appendValue(JsonToken type,
                    Object value)

Similar to `_append(JsonToken,Object)` but also updates context with knowledge that a scalar value was written

Since:
  2.6.4

#### _appendRaw

    protected final void _appendRaw(int rawType,
                  Object value)

#### _reportUnsupportedOperation

    protected void _reportUnsupportedOperation()

Overrides:
  `_reportUnsupportedOperation` in class `JsonGenerator`