jsonv::tokenizer Class Reference

Splits input into tokens, allowing traversal of JSON without verification. More...

#include <jsonv/tokenizer.hpp>

Classes
struct	token
	A representation of what this tokenizer has. More...

Public Types
using	size_type = std::vector< char >::size_type

Public Member Functions
	tokenizer (string_view input)
	Construct a tokenizer to read the given non-owned input.
	tokenizer (std::istream &input)
	Construct a tokenizer from the provided input.
const string_view &	input () const
	Get the input this instance is reading from.
bool	next ()
	Attempt to go to the next token in the input stream. More...
const token &	current () const
	Get the current token and its associated token_kind. More...
void	buffer_reserve (size_type sz)

Static Public Member Functions
static size_type	min_buffer_size ()
static void	set_min_buffer_size (size_type sz)

Detailed Description

Splits input into tokens, allowing traversal of JSON without verification.

This is the basis for JSON parsers.

An important thing to remember is a tokenizer does not perform any real validation of any kind beyond emitting a token_kind::unknown when it encounters complete garbage. What does this mean? Given the input string:

[{]{{{{{{}}{{]]]]][][]]][[[[]]]"fdsadf"]]]}}}}}}}}]]]

A tokenizer will emit token_kind::array_begin, token_kind::object_begin, token_kind::array_end and so on, even though it is illegal JSON. It is up to a higher-level construct to detect such failures.

Examples:

parse(tokenizer&, const parse_options&).

Definition at line 99 of file tokenizer.hpp.

Member Function Documentation

void jsonv::tokenizer::buffer_reserve

(

size_type

sz

)

Deprecated:

Calling this function has no effect and will be removed in 2.0.

const token& jsonv::tokenizer::current

(

)

const

Get the current token and its associated token_kind.

Returns

The current token.

Exceptions

std::logic_error

if next has not been called or if it returned false.

static size_type jsonv::tokenizer::min_buffer_size ( )

static

Deprecated:

See buffer_reserve.

bool jsonv::tokenizer::next

(

)

Attempt to go to the next token in the input stream.

The contents of current will be cleared.

Returns

true if another token was obtained; false if we reached EOF or an I/O failure. Check input to see which.

static void jsonv::tokenizer::set_min_buffer_size ( size_type  sz )

static

Deprecated:

See buffer_reserve.

---

The documentation for this class was generated from the following file:

• jsonv/tokenizer.hpp