Graphviz: lib/cgraph/tokenize.h File Reference

Searching...

No Matches

String tokenization. More...

#include <assert.h>
#include <cgraph/strview.h>
#include <stddef.h>
#include <string.h>

Include dependency graph for tokenize.h:

This graph shows which files directly or indirectly include this file:

Data Structures
struct	tok_t
	state for an in-progress string tokenization More...

Functions
static tok_t	tok (const char input, const char separators)
	begin tokenization of a new string

static bool	tok_end (const tok_t *t)
	is this tokenizer exhausted?

static strview_t	tok_get (const tok_t *t)
	get the current token

static void	tok_next (tok_t *t)
	advance to the next token in the string being scanned

Detailed Description

This is essentially equivalent to strtok but with two main improvements:

The input string is not modified. This means, if you have a const string, you do not need to strdup it in order to tokenize it. This (combined with other properties like no opaque struct pointers) enables you to tokenize a string with no heap allocation.
No global state. All the state for tokenization is contained in the tok_t struct.

The above two properties are intended to make string tokenization scalable (no locks, no thread-shared state) and transparent to the compiler (a good optimizing compiler implements all the string.h functions we use as built-ins and, if separators is a compile-time literal, can typically flatten everything into a tight loop with no function calls).

Sample usage:

const char my_input[] = "foo; bar:/baz"; for (tok_t t = tok(my_input, ";:/"); !tok_end(&t); tok_next(&t)) { strview_t s = tok_get(&t); printf("%.*s\n", (int)s.size, s.data); } // prints “foo”, “ bar”, “baz”

Definition in file tokenize.h.