packaging._tokenizer.Tokenizer Class Reference

Public Member Functions
None	__init__ (self, str source, *dict[str, str|re.Pattern[str]] rules)
None	consume (self, str name)
bool	check (self, str name, *bool peek=False)
Token	expect (self, str name, *str expected)
Token	read (self)
NoReturn	raise_syntax_error (self, str message, *int|None span_start=None, int|None span_end=None)
Iterator[None]	enclosing_tokens (self, str open_token, str close_token, *str around)

Public Attributes
	source
	position
	rules
	next_token

Detailed Description

Context-sensitive token parsing.

Provides methods to examine the input stream to check whether the next token
matches.

Member Function Documentation

check()

bool packaging._tokenizer.Tokenizer.check	(		self,
		str	name,
		*bool	peek = False
	)

Check whether the next token has the provided name.

By default, if the check succeeds, the token *must* be read before
another check. If `peek` is set to `True`, the token is not loaded and
would need to be checked again.

consume()

None packaging._tokenizer.Tokenizer.consume	(		self,
		str	name
	)

Move beyond provided token name, if at current position.

expect()

Token packaging._tokenizer.Tokenizer.expect	(		self,
		str	name,
		*str	expected
	)

Expect a certain token name next, failing with a syntax error otherwise.

The token is *not* read.

raise_syntax_error()

NoReturn packaging._tokenizer.Tokenizer.raise_syntax_error	(		self,
		str	message,
		*int | None	span_start = None,
		int | None	span_end = None
	)

Raise ParserSyntaxError at the given position.

read()

Token packaging._tokenizer.Tokenizer.read

(

self

)

Consume the next token and return it.

---

The documentation for this class was generated from the following file:

• docs/help/help-venv/lib/python3.12/site-packages/packaging/_tokenizer.py