docutils.utils.code_analyzer.Lexer Class Reference

Public Member Functions
	__init__ (self, code, language, tokennames='short')
	merge (self, tokens)
	__iter__ (self)

Public Attributes
	code
	language
	tokennames
	lexer

Detailed Description

Parse `code` lines and yield "classified" tokens.

Arguments

  code       -- string of source code to parse,
  language   -- formal language the code is written in,
  tokennames -- either 'long', 'short', or 'none' (see below).

Merge subsequent tokens of the same token-type.

Iterating over an instance yields the tokens as ``(tokentype, value)``
tuples. The value of `tokennames` configures the naming of the tokentype:

  'long':  downcased full token type name,
  'short': short name defined by pygments.token.STANDARD_TYPES
           (= class argument used in pygments html output),
  'none':  skip lexical analysis.

Constructor & Destructor Documentation

__init__()

docutils.utils.code_analyzer.Lexer.__init__	(		self,
			code,
			language,
			tokennames = 'short'
	)

Set up a lexical analyzer for `code` in `language`.

Member Function Documentation

__iter__()

docutils.utils.code_analyzer.Lexer.__iter__

(

self

)

Parse self.code and yield "classified" tokens.

merge()

docutils.utils.code_analyzer.Lexer.merge	(		self,
			tokens
	)

Merge subsequent tokens of same token-type.

   Also strip the final newline (added by pygments).

---

The documentation for this class was generated from the following file:

• docs/help/help-venv/lib/python3.12/site-packages/docutils/utils/code_analyzer.py