pip._vendor.distlib.locators.SimpleScrapingLocator Class Reference

Inheritance diagram for pip._vendor.distlib.locators.SimpleScrapingLocator:

Collaboration diagram for pip._vendor.distlib.locators.SimpleScrapingLocator:

Public Member Functions
	__init__ (self, url, timeout=None, num_workers=10, **kwargs)
	get_page (self, url)
	get_distribution_names (self)
Public Member Functions inherited from pip._vendor.distlib.locators.Locator
	get_errors (self)
	clear_errors (self)
	clear_cache (self)
	get_project (self, name)
	score_url (self, url)
	prefer_url (self, url1, url2)
	split_filename (self, filename, project_name)
	convert_url_to_download_info (self, url, project_name)
	locate (self, requirement, prereleases=False)

Public Attributes
	base_url
	timeout
	skip_externals
	num_workers
	platform_check
	result
	project_name
	excluded_extensions
Public Attributes inherited from pip._vendor.distlib.locators.Locator
	opener
	matcher
	errors
	downloadable_extensions

Static Public Attributes
dict	decoders
	platform_dependent
Static Public Attributes inherited from pip._vendor.distlib.locators.Locator
tuple	source_extensions = ('.tar.gz', '.tar.bz2', '.tar', '.zip', '.tgz', '.tbz')
tuple	binary_extensions = ('.egg', '.exe', '.whl')
tuple	excluded_extensions = ('.pdf',)
	wheel_tags = None
tuple	downloadable_extensions = source_extensions + ('.whl',)

Protected Member Functions
	_prepare_threads (self)
	_wait_threads (self)
	_get_project (self, name)
	_is_platform_dependent (self, url)
	_process_download (self, url)
	_should_queue (self, link, referrer, rel)
	_fetch (self)
Protected Member Functions inherited from pip._vendor.distlib.locators.Locator
	_get_scheme (self)
	_set_scheme (self, value)
	_get_digest (self, info)
	_update_version_data (self, result, info)

Protected Attributes
	_page_cache
	_seen
	_to_fetch
	_bad_hosts
	_lock
	_gplock
	_threads
Protected Attributes inherited from pip._vendor.distlib.locators.Locator
	_cache
	_scheme

Static Protected Attributes
	_distname_re = re.compile('<a href=[^>]*>([^<]+)<')

Additional Inherited Members
Properties inherited from pip._vendor.distlib.locators.Locator
	scheme = property(_get_scheme, _set_scheme)

Detailed Description

A locator which scrapes HTML pages to locate downloads for a distribution.
This runs multiple threads to do the I/O; performance is at least as good
as pip's PackageFinder, which works in an analogous fashion.

Constructor & Destructor Documentation

__init__()

pip._vendor.distlib.locators.SimpleScrapingLocator.__init__	(		self,
			url,
			timeout = None,
			num_workers = 10,
		**	kwargs
	)

Initialise an instance.
:param url: The root URL to use for scraping.
:param timeout: The timeout, in seconds, to be applied to requests.
                This defaults to ``None`` (no timeout specified).
:param num_workers: The number of worker threads you want to do I/O,
                    This defaults to 10.
:param kwargs: Passed to the superclass.

Reimplemented from pip._vendor.distlib.locators.Locator.

Member Function Documentation

_fetch()

pip._vendor.distlib.locators.SimpleScrapingLocator._fetch (   self )

protected

Get a URL to fetch from the work queue, get the HTML page, examine its
links for download candidates and candidates for further scraping.

This is a handy method to run in a thread.

_get_project()

pip._vendor.distlib.locators.SimpleScrapingLocator._get_project (   self,   name  )

protected

For a given project, get a dictionary mapping available versions to Distribution
instances.

This should be implemented in subclasses.

If called from a locate() request, self.matcher will be set to a
matcher for the requirement to satisfy, otherwise it will be None.

Reimplemented from pip._vendor.distlib.locators.Locator.

_is_platform_dependent()

pip._vendor.distlib.locators.SimpleScrapingLocator._is_platform_dependent (   self,   url  )

protected

Does an URL refer to a platform-specific download?

_prepare_threads()

pip._vendor.distlib.locators.SimpleScrapingLocator._prepare_threads (   self )

protected

Threads are created only when get_project is called, and terminate
before it returns. They are there primarily to parallelise I/O (i.e.
fetching web pages).

_process_download()

pip._vendor.distlib.locators.SimpleScrapingLocator._process_download (   self,   url  )

protected

See if an URL is a suitable download for a project.

If it is, register information in the result dictionary (for
_get_project) about the specific version it's for.

Note that the return value isn't actually used other than as a boolean
value.

_should_queue()

pip._vendor.distlib.locators.SimpleScrapingLocator._should_queue (   self,   link,   referrer,   rel  )

protected

Determine whether a link URL from a referring page and with a
particular "rel" attribute should be queued for scraping.

_wait_threads()

pip._vendor.distlib.locators.SimpleScrapingLocator._wait_threads (   self )

protected

Tell all the threads to terminate (by sending a sentinel value) and
wait for them to do so.

get_distribution_names()

pip._vendor.distlib.locators.SimpleScrapingLocator.get_distribution_names

(

self

)

Return all the distribution names known to this locator.

Reimplemented from pip._vendor.distlib.locators.Locator.

get_page()

pip._vendor.distlib.locators.SimpleScrapingLocator.get_page	(		self,
			url
	)

Get the HTML for an URL, possibly from an in-memory cache.

XXX TODO Note: this cache is never actually cleared. It's assumed that
the data won't get stale over the lifetime of a locator instance (not
necessarily true for the default_locator).

Member Data Documentation

decoders

dict pip._vendor.distlib.locators.SimpleScrapingLocator.decoders

static

Initial value:

=  {
        'deflate': zlib.decompress,
        'gzip': lambda b: gzip.GzipFile(fileobj=BytesIO(b)).read(),
        'none': lambda b: b,
    }

platform_dependent

pip._vendor.distlib.locators.SimpleScrapingLocator.platform_dependent

static

Initial value:

=  re.compile(r'\b(linux_(i\d86|x86_64|arm\w+)|'
                                    r'win(32|_amd64)|macosx_?\d+)\b', re.I)

---

The documentation for this class was generated from the following file:

• docs/help/help-venv/lib/python3.12/site-packages/pip/_vendor/distlib/locators.py