:py:mod:`shards` ================ .. py:module:: conda._private.shards .. toctree:: :hidden: :titlesonly: :maxdepth: 3 cache/index.rst decompression/index.rst misc/index.rst shards/index.rst subset/index.rst typing/index.rst Classes ------- .. autoapisummary:: conda._private.shards.ShardBase conda._private.shards.ShardLike conda._private.shards.Shards Functions --------- .. autoapisummary:: conda._private.shards.fetch_channels conda._private.shards.fetch_shards_index .. py:class:: ShardBase Bases: :py:obj:`abc.ABC` Abstract base class for shard-like objects. Defines the common interface for both sharded repodata (Shards) and monolithic repodata presented as shards (ShardLike). .. py:attribute:: url :type: str .. py:attribute:: repodata_no_packages :type: conda._private.shards.typing.RepodataDict .. py:attribute:: visited :type: dict[str, conda._private.shards.typing.ShardDict | None] .. py:attribute:: _base_url :type: str .. py:property:: package_names :type: collections.abc.KeysView[str] :abstractmethod: Return the names of all packages available in this shard collection. .. py:property:: base_url :type: str Return self.url joined with base_url from repodata, or self.url if no base_url was present. Packages are found here. Note base_url can be a relative or an absolute url. Uses _safe_urljoin_with_slash to handle non-HTTP schemes (s3://, etc.). .. py:method:: __contains__(package: str) -> bool Check if a package is available in this shard collection. .. py:method:: shard_url(package: str) -> str :abstractmethod: Return shard URL for a given package. For monolithic repodata, should not be fetched but is a unique identifier. Raise KeyError if package is not in the index. .. py:method:: shard_loaded(package: str) -> bool :abstractmethod: Return True if the given package's shard is in memory. .. py:method:: visit_package(package: str) -> conda._private.shards.typing.ShardDict Return a shard that is already loaded in memory and mark as visited. .. py:method:: visit_shard(package: str, shard: conda._private.shards.typing.ShardDict) Store new shard data in the visited dict. .. py:method:: build_repodata() -> conda._private.shards.typing.RepodataDict Return monolithic repodata including all visited shards. Does not return "v3" repodata. Prefer iter_records_v3() over this method. .. py:method:: iter_records() -> collections.abc.Iterable[tuple[str, dict]] Yield (key, record) tuples for all packages in visited shards, including v3 records. For classic packages, *key* is the filename. For v3 packages, *key* is the shard key (not necessarily the download filename; see ``record["fn"]``). Prefer :meth:`iter_records_v3` when section information is needed. .. py:method:: iter_records_v3() -> collections.abc.Iterable[tuple[tuple[str, str], dict]] Yield ((key, section), record) tuples for all packages in visited shards. Section can be: "packages" for .tar.bz2 packages, "packages.conda" for .conda packages, "v3.whl", "v3.conda", "v3.tar.bz2" for v3 packages. key is the same as the filename for "packages", "packages.conda" but is different from the filename for v3 packages. .. py:class:: ShardLike(repodata: conda._private.shards.typing.RepodataDict, url: str = '') Bases: :py:obj:`ShardBase` Present a "classic" repodata.json as per-package shards. url: must be unique for all ShardLike used together. .. py:attribute:: repodata_no_packages :type: conda._private.shards.typing.RepodataDict .. py:attribute:: url :value: '' .. py:attribute:: shards :type: dict[str, conda._private.shards.typing.ShardDict] .. py:attribute:: visited :type: dict[str, conda._private.shards.typing.ShardDict | None] .. py:method:: __repr__() .. py:property:: package_names :type: collections.abc.KeysView[str] Return the names of all packages available in this shard collection. .. py:method:: shard_url(package: str) -> str Return shard URL for a given package. Raise KeyError if package is not in the index. .. py:method:: shard_loaded(package: str) -> bool Return True if the given package's shard is in memory. .. py:method:: visit_package(package: str) -> conda._private.shards.typing.ShardDict Return a shard that is already in memory and mark as visited. .. py:class:: Shards(shards_index: conda._private.shards.typing.ShardsIndexDict, url: str) Bases: :py:obj:`ShardBase` Handle repodata_shards.msgpack.zst and individual per-package shards. :param shards_index: raw parsed msgpack dict. Don't change it or base_url, :param shards_base_url will be wrong.: :param url: URL of repodata_shards.msgpack.zst .. py:attribute:: _shards_base_url :type: str .. py:attribute:: shards_index .. py:attribute:: url .. py:attribute:: _base_url .. py:attribute:: session .. py:attribute:: repodata_no_packages .. py:attribute:: visited :type: dict[str, conda._private.shards.typing.ShardDict | None] .. py:attribute:: _shard_url_cache :type: dict[str, str] .. py:property:: package_names Return the names of all packages available in this shard collection. .. py:property:: packages_index .. py:property:: shards_base_url :type: str Return self.url joined with shards_base_url. Note shards_base_url can be a relative or an absolute url. .. py:method:: shard_url(package: str) -> str Return shard URL for a given package. Raise KeyError if package is not in the index. .. py:method:: shard_loaded(package: str) -> bool Return True if the given package's shard is in memory. .. py:method:: visit_package(package: str) -> conda._private.shards.typing.ShardDict Return a shard that is already in memory and mark as visited. .. py:function:: fetch_channels(url_to_channel: dict[str, conda.models.channel.Channel], *, require_shards: bool = True) -> dict[str, ShardBase] | None :param url_to_channel: not modified, must already be expanded to subdirs. :param require_shards: Return None unless at least one channel provides shards. Attempt to fetch the sharded index first and then fall back to retrieving a monolithic `repodata.json` file. :returns: A dict mapping channel URLs to `Shard` or `ShardLike` objects. If `require_shards` is true, return None when no channels have shards. This dict preserves the key order of the input `url_to_channel`. .. py:function:: fetch_shards_index(sd: conda.core.subdir_data.SubdirData) -> Shards | None Check a SubdirData's URL for shards. Return shards index bytes from cache or network. Return None if not found; caller should fetch normal repodata. TODO: If this function fails to retrieve the sharded repodata index file, it will mark it is as not supporting this feature in cache. This can problematic because sometimes server errors can happen which will lead it to wrongly assuming the channel doesn't support sharding. We need to rethink our logic for determining shard support.