shards#

Interface to sharded repodata code.

Classes#

RepodataSubset

Build a subset of repodata by traversing all packages that are dependencies

BuildRepodataSubset

Protocol for build_repodata_subset callable.

Functions#

build_repodata_subset(root_packages, channels[, ...]) → dict[str, conda._private.shards.shards.ShardBase] | None

Retrieve all necessary information to build a repodata subset.

class RepodataSubset(shardlikes: collections.abc.Iterable[conda._private.shards.shards.ShardBase], spec_to_package_name: collections.abc.Callable[[str], str] = spec_to_package_name, repodata_version: int = 1, depth: int = sys.maxsize)#

Build a subset of repodata by traversing all packages that are dependencies and transitive dependencies of a root set of packages.

shardlikes: collections.abc.Sequence[conda._private.shards.shards.ShardBase]#
DEFAULT_STRATEGY = 'pipelined'#
_nodes: dict[NodeId, Node]#
_use_only_tar_bz2: bool#
_add_pip_as_python_dependency: bool#
_spec_to_package_name: collections.abc.Callable[[str], str]#
_repodata_version = 1#
depth = 9223372036854775807#
property node_count: int#

Number of (channel, package) nodes discovered during traversal.

classmethod has_strategy(strategy: str) → bool#

Return True if this class provides the named shard traversal strategy.

_expansion_specs(node: Node, shard: conda._private.shards.typing.ShardDict) → collections.abc.Iterator[tuple[str, frozenset[str]]]#

Yield (package name, extras requested by the spec) for everything reachable from shard that has not been traversed for node yet.

On the first expansion that is the shard's depends plus every group in node.requested_extras. When the node is revisited because another dependency activated additional groups, only those new groups are traversed. Records the groups as expanded on the node.

_neighbors(node: Node) → collections.abc.Iterator[Node]#

Retrieve all unvisited neighbors of a node, plus already visited neighbors that need to be revisited because this node's dependency specs activate additional extras groups on them.

Neighbors in the context are dependencies of a package.

NOTE: This method assumes that the required shards have already been retrieved from the network via batch_retrieve_from_network() before neighbors() is called. It uses visit_package() to access already-loaded shards.

_outgoing(node: Node)#

All nodes that can be reached by this node, plus cost.

reachable(root_packages, *, strategy=DEFAULT_STRATEGY, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None) → None#

Run named reachability strategy or the default.

Update self.shardlikes with reachable package records. Later, [shardlike.build_repodata() for shardlike in shardlikes] can be used to generate repodata.json-format subsets of each channel.

Parameters:
  • root_packages -- installed and requested package names.

  • strategy -- traversal algorithm, "bfs" or "pipelined".

  • root_extras -- optional mapping of root package name to an iterable of CEP 44 extras requested for it (e.g. {"package": ["extras_group"]}). Extras explicitly set on dependency specs of fetched packages (e.g. pkg-u[extras=more]) are followed transitively; groups that nobody requested stay inactive.

reachable_bfs(root_packages, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None)#

Fetch all packages reachable from root_packages' by following dependencies using the "breadth-first search" algorithm.

Update associated self.shardlikes to contain enough data to build a repodata subset.

_reachable_bfs(root_packages, shard_cache: conda._private.shards.cache.ShardCache, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None)#

Inner reachable_bfs() implementation.

reachable_pipelined(root_packages, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None)#

Fetch all packages reachable from root_packages' by following dependencies.

Build repodata subset using concurrent threads to follow dependencies, fetch from cache, and fetch from network.

_reachable_pipelined(root_packages, network_worker: collections.abc.Callable[[queue.SimpleQueue[collections.abc.Sequence[NodeId] | None], queue.SimpleQueue[list[tuple[NodeId, conda._private.shards.typing.ShardDict] | Exception]], RepodataSubset._reachable_pipelined.cache, collections.abc.Sequence[conda._private.shards.shards.ShardBase]], None], cache: RepodataSubset._reachable_pipelined.cache, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None)#

Set up queues and threads for shard traversal with a configurable network_worker. Called by reachable_pipelined()

_pipelined_traversal(root_packages, cache_in_queue: queue.SimpleQueue[list[NodeId] | None], shard_out_queue: queue.SimpleQueue[list[tuple[NodeId, conda._private.shards.typing.ShardDict]] | Exception], cache_thread: threading.Thread, network_thread: threading.Thread, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None)#

Run reachability algorithm given queues to submit and receive shards.

_visit_node(parent_node: Node, mentioned_packages: collections.abc.Iterable[tuple[str, frozenset[str]]], extras_by_package: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None) → collections.abc.Iterable[NodeId]#

Broadcast mentioned packages across channels. yield pending NodeId's.

mentioned_packages are (name, extras) pairs; extras are the CEP 44 groups requested for that package by the spec that mentions it. A package that already has a node only yields again if the extras add groups to a node whose shard was already processed.

_drain_pending(pending: set[NodeId], shardlikes_by_url: dict[str, conda._private.shards.shards.ShardBase]) → tuple[list[tuple[NodeId, conda._private.shards.typing.ShardDict]], list[NodeId]]#

Check pending for in-memory shards. Clear pending.

Return a list of shards we have and shards we need to fetch.

build_repodata_subset(root_packages: collections.abc.Iterable[str], channels: dict[str, conda.models.channel.Channel], algorithm: Literal['bfs', 'pipelined'] = RepodataSubset.DEFAULT_STRATEGY, spec_to_package_name_func: collections.abc.Callable[[str], str] = spec_to_package_name, repodata_version: int = 1, depth: int = sys.maxsize, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None) → dict[str, conda._private.shards.shards.ShardBase] | None#

Retrieve all necessary information to build a repodata subset.

This function implements the conda.gateways.shards.BuildRepodataSubset protocol, allowing it to be passed to solvers that support sharded repodata optimization.

Params:

root_packages: iterable of installed and requested package names channels: Channel objects; dict form preferred. algorithm: desired traversal algorithm ("bfs" or "pipelined") spec_to_package_name_func: callable to convert package specs to names.

Defaults to the standard spec_to_package_name.

repodata_version: repodata format version (1 = classic, 3 = v3). depth: the maximum depth of dependant packages to include in the repodata

subset.

root_extras: optional mapping of root package name to an iterable of

CEP 44 extras requested for it (e.g. {"package": ["extras_group"]}). Extras on dependency specs are followed transitively; see RepodataSubset.reachable.

Returns:

None if there are no shards available, or a mapping of channel URL's to ShardBase objects where build_repodata() returns the computed subset.

class BuildRepodataSubset#

Bases: Protocol

Protocol for build_repodata_subset callable.

This function is used by solvers to construct a minimal subset of repodata based on the root packages that might be installed and the available channels. It traverses package dependencies to discover all reachable (channel, package) tuples, which are then used by the solver to reduce search space.

__call__(root_packages: collections.abc.Iterable[str], channels: dict[str, Any], algorithm: Literal['bfs', 'pipelined'] = 'pipelined', repodata_version: int = 1, root_extras: collections.abc.Mapping[str, collections.abc.Iterable[str]] | None = None) → dict[str, Shards] | None#

Retrieve a minimal subset of repodata based on root packages.

Parameters:
  • root_packages -- Iterable of installed and requested package names

  • channels -- Dictionary mapping channel URLs to Channel objects

  • algorithm -- Traversal algorithm to use ("bfs" or "pipelined")

  • repodata_version -- repodata format version (1 = classic, 3 = v3).

  • root_extras -- Optional mapping of root package name to an iterable of CEP 44 extras requested for it (e.g. {"httpx": ["cli"]}), so that shards for its requested extra_depends groups are also fetched. Extras explicitly requested by dependency specs of fetched packages are followed transitively.

Returns:

A dictionary mapping channel URLs to Shards objects containing the subset of packages needed, or None if shards are unavailable