Iteration#

This package contains the trees used for data iteration.

The DataTree stores an actual data point, which consists of nested dict and list objects, with DataLiteral leaves.

Then, the IterationTree provides a framework to build complex searches over data trees. Its leaves consist of literal values, or data iterators. Those leaves can simply be iterated over, but they can also be used to build more complex iteration trees.

First, they can be combined with IterationMethod nodes, which provide a way to iterate over multiple data sources. Then, Transform nodes can be inserted in those trees in order to modify the data trees generated while iterating.

class phileas.iteration.Accumulator(child: IterationTree, recursive: bool = False, start_value: dict[phileas.iteration.base.Key, phileas.iteration.base.DataTree] | None = None)[source]#

Transform node that accumulates its inputs, as a kind of unlazifying transform:

if its successive inputs are dictionaries, merge them using the union operator (recursively or not), and return the results;
otherwise, leave its inputs untouched.

recursive: bool = False#: Specify if the accumulation must be done recursively or not. For example, accumulating values {"a": 1, "b": {"ba": 1}} and {"a": 2, "b": {"bb": 2}} recursively will return {"a": 2, "b": {ba": 1, "bb": 2}}, whereas doing it non-recursively will return {"a": 2, "b": {"bb": 2}}.

start_value: dict[Key, DataTree] | None = None#: Start value of the accumulator, which must either be a dictionary, or None.

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.CartesianProduct(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, snake: bool = False)[source]#

Iteration over the cartesian product of the children. The iteration order is the same as itertools.product(). In other words, iteration will behave roughly as

for v1 in c1:
    for v2 in c2:
        for v3 in c3:
            yield [v1, v2, v3]

If an order is specified, its first element will correspond to the outermost loop, and its last to the innermost one.

snake: bool = False#: Enable snake iteration, which guarantees that successive yielded elements differ by only one key at most (a la Gray code).

class phileas.iteration.Configurations(children: dict[Key, IterationTree], order: list[Key] | None = None, lazy: bool = False, default_configuration: Key | None = None, move_up: bool = False, insert_name: bool = False)[source]#

Represents a set of named configurations that can be invoked using IterationTree.get_configuration(). When it is called, with argument name, all the Configurations nodes that have a matching configuration will be replaced by it. Other ones will be replaced by their default value.

This allows to escape from the recursive and local nature of trees: a single call can modify nodes throughout a whole tree. However, it is often convenient to convert configurable trees back to classical, non-configurable trees. This is done by IterationTree.unroll_configurations().

The Configurations node holds a set of iteration trees, called configurations, which are identified by their name. Two parameters, move_up and insert_name, control how IterationTree.get_configuration() behaves.

By default, move_up == insert_name == False. The Configuration node is simply replaced by the content of the requested configuration.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     })
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': {'param1': '1-1', 'param2': '1-2'}}

If move_up == True, the content of the chosen configuration is moved one level up, so that it is at the same level as the Configurations node. This requires it to have an IterationMethod parent with dict children. This can be used to factorize configurations.

>>> tree = CartesianProduct({
...     "_": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...         })
...     }, move_up=True),
...     "param2": IterationLiteral(value="2")
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'param1': '1-1', 'param2': '2'}

If insert_name == True, the name of the requested configuration is inserted in the resulting tree. If the requested configuration allows it (_ie_. it is an IterationMethod with dict children), the name is inserted into itself, with the key _configuration.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     }, insert_name=True)
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': {'_configuration': 'config1', 'param1': '1-1', 'param2': '1-2'}}

If, additionally, move_up == True, the name of the configuration is inserted instead of the Configurations node.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     }, insert_name=True, move_up=True)
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': 'config1', 'param1': '1-1', 'param2': '2'}

However, if the requested configuration is not an IterationMethod, its name is inserted as a sibling, assigned to the key f"_{name}_configuration".

>>> tree = CartesianProduct({
...     "param": Configurations({
...         "config1": IterationLiteral(value="1"),
...         "config2": IterationLiteral(value="2"),
...     }, insert_name=True),
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'_param_configuration': 'config1', 'param': '1'}

This means that the content of the configurations affects how they are handled. Although this might change in the future, it is made to support most situations. It is recommended to keep all the configurations with the same shape.

insert_name is not necessary. It is possible to replace it with the following kind of tree:

>>> tree = CartesianProduct({
...     "param": Configurations({
...         "config1": IterationLiteral(value="1"),
...         "config2": IterationLiteral(value="2"),
...     },
...     "config": Configurations({
...         "config1": IterationLiteral(value="config1"),
...         "config2": IterationLiteral(value="config2"),
...     })),
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'param': '1', 'config': 'config1'}

children: dict[Key, IterationTree]#: The children of the node. It must not be empty.

default_configuration: Key | None = None#: Key of the default configuration, which must be in the set of keys of children.

move_up: bool = False#

If set, the content of a requested configuration is moved up, at the parent level of the current node. In other words, it becomes a sibling of the current Configurations node.

Otherwise, the content is inserted at the same level as the configurations themselves.

insert_name: bool = False#: If set, insert the name of the requested configuration when calling get_configuration(). If move_up, then the Configurations node is replaced by this name. Otherwise, a "_configuration" sibling node is inserted.

class phileas.iteration.First(child: IterationTree, size: int | None)[source]#

Return only the first elements of its child.

size: int | None#: Number of elements to keep. If it is None, or is bigger than the child size, the same number of elements as the child are iterated over.

class phileas.iteration.FunctionalTranform(child: IterationTree, f: Callable[[phileas.iteration.base.DataTree], phileas.iteration.base.DataTree])[source]#

Transform node using its function attribute to modify its child.

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

Wrapper around a generator function, which can be used in order not to have to implement a new iteration leave, and its iterator. Note that only continuous forward iteration is supported by the node.

size: int | None = None#: Size of the tree. If the generator can provide more elements, only the first size ones are returned. If it cannot generate enough, a StopIteration is raised during iteration. None represents an infinite generator.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.GeometricRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, steps: int = 2)[source]#: Generate steps values geometrically spaced between start and end, both included.

exception phileas.iteration.InfiniteLength[source]#: Exception raised when requesting the length of an infinite collection.

class phileas.iteration.IntegerRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, step: int = 1)[source]#: Generate integer values step spaced, between start and end, both included. start must be an int, but end can also be math.inf or -math.inf. In these cases, the range is infinite.

class phileas.iteration.IterationLeaf[source]#

class phileas.iteration.IterationLiteral(value: DT)[source]#

Wrapper around a data tree.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.IterationMethod(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False)[source]#

Iteration node having multiple children, supplied either as a list or dictionary.

In order to implement a concrete iteration method, you should sub-class IterationMethod and implement a corresponding IterationMethodIterator, which is returned by _iter().

This should remain the only node in an iteration tree that can hold dict and list children. If you are tempted to create another node doing so, you should verify that it cannot be done by sub-classing IterationMethod instead.

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

order: list[Key] | None = None#: Order of iteration over the children. How it is used depends on the concrete iteration method implementation. It must be a permutation of the set of keys of children.

lazy: bool = False#

Notify the iteration method to be lazy. For now, this feature is only supported for dict children. In this case, lazy iteration will just yield the keys that have changed at each step.

Note that concrete iteration method classes are not required to actually implement lazy iteration, and if they don’t, they will probably do so silently. Refer to their documentation or their implementation.

to_pseudo_data_tree() → phileas.iteration.base.PseudoDataTree[source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.IterationTree[source]#

Represents a set of data trees, as well as the way to iterate over them. In order to be able to get a single data tree from an iteration tree, they are able to build a default data tree, which (usually) has the same shape as the generated data tree.

An iteration tree cannot be modified, as it is a frozen dataclass. Instead, a new one must be created from this one.

configurations: frozenset[Key]#: Names of the available configurations.

get_configuration(key: phileas.iteration.base.Key) → IterationTree[source]#: Returns a given configuration of the tree, if it exists. Otherwise, raises a KeyError. See Configurations node for more details on the behavior of this method.

unroll_configurations() → IterationTree[source]#

Transforms a configurable tree, ie. one with configurations != {}, to a non-configurable tree. It requests all the available configurations, and gather them into a Union node with reset=False. A MoveUpTransform is used to get rid of the name of the configurations.

If the tree is not configurable, return it.

safe_len() → int | None[source]#: Return the number of data trees represented by the iteration tree. If it is finite, it is the same as the number of elements yielded by __iter__(). Otherwise, return None.

iterate() → TreeIterator[source]#: Implementation of __len__() which assumes that self does not have any configuration.

abstract to_pseudo_data_tree() → phileas.iteration.base.PseudoDataTree[source]#: Converts the iteration tree to a pseudo data tree.

default(no_default_policy: NoDefaultPolicy = NoDefaultPolicy.ERROR) → DataTree | _NoDefault[source]#: Returns a default data tree. If the tree does not have a default value, follows the behavior dictated by no_default_policy.

with_params(path: list[bool | str | int | float | _Child] | None = None, **kwargs) → Self[source]#: Returns a similar iteration tree, where the node at path is assigned the given keyword parameters. If path is not specified, modifies the root of the tree directly.

get(path: list[bool | str | int | float | _Child]) → IterationTree[source]#: Get a node inside a tree. It should not be used to modify the tree.

Insert a child anywhere in the tree, whose location is specified by path argument. Return the newly created tree.

If there is already a node at this location, it will be replaced.

If the specified tree is None, a node is supposed to exist at this location (otherwise, a KeyError is raised), and will be removed if possible. Only iteration methods nodes will support child removal, see their implementation of _remove_child(). In any other case, a TypeError is raised.

Note that the root of a tree cannot be removed, so specifying an empty path with a None tree will raise a KeyError.

remove_child(path: list[bool | str | int | float | _Child]) → IterationTree[source]#: Remove a node in a tree. It is equivalent to insert_child(path, None).

insert_unary(path: ChildPath, Parent: type[UnaryNode], *args, **kwargs) → IterationTree[source]#

Insert a parent to the node at the given path, parent which is necessarily a transform node, as it will only have a single child. The parent is built using the Parent class, and the supplied arguments.

The newly created tree is returned.

replace_node(path: list[bool | str | int | float | _Child], Node: type[IterationTree], *args, **kwargs) → IterationTree[source]#

Replace the node at the given path with another one. The other node is built using its type, Node, and the args and kwargs arguments. Note that the sub-tree of the replaced node is not modified.

This requires Node to be of the same kind of the node that is being replaced: a transform for a transform, an iteration method for an iteration method, a leaf for a leaf.

depth_first_modify(modifier: Callable[[IterationTree, list[bool | str | int | float | _Child]], IterationTree]) → IterationTree[source]#: Using a post-fix depth-first search, replace each node of the tree, located at path, with modifier(node, path).

class phileas.iteration.Lazify(child: IterationTree)[source]#

Transform node that only returns the elements that were updated in its children values (for dictionary values), or leaves its inputs untouched (for other types).

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.LinearRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, steps: int = 2)[source]#: Generate steps values linearly spaced between start and end, both included.

Indicates that IterationTree.default() has been called on an iteration tree where a node does not have a default value.

path: list[bool | str | int | float | _Child]#: Path of a child without a default value.

class phileas.iteration.NoDefaultPolicy(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Behavior of IterationTree.default() for trees not having a default value.

ERROR = 'ERROR'#: Raise a NoDefaultError if any of the nodes in the tree does not have a default value.

SENTINEL = 'SENTINEL'#: Return the _NoDefault sentinel if any of the nodes in the tree does not have a default value.

SKIP = 'SKIP'#

Skip elements without a default element. If the root of the tree does not have a default value, return a _NoDefault sentinel.

Note that this is not supported by iteration method nodes with list children.

FIRST_ELEMENT = 'FIRST_ELEMENT'#: Return the first element of iteration leaves without a default value.

class phileas.iteration.NumericRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>)[source]#

Represents a range of numeric values.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

Random iteration leaf based on the RNG of numpy.

Note:

Iteration is based on NumpyRNGIterator. It works by seeding a PRNG with a value which depends on the requested iteration position. This might introduce a bias on the generated random values. This is done as a way to provide a random-access PRNG, although an actual random-access PRNG would be preferable.

Alternatively, the iterator could be built from a usual iterative PRNG, and caching would be used to provide access to previous values. This would impact iteration performances (especially for cache misses, which can induce unbound time delays).

distribution(size=None, dtype=<class 'numpy.float64'>, out=None)#: Which distribution to use for the node. It must be a distribution method of numpy.random.Generator.

args: list#: Arguments list to pass to the distribution.

kwargs: dict[str, Any]#: Keyword arguments to pass to the distribution.

class phileas.iteration.Pick(children: list[IterationTree] | dict[Key, IterationTree], order: list[Key] | None = None, lazy: bool = False, seed: Seed | None = None, default_child: Key | None = None)[source]#

Randomly pick and return one child at a time. For finite trees, it behaves in a way similar to composing Shuffle and Union (reset=False) nodes. It can additionally handle infinite children, whereas Shuffle cannot.

default_child: Key | None = None#: Key of the child to use as a default value.

Random iteration leaf generating prime numbers.

Generation is done in two steps:

a uniform integer is uniformly drawn from the interval [low, high];
sympy nextprime() and prevprime() are used to find the closest prime.

high: int = 0#: Lower generation bound, inclusive.

low: int = 255#: Upper generation bound, inclusive.

Deterministic pseudo-random elements generator.

size: None | int = None#: Number of elements generated by the leaf. If None, the leaf is infinite.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

Seed of a random iteration node, used by its RNG.

path: list[bool | str | int | float | _Child]#: Path of the node in the biggest tree that used it.

salt: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree]#: Salt value, to customize iteration independently of the shape of the iteration tree.

to_bytes() → bytes[source]#: Convert the seed to bytes, for RNG seeding.

Non-empty sequence of data trees.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.Shuffle(child: IterationTree, seed: Seed | None = None)[source]#

Unary node that shuffles the order of its child. In other words, it iterates over a random permutation of its children values.

Shuffling has a constant memory cost. The permutation of the order of the child tree is obtained through the use of a cypher, which means that it is entirely represented by a short key.

To shuffle a tree with size \(n\), a symmetric cypher on \(\{ 0, \dots , n-1 \}\) is used. It is built with two components:

a block cypher working on \(2 \lceil \log_2 n / 2 \rceil\)-bit words. It consists in a 3-round Feistel network which uses SipHash 1-3 as a round function.
Cycle walking is used to restrict the message space to \(\{ 0, \dots, n - 1 \}\). The block cypher is iterated until its output is valid.

class phileas.iteration.Transform(child: IterationTree)[source]#

Node that modifies the data trees generated by its child during iteration.

If you want to transform a list or dict of iteration trees, you should wrap them in an IterationMethod object first.

abstract transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.TreeIterator(tree: T)[source]#

Iteration tree iterator.

Compared to a usual iterator, it supports forward and backward iteration, and is endlessly usable. This means that, whenever it is “exhausted” (__next__() raises StopIteration), it can either be reset to its starting position with reset(), or its iteration direction can be switched using reverse().

Additionally, it supports random access with the __getitem__() method, which uses update() under the hood.

position: int#

Position of the last value that was yielded. Valid values start at -1 (backward-exhausted iterator, or forward iteration start), and go up to size (forward-exhausted iterator, or backward iteration start).

It can be directly modified by update().

tree: T#: Reference to the tree being iterated over.

size: int | None#: Cached size of the iterated tree. Consider using this instead of len (self.tree), as computing the length of a tree can be an expensive operation.

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

is_forward() → bool[source]#: Returns whether the iterator is going forward or not.

reverse()[source]#

Reverse the iteration direction of the iterator, but stay at the same position. Thus, if it is any TreeIterator, the following behavior is expected:

>>> it.reset()
>>> it.reverse()
>>> list(it)
[]

If you want to iterate over an iteration tree tree backward, you have to reset the iterator after having reversed it:

>>> it = iter(tree)
>>> it.reverse()
>>> it.reset()

update(position: int)[source]#

Update the position of the iterator to any supported position. This includes the positions ranging from 0 to self.size - 1, included, as well as - -1, which represents a reset forward iterator and - self.size, which represents a reset backward iterator.

If an invalid position is requested, an IndexError is raised, and the state of the iterator remains unchanged.

Random iteration leaf generating arbitrarily big integers. It works by iteratively repeatedly using a Numpy RNG.

By default returns a byte (integer larger or equal to 0 and smaller than 256).

low: int = 0#: The inclusively lowest value that may be taken by the output.

high: int = 255#: The inclusively highest value that may be taken by the output.

class phileas.iteration.Union(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, preset: Literal['first'] | Literal['default'] | None = 'first', common_preset: bool = False, reset: Literal['first'] | Literal['last'] | Literal['default'] | None = 'first')[source]#

Iteration over one child at a time, starting with the first one (or the first one of the order, if specified). Children that are not being iterated over have

their default value if it exists and
their first value otherwise.

preset: Literal['first'] | Literal['default'] | None = 'first'#: Defines which value to use before starting the iteration over a child. “first” uses the child’s first value and “default” its default value. None does not set the children values before iteration. Thus, it is only applicable to dict children.

common_preset: bool = False#: You can only set it when preset == "first". Then, children are all set to their first value at the first iteration. When iterating over them, they start directly at their second value.

reset: Literal['first'] | Literal['last'] | Literal['default'] | None = 'first'#: Defines which value to use after ending the iteration over a child. “first” resets it to its starting value, “last” leaves it unchanged, and “default” resets it to its default value. None does not reset the child after iteration. Thus, it is only applicable to dict children.

class phileas.iteration.Zip(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, stops_at: Literal['longest', 'shortest'] = 'shortest', ignore_fixed: bool = True)[source]#

Iteration over all of the children of the nodes at the same time, in a way similar to zip().

stops_at: Literal['longest'] | Literal['shortest'] = 'shortest'#: If shortest, iteration stops whenever the first child is exhausted. If longest, iteration continues until the last child is exhausted. Note that longest is only supported for dict children.

ignore_fixed: bool = True#: Ignore children with a fixed value, ie those with length 1. When set, they do not restrict the node size to 1.

phileas.iteration.generate_seeds(tree: IterationTree, salt: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree] = None) → IterationTree[source]#: Populate the seeds of the random nodes and leaves of a tree, using the given salt value.

phileas.iteration.restrict_leaves_sizes(tree: IterationTree, policy: RestrictionPolicy = RestrictionPolicy.FIRST_LAST) → IterationTree[source]#: Restrict the size of the iteration leaves of the tree, depending on the restriction policy. This is useful for troubleshooting, or to verify that the full range of the leaves is supported by something.

This module contains the definition of the base type and classes used for iteration (data tree, pseudo data tree and iteration tree).

phileas.iteration.base.DataLiteral#

Data values that can be used as data tree leaves

phileas.iteration.base.Key = bool | str | int | float#: Dictionary keys

phileas.iteration.base.DataTree#

A data tree consists of literal leaves, and dictionary or list nodes

phileas.iteration.base.PseudoDataLiteral#

A leave of a pseudo data tree is either a data tree leave, or a non-trivial iteration tree leave.

phileas.iteration.base.PseudoDataTree#

A pseudo data tree is a data tree whose leaves can be non literal iteration leaves.

class phileas.iteration.base.DefaultIndex[source]#

Bases: Sentinel

Index of the default value of an iteration tree.

class phileas.iteration.base.TreeIterator(tree: T)[source]#

Bases: ABC, Generic[T]

Iteration tree iterator.

Compared to a usual iterator, it supports forward and backward iteration, and is endlessly usable. This means that, whenever it is “exhausted” (__next__() raises StopIteration), it can either be reset to its starting position with reset(), or its iteration direction can be switched using reverse().

Additionally, it supports random access with the __getitem__() method, which uses update() under the hood.

position: int#

Position of the last value that was yielded. Valid values start at -1 (backward-exhausted iterator, or forward iteration start), and go up to size (forward-exhausted iterator, or backward iteration start).

It can be directly modified by update().

tree: T#: Reference to the tree being iterated over.

size: int | None#: Cached size of the iterated tree. Consider using this instead of len (self.tree), as computing the length of a tree can be an expensive operation.

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

is_forward() → bool[source]#: Returns whether the iterator is going forward or not.

reverse()[source]#

Reverse the iteration direction of the iterator, but stay at the same position. Thus, if it is any TreeIterator, the following behavior is expected:

>>> it.reset()
>>> it.reverse()
>>> list(it)
[]

If you want to iterate over an iteration tree tree backward, you have to reset the iterator after having reversed it:

>>> it = iter(tree)
>>> it.reverse()
>>> it.reset()

update(position: int)[source]#

Update the position of the iterator to any supported position. This includes the positions ranging from 0 to self.size - 1, included, as well as - -1, which represents a reset forward iterator and - self.size, which represents a reset backward iterator.

If an invalid position is requested, an IndexError is raised, and the state of the iterator remains unchanged.

class phileas.iteration.base.OneWayTreeIterator[source]#

Bases: object

Iterator used in cases where random access iteration is too cumbersome to implement. It only requires implementing the _next() method.

Random access is obtained by using a cache. Thus, this method is usually more time and memory expensive than classical random access iterators, so it should only be used as a last resort. For now, the cache size is unbound, as it stores all the iterated values. This might change in the future.

This is not a subclass of TreeIterator in order to prevent diamond inheritance issues. Child classes must thus inherit from this, alongside TreeIterator. It is recommended to place OneWayTreeIterator() first, so that its _current_value() implementation is used.

class phileas.iteration.base.NoDefaultPolicy(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: Enum

Behavior of IterationTree.default() for trees not having a default value.

ERROR = 'ERROR'#: Raise a NoDefaultError if any of the nodes in the tree does not have a default value.

SENTINEL = 'SENTINEL'#: Return the _NoDefault sentinel if any of the nodes in the tree does not have a default value.

SKIP = 'SKIP'#

Skip elements without a default element. If the root of the tree does not have a default value, return a _NoDefault sentinel.

Note that this is not supported by iteration method nodes with list children.

FIRST_ELEMENT = 'FIRST_ELEMENT'#: Return the first element of iteration leaves without a default value.

Bases: Exception

Indicates that IterationTree.default() has been called on an iteration tree where a node does not have a default value.

path: list[bool | str | int | float | _Child]#: Path of a child without a default value.

phileas.iteration.base.no_default = _NoDefault()#: You can store this value - instead of an actual default value - in instances of classes that can have a default value, but don’t.

exception phileas.iteration.base.InfiniteLength[source]#

Bases: BaseException

Exception raised when requesting the length of an infinite collection.

class phileas.iteration.base.IterationTree[source]#

Bases: ABC

Represents a set of data trees, as well as the way to iterate over them. In order to be able to get a single data tree from an iteration tree, they are able to build a default data tree, which (usually) has the same shape as the generated data tree.

An iteration tree cannot be modified, as it is a frozen dataclass. Instead, a new one must be created from this one.

configurations: frozenset[Key]#: Names of the available configurations.

get_configuration(key: phileas.iteration.base.Key) → IterationTree[source]#: Returns a given configuration of the tree, if it exists. Otherwise, raises a KeyError. See Configurations node for more details on the behavior of this method.

unroll_configurations() → IterationTree[source]#

Transforms a configurable tree, ie. one with configurations != {}, to a non-configurable tree. It requests all the available configurations, and gather them into a Union node with reset=False. A MoveUpTransform is used to get rid of the name of the configurations.

If the tree is not configurable, return it.

safe_len() → int | None[source]#: Return the number of data trees represented by the iteration tree. If it is finite, it is the same as the number of elements yielded by __iter__(). Otherwise, return None.

iterate() → TreeIterator[source]#: Implementation of __len__() which assumes that self does not have any configuration.

abstract to_pseudo_data_tree() → phileas.iteration.base.PseudoDataTree[source]#: Converts the iteration tree to a pseudo data tree.

default(no_default_policy: NoDefaultPolicy = NoDefaultPolicy.ERROR) → DataTree | _NoDefault[source]#: Returns a default data tree. If the tree does not have a default value, follows the behavior dictated by no_default_policy.

with_params(path: list[bool | str | int | float | _Child] | None = None, **kwargs) → Self[source]#: Returns a similar iteration tree, where the node at path is assigned the given keyword parameters. If path is not specified, modifies the root of the tree directly.

get(path: list[bool | str | int | float | _Child]) → IterationTree[source]#: Get a node inside a tree. It should not be used to modify the tree.

Insert a child anywhere in the tree, whose location is specified by path argument. Return the newly created tree.

If there is already a node at this location, it will be replaced.

If the specified tree is None, a node is supposed to exist at this location (otherwise, a KeyError is raised), and will be removed if possible. Only iteration methods nodes will support child removal, see their implementation of _remove_child(). In any other case, a TypeError is raised.

Note that the root of a tree cannot be removed, so specifying an empty path with a None tree will raise a KeyError.

remove_child(path: list[bool | str | int | float | _Child]) → IterationTree[source]#: Remove a node in a tree. It is equivalent to insert_child(path, None).

insert_unary(path: ChildPath, Parent: type[UnaryNode], *args, **kwargs) → IterationTree[source]#

Insert a parent to the node at the given path, parent which is necessarily a transform node, as it will only have a single child. The parent is built using the Parent class, and the supplied arguments.

The newly created tree is returned.

replace_node(path: list[bool | str | int | float | _Child], Node: type[IterationTree], *args, **kwargs) → IterationTree[source]#

Replace the node at the given path with another one. The other node is built using its type, Node, and the args and kwargs arguments. Note that the sub-tree of the replaced node is not modified.

This requires Node to be of the same kind of the node that is being replaced: a transform for a transform, an iteration method for an iteration method, a leaf for a leaf.

depth_first_modify(modifier: Callable[[IterationTree, list[bool | str | int | float | _Child]], IterationTree]) → IterationTree[source]#: Using a post-fix depth-first search, replace each node of the tree, located at path, with modifier(node, path).

class phileas.iteration.base.IterationLeaf[source]#: Bases: IterationTree

This module defines abstract and concrete iteration leaves, which are the actual data sources of an iteration tree, alongside their iterators.

class phileas.iteration.leaf.IterationLiteral(value: DT)[source]#

Bases: IterationLeaf, Generic[DT]

Wrapper around a data tree.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.leaf.LiteralIterator(tree: T)[source]#: Bases: TreeIterator[IterationLiteral]

Bases: IterationLeaf

Wrapper around a generator function, which can be used in order not to have to implement a new iteration leave, and its iterator. Note that only continuous forward iteration is supported by the node.

size: int | None = None#: Size of the tree. If the generator can provide more elements, only the first size ones are returned. If it cannot generate enough, a StopIteration is raised during iteration. None represents an infinite generator.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.leaf.GeneratorWrapperIterator(tree: GeneratorWrapper)[source]#: Bases: OneWayTreeIterator, TreeIterator[GeneratorWrapper]

class phileas.iteration.leaf.NumericRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>)[source]#

Bases: IterationLeaf, Generic[T]

Represents a range of numeric values.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.leaf.LinearRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, steps: int = 2)[source]#

Bases: NumericRange[float]

Generate steps values linearly spaced between start and end, both included.

class phileas.iteration.leaf.GeometricRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, steps: int = 2)[source]#

Bases: NumericRange[float]

Generate steps values geometrically spaced between start and end, both included.

class phileas.iteration.leaf.IntegerRange(start: ~phileas.iteration.leaf.T, end: ~phileas.iteration.leaf.T, default_value: ~phileas.iteration.leaf.T | ~phileas.iteration.base._NoDefault = <factory>, step: int = 1)[source]#

Bases: NumericRange[int | float]

Generate integer values step spaced, between start and end, both included. start must be an int, but end can also be math.inf or -math.inf. In these cases, the range is infinite.

class phileas.iteration.leaf.IntegerRangeIterator(tree: IntegerRange)[source]#: Bases: TreeIterator[IntegerRange]

Bases: IterationLeaf

Non-empty sequence of data trees.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.leaf.SequenceIterator(tree: T)[source]#: Bases: TreeIterator[Sequence]

Bases: IterationLeaf, RandomTree

Deterministic pseudo-random elements generator.

size: None | int = None#: Number of elements generated by the leaf. If None, the leaf is infinite.

to_pseudo_data_tree() → None | bool | str | int | float | _NoDefault | IterationLeaf | dict[bool | str | int | float, phileas.iteration.base.PseudoDataTree] | list[phileas.iteration.base.PseudoDataTree][source]#: Converts the iteration tree to a pseudo data tree.

configurations: frozenset[Key]#: Names of the available configurations.

Bases: RandomIterationLeaf

Random iteration leaf based on the RNG of numpy.

Note:

Iteration is based on NumpyRNGIterator. It works by seeding a PRNG with a value which depends on the requested iteration position. This might introduce a bias on the generated random values. This is done as a way to provide a random-access PRNG, although an actual random-access PRNG would be preferable.

Alternatively, the iterator could be built from a usual iterative PRNG, and caching would be used to provide access to previous values. This would impact iteration performances (especially for cache misses, which can induce unbound time delays).

distribution(size=None, dtype=<class 'numpy.float64'>, out=None)#: Which distribution to use for the node. It must be a distribution method of numpy.random.Generator.

args: list#: Arguments list to pass to the distribution.

kwargs: dict[str, Any]#: Keyword arguments to pass to the distribution.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.leaf.NumpyRNGIterator(tree: NumpyRNG)[source]#

Bases: TreeIterator[NumpyRNG]

Iterator that generates random numbers by reseeding a numpy bit generator, and getting its first returned values.

Bases: RandomIterationLeaf

Random iteration leaf generating arbitrarily big integers. It works by iteratively repeatedly using a Numpy RNG.

By default returns a byte (integer larger or equal to 0 and smaller than 256).

low: int = 0#: The inclusively lowest value that may be taken by the output.

high: int = 255#: The inclusively highest value that may be taken by the output.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.leaf.UniformBigIntegerRngIterator(tree: UniformBigIntegerRng)[source]#

Bases: TreeIterator[UniformBigIntegerRng]

Iterator that generates random numbers by reseeding a numpy byte generator.

Bytes are repeatedly sampled until the generated number fits in the required bounds.

Bases: RandomIterationLeaf

Random iteration leaf generating prime numbers.

Generation is done in two steps:

a uniform integer is uniformly drawn from the interval [low, high];
sympy nextprime() and prevprime() are used to find the closest prime.

high: int = 0#: Lower generation bound, inclusive.

low: int = 255#: Upper generation bound, inclusive.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.leaf.PrimeRngIterator(tree: PrimeRng)[source]#

Bases: TreeIterator[PrimeRng]

Iterator that generates random prime numbers by uniformly generating an number with the big integer RNG, before finding a neighboring prime with sympy prevprime() and nextprime().

Details on the process:

Generate a big integer,
find the next and previous primes with Sympy,
return the closest one inside the range; if they are equidistant, pick one randomly.

This module defines abstract and concrete iteration tree nodes, which are iteration methods and transform nodes, as well as their iterators.

class phileas.iteration.node.IterationMethod(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False)[source]#

Bases: IterationTree

Iteration node having multiple children, supplied either as a list or dictionary.

In order to implement a concrete iteration method, you should sub-class IterationMethod and implement a corresponding IterationMethodIterator, which is returned by _iter().

This should remain the only node in an iteration tree that can hold dict and list children. If you are tempted to create another node doing so, you should verify that it cannot be done by sub-classing IterationMethod instead.

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

order: list[Key] | None = None#: Order of iteration over the children. How it is used depends on the concrete iteration method implementation. It must be a permutation of the set of keys of children.

lazy: bool = False#

Notify the iteration method to be lazy. For now, this feature is only supported for dict children. In this case, lazy iteration will just yield the keys that have changed at each step.

Note that concrete iteration method classes are not required to actually implement lazy iteration, and if they don’t, they will probably do so silently. Refer to their documentation or their implementation.

to_pseudo_data_tree() → phileas.iteration.base.PseudoDataTree[source]#: Converts the iteration tree to a pseudo data tree.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.IterationMethodIterator(tree: T)[source]#

Bases: TreeIterator[T]

Base class used to implement concrete IterationMethod nodes iterators, and providing helper attributes to do so.

keys: list[Key | int]#

Access keys for the children iterators, such that iterators[i] is an iterator of tree.children[keys[i]].

Note that the previous statement is not properly typed, as the current type hints do not state that keys contains valid keys for children. However, the constructor takes care of the validity of the runtime types. Because of that, ignoring type checks can be required when implementing concrete iteration methods.

iterators: list[TreeIterator]#

Children iterators stored in a list. This, with keys, allows child-class to only implement iteration over lists.

If the iteration tree does specify an order, use it. Otherwise, dictionaries are sorted by their key value, and lists keep the same order.

positions: Sequence[int | DefaultIndex | None]#: Last returned positions of the child iterators.

sizes: list[int | None]#: Size of each of the iterators. None represents infinite trees.

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

reverse()[source]#

Reverse the iteration direction of the iterator, but stay at the same position. Thus, if it is any TreeIterator, the following behavior is expected:

>>> it.reset()
>>> it.reverse()
>>> list(it)
[]

If you want to iterate over an iteration tree tree backward, you have to reset the iterator after having reversed it:

>>> it = iter(tree)
>>> it.reverse()
>>> it.reset()

class phileas.iteration.node.CartesianProduct(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, snake: bool = False)[source]#

Bases: IterationMethod

Iteration over the cartesian product of the children. The iteration order is the same as itertools.product(). In other words, iteration will behave roughly as

for v1 in c1:
    for v2 in c2:
        for v3 in c3:
            yield [v1, v2, v3]

If an order is specified, its first element will correspond to the outermost loop, and its last to the innermost one.

snake: bool = False#: Enable snake iteration, which guarantees that successive yielded elements differ by only one key at most (a la Gray code).

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.CartesianProductIterator(product: CartesianProduct)[source]#

Bases: IterationMethodIterator[CartesianProduct]

cumsizes: list[int | None]#: Backward cumulated products of sizes, with size + 1 elements, and ending with a 1. Before, and including, the last infinite child, it only contains None.

class phileas.iteration.node.Union(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, preset: Literal['first'] | Literal['default'] | None = 'first', common_preset: bool = False, reset: Literal['first'] | Literal['last'] | Literal['default'] | None = 'first')[source]#

Bases: IterationMethod

Iteration over one child at a time, starting with the first one (or the first one of the order, if specified). Children that are not being iterated over have

their default value if it exists and
their first value otherwise.

preset: Literal['first'] | Literal['default'] | None = 'first'#: Defines which value to use before starting the iteration over a child. “first” uses the child’s first value and “default” its default value. None does not set the children values before iteration. Thus, it is only applicable to dict children.

common_preset: bool = False#: You can only set it when preset == "first". Then, children are all set to their first value at the first iteration. When iterating over them, they start directly at their second value.

reset: Literal['first'] | Literal['last'] | Literal['default'] | None = 'first'#: Defines which value to use after ending the iteration over a child. “first” resets it to its starting value, “last” leaves it unchanged, and “default” resets it to its default value. None does not reset the child after iteration. Thus, it is only applicable to dict children.

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.UnionIterator(tree: Union)[source]#

Bases: IterationMethodIterator[Union]

cumsizes: list[int | float]#: Cumulated sum of the iterated sizes of the children, containing int or math.inf values. After, and including, the first infinite children, it only contains math.inf values. Its size is len(self.sizes) + 1, as it is prefixed with a 0.

initial_value: int | DefaultIndex | None#: Value of a child before iteration over it starts

final_value: int | DefaultIndex | None#: Value of a child after iteration over it ends

class phileas.iteration.node.Zip(children: list[IterationTree] | dict[phileas.iteration.base.Key, IterationTree], order: list[phileas.iteration.base.Key] | None = None, lazy: bool = False, stops_at: Literal['longest', 'shortest'] = 'shortest', ignore_fixed: bool = True)[source]#

Bases: IterationMethod

Iteration over all of the children of the nodes at the same time, in a way similar to zip().

stops_at: Literal['longest'] | Literal['shortest'] = 'shortest'#: If shortest, iteration stops whenever the first child is exhausted. If longest, iteration continues until the last child is exhausted. Note that longest is only supported for dict children.

ignore_fixed: bool = True#: Ignore children with a fixed value, ie those with length 1. When set, they do not restrict the node size to 1.

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.ZipIterator(tree: T)[source]#: Bases: IterationMethodIterator[Zip]

class phileas.iteration.node.Pick(children: list[IterationTree] | dict[Key, IterationTree], order: list[Key] | None = None, lazy: bool = False, seed: Seed | None = None, default_child: Key | None = None)[source]#

Bases: RandomTree, IterationMethod

Randomly pick and return one child at a time. For finite trees, it behaves in a way similar to composing Shuffle and Union (reset=False) nodes. It can additionally handle infinite children, whereas Shuffle cannot.

default_child: Key | None = None#: Key of the child to use as a default value.

children: list[IterationTree] | dict[Key, IterationTree]#: The children of the node. It must not be empty.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.PickIterator(tree: Pick)[source]#: Bases: OneWayTreeIterator, IterationMethodIterator

class phileas.iteration.node.UnaryNode(child: 'IterationTree')[source]#

Bases: IterationTree

to_pseudo_data_tree() → phileas.iteration.base.PseudoDataTree[source]#: Converts the iteration tree to a pseudo data tree.

class phileas.iteration.node.Shuffle(child: IterationTree, seed: Seed | None = None)[source]#

Bases: RandomTree, UnaryNode

Unary node that shuffles the order of its child. In other words, it iterates over a random permutation of its children values.

Shuffling has a constant memory cost. The permutation of the order of the child tree is obtained through the use of a cypher, which means that it is entirely represented by a short key.

To shuffle a tree with size \(n\), a symmetric cypher on \(\{ 0, \dots , n-1 \}\) is used. It is built with two components:

a block cypher working on \(2 \lceil \log_2 n / 2 \rceil\)-bit words. It consists in a 3-round Feistel network which uses SipHash 1-3 as a round function.
Cycle walking is used to restrict the message space to \(\{ 0, \dots, n - 1 \}\). The block cypher is iterated until its output is valid.

class phileas.iteration.node.ShuffleIterator(tree: Shuffle)[source]#

Bases: TreeIterator

rounds: ClassVar[int] = 3#: Number of rounds of the generalized Feistel network.

keys: list[tuple[uint64, uint64]]#: Round keys of the generalized Feistel network. They are used for SipHash 1-3, so they consist in two 64 bit integers. The size of this list must be rounds.

class phileas.iteration.node.First(child: IterationTree, size: int | None)[source]#

Bases: UnaryNode

Return only the first elements of its child.

size: int | None#: Number of elements to keep. If it is None, or is bigger than the child size, the same number of elements as the child are iterated over.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.FirstIterator(tree: First)[source]#: Bases: TreeIterator[First]

class phileas.iteration.node.Transform(child: IterationTree)[source]#

Bases: UnaryNode

Node that modifies the data trees generated by its child during iteration.

If you want to transform a list or dict of iteration trees, you should wrap them in an IterationMethod object first.

abstract transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.node.TransformIterator(tree: U)[source]#

Bases: TreeIterator[U]

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

reverse()[source]#

Reverse the iteration direction of the iterator, but stay at the same position. Thus, if it is any TreeIterator, the following behavior is expected:

>>> it.reset()
>>> it.reverse()
>>> list(it)
[]

If you want to iterate over an iteration tree tree backward, you have to reset the iterator after having reversed it:

>>> it = iter(tree)
>>> it.reverse()
>>> it.reset()

class phileas.iteration.node.FunctionalTranform(child: IterationTree, f: Callable[[phileas.iteration.base.DataTree], phileas.iteration.base.DataTree])[source]#

Bases: Transform

Transform node using its function attribute to modify its child.

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.node.Accumulator(child: IterationTree, recursive: bool = False, start_value: dict[phileas.iteration.base.Key, phileas.iteration.base.DataTree] | None = None)[source]#

Bases: Transform

Transform node that accumulates its inputs, as a kind of unlazifying transform:

if its successive inputs are dictionaries, merge them using the union operator (recursively or not), and return the results;
otherwise, leave its inputs untouched.

recursive: bool = False#: Specify if the accumulation must be done recursively or not. For example, accumulating values {"a": 1, "b": {"ba": 1}} and {"a": 2, "b": {"bb": 2}} recursively will return {"a": 2, "b": {ba": 1, "bb": 2}}, whereas doing it non-recursively will return {"a": 2, "b": {"bb": 2}}.

start_value: dict[Key, DataTree] | None = None#: Start value of the accumulator, which must either be a dictionary, or None.

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

configurations: frozenset[Key]#: Names of the available configurations.

class phileas.iteration.node.AccumulatorIterator(tree: Accumulator)[source]#

Bases: TransformIterator[Accumulator]

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

class phileas.iteration.node.Lazify(child: IterationTree)[source]#

Bases: Transform

Transform node that only returns the elements that were updated in its children values (for dictionary values), or leaves its inputs untouched (for other types).

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.node.LazifyIterator(tree: Lazify)[source]#

Bases: TransformIterator[Lazify]

accumulated_value: dict[Key, DataTree] | None#: Accumulation of the values yielded by the tree iterator. If it generates a non-dict value, then stores None.

reset()[source]#: Reset the internal state of the iterator, so that its next value will be the first iterated value in the current direction. It takes into account the value of forward, going either to the start or the end of the iterated collection.

class phileas.iteration.node.MoveUpTransform(child: 'IterationTree', insert_name: 'bool | str' = False)[source]#

Bases: Transform

transform(data_tree: phileas.iteration.base.DataTree) → phileas.iteration.base.DataTree[source]#: Method implemented by concrete sub-classes to modify the data tree generated by the child tree.

class phileas.iteration.node.Configurations(children: dict[Key, IterationTree], order: list[Key] | None = None, lazy: bool = False, default_configuration: Key | None = None, move_up: bool = False, insert_name: bool = False)[source]#

Bases: IterationMethod

Represents a set of named configurations that can be invoked using IterationTree.get_configuration(). When it is called, with argument name, all the Configurations nodes that have a matching configuration will be replaced by it. Other ones will be replaced by their default value.

This allows to escape from the recursive and local nature of trees: a single call can modify nodes throughout a whole tree. However, it is often convenient to convert configurable trees back to classical, non-configurable trees. This is done by IterationTree.unroll_configurations().

The Configurations node holds a set of iteration trees, called configurations, which are identified by their name. Two parameters, move_up and insert_name, control how IterationTree.get_configuration() behaves.

By default, move_up == insert_name == False. The Configuration node is simply replaced by the content of the requested configuration.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     })
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': {'param1': '1-1', 'param2': '1-2'}}

If move_up == True, the content of the chosen configuration is moved one level up, so that it is at the same level as the Configurations node. This requires it to have an IterationMethod parent with dict children. This can be used to factorize configurations.

>>> tree = CartesianProduct({
...     "_": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...         })
...     }, move_up=True),
...     "param2": IterationLiteral(value="2")
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'param1': '1-1', 'param2': '2'}

If insert_name == True, the name of the requested configuration is inserted in the resulting tree. If the requested configuration allows it (_ie_. it is an IterationMethod with dict children), the name is inserted into itself, with the key _configuration.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     }, insert_name=True)
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': {'_configuration': 'config1', 'param1': '1-1', 'param2': '1-2'}}

If, additionally, move_up == True, the name of the configuration is inserted instead of the Configurations node.

>>> tree = CartesianProduct({
...     "instrument": Configurations({
...         "config1": CartesianProduct({
...             "param1": IterationLiteral(value="1-1"),
...             "param2": IterationLiteral(value="1-2"),
...         }),
...         "config2": CartesianProduct({
...             "param1": IterationLiteral(value="2-1"),
...             "param3": IterationLiteral(value="2-3"),
...         })
...     }, insert_name=True, move_up=True)
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'instrument': 'config1', 'param1': '1-1', 'param2': '2'}

However, if the requested configuration is not an IterationMethod, its name is inserted as a sibling, assigned to the key f"_{name}_configuration".

>>> tree = CartesianProduct({
...     "param": Configurations({
...         "config1": IterationLiteral(value="1"),
...         "config2": IterationLiteral(value="2"),
...     }, insert_name=True),
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'_param_configuration': 'config1', 'param': '1'}

This means that the content of the configurations affects how they are handled. Although this might change in the future, it is made to support most situations. It is recommended to keep all the configurations with the same shape.

insert_name is not necessary. It is possible to replace it with the following kind of tree:

>>> tree = CartesianProduct({
...     "param": Configurations({
...         "config1": IterationLiteral(value="1"),
...         "config2": IterationLiteral(value="2"),
...     },
...     "config": Configurations({
...         "config1": IterationLiteral(value="config1"),
...         "config2": IterationLiteral(value="config2"),
...     })),
... })
>>> tree.get_configuration("config1").to_pseudo_data_tree()
{'param': '1', 'config': 'config1'}

configurations: frozenset[Key]#: Names of the available configurations.

children: dict[Key, IterationTree]#: The children of the node. It must not be empty.

default_configuration: Key | None = None#: Key of the default configuration, which must be in the set of keys of children.

move_up: bool = False#

If set, the content of a requested configuration is moved up, at the parent level of the current node. In other words, it becomes a sibling of the current Configurations node.

Otherwise, the content is inserted at the same level as the configurations themselves.

insert_name: bool = False#: If set, insert the name of the requested configuration when calling get_configuration(). If move_up, then the Configurations node is replaced by this name. Otherwise, a "_configuration" sibling node is inserted.

Bases: object

Seed of a random iteration node, used by its RNG.

path: list[bool | str | int | float | _Child]#: Path of the node in the biggest tree that used it.

salt: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree]#: Salt value, to customize iteration independently of the shape of the iteration tree.

to_bytes() → bytes[source]#: Convert the seed to bytes, for RNG seeding.

class phileas.iteration.random.RandomTree(seed: Seed | None = None)[source]#

Bases: object

Additional base class of random iteration trees.

seed: Seed | None = None#

Seed of the generator used by the random tree. It must be guaranteed that successive iteration values only depend on the seed value.

For iteration to be possible, the seed must be set. If you don’t want to manually specify the seed, you can use generate_seeds().

phileas.iteration.random.generate_seeds(tree: IterationTree, salt: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree] = None) → IterationTree[source]#: Populate the seeds of the random nodes and leaves of a tree, using the given salt value.

This module defines utility functions related to iteration.

class phileas.iteration.utility.RestrictionPolicy(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]#

Bases: Enum

FIRST_LAST = 'FIRST_LAST'#: Iteration leaves only keep their first and last values.

FIRST_SECOND = 'FIRST_SECOND'#: Iteration leaves only keep their two first values.

COMBINED = 'COMBINED'#: Iteration leaves keep their first, second and last values.

phileas.iteration.utility.restrict_leaves_sizes(tree: IterationTree, policy: RestrictionPolicy = RestrictionPolicy.FIRST_LAST) → IterationTree[source]#: Restrict the size of the iteration leaves of the tree, depending on the restriction policy. This is useful for troubleshooting, or to verify that the full range of the leaves is supported by something.

phileas.iteration.utility.recursive_union(tree1: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree], tree2: None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree]) → None | bool | str | int | float | _NoDefault | dict[bool | str | int | float, DataTree] | list[DataTree][source]#: Return the recursive union of two datatrees. If any of those is not a dictionary, it returns the latter. Otherwise, it recursively applies the union operator of dictionaries.

phileas.iteration.utility.is_transformed_iteration_leaf(tree: IterationTree) → bool[source]#

A transformed iteration leaf is an iteration tree that only contained Transform and IterationLeaf nodes. That is, it does not contain IterationMethod nodes.

This function checks if a tree is a transformed iteration leaf.

Transform nested dict and list objects to a single-level dict. DataTree returns a DataTree, and flattening a PseudoDataTree returns a PseudoDataTree.

Keys are converted to str, and concatenated using the specified separator. list objects are considered as int-keyed dict.

>>> tree = {
...     "key1": {
...         "key1-1": 1
...     },
...     "key2": [1, 2],
...     "key3": "value"
... }
>>> flatten_datatree(tree)
{'key1.key1-1': 1, 'key2.0': 1, 'key2.1': 2, 'key3': 'value'}

Generate the arguments required to build an xarray.DataArray or xarray.DataFrame. You can then modify them, if needed, and build the xarray container used to store the results of your experiment.

This function can only be applied to trees that only have finite leaves.

>>> import numpy as np
>>> import xarray as xr
>>> coords, dims_name, dims_shape = iteration_tree_to_xarray_parameters(tree)
>>> # Single data to be recorded for each iteration point
>>> xr.DataArray(data=np.empty(dims_shape), coords=coords, dims=dims_name)
>>> # Multiple data for each iteration point
>>> xr.Dataset(
>>>     data_vars=dict(
>>>             measurement_1=(dims_name, np.full(dims_shape, np.nan)),
>>>             measurement_2=(dims_name, np.full(dims_shape, np.nan)),
>>>     ),
>>>     coords=coords,
>>> )

Caution

This function is tested against trees whose iteration methods are cartesian products only, as they represent the coordinates of full gridded datasets. You can use it on other kinds of trees, but then make sure to verify that its behaves properly.

Iteration#

This Page