Covalent API#
The following API documentation describes how to use Covalent.
Local Executor is used to execute electrons locally
File and Data Transfer is used to queue remote or local file transfer operations prior or post electron execution.
File transfer strategies are used to perform download/upload/copy operations over various protocols.
๐ Dask Executor is used to execute electrons in a Dask cluster
Dependencies are used to specify any kind of electron dependency
Pip Dependencies are used to specify PyPI packages that are required to run an electron
Bash Dependencies are used to specify optional pre-execution shell commands for an electron
Call Dependencies are used to specify functions or dependencies that are called in an electronโs execution environment
Results is used for collecting and manipulating results
Dispatcher is used for dispatching workflows
The Covalent CLI Tool is used for interfacing with the Covalent server
Electron#
- class covalent._workflow.electron.Electron(function, node_id=None, metadata=None)[source]#
An electron (or task) object that is a modular component of a work flow and is returned by
electron
.- function#
Function to be executed.
- node_id#
Node id of the electron.
- metadata#
Metadata to be used for the function execution.
- kwargs#
Keyword arguments if any.
Methods:
add_collection_node_to_graph
(graph, prefix)Adds the node to latticeโs transport graph in the case where a collection of electrons is passed as an argument to another electron.
connect_node_with_others
(node_id, โฆ)Adds a node along with connecting edges for all the arguments to the electron.
get_metadata
(name)Get value of the metadata of given name.
get_op_function
(operand_1, operand_2, op)Function to handle binary operations with electrons as operands.
set_metadata
(name, value)Function to add/edit metadata of given name and value to electronโs metadata.
wait_for
(electrons)Waits for the given electrons to complete before executing this one.
- add_collection_node_to_graph(graph, prefix)[source]#
Adds the node to latticeโs transport graph in the case where a collection of electrons is passed as an argument to another electron.
- Parameters
graph (_TransportGraph) โ Transport graph of the lattice
prefix (
str
) โ Prefix of the node
- Returns
Node id of the added node
- Return type
node_id
- connect_node_with_others(node_id, param_name, param_value, param_type, arg_index, transport_graph)[source]#
Adds a node along with connecting edges for all the arguments to the electron.
- Parameters
node_id (
int
) โ Node number of the electronparam_name (
str
) โ Name of the parameterparam_value (
Union
[Any
,ForwardRef
]) โ Value of the parameterparam_type (
str
) โ Type of parameter, positional or keywordtransport_graph (_TransportGraph) โ Transport graph of the lattice
- Returns
None
- get_metadata(name)[source]#
Get value of the metadata of given name.
- Parameters
name (
str
) โ Name of the metadata whose value is needed.- Returns
Value of the metadata of given name.
- Return type
value
- Raises
KeyError โ If metadata of given name is not present.
- get_op_function(operand_1, operand_2, op)[source]#
Function to handle binary operations with electrons as operands. This will not execute the operation but rather create another electron which will be postponed to be executed according to the default electron configuration/metadata.
This also makes sure that if these operations are being performed outside of a lattice, then they are performed as is.
- Parameters
- Returns
- Electron object corresponding to the operation execution.
Behaves as a normal function call if outside a lattice.
- Return type
electron
- set_metadata(name, value)[source]#
Function to add/edit metadata of given name and value to electronโs metadata.
- Parameters
name (
str
) โ Name of the metadata to be added/edited.value (
Any
) โ Value of the metadata to be added/edited.
- Return type
None
- Returns
None
Lattice#
- class covalent._workflow.lattice.Lattice(workflow_function, transport_graph=None)[source]#
A lattice workflow object that holds the work flow graph and is returned by
lattice
decorator.- workflow_function#
The workflow function that is decorated by
lattice
decorator.
- transport_graph#
The transport graph which will be the basis on how the workflow is executed.
- metadata#
Dictionary of metadata of the lattice.
- post_processing#
Boolean to indicate if the lattice is in post processing mode or not.
- kwargs#
Keyword arguments passed to the workflow function.
- electron_outputs#
Dictionary of electron outputs received after workflow execution.
Methods:
build_graph
(*args, **kwargs)Builds the transport graph for the lattice by executing the workflow function which will trigger the call of all underlying electrons and they will get added to the transport graph for later execution.
dispatch
(*args, **kwargs)DEPRECATED: Function to dispatch workflows.
dispatch_sync
(*args, **kwargs)DEPRECATED: Function to dispatch workflows synchronously by waiting for the result too.
draw
(*args, **kwargs)Generate lattice graph and display in UI taking into account passed in arguments.
get_metadata
(name)Get value of the metadata of given name.
set_metadata
(name, value)Function to add/edit metadata of given name and value to latticeโs metadata.
- build_graph(*args, **kwargs)[source]#
Builds the transport graph for the lattice by executing the workflow function which will trigger the call of all underlying electrons and they will get added to the transport graph for later execution.
Also redirects any print statements inside the lattice function to null and ignores any exceptions caused while executing the function.
GRAPH WILL NOT BE BUILT AFTER AN EXCEPTION HAS OCCURRED.
- Parameters
*args โ Positional arguments to be passed to the workflow function.
**kwargs โ Keyword arguments to be passed to the workflow function.
- Return type
None
- Returns
None
- dispatch(*args, **kwargs)[source]#
DEPRECATED: Function to dispatch workflows.
- Parameters
*args โ Positional arguments for the workflow
**kwargs โ Keyword arguments for the workflow
- Return type
str
- Returns
Dispatch id assigned to job
- dispatch_sync(*args, **kwargs)[source]#
DEPRECATED: Function to dispatch workflows synchronously by waiting for the result too.
- Parameters
*args โ Positional arguments for the workflow
**kwargs โ Keyword arguments for the workflow
- Return type
Result
- Returns
Result of workflow execution
- draw(*args, **kwargs)[source]#
Generate lattice graph and display in UI taking into account passed in arguments.
- Parameters
*args โ Positional arguments to be passed to build the graph.
**kwargs โ Keyword arguments to be passed to build the graph.
- Return type
None
- Returns
None
Local Executor#
Executing tasks (electrons) directly on the local machine
File Transfer#
File Transfer from (source) and to (destination) local or remote files prior/post electron execution. Instances are are provided to files keyword argument in an electron decorator.
- class covalent._file_transfer.file.File(filepath=None, is_remote=False, is_dir=False, include_folder=False)[source]#
File class to store components of provided URI including scheme (s3://, file://, ect.) determine if the file is remote, and acts a facade to facilitate filesystem operations.
- filepath#
File path corresponding to the file.
- is_remote#
Flag determining if file is remote (override). Default is resolved automatically from file scheme.
- is_dir#
Flag determining if file is a directory (override). Default is determined if file uri contains trailing slash.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
- class covalent._file_transfer.folder.Folder(filepath=None, is_remote=False, is_dir=True, include_folder=False)[source]#
Folder class to store components of provided URI including scheme (s3://, file://, ect.), determine if the file is remote, and act as facade to facilitate filesystem operations. Folder is a child of the File class which sets is_dir flag to True.
- include_folder#
Flag that determines if the folder should be included in the file transfer, if False only contents of folder are transfered.
- class covalent._file_transfer.file_transfer.FileTransfer(from_file=None, to_file=None, order=<Order.BEFORE: 'before'>, strategy=None)[source]#
FileTransfer object class that takes two File objects or filepaths (from, to) and a File Transfer Strategy to perform remote or local file transfer operations.
- from_file#
Filepath or File object corresponding to the source file.
- to_file#
Filepath or File object corresponding to the destination file.
- order#
Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution.
- strategy#
Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.
- covalent._file_transfer.file_transfer.TransferFromRemote(from_filepath, to_filepath=None, strategy=None, order=<Order.BEFORE: 'before'>)[source]#
Factory for creating a FileTransfer instance where from_filepath is implicitly created as a remote File Object, and the order (Order.BEFORE) is set so that this file transfer will occur prior to electron execution.
- Parameters
from_filepath (
str
) โ File path corresponding to remote file (source).to_filepath (
Optional
[str
]) โ File path corresponding to local file (destination)strategy (
Optional
[FileTransferStrategy
]) โ Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order
) โ Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is BEFORE
- Return type
- Returns
FileTransfer instance with implicit Order.BEFORE enum set and from (source) file marked as remote
- covalent._file_transfer.file_transfer.TransferToRemote(to_filepath, from_filepath=None, strategy=None, order=<Order.AFTER: 'after'>)[source]#
Factory for creating a FileTransfer instance where to_filepath is implicitly created as a remote File Object, and the order (Order.AFTER) is set so that this file transfer will occur post electron execution.
- Parameters
to_filepath (
str
) โ File path corresponding to remote file (destination)from_filepath (
Optional
[str
]) โ File path corresponding to local file (source).strategy (
Optional
[FileTransferStrategy
]) โ Optional File Transfer Strategy to perform file operations - default will be resolved from provided file schemes.order (
Order
) โ Order (enum) to execute the file transfer before (Order.BEFORE) or after (Order.AFTER) electron execution - default is AFTER
- Return type
- Returns
FileTransfer instance with implicit Order.AFTER enum set and to (destination) file marked as remote
File Transfer Strategies#
A set of classes with a shared interface to perform copy, download, and upload operations given two (source & destination) File objects that support various protocols.
- class covalent._file_transfer.strategies.transfer_strategy_base.FileTransferStrategy[source]#
Base FileTransferStrategy class that defines the interface for file transfer strategies exposing common methods for performing copy, download, and upload operations.
- class covalent._file_transfer.strategies.rsync_strategy.Rsync(user='', host='', private_key_path=None)[source]#
Implements Base FileTransferStrategy class to use rsync to move files to and from remote or local filesystems. Rsync via ssh is used if one of the provided files is marked as remote.
- user#
(optional) Determine user to specify for remote host if using rsync with ssh
- host#
(optional) Determine what host to connect to if using rsync with ssh
- private_key_path#
(optional) Filepath for ssh private key to use if using rsync with ssh
Dask Executor#
Executing tasks (electrons) in a Dask cluster
Dependencies#
Generic dependencies for an electron
- class covalent._workflow.deps.Deps(apply_fn=None, apply_args=[], apply_kwargs={}, *, retval_keyword='')[source]#
Generic dependency class used in specifying any kind of dependency for an electron.
- apply_fn#
function to be executed in the backend environment
- apply_args#
list of arguments to be applied in the backend environment
- apply_kwargs#
dictionary of keyword arguments to be applied in the backend environment
Methods:
apply
()Encapsulates the exact function and args/kwargs to be executed in the backend environment.
- apply()[source]#
Encapsulates the exact function and args/kwargs to be executed in the backend environment.
- Parameters
None โ
- Return type
Tuple
[TransportableObject
,TransportableObject
,TransportableObject
,str
]- Returns
A tuple of transportable objects containing the function and optional args/kwargs
Pip Dependencies#
PyPI packages to be installed before executing an electron
- class covalent._workflow.depspip.DepsPip(packages=[], reqs_path='')[source]#
PyPI packages to be installed before executing an electron
A specification of Pip packages to be installed
- packages#
A list of PyPI packages to install
- reqs_path#
Path to requirements.txt (overrides packages)
These packages are installed in an electronโs execution environment just before the electron is run.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
Bash Dependencies#
Shell commands to run before an electron
- class covalent._workflow.depsbash.DepsBash(commands=[])[source]#
Shell commands to run before an electron
Deps class to encapsulate Bash dependencies for an electron.
The specified commands will be executed as subprocesses in the same environment as the electron.
- commands#
A list of bash commands to execute before the electron runs.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
Call Dependencies#
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electronโs execution environment
- class covalent._workflow.depscall.DepsCall(func=None, args=[], kwargs={}, *, retval_keyword='', override_reserved_retval_keys=False)[source]#
Functions, shell commands, PyPI packages, and other types of dependencies to be called in an electronโs execution environment
Deps class to encapsulate python functions to be called in the same execution environment as the electron.
- func#
A callable
- args#
args list
- kwargs#
kwargs dict
- retval_keyword#
An optional string referencing the return value of func.
If retval_keyword is specified, the return value of func will be passed during workflow execution as an argument to the electron corresponding to the parameter of the same name.
Notes
Electron parameters to be injected during execution must have default parameter values.
It is the userโs responsibility to ensure that retval_keyword is actually a parameter of the electron. Unexpected behavior may occur otherwise.
Methods:
from_dict
(object_dict)Rehydrate a dictionary representation
to_dict
()Return a JSON-serializable dictionary representation of self
Dispatcher#
Dispatching jobs to the server
Results#
Collecting and managing results
Covalent CLI Tool#
This Command Line Interface (CLI) tool is used to manage Covalent server.