Maestro¶

Maestro is the command-line interface for the computer grid at the Signal Processing Laboratory @ Federal University of Rio de Janeiro (LPS/UFRJ).

It allows you to manage datasets and tasks, providing everything you need for best usage of the Orchestra Cluster, that aggregates multiple computer grids in one, increasing your own productivity.

Installation (Python API)¶

Disclaimer: Python 2.x versions aren’t necessarily supported, please upgrade to Python 3.x or use it at your own risk. If you really need support for those versions and it doesn’t work, please feel free to implement it and open a pull request on the Maestro repository <https://github.com/gabriel-milan/maestro>

For the Python API, you must install it from PyPI using

After that, you can import it on your Python 3.x script:

Installation (CLI)¶

Install Maestro by running:

Clone the Maestro repository

git clone https://github.com/gabriel-milan/maestro
Go to the cloned directory:

cd maestro
Run the setup script:

source setup.sh
Done! You can test your installation by running:

maestro –help

Getting started (Python API)¶

Disclaimer 1: for using Maestro Python API, you’ll need credentials on the Orchestra Cluster. If you don’t have it, please register here using your LPS mail or contact us. More information on contact on section Support.

Disclaimer 2: Internet access is mandatory.

Maestro has three main modules:

authenticate for generating your authentication token that will be stored at $HOME/.maestro_credentials
castor for managing datasets (uploading, downloading, deleting and listing)
task for managing workload tasks (creating, retrying, deleting, listing and killing)

maestro.authenticate (username, password)¶

This is the simplest module on Maestro and it’s necessary so you can access the other two. By running:

you will attempt to authenticate with the Orchestra API and, if succeeded, a token will be generated and stored at $HOME/.maestro_credentials. After this, you’ll no longer need to worry about authentication, if your credentials file remains untouched.

If you’ve changed passwords or wish to authenticate with another account, just run this command again and your credentials will be overwritten.

maestro.castor¶

Castor module has four commands:

upload for uploading new datasets
download for downloading datasets from the Orchestra Cluster
delete for deleting datasets on the Orchestra Cluster
list for listing datasets related to a given username

maestro.castor.upload (datasetname, path)¶

maestro.castor.download (datasetname)¶

maestro.castor.delete (datasetname)¶

maestro.castor.list (username, cli=False)¶

If you let cli = False, as default, it will return a list object containing the datasets related to the username provided. If set to True, it will print, in a pretty way, a table containing the same datasets (better for visualization).

maestro.task¶

Task module has five commands:

create for deploying workload
retry for retrying tasks that either failed or got killed
delete for deleting tasks on the Orchestra Cluster
list for listing tasks related to a given username
kill for stopping execution of tasks

maestro.task.create (taskname, dataFile, configFile, execCommand, containerImage, queue=’cpu_small’, secondaryDS=None, et=None, eta=None, dry_run=False)¶

maestro.task.retry (taskname)¶

maestro.task.delete (taskname)¶

maestro.task.list (username, cli=False)¶

maestro.task.kill (username, taskname)¶

Getting started (CLI)¶

Disclaimer 1: for using Maestro CLI, you’ll need credentials on the Orchestra Cluster. If you don’t have it, please register here using your LPS mail or contact us. More information on contact on section Support.

Disclaimer 2: Internet access is mandatory.

Maestro has three main modules:

authenticate for generating your authentication token that will be stored at $HOME/.maestro_credentials
castor for managing datasets (uploading, downloading, deleting and listing)
task for managing workload tasks (creating, retrying, deleting, listing and killing)

maestro authenticate¶

This is the simplest module on Maestro and it’s necessary so you can access the other two. By running:

maestro authenticate –user <username> –password <password>

or

maestro authenticate -u <username> -p <password>

you will attempt to authenticate with the Orchestra API and, if succeeded, a token will be generated and stored at $HOME/.maestro_credentials. After this, you’ll no longer need to worry about authentication, if your credentials file remains untouched.

If you’ve changed passwords or wish to authenticate with another account, just run this command again and your credentials will be overwritten.

maestro castor¶

Castor module has four commands:

upload for uploading new datasets
download for downloading datasets from the Orchestra Cluster
delete for deleting datasets on the Orchestra Cluster
list for listing datasets related to a given username

maestro castor upload¶

usage: maestro upload [-h] -d DATASETNAME -p PATH

optional arguments:
-h, --help                                show this help message and exit

-d DATASETNAME, --dataset DATASETNAME     The dataset name that will be registered on the
                                          database (e.g: user.jodafons...)

-p PATH, --path PATH                      The path to the dataset file

maestro castor download¶

usage: maestro download [-h] -d DATASETNAME

optional arguments:
-h, --help                                show this help message and exit

-d DATASETNAME, --dataset DATASETNAME     The dataset name to be downloaded

maestro castor delete¶

usage: maestro delete [-h] -d DATASETNAME

optional arguments:
-h, --help                                show this help message and exit

-d DATASETNAME, --dataset DATASETNAME     The dataset name to be removed

maestro castor list¶

usage: maestro list [-h] -u USERNAME

optional arguments:
-h, --help                                show this help message and exit

-u USERNAME, --user USERNAME              List all datasets for a selected user

maestro task¶