Custom Benchmark Scripts

The python -m taps.run CLI has limited support for repeating benchmarks while varying parameters (e.g., executor types, application parameters, etc.). More sophisticated benchmarks can be performed through custom benchmark scripts that utilize the TaPS API.

This guide provides some examples scripts and pointers to get started. Each example will use the Cholesky Factorization application and Python's ProcessPoolExecutor for task execution.

Running an App

A TaPS App can be executed directly using App.run() which requires an Engine and a run directory (required, but not actually used by all apps). This is the most direct and manual means of running an application.

Note

This example does not configure the Engine with a RecordLogger. A RecordLogger, such as JSONRecordLogger, is necessary to take advantage of the task execution information collection of the Engine.

import contextlib
import pathlib
from concurrent.futures import ProcessPoolExecutor
from datetime import datetime

from taps.apps.cholesky import CholeskyApp
from taps.engine import Engine
from taps.executor.utils import FutureDependencyExecutor
from taps.logging import init_logging


def main() -> int:
    init_logging()

    app = CholeskyApp(matrix_size=100, block_size=25)
    executor = FutureDependencyExecutor(ProcessPoolExecutor(max_workers=4))
    timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')
    run_dir = pathlib.Path.cwd() / 'runs' / timestamp

    with contextlib.closing(app), Engine(executor) as engine:
        app.run(engine, run_dir)

    return 0


if __name__ == '__main__':
    raise SystemExit(main())

This script can be executed directly.

python app_example.py

TaPS Run Helper

Internally, the python -m taps.run CLI calls the run() function which handles creating all of the benchmarking objects based on a Config and executing the benchmark. Thus, run() is easier to use but is also higher level and therefore less customizable.

import pathlib
from datetime import datetime

from taps.apps.configs.cholesky import CholeskyConfig
from taps.engine import EngineConfig
from taps.executor.python import ProcessPoolConfig
from taps.logging import init_logging
from taps.run.config import Config
from taps.run.main import run
from taps.run.utils import change_cwd


def main() -> int:
    init_logging()

    config = Config(
        app=CholeskyConfig(matrix_size=100, block_size=25),
        engine=EngineConfig(executor=ProcessPoolConfig(max_processes=4)),
    )
    timestamp = datetime.now().strftime('%Y-%m-%d-%H-%M-%S')

    with change_cwd(pathlib.Path.cwd() / 'runs' / timestamp) as run_dir:
        run(config, run_dir)

    return 0


if __name__ == '__main__':
    raise SystemExit(main())

This script can be executed directly.

python run_example.py

A matrix of benchmark configurations can be performed, for example, by altering the Config each time and creating a new run directory (to avoid overwriting previous configurations).

Warning

A new Engine, and therefore executor, will be created each time run() is invoked. This can be inefficient for certain executors which take extended time to startup or shutdown.

Parameter Matrix Example

This example shows how to repeat benchmarks with the Cholesky Factorization application for a matrix of application parameters (matrix_size and block_size) using itertools.product. Here, we reuse the same Engine across all runs since the engine parameters are not being changed.

import contextlib
import itertools
import logging
import pathlib
from concurrent.futures import ProcessPoolExecutor

from taps.apps.cholesky import CholeskyApp
from taps.engine import Engine
from taps.executor.utils import FutureDependencyExecutor
from taps.logging import init_logging
from taps.logging import RUN_LOG_LEVEL

logger = logging.getLogger(__name__)


def main() -> int:
    init_logging()

    matrix_sizes = [100, 200, 300]
    blocks = [1, 2, 4]
    executor = FutureDependencyExecutor(ProcessPoolExecutor(max_workers=4))

    with Engine(executor) as engine:
        for matrix_size, n_blocks in itertools.product(matrix_sizes, blocks):
            logger.log(
                RUN_LOG_LEVEL,
                'Starting new run '
                f'(matrix_size={matrix_size}, blocks={n_blocks})',
            )

            app = CholeskyApp(
                matrix_size=matrix_size,
                block_size=matrix_size // n_blocks,
            )
            run_dir = f'matrix-{matrix_size}x{matrix_size}-blocks-{n_blocks}'
            run_dir = pathlib.Path.cwd() / 'runs' / run_dir

            with contextlib.closing(app):
                app.run(engine, run_dir)

    return 0


if __name__ == '__main__':
    raise SystemExit(main())

This script can be executed directly.

python matrix_example.py