Skip to content

taps.apps.configs.mapreduce

MapreduceConfig

Bases: AppConfig

Mapreduce application configuration.

Parameters:

  • name (Literal[str], default: 'mapreduce' ) –

    Application name.

  • data_dir (Path) –

    Text file directory.

  • map_tasks (int | None, default: 32 ) –

    Maximum number of map tasks (None uses one map task per input file).

  • generate (bool, default: False ) –

    Generate random text files in data-dir rather than reading existing files.

  • generated_files (int, default: 10 ) –

    Number of text files to generate.

  • generated_words (int, default: 10000 ) –

    Number of words to generate per file.

get_app 

get_app() -> App

Create an application instance from the config.

Source code in taps/apps/configs/mapreduce.py 
46
47
48
49
50
51
52
53
54
55
56
def get_app(self) -> App:
    """Create an application instance from the config."""
    from taps.apps.mapreduce import MapreduceApp

    return MapreduceApp(
        data_dir=pathlib.Path(self.data_dir),
        map_tasks=self.map_tasks,
        generate=self.generate,
        generated_files=self.generated_files,
        generated_words=self.generated_words,
    )