PipelineOrchestrator

Role and Responsibilities

The PipelineOrchestrator is a key component of the EZStitcher architecture. For an overview of the complete architecture, see Architecture Overview.

The PipelineOrchestrator is the top-level component that manages all plate-specific operations and coordinates the execution of pipelines. It serves as an abstraction layer between the plate-specific details and the pipeline steps.

Orchestrator and Pipeline Relationship — The relationship between PipelineOrchestrator and Pipeline components.

Key responsibilities:

Plate Management: - Plate and well detection - Microscope handler initialization (specific to each plate type) - Image locator configuration
Workspace Initialization: - Creates a workspace by mirroring the plate folder path structure - Creates symlinks to the original images in this workspace - Ensures that modifications happen on workspace copies, not original data - Provides this workspace as the input for pipelines

For detailed information about directory structure, see Directory Structure.

Pipeline Execution: - Multithreaded execution across wells - Error handling and logging
Specialized Services: - Provides configured Stitcher instances suitable for the plate - Manages position generation specific to the plate format - Abstracts plate-specific operations that depend on the microscope handler

The orchestrator acts as a “plate manager” that knows how to handle the specific details of different plate formats, allowing the pipeline steps to focus on their image processing tasks without needing to know about the underlying plate structure.

Note

While a pipeline defines what processing to perform, the orchestrator controls how and where that processing is applied across a plate.

Creating an Orchestrator

from ezstitcher.core.config import PipelineConfig
from ezstitcher.core.processing_pipeline import PipelineOrchestrator

# Create configuration
config = PipelineConfig(
    num_workers=2  # Use 2 worker threads
)

# Create orchestrator
orchestrator = PipelineOrchestrator(
    config=config,
    plate_path="path/to/plate"
)

Plate-Specific Services

The orchestrator provides several plate-specific services that abstract away the details of different plate formats:

Workspace and Original Data Protection:

The orchestrator creates a workspace to protect original data:

# Create an orchestrator with a plate path
orchestrator = PipelineOrchestrator(
    config=config,
    plate_path="path/to/plate"  # Original plate path
)

# Access the workspace path (contains symlinks to original images)
workspace_path = orchestrator.workspace_path

Used by: Pipelines and steps use this workspace path as their input directory, ensuring that original data is protected from modification.

Microscope Handler: Understands the specific plate format and how to parse filenames

# The microscope handler knows how to interpret filenames for the specific plate type
microscope_handler = orchestrator.microscope_handler

# Parse a filename to extract components (channel, z-index, site, etc.)
components = microscope_handler.parser.parse_filename("image_c1_z3_s2.tif")

# Generate patterns for finding images
patterns = microscope_handler.auto_detect_patterns(input_dir)

Used by: The get_stitcher() method uses the microscope handler’s parser to configure the stitcher. The stitch_images() and generate_positions() methods use it to understand the plate format and parse filenames.

Position Generation: Generates position files for stitching
```
# Generate positions for a specific well
positions_file, _ = orchestrator.generate_positions(
    well="A01",
    input_dir=input_dir,
    positions_dir=positions_dir
)
```
Used by: The PositionGenerationStep calls this method to generate position files for stitching. Internally, this method uses the microscope handler and a stitcher instance obtained via get_stitcher().
Image Stitching: Stitches images using position files
```
# Stitch images for a specific well
orchestrator.stitch_images(
    well="A01",
    input_dir=input_dir,
    output_dir=output_dir,
    positions_file=positions_file
)
```
Used by: The ImageStitchingStep calls this method to stitch images. Internally, this method uses the microscope handler and a stitcher instance obtained via get_stitcher().
Thread-Safe Stitcher Creation:

The get_stitcher() method creates a new Stitcher instance configured for the plate:
```
# Get a thread-safe stitcher instance
stitcher = orchestrator.get_stitcher()
```
Used by: The stitch_images() and generate_positions() methods call this internally to get a thread-safe stitcher instance. Steps don’t need to call this directly.

Running Pipelines

For detailed API documentation, see PipelineOrchestrator.

The orchestrator can run one or more pipelines:

# Run a single pipeline
orchestrator.run(pipelines=[pipeline])

# Run multiple pipelines in sequence
orchestrator.run(pipelines=[pipeline1, pipeline2, pipeline3])

When multiple pipelines are provided, they are executed in sequence for each well. If num_workers is greater than 1, multiple wells are processed in parallel.

Pipeline Execution Flow — Pipeline execution flow with multiple wells and pipelines.

The execution flow is:

For each well in the plate (or well_filter if specified): a. Create a thread to process the well (if multithreading is enabled) b. For each pipeline in the pipelines list:
1. Run the pipeline on the well
2. Wait for completion before starting the next pipeline

This approach ensures that: - Multiple wells can be processed concurrently (controlled by num_workers) - Pipelines are executed in sequence for each well - Each pipeline has access to the results of previous pipelines

For more information on how pipelines are executed, see Running a Pipeline.

Orchestrator-Pipeline Relationship

The relationship between the PipelineOrchestrator and Pipeline is a key aspect of EZStitcher’s architecture:

Orchestrator, Pipeline, and Steps Relationship — The hierarchical relationship between Orchestrator, Pipeline, and Steps.

Responsibilities:

PipelineOrchestrator: Manages plate-level operations and multithreaded execution
Pipeline: Manages a sequence of processing steps and their execution
Step: Performs a specific processing operation on images

Communication Flow:

The orchestrator provides plate-specific services to pipelines
Pipelines use these services to execute their steps
Steps access the orchestrator through the pipeline’s context

Key Interactions:

The orchestrator creates a ProcessingContext for each pipeline
The context includes a reference to the orchestrator
Steps can access the orchestrator through this context reference
Specialized steps (like PositionGenerationStep) use orchestrator services

For more information on pipelines and their structure, see Pipeline.