Architecture

Exploring what kind of architecture is the most applicable.

The pipeline consists of various analyses and therefore forms of data that are sequentially manipulated and forwarded to the next part of the pipeline. After looking at the various data flow architectures, the most immediately applicable architecture seems to be Pipes and Filters.

Pipes & Filters

Pipes and filters architecture is commonly used in data manipulation. Data passes through a set of services that manipulate or use the data (filters) using pipes (such as HTTP calls or files). This makes a kind of chain that allows for easy tracking of data throughout the pipeline. A benefit here is the modularity of the architecture: certain filters can be swapped out or changed without affecting other filters. Conceptually, it looks like this:

Source: Microsoft

Implementation

By splitting the pipeline into different services, the same structure as the above image can be implemented in the Avatar Validation pipeline too. More info here.