Building Filter Pipelines in C#

Overview

During development of our .NET Core generic host extension Hosuto we had the requirement to make it extensible and configureable. So I was thinking about how to implement a chain of filters that can be used to add new features without changing the core implementation.

The basic parts of this implementation are explained here. If you would like to see how it used in the real world see Hosuto's github repo. It was mainly inspired by the AspNet.Core middleware pipeline - which is actually nothing more than a filter pipeline.

Design pattern

This is not the Pipe and Filters Pattern, as the pipeline built here doesn't return the result of the processing. The AspNet.Core Middleware Pipeline is actually more a Chain of Responsibility design pattern implementation — and that's what we will also create here.

Filter it!

We will start here with the Visual Studio template for a Console application. You know: Hello world!

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hello World!");
    }
}

Next we declare a interface for filter:

public interface IFilter<T1>
{
    Action<T1> Invoke(Action<T1> next);

}

The input and the result of the method "Invoke" is an Action, which has a generic value as input. If the method is called it can process the value in the returned action. As long as the type T is immutable, it cannot change the value, only replace it.

It can also decide whether to stop processing or call the next action with the value to continue the pipeline.

Here two simple filters that can be used as string filters:

public class ToUpperFilter : IFilter<string>
{
    public Action<string> Invoke(Action<string> next)
    {
        return stringValue =>
        {
            next(stringValue.ToUpper());
        };
    }
}

public class StopHelloWorldFilter : IFilter<string>
{
    public Action<string> Invoke(Action<string> next)
    {
        return stringValue =>
        {
            if (stringValue != "Hello world!")
                next(stringValue);
        };
    }
}

Building the pipeline

To create a pipeline each of the filters have to be called one after the other, where the input of next filter will be the action result of the previous filter.

But how do you code a pipeline of these filters?

Here it comes:

public static class Filters
{

    public static Action<T1> BuildFilterPipeline<T1>(
        IEnumerable<IFilter<T1>> filters,
        Action<T1> filteredDelegate)
    {
        if (filteredDelegate == null) throw
            new ArgumentNullException(nameof(filteredDelegate));

        return (p1 =>
        {
            var filterList = filters.Reverse().ToList();
            if (filterList.Count == 0)
            {
                filteredDelegate(p1);
                return;
            }

            var pipeline = filterList.Aggregate(
                filteredDelegate, (current, filter)
                    => filter.Invoke(current));

            pipeline(p1);

        });
    }
}

This method first reverses the order of the filters and converts them to a list (see below why we reverse the order). If there are no filters, only the delegate is called. If not, the filters are chained with IEnumerable.Aggregate.

Instead of method Aggregate you could also use a foreach loop to do the same:


            var pipeline = filteredDelegate;
            foreach (var filter in filterList)
            {
                pipeline = filter.Invoke(pipeline);
            }

            pipeline(p1);

Hello world 2.0

You will probably already guess what comes next We will use the filters to influence how programs prints 'Hello world'!

static void Main(string[] args)
{
    var filters = new IFilter<string>[]
    {
        new ToUpperFilter(),
        new StopHelloWorldFilter()
    };

    Filters.BuildFilterPipeline(filters, Console.WriteLine)
        ("Hello world!");
    Filters.BuildFilterPipeline(filters, Console.WriteLine)
        ("Some new greeting!");
}

If you run this program output will be like this:

HELLO WORLD!
SOME NEW GREETING!

Ohh, what happened here? The first filter changed string to uppercase, and as second filter is case sensitive...

The output will change if we change the order of filters:

SOME NEW GREETING!

So the order of filters is important!

They are chained by the pipeline from the last to the first. This means that the first is called first, then the second, and so on. If we would not reverse the filter order in method BuildFilterPipeline the last one would be called first.

Use it!

So this is already a quite good result and works exactly like the AspNetCore middleware. However, we can extend it a bit more.

So imagine that you have to build a output processor for strings. It will just take a string as input and writes it:

public class OutputProcessor
{
    public void WriteLine(string input)
    {
        Console.WriteLine(input);
    }
}

Of course, the output processor could be a bit more flexible. So let's use our filters to make the output configurable with some fluent code:

public class OutputProcessor
{
    private readonly List<IFilter<string>> _filters =
        new List<IFilter<string>>();

    public OutputProcessor UseFilter(IFilter<string> filter)
    {
        _filters.Add(filter);
        return this;
    }

    public void WriteLine(string input)
    {
        Filters.BuildFilterPipeline(
            _filters,
            Console.WriteLine)(input);
    }
}

static void Main(string[] args)
{
    var outputProcessor = new OutputProcessor()
        .UseFilter(new StopHelloWorldFilter())
        .UseFilter(new ToUpperFilter());

    outputProcessor.WriteLine("Hello world!");
    outputProcessor.WriteLine("Some new greeting!");
}

Use it! functional

Ok, that will also work quite well. But some days later a collegue from the AspNet.Core dev team ask you to add a filter like the IApplicationBuilder.Use method - a filter that can be declared without a type, just with a function.

You agree - as functional programming style is always better - so add it:


public class UseDelegateFilter<T> : IFilter<T>
{
    private readonly Func<Action<T>, Action<T>> _filterFunc;

    public UseDelegateFilter(
        Func<Action<T>, Action<T>> filterFunc)
    {
        _filterFunc = filterFunc;
    }

    public Action<T> Invoke(Action<T> next)
        => _filterFunc(next);
}

public class OutputProcessor
{
    private readonly List<IFilter<string>> _filters =
        new List<IFilter<string>>();

    public OutputProcessor UseFilter(IFilter<string> filter)
    {
        _filters.Add(filter);
        return this;
    }

    public OutputProcessor Use(
        Func<Action<string>, Action<string>> filter)
    {
        _filters.Add(new UseDelegateFilter<string>(filter));
        return this;
    }

    public void WriteLine(string input)
    {
        Filters.BuildFilterPipeline(
            _filters,
            Console.WriteLine)(input);
    }
}

// Main:

static void Main(string[] args)
{
    var outputProcessor = new OutputProcessor()
        .Use(next => s =>
        {
            if (s != "Hello world!")
                next(s);
        })
        .Use(next => s =>
        {
            next(s.ToUpper());
        });

        // ...

As you can see we no longer have to use the filter types. Instead everything is declared in functions.

With more features!

Finally someone ask you to extend the processor further:

It should be possible to add additional output processors
They should be processed after filtering, but before writing to the console.

Ok, more features are always a good thing. Let's implement that, too:

public class OutputProcessor
{
    private readonly List<IFilter<string>> _filters =
        new List<IFilter<string>>();
    private readonly List<IFilter<string>> _actions =
        new List<IFilter<string>>();

    public OutputProcessor UseFilter(IFilter<string> filter)
    {
        _filters.Add(filter);
        return this;
    }

    public OutputProcessor Use(
        Func<Action<string>, Action<string>> filter)
    {
        _filters.Add(new UseDelegateFilter<string>(filter));
        return this;
    }

    public OutputProcessor With(Action<string> action)
    {
        _actions.Add(new UseDelegateFilter<string>(
            next =>
                s =>
                {
                    action(s);
                    next(s);
                }));

        return this;
    }

    public void WriteLine(string input)
    {
        Filters.BuildFilterPipeline(
            _filters.Concat(_actions),
            Console.WriteLine)(input);

    }
}


static void Main(string[] args)
{
     var outputProcessor = new OutputProcessor()
        .With(s =>
        {
            Console.WriteLine(
                $"I cannot prevent it, but I know you will write string '{s}'");
        })
        .Use(next => s =>
        {
            if (s != "Hello world!")
                next(s);
        })
        .Use(next => s =>
        {
            next(s.ToUpper());
        });

    outputProcessor.WriteLine("Hello world!");
    outputProcessor.WriteLine("Some new greeting!");
}

If we now run the program again output will be:

I cannot prevent it, but I know you will write string 'SOME NEW GREETING!'
SOME NEW GREETING!

Conclusion

As you can see, building your own middleware or filter pipeline is not as complicated as it may seem.

You can use it for middlewares, extensible builders, dependency injection and much more. All you need at the very beginning is this interface:

public interface IFilter<T1>
{
    Action<T1> Invoke(Action<T1> next);
}