Creating a Processing Pipeline with Coroutines

Snippets of this code a have already been covered, now lets put it all together to make a processing pipeline in python.  There has to be atleast three parts to a make the pipeline useable, a datasource, a filter, and a sink.  The datasource will be a tail -f function and the filter will be a grep function, both of which I already covered in an older post.  The sink in this case will just be a function that prints the filtered out data to standard out. This example is covered in David Beazley’s example copipe.py

dabeaz.com

import time 

def coroutine(func):
    def start(*args,**kwargs):
        cr = func(*args,**kwargs)
        cr.__next__()
        return(cr)
    return(start)

def monitor(file, target):
    file.seek(0,2)      
    while True:
         line = file.readline()
         if not line:
             time.sleep(0.1)    
             continue
         target.send(line)

@coroutine
def grep(pattern,target):
    while True:
        line = (yield)
        if pattern in line:
            target.send(line)    

@coroutine
def printer():
    while True:
         line = (yield)
         print(line)

if __name__ == '__main__':
    monitor( open("aFile.txt"),
           grep('python',
           printer()))
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s