Carl Trachte: Using Generators and Coroutines to Merge Tabular Data (Drill Holes)

I have some mining drill hole data that I need to merge into an old vendor FORTRAN input format.  Basically I do a series of SQL pulls from the drillhole database to csv files, then merge the data.  My methodology has been a bit brute force in matching the separate parts of the drill hole data (lists, opening and closing of files to find matching holes, etc.).  My thought was that I could do this more elegantly and efficiently by iterating through the files with generators.

The ability of generators to communicate with each other via the send() method intrigued me.  I had always been a bit shy about using this language feature.  My csv problem gave me a justification for checking it out.

The reference I used was Dr. Dave Beazley's 2009 Pycon Tutorial.  He does a nice job of explaining things as well as dispatching good advice.  (I disobeyed the good advice in the interest of shoehorning coroutines into my solution; I'll cover this below.)  Beazley defines a coroutine in the sense of generators and the "yield" keyword as generators where "yield" is used more generally.  That is the context I'm using the word "coroutine" in this post.

Given my problem of a one (drill hole start survey) to many (drill hole interval values) relationship, I attempted a very simple (perhaps oversimplified) toy program demo of what I wanted to do with real data:

def coroutinex(subgenerator):
    """
    Generator function that consumes
    a key value sent from a higher
    level generator.  This generator
    yields two tuples of the form
    (<boolean>, data).  The boolean
    value indicates whether the key
    matches the data.

    Returns a generator.
    """
    while True:
        # One entry point for send()/reset.
        keyx = yield
        subdatatop = next(subgenerator)
        if subdatatop[0] == keyx:
            yield (True, subdatatop)
            for subdataloop in subgenerator:
                if subdataloop[0] == keyx:
                    yield (True, subdataloop)
                else:
                    yield (False, subdataloop)
                    break
 
def toplevelgen(topleveliter, coroutinex):
    """
    Top level generator function.

    subgenerator is a generator
    that this generator sends
    a key value to.  The
    subgenerator yields a two
    tuple that communicates if
    the key matches or not.

    Returns a generator.
    """
    # Get sub generator/coroutine initialized.
    coroutinex.send(None)
    # Variable for dealing with return
    # from sub-generator/coroutine.
    subvalue = False
    for keyx in topleveliter:
        yield keyx
        if subvalue:
            yield subvalue
        subvalue = coroutinex.send(keyx)
        # Get sub generator/coroutine re-initialized
        # after send() reset.
        if subvalue is None:
            # XXX - hack
            subvalue = coroutinex.send(keyx)
        yield subvalue
        for submessage in coroutinex:
            # XXX - another hack to deal with yield of None.
            if not submessage:
                continue
            subvalue = submessage
            # if submessage[0] is True, kick it out.
            if submessage[0]:
                yield submessage
            else:
                # Keep subvalue for after keyvalue
                # yield at top.
                break

topleveliter = range(44, 55)
keysx = [44, 44, 44, 45, 45, 45, 45, 45,
         46, 46, 46, 46, 46, 46, 46, 46,
         47, 47, 47, 48, 48, 48, 48, 48,
         49, 49, 49, 49, 49, 49, 50, 50,
         51, 51, 51, 51, 51, 51, 51, 51,
         52, 52, 52, 52, 52, 52, 52, 52,
         53, 53, 53, 53, 53, 53, 53, 53,
         54, 54, 54, 54, 54, 54, 54, 54]

sequencex = range(1, len(keysx) + 1)
subgenerator = zip(keysx, sequencex)

gensub = coroutinex(subgenerator)
genmain = toplevelgen(topleveliter, gensub)

for x in genmain:
     print(x)


Output:

44
(True, (44, 1))
(True, (44, 2))
(True, (44, 3))
45
(False, (45, 4))
(True, (45, 5))
(True, (45, 6))
(True, (45, 7))
(True, (45, 8))
46
(False, (46, 9))
(True, (46, 10))
(True, (46, 11))
(True, (46, 12))
(True, (46, 13))
(True, (46, 14))
(True, (46, 15))
(True, (46, 16))
47
(False, (47, 17))
(True, (47, 18))
(True, (47, 19))
48
(False, (48, 20))
(True, (48, 21))
(True, (48, 22))
(True, (48, 23))
(True, (48, 24))
49
(False, (49, 25))
(True, (49, 26))
(True, (49, 27))
(True, (49, 28))
(True, (49, 29))
(True, (49, 30))
50
(False, (50, 31))
(True, (50, 32))
51
(False, (51, 33))
(True, (51, 34))
(True, (51, 35))
(True, (51, 36))
(True, (51, 37))
(True, (51, 38))
(True, (51, 39))
(True, (51, 40))
52
(False, (52, 41))
(True, (52, 42))
(True, (52, 43))
(True, (52, 44))
(True, (52, 45))
(True, (52, 46))
(True, (52, 47))
(True, (52, 48))
53
(False, (53, 49))
(True, (53, 50))
(True, (53, 51))
(True, (53, 52))
(True, (53, 53))
(True, (53, 54))
(True, (53, 55))
(True, (53, 56))
54
(False, (54, 57))
(True, (54, 58))
(True, (54, 59))
(True, (54, 60))
(True, (54, 61))
(True, (54, 62))
(True, (54, 63))
(True, (54, 64))

Back to Dr. Beazley's advice - he doesn't recommend this - even though "yield" is the keyword, it means two different things in two different contexts.  Do not mix generator and coroutine functionality.  I'm going ahead in this post and doing it anyway.  I don't have an excuse.  It does remind me of some old Bob Dylan lyrics: