### The Idea

A little while ago, my friend posted this amazing picture to Facebook:

A long thought process followed, but it ended with a thought that boiled down to

```What if I took these charts, converted them to coordinate sets, and wrote
a 26-part vocal piece in which each performer was assigned a phoneme for
a letter of the alphabet and they recited their phonemes in rhythms
that matched the relative frequencies for each point on the X axes?
```

For example, if `A` appears at the beginning of a word twice as often as `B` and three times as often as `D` (which they don’t but it makes the example shorter), the opening of the piece would have one vocalist repeat `/æ/` 6 times, another repeat `/b/` three times, and a third vocalist `/d/` twice.

Obviously there’s a rather long line between a .jpg on Facebook and a complete musical composition, but broken down into what I hope are manageable (and discretely bloggable) steps, it looks something like this:

1. extract the 26 charts from the original image
2. convert each chart into a coordinate set
3. parse the 26 coordinate sets into a set of polyrhythms for each point on the X axis
4. determine the musicality of the resulting polyrhythm sets and add dynamics/pitches/vowel sounds to the phonemes
5. ???
6. profit

And what better place to start that at the beginning?

### Step 1 - extract the 26 charts from the original image

This was easy enough. Using the wonderful Acorn image editor, I was able to crop and extract the graphs into 26 images of identical width (important for generating identically sized coordinate sets for each letter).

I ended up doing this manually with a lot of cropping and copying and pasting and exporting. There may well be easier ways to do this – processing the image, detecting horizontal swathes of blank space of a certain height, and splitting the image at them – but this didn’t actually take that long, and it wasn’t the most interesting problem to solve in this project. I ended up with 26 images that looked similar to this:

Simple enough. On to step 2.

### Step 2 - convert each chart into a coordinate set

I was looking forward to this step. I’ve played around with Processing some in the past, often as a canvas for attempting to build mazes out of the texts from Italo Calvino’s Invisible Cities (which is a topic for a whole other blog series). The task at hand – parsing the pixels of an image to try to find the coordinates for each graph – seemed like something Processing was suited for.

A bit of experimentation later, I had what seemed like a working function to find and mark the graph line on one of the letter charts. My plan (annotated below) was to work through each vertical row of pixels and find the darkest pixel in the column (eyeballing it suggested this should be the top line of the graph).

Running this on the chart for `A`, we get:

Hey! Not bad for a first try. But, if we look closely, we can see that the very leftmost green pixel is down in the middle of the graph, and over on the right the highlighted line actualy drops below the X axis.

And at a few points it looks like some of the chart line is above the line we’ve drawn. This appears to be because the line on the original image is not a single color, but has a darker band in the center. I would assume this has something to do with the quality of the original image, and the subsequent, aforementioned, copying and pasting and exporting. If I knew anything about how digital graphics work, I’m sure I’d have a better explanation, but I don’t. If you do, I’d love to learn.

As it stands, these are issues that could be handled by massaging the resulting data by hand after the fact, but this has been fun, and I bet I can get Processing to do better. So let’s go.

My second thought was to take the same pixel averages as above, but chunk them into groups, find the group with the lowest average, and use the highest pixel in that chunk, but the fact that that graphs are filled in gave me some trouble with this.

I briefly played around with comparing the average of each chunk with the chunk above, looking for the first large jump in value (i.e. the switch from the white-ish space above the graph to the top of the line). And hey, look at that!

Now we just need to save the coordinates we’ve found.

We’re almost done, but there’s one small issue with the values we have for the Y coordinates. What we’re storing here is the distance from the top of the image to the line (remember, [0, 0] is the top-left corner). What we really want is the distance up from the X axis of the graph to that point. Of course, this means we need to find that X axis line.

We can reverse the formula we used to find the top of the graph, starting from the bottom and working up until we find a drastic change. Unfortunately, the bottom of the graph is harder to detect, since the largest color difference already occurs between the top of the graph and the blank space above it. The problem, again, stems from the impurity of the colors due to the quality of the image. What we need is a way to reduce the number of subtle shade variations in the image.

And hey, I bet we can find one of those. What if we could convert the graph image to pure black and white, making it much easier to find the edges.

And we have:

It’s not perfect, but the bottom axis looks right, which is all we care about here. Now we can iterate up from the bottom and find the first black pixel from the bottom. Since the X axis is a straight line, we only need to find the value for one column.

There we go!

Now the last step is to combine everything we’ve done above into the final coordinate set.

We know the Y value of the bottom of the graph, so for each coordinate in the graph line we can subtract from it that Y value to get a value we can graph on a standard Cartesian graph:

Now we just do that 25 more times, and we’ll be back next time to start turning this into music!

Thanks for reading! If you liked this post, and want to know when the next one is coming out, follow me on Twitter (link below)!