(0 comments, 4 posts)

This user hasn't shared any profile information

Posts by rachellopatin

Musical Mappings final post


Sam King and Rachel Lopatin

We started off our final project brainstorm by determining that we did NOT want to make a product that taught kids about science. Rachel  in particular noticed that most of educational tools and scaffolding discussed over the course of the quarter were designed to teach kids math and science concepts. What about the “fuzzier” stuff?  Rachel decided music would strike a good balance because it can be very mathematical (rhythm, etc.) but also has an emotional and artsy component that math/science doesn’t. Sam agreed to work with her on something related to music.

We brainstormed the first week. We were inspired by the Mario Teaches Typing game and wondered if we could make a similar game, but use notes to control Mario instead of keyboard keys.  We thought that trying to reimplement the Mario game with music, however, would be pretty hard.  We also thought about an interactive computer interface where you could take care of a virtual plant over time and play different notes to water it, give it fertilizer, etc.   That also led into our final idea: instead of trying to create a particular game or interface where you use music to make stuff happen on the screen, create a general game controller that used pitch instead of the normal controls that can be used for any game.

We decided that we’d need to work on a few components: the pitch recognition library so that we could determine what note the user actually inputs, a program that would take the note and simulate a key press, a window so that the user could set the mappings between notes and controls, and a feedback window to let the user know how close he/she got to a particular note (this would be based on the input frequency).

[design scribbles]

Sam already knew about AutoHotkey, which simulates keypresses, and wrote a nice script for it over the weekend.  Check.  Because we didn’t have the background knowledge to create our own pitch detection library (and because time was short enough that writing one from scratch would take 2 weeks by itself), we searched for open source libraries.  Surprisingly, despite pitch recognition being used even in simple mobile apps, we couldn’t find code online…or we found code but no documentation so we couldn’t figure out how to use it…or the library hadn’t been updated in 3 years and we weren’t even sure it was correct.  Finally we decided on FMOD, a framework that contains an API for pitch detection and manipulation.  Awesome!  The library was written in C, so our GUI (i.e., the windows to have the user set the mappings and give the user feedback on notes) would also have to be written in C or C++ so that we could combine the library with the GUI into 1 application.  Not so hard, right?  We did some Google searching and pretty much everyone said that Qt, a framework designed to streamline the prototyping and testing of GUIs, was the way to go.  We were on our way, this would be smooth sailing…

…until we realized that just because Qt was the best option out there, it did not mean it was actually good.  The documentation existed, but none of it was designed for complete beginners like us.  We spent 2-3 days installing the software (Rachel’s computer also conveniently died at this point so we had to get a loaner computer) and trying to integrate it with Visual Studio 2010.  The Qt website explicitly said that the newest version of Qt is compatible with VS 2010, but we had linking problems, unrecognized libraries, required .dlls that just didn’t get imported, etc etc etc.  Meanwhile, Rachel was able to use Qt on its own–kind of.  The plus side was that it was really easy to use Qt’s “Design” feature to create a GUI window and drag and drop the buttons and text labels you want on there.  The minus side was that it didn’t compile like it was supposed to, so no GUI window ever launched.  Rachel spent 2 days reading forum posts.  First, she realized that MinGW, a program that simulates a Linux compiler for Windows, was required.  It was listed as an option in the Qt installation, but it was under the “miscellaneous” tab, and nothing ever said it was necessary to make Qt work.  Grrrrrrr.  Okay, so once MinGW was installed, it still didn’t work.  Getting an error message like “Error 2: File not found: ” was not helpful, especially since there weren’t any files missing and it was a compiler issue.  Eventually Rachel, frustrated, decided to just start another project from scratch to see if that would compile.  This one worked!  What happened?  Still have no clue but best guess is because the original GUI was created under the “Quick project” option, which screwed something up.  By contrast, the second project was created under the “New widget” option.  Beware!!!  Very subtle bugs!!!

Sam was making no progress on the Visual Studio end, so decided to uninstall VS2010 and install VS2008 instead.  Lo and behold, 2008 works very smoothly with the Qt plugin.  We’re not sure why 2010 was so broken; it might actually be Microsoft’s fault rather than Qt’s.  Even so, a step-by-step configuration guide to install the Qt plugin and get it to work with VS2010 would have been helpful so we didn’t waste 10 hours troubleshooting it.

From there, it was a simpler case of coding the interfaces.  We already had the “look” of the GUIs through the drag and drop feature, but the buttons weren’t interactive yet.  By Googling a LOT of tutorials and searching through layers and layers of (mostly unhelpful) documentation on the Qt website, we were able to write functions that retrieved text from the input box and learned about slots and signals, Qt’s version of using callback functions.  While we understand why slots/signals are good design strategy, it was confusing at first to figure out how they worked, and in particular the syntax to invoke them.

The final challenge was figuring out how to make shapes be drawn on the screen using code.  We needed some sort of dynamically changing indicator in our feedback window so we could show people what note they actually hit (like, 2/3 of the way from an A to a B).  The drag and drop interface was wonderfully easy to figure out for the buttons in the Qt library, but you couldn’t drag a circle onto the screen, for example, and once placed on the screen it was static (couldn’t be animated/moved around).  Because putting a circle on the screen was a huge headache to program (although we thought it was the superior option from a design standpoint), we settled for a horizontal line with labeled increments for the different notes, and a bold vertical line + a label that would move to show you what note you just hit.


After that, Sam did some awesome ninja coding to get the library to work in conjunction with the GUI, and we did some testing and searched for optimal programs to showcase our mappings.  We wanted programs where you didn’t have to move too fast, because it’s pretty hard to get the pitch correct at first.  We chose a Neopets game and a paint program.  Both had one-key shortcuts that we could control with pitch.  We would have liked to be able to support two-key controls, but AutoHotkey doesn’t support them without a lot of tweaking.  We decided it was beyond the scope of this project but would be a cool extension.

Overall, we thought it was a pretty cool project.  Although we didn’t get a chance to test it with any kids, we did notice that Sam, who had no musical background, got WAY better at hitting pitches after practicing with our program for a couple days.


tl;dr Qt sucks but we managed to make a program that lets you control computer programs using pitch instead of the keyboard.


Finished program pics:




Sketching and design along the way:

interface sketch 1

interface sketch 2

code design 1

code design 2

Geographix: A Slate Toolkit


Geographix is a tangible toolkit designed to teach children geography. The pieces are made out of colored acrylic. We chose the South Africa region because the pieces are fairly regular in size, which is necessary when using the Slate backings. This region also works well for puzzle solving because the country borders are not straight lines (unlike in the US). This provides a scaffolding for learning what each country looks like on a map, because each country has a clearly unique shape. It also helps children learn where countries are positioned in relation to each other. We added the name of each country in each vinyl to the acrylic piece, which provides more scaffolding because this way they can realize which way is up, and can also connect the country name with the shape.

We prototyped our design in cardboard, which made us realize that our pieces were too small; Zimbabwe was not wide enough to encompass of the backing. We then enlarged our design by 150% and cut the pieces using the laser cutter.

[we would load images but we reached the upload limit…]

Our toolkit is suited to Slate’s vertical surface because it mimics real maps, which are traditionally displayed on walls instead of desks, and they are drawn so up and down correspond to north and south. In addition, it is easier to collaborate on puzzle solving when the pieces are mounted vertically. Finally, it’s easier to track your progress because you can stand back from the surface and still see the whole thing.

When people feel puzzle pieces, they will gain an appreciation for each country’s distinctive shape. For example, Namibia has a distinctive “dongle” on its northeastern border.

There were a few resources missing from Slate. There were no easily programmable objects, nor were we able to make relationships between objects. For example, we originally wanted to make a simulation of a plant growing, where the tangible parts of the toolkit included a watering can, clouds, fertilizer, etc. We soon realized there was no way to make a virtual plant grow, nor was there a way to make that virtual plant react to the watering can. We also felt limited by the criterion of vertical affordance: apart from physics simulations that require gravity, there is very little that is distinctive about a vertical surface. The vertical surface worked well for Mechanix, but limited our create options when designing a toolkit meant to work with Slate.

If we had more time, we would like to add extensions to Geographix. Our challenge mode would be a timed mode, to see how fast you can put together the puzzle pieces. Another challenge possibility is to add additional, fake country pieces to the set. Children would have to sift out the fake pieces from the real pieces to put together the puzzle. For our free play mode, we would like to create a custom background that has the outline of the African continent, as well as blue for the Atlantic and Indian Oceans, as further educational scaffolding. Lastly, we’d like to add support for a 2-player mode. This mode would include pieces for 2 continents, and each player would receive a random assortment of country pieces, some from each continent. The two players would have to work together to fit their pieces in with their partner’s.

Geographix demo

change initial distance

NetLogo Model: Sam and Rachel






This lets you generate regular geometric shapes. The inspiration is a snowflake, with the idea of a snowflake being generated regularly and recursively.


At each tick, all particles in the snowflake will create num-branches children around it. Each child is at a distance of branch-distance, which will get smaller with each tick. Each child is spaced regularly around its parent with a random number of degrees specified by the upper and lower bounds that the user chooses. It continues until max-order ticks have happened.


setup creates a particle in the center go runs everything max-order is the number of times children will spawn num-branches is the number of children that will spawn around each parent initial-branch-distance is the distance between the initial particle and its children.
lines? if true, will draw lines between children and their parents colors-change? if true, each child is darker than its parent. If order is high enough, the colors can wrap around to white again. if false, everything is white.
random-min-degrees, random-max-degrees: if both of these are 0, then children will be regularly spaced around the parent. When they are not 0, then they are used as a lower and upper bound on a random number of degrees added to the regular spacing for each child. When the lower bound is 0 and the upper bound is 360, the spacing will be completely random. branch-distance-drop-factor: if the initial branch distance is 100 and the drop factor is 2, then the initial particle and its children will will be 100 apart. The next generation will be 50 apart. The fourth generation will be 25 apart, etc.


Getting a Beautiful Snowflake

After setting up, the snowflake starts with one particle at the center:

The above setup doesn’t look too interesting because it’s a completely random snowflake:

However, with the variables at reasonable values, it begins to resemble a snowflake (or at least an interesting geometric shape):

One quick realization was that we were exponential. The number of particles in the last iteration of snowflake with order O and B branches was B^O. Thus, even when the variables were pretty small, the computation grew slow and the snowflake became solid ice. Order 4 could only really take 7 or 8 branches, and order 5 could only really take 3 branches before the shape seemed cluttered. Here is what we get with order 4 and 10 branches:

Here is one with more reasonable variables:

This one shows how cluttered it can get with high orders:

With order 4, everything looks too regular, so it gets pretty cheesy:

However, by changing the distance drop factor, we can make it spread out a little more, and it looks very nice:

Changing the initial distance doesn’t change the shape at all (since the shape is all regularly generated), but it makes everything more compact, so it’s harder to see the spaces in between the particles of ice, and it’s harder to appreciate the space as a whole.

If we make the image all white instead of having each successive generation be bluer, then the image also seems worse. Even though the earlier images practically seem all teal, the little spots of white really do matter. Having a monochromatic image makes it much harder to see the complexity within each generation:

Also, it appears as though much of the complexity comes with the regularity. Even 10 degrees of randomness at in each child can make the image as a whole seem very scattered:

30 degrees makes the snowflake devolve into a discordant vortex:

50 degrees makes it unclear even how many branches there are:

With a sufficiently small drop factor and a sufficiently large branch distance, it was possible to appreciate the maze-like repetitions within even high-order, high-branch combinations:

But again, it seems as though the snowflake-ness of a shape depends on the empty space:

There was also a big odd/even divide. With an odd number of branches, the images seemed less like snowflakes and more like circularish blobs:

However, lowering the drop-factor even more proved fruitful, even for an odd number of branches:

When the drop factor is eliminated, it looks even cooler: every child is as far from its parent, which means that the density of the snowflake decreases exponentially the farther it gets from the center, and because of a slight offset , there are a bunch of very fine and intricate lines:

Last, if we eliminate the lines connecting particles, the snowflake becomes different; neither better nor worse. It loses intricacies in some ways and gains them in others.

As may be evident by this point, the emergence that we were studying was the aesthetic value that came from the regular reproduction of similar agents. Most of the time we spent was trying to get pretty particles and make things that looked more like snowflakes. We made several changes to our model to help with this. For instance, we started with the distance between a parent and child particle set at a particular interval of the initial branch distance. That proved very ugly, so we changed it to a factor such that the distance would decay geometrically. This proved much nicer, especially with high orders. Another big change was to make it so that every child turned 45 degrees. This made it so that its children didn’t wind up on top of the link with its parent.

The best part was that, by programming a relatively simple model, most of the experimentation didn’t even have to do with the programming. We just needed to play around with the variables. Even though the variables might not have much of a correspondence to physical properties of snowflakes in the natural world, it still allowed an exploration of aesthetics and the discovery of the importance of variations, their regularity, variations of color, and variations of space.

breed [particles particle]

order ;;order 0 will make no children. Order n will make order n-1 children.

to setup
set-default-shape particles “dot”
;;ask patches [set pcolor black] ;;sets the background to black
create-particles 1
setxy 0 0
set color 89.9
set order max-order
set branch-distance initial-branch-distance

to go
repeat max-order
wait 1

to create-new-order
ask particles with [order > 0]
set order 0

to make-kids
left 45 ;;so that my child and my parent are not in the same line
let i 0
repeat num-branches
hatch 1
left (i * (360 / num-branches ) )
left bounded-random random-min-degrees random-max-degrees
forward branch-distance
set order (order – 1)
set branch-distance (branch-distance / branch-distance-drop-factor)
if colors-change? [set color (color – 1)]
if lines? [create-link-with myself]
set i (i + 1)

to-report bounded-random [lower upper]
let range (upper – lower)
let rand-num random range
report lower + rand-num


rachellopatin's RSS Feed
Go to Top