It tells me that BAC, KEY, MI, RF, SNV, and STI are related. They’re all acronyms, yes, but more than that. They are all banking stock symbols. The node that the arrow points to contains these values. All banks. The nodes on either side contain exclusively real estate holding companies and home builders respectively.

This puts a huge grin on my face.

If you’ve talked to me in the last several months, I’ll have probably mentioned at some point that I’ve been learning some machine learning concepts. I’ve been watching and working through the examples in the Stanford machine learning course by Andrew Ng. On the one hand, the course is excellent. Andrew clearly knows his stuff and teaches toward underlying theory and principle. His goals are exactly how I like to approach a new area of study; always asking why not how. The concepts are compelling but on the other hand the math is difficult for me and it is lacking the kind of “proof is in the pudding” mentality that I’m used to as a programmer. I decided that I also need to approach this topic from the practical side. I’ve just discovered RapidMiner and have been playing with it recently.

My chosen problem set is stock data. I love the uncertainty inherent in the market, its a mass of data, action and reaction. One problem (an easy one to start with) that I’ve always wanted to work on is stock correlation. Simply put, stocks that are similar move together. If you have two businesses that are similar in industry and size, they will likely move together as they have similar economic environment. News that affects one is much more likely to affect the other than a third unrelated company in another industry. This relationship can be coaxed out of the data. For each stock, you should be able to calculate a web of close “neighbors” that move similarly, and moving out from there you may approach another “neighborhood” of related stocks. In machine learning this problem could be approached as a time series or as clustering. Since we don’t know the labels (names of the clusters so to speak) it’s not classification.

After many false starts, I grabbed some stock data and loaded it into RM with the built in jdbc tools they have. I then selected stocks for the last two years without missing data points on days where volume was greater than zero and pulled the set into a hierarchical clustering algorithm. The hierarchical clusterer uses an internal simpler one level cluster (kmeans) and applies it it recursively and in parallel. I also had some promising results with a correlation matrix which showed for example that INTC (Intel) and MU (Micron) are related much more closely than MU and KFT (Kraft).

Its great to be able to test out some of the things that I’ve been learning about in an environment that lets me try a lot of things in a relatively short amount of time. Up next: Better clustering algorithms, and using class labels from my clustering to train a model for prediction.

https://www.joshuafrankamp.com/svn/incubator/simpleGroovyCalendar/src/

I read up on groovy’s site, about google data support but I actually couldn’t find the referenced code anywhere… I’d like to learn more about ‘use’ and other aop/mixin techniques.

I knew groovy collections supported an “each” function that takes a closure. Groovy syntax allows for omission of parentheses when there is at least one argument, so it can look like this:

myCollection.each{ closure } not myCollection.each({ closure })

it would also allow you do things like:

myCollection.add object  (look no parens!)

which creeps me out right now, so I’m not going to think about it.

I digress… Here is my method, that I thought was a short implementation:

int getDuration(){
	int hours = 0;
	getDaySizeList().each{dayHourMap -> hours += dayHourMap.hours}
	return hours
}

I was excited, no visible loop, just something to do “each” for the members of the collection, drilling down into the hours property of the map in each collection item, summing them into a waiting “hours” int, then returning it. Slick.

Barney then replied that I should have (duh) used the sum closure-taking builtin in groovy… which turns the above method into this:

int getDuration(){
	return getDaySizeList().sum{it.hours}
}

“it” is the default single argument to the closure.  You can see I only kept the end of the closure, to the right of the former +=.

Magic.

I have most of the front end of the app done, but I needed a good calendar view to show the events. I decided I would use google calendars, so I wrote a little calender wrapper. Here is a snippet of the test driver class. GoogleCalendar is mine, the *Entry items are part of the api.

This is groovy code, not java…

//omitting calendar name assumes the default users calendar
GoogleCalendar gc1 = new GoogleCalendar(username,password)
// adds an event 3 days in duration starting 3 days from now, with title and desc here
CalendarEventEntry entry = gc1.insertEvent("DELETE ME FTO","FTO",new Date()+3,new Date()+6)
gc1.deleteEvent(entry)

println("List of users owned calendars:")
for (CalendarEntry cal : gc1.getCalendars()) {
	println("\t" + cal.getTitle().getPlainText());
}

//this demonstrates the mixed constructor that is allowed in the groovy wrapper.
//i define a calendar name, and it looks it up and sets that as the calendar to work with
GoogleCalendar gc2 = new GoogleCalendar(username,password,"Web Services Team")
//adds an event 3 days in duration starting 2 days from now, with title and desc here on the WST calendar
gc2.insertEvent("Joshua FTO","FTO",new Date()+2,new Date()+5)

Such fun.

This will allow me to have a nice visual display of everyone’s scheduled vacation calendar, and have an easy way to share it with other apps/users such as our wiki.