One of my favorite quotes is this:

It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself for a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat – Roosevelt

What this means to me is that I am getting comfortable not with success, but with failure. And with failure, slowly, haltingly, comes some measure of success.

cloud surfing

At the beginning of 2010 at my job at Mentor I enjoyed moving some more systems to AWS from an acquisition’s internal server farms. I closed out six years at Mentor; an excellent chapter in my life that I was sad to see end. I miss my co-workers, and agree with Barney “Twitter is such a poor excuse for seeing them every day.”  When I put my notice in, most were happy for me, some could hardly believe what I was doing. Others didn’t actually believe that I did not have a position to go to at another safe corporation. They kept asking what my real plan was, and I kept replying: “I’m taking a year off to study, to grow, to try new things.”

Technologies used: Bash, Railo, AWS API’s, Python

life changes. no really it does!

I
quit my job in March,
moved out of down town in June,
rented a big old house a week later,
became a foster parent in July,
and enrolled as a fake student at PSU.

it slices, it dices. well no, actually it only slices.

I had a poor experience with a cloud dashboard company and built autosnappy.com as a response. It makes snapshots for AWS volumes on a schedule. So simple. Happy dance. It broke even almost immediately, and although it doesn’t make tons of money, there is a lot of room here for growth. A customer is asking to pay for me to develop new features for it currently so I may revisit it and roll out new things. I wanted to build this as compartmentalized as possible. One of my design goals was that the front end know as little as possible about the backing AWS services as it could. So it talks exclusively to the middle python tier, even though it is powerful enough to accomplish both functions. This means that if I needed to I could scale those components separately, and keep the user facing process in a separate linux user/group as the process that talks to AWS and has the security keys.

Technologies used: Java Magnolia and Railo templating front end, python web service backend,  AWS SimpleDb storage. Scaleable! Stateless! Cloud!

on being a fake student

Being a fake student is the best! When you take a one credit class at PSU you have access to super high speed internet, a research library, and a place to go work. I spent a large part of my year hacking code in Food For Thought cafe, next to artists, musicians, and hippies. But that’s not all. I also have access to the new(ish) Rec Center which has a pool, spa, rock wall and all the normal workout stuff.

Oh, and my one credit class? Yoga. So stressful.

Next post: Basic Tools of Science and Flying Helicopters Upside Down

Algorithm Notes

This video is a bunch of visualizations put back to back of the process described below. The algorithm is run against piece 1 of puzzle 1 of the challenge, shown below. You’ll note in my visualization it is upside down, this is simply due to the start of x,y in the graphing application is at the lower left, while the image library gave me x and y starting from upper left.

Edging

Since I had very little time, I wanted to use as much existing code as possible. To find edges I used gimp/python-fu to posterize each piece 5 times. Each time that ran, it simplified the colors of the image. I then saved out the green channel. This made the edge finding as simple as finding the uniform green edge pieces and rolling around each piece. The edge I found is shown in the right hand side as black pixels.

Segmentation

Because each piece could match on only part of another piece, I wanted to find good matches based on local shape part matching. I chose to segment the piece by rolling through each location with surrounding bits by distance. The red X represents the current location, surrounded on either side by an arbitrary sized window shown as blue highlight.

Rotation Invariance

Each piece is given to you in a random rotation. So the segments between pieces needed to be fitted together in such a way as to ignore which way they were originally rotated. So next I found a ‘landscape horizon’ using a linear average of the segment. I chose PCA because of the x/y arbitraryness in this problem. This is because ordinary least squares only penalizes on the y term, and I needed total least squares so it would penalize on both terms equally. This is shown as a red line.

Rotate!

After finding the horizon, this becomes the new relative x axis for that segment. I use a linear transformation to rotate to a normalized inward facing view, and then transpose it to be centered at zero zero. This is the lower left portion of the video. Most flat sides are nearly equal to the x axis at that point, with variation easily seen. If you watch this window alone as you see the video it is as if a camera is going around the piece looking in on it with smooth transitions.

Representing Shape

Next I built a representation of surrounding shape at every edge location based on the new normalized x,y from the previous step. Surrounding shape should include other markings page lines and pen marks, but I haven’t included them yet. I converted the rotated Cartesian coordinates to a log polar system, and then used binning to produce a log polar 2d histogram of the surrounding shape. This is the upper right picture. I adjusted my binning to be more sensitive to changes in y at the horizon rather than in the middle because most shapes are fairly close to the x axis after they have been rotated. Each histogram results in 100 total bins, each with a count of the items in the segment. The degrees are left to right, while distance is the vertical dimension. Inspiration for this shape representation came from Shape Matching and Object Recognition.

Cluster

Using the 100 integer histogram arrays of shape I then clustered them using K means so that I could simplify the comparisons between groups of ordered segments. The result is that I can classify a segment’s shape as a single cluster membership, and an array of segments (a side of a piece) as an ordered list of those cluster assignments. (Not Shown)

Fuzzy Segment-Cluster-Array Matching and Reassembly

The next task is to match segment cluster arrays with others that are closest, indicating a general shape match. (Not Shown) This produces lists of candidate matches to aid in a manual or automatic assembly.

Running Out Of Time

At about this point the contest ended. I am still working on getting the first puzzle completely automatically assembled. I learned a lot, and had a good time but the primary takeaway was this: visualize, visualize and then when in doubt, visualize. Visualization costs a lot, and building pretty graphs for yourself is a complete waste of time if you are a perfect coder. Countless times throughout this process I had implemented a part of it, and moved on but only later came back and visualized what I had actually done. Often that immediately showed what went wrong. One mistake that it helped me find was that the rotation and transposition that I had originally done were rotated correctly but transposed based on my original center, not the rotated center, so the histograms were erratic. Once I could see that the middle was not 0,0 (in the lower left picture) I immediately knew what the issue was. Slowly I came to learn that every minute spent visualizing a problem like this pays off immediately.

easy_install is the easiest thing since sliced bread. What does it do? Everything. Its so magic it probably installs itself recursively just for fun.

You want a package?

Ok just type this: easy_install sqlalchemy (for the awesome ORM package for python)

It magically goes and finds sqlalchemy, and installs it INTO your system python installed path.

Why is the standard assumption that if I want to use a python package that is say a dependency for my project, that I want to INSTALL IT INTO PYTHON running on my system?

What kind of crazy idea is this? It causes all kinds of issues. The first and most obvious is: What If I have two programs that expect different versions of a given package? Since the packages are installed in to the runtime and not my app, you have to know about this issue and work around it.

If packages were managed the java way, the assumption would be that I want to install the package in the app that I am working on, not into /systemjdk/extensions/somePackage

The only argument FOR doing it this way that I can think of is saving disk space. Disk space is cheap.

/rant.

Ok so honestly, can anyone tell me why this is?

Comparing Crawled Pages Across GSA’s (or mini’s)

You could use this to compare any two url xml files from the Crawl Diagnostics –> Export All Pages To a File
I could have called them A and B, but I was comparing a mini and a gsa at the time, so the naming convention in the file goes. It uses simple python sets to see what is crawled in one machine, but not the other etc. Expects two local files, mini-urls.xml and gsa-urls.xml; either could be a gsa or mini export.

import xml.dom.minidom

def main():
    miniUrlSet = extractUrls(xml.dom.minidom.parse('mini-urls.xml'))
    gsaUrlSet = extractUrls(xml.dom.minidom.parse('gsa-urls.xml'))
    print 'mini', len(miniUrlSet)
    print 'gsa', len(gsaUrlSet)
    print 'intersections', len(miniUrlSet & gsaUrlSet)
    print 'mini is sub of gsa?', miniUrlSet <= gsaUrlSet
    gsaNotMini = gsaUrlSet - miniUrlSet
    print 'things in gsa but not mini:', len(gsaNotMini)
    for i in gsaNotMini:
        print i
    miniNotGsa = miniUrlSet - gsaUrlSet
    print 'things in mini but not gsa', len(miniNotGsa)
    for i in miniNotGsa:
        print i

def extractUrls(dom):
    nodelist = dom.getElementsByTagName("loc")
    urls = set()
    for node in nodelist:
        urls.add(node.firstChild.data) #i know all loc nodes have a single child text node, text nodes have a data property
    return urls

main()

Calculating Search Keywords Density Over Time

This is calculated against an export from the search logs feature under status and reports. You export the timeframe you want to compare over, and then run this against the log file. You get the top 100 keywords that people searched for, and the counts of how many times they were searched. Expects a local file log.log.

from datetime import datetime
from operator import itemgetter

def getQueryCounts(f):
       import re
       words = {}
       qReg = re.compile('.*?&q=(.*?)&')
       for l in f:
              keyword = qReg.findall(l)
              if(len(keyword) and len(keyword[0])):
                  words[keyword[0]] = words.get(keyword[0], 0) + 1
       return words

start = datetime.now()
f=open('log.log')
words = getQueryCounts(f)
f.close()
top = sorted(words.iteritems(),key=itemgetter(1),reverse=True)[:100]
print 'Top Words'
print '---------'
for word, num in top:
       print word, num
print 'runtime:', datetime.now() - start
raw_input("press enter")

I decided to go with python, which has a plethora of tools. Pylons is a piecemeal web framework that is closest to my liking, migrate is a library for schema migration, which works nicely with sqlalchemy, a monster orm.

sqlalchemy is cool because you can use parts of it totally independently. Coming from a CF background I am used to having nice named/pooled connections that I don’t have to think about. The base layer of sqlalchemy is that, a database type abstraction and pooling. Then you are free to go crazy with ORMish things or not, its up to you.

It is so reusable many people have written layers on top of it for even more magical coding… but its nice to have all the options.

Migrate, a RoR knockoff is the real find though, it looks young (as far as a project goes) but I watched a demo of it used in another python framework and it was exactly as I expected, like something we use at work for CF.  It has a schema version table, that holds app state version, and version files with ‘up/down’ methods. My main issue with many of these ’scafolmagic’ things is that no one bothered to mention how you get from one version to the next… or back again. You can’t build the model right the first time, and iterative programming is a fact of life. This library addresses that.