jfrank

How do you make very hard glass ceramics with low coefficient of thermal expansion (CTE)?

Answer: Take a combination of SiO2+Al2O3 (about 80%-90%) some TiO2+ZrO2 (about 4.5%) and LiO2 (also about 5%). Heat it to 1650 C.

Anneal it at 650C for two hours.

Raise the temp slowly to about 680C and hold it for an hour while crystal nucleation begins, then up to about 850 to transform the glass into a few different crystals, ceramizing it.

Now the CTE of the final product will be the amalgam of the CTE’s of the component parts, possibly even producing a material with a negative coefficient! Imagine something that shrinks when you heat it.

I’d love to make some some day, however the temperatures involved are extreme.

Microstructure and Properties of Li2O-Al2O3-SiO2-P2O5 Glass-Ceramics

“I want to make my cat shoot acorns! Ok?” a grinning sixth grade girl proclaims while pointing to her laptop.

“Sure thing.” I reply, smiling back.

I spent the last three months teaching a class about making video games for Saturday Academy. Working from two schools in north Portland that participate in the SUN after school program, I got to hold conversations like these during each class. During the end of 2011 I began wanting to teach a high school programming course, and when I heard about this opportunity via twitter I knew it wasn’t exactly what I wanted but it was close.

The software environment for the class was pre determined to be Game Maker from YoYo Games. I was skeptical at first wanting a cross platform open system, but it turned out to be good for getting students into game programming AND design all in one environment. I started by having each student sketch a character in four positions, run1, run2, stand, and jump. Using these as different sprites for the same object, we animated the characters and built basic platforming games over the course of 8 weeks.

This video is a montage of various game concepts that students came up with. Enjoy!

linked in left out

I had an thought and built a… sketch of the concept at www.repcard.org. The idea is basically linked-in for people experiencing homelessness. Instead of professional high fives, they would exchange bits of reputation, like ebay’s based on how many positive impacts they make in the community big and small. Non monetary, since credit/capital isn’t usually available to that population. The card could function as some sort of ad-hoc resume or a way to gauge trust in someone you meet. I’m not an expert in this area so I don’t know if its a good idea, but I wanted to submit it as a sort of working sketch to show it to people in the social work field.

Technologies Used: Google App Engine, Gaelyk, Groovy, Java

(un)shredding like a boss

look no hands, brain surgery for a hotplate

In part one I moved and started new things… here is what else I did.

helicopters that fly upside down

One of my big goals this year was to study machine learning. I worked my way though Andrew Ng’s Stanford Machine Learning course. One of his demos is flying a helicopter upside down using reinforcement learning.

Technologies used: Python’s NumPy, Octave

transformations in 3d

I quickly learned that in order to grasp most machine learning algorithms, an understanding of linear algebra was a must. I put Machine Learning on hold and took Gilbert Strang’s MIT Linear Algebra course next. This guy can describe your haircut before and after in terms of a matrix operation. I earned 83,802 Khan Academy points mostly working through semi-advanced mathematics.

Technologies used: Python’s NumPy, Octave

the basic tools of science

I think everyone should own the basic tools of science, that’s why this year I set out to buy them. I acquired a 1600x microscope, a digital thermometer, a magnetic stirring hotplate, lab glassware and a balance that is sensitive down to a thousandth of a gram. Somehow in modern media portrayals home science equipment has nothing but bad connotations. I wholly reject that idea. Science belongs at home! In a great article Bill Nye put it best

People who want to make meth will find ways to do it that don’t require an Erlenmeyer flask. But raising a generation of people who are technically incompetent is a recipe for disaster

combine at 80 degrees celsius

Women’s deodorant is bad. Not because it doesn’t work, it does, but because it contains aluminum. For those with sensitive skin, its awful. There are also a few potential hazards with applying nano sized metals that are easily absorbable into the human body.  My brother is a PhD chemist, so he helped me with my lab technique. I asked a large chemical company for a sample of sodium sterate (they sent me 2 kilos free!), bought some propylene glycol locally, and made a variant on old spice. Colorless, odorless. My wife loved it, but it needs some work.

wingdings and matisse

An Android app that uses Wingdings AND Matisse? You’re thinking “No you didn’t” but in fact I did. I did. In case you forgot these fonts.. let me give you a refresher:

I built a task manager with Android push notifications for a client who really really wanted one. I wanted to build an Android app, so we had a deal. A few weeks later it was done. I don’t know why people complain about Android development, I found it fairly painless.

Technologies Used: Java, Android SDK, Eclipse, PHP (client backend), Google’s Android Push API

no pie for you

A friend of a friend approached me with an idea, and I took it on as a co-founder. We built out a QR powered profile site for hot-rods called www.qrmyride.com. After printing up some stickers we got some enthusiastic responses. We even applied for PIE, an incubator here in Portland but we were denied. There was stiff competition, but perhaps they were right. This site never really took off as planned. It costs next to nothing to run, so my partner and I will continue to let it run, but it was a good lesson in the cost of prototyping. I still think the idea is good, especially if adopted by a whole car club for the purposes of a unified car show.

Technologies Used: Google App Engine, Java, Groovy, QR codes

Next Post: Un-Shredding and Electronic Brain Surgery

This has been a big year for me. So big that you all get a year-end recap of it because you are here, reading my blog. Except it’s too big to write in one post. So if you didn’t think you were going to get personal stuff mixed in to this mostly tech blog, now is the time to unsubscribe.

One of my favorite quotes is this:

It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself for a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat – Roosevelt

What this means to me is that I am getting comfortable not with success, but with failure. And with failure, slowly, haltingly, comes some measure of success.

cloud surfing

At the beginning of 2010 at my job at Mentor I enjoyed moving some more systems to AWS from an acquisition’s internal server farms. I closed out six years at Mentor; an excellent chapter in my life that I was sad to see end. I miss my co-workers, and agree with Barney “Twitter is such a poor excuse for seeing them every day.”  When I put my notice in, most were happy for me, some could hardly believe what I was doing. Others didn’t actually believe that I did not have a position to go to at another safe corporation. They kept asking what my real plan was, and I kept replying: “I’m taking a year off to study, to grow, to try new things.”

Technologies used: Bash, Railo, AWS API’s, Python

life changes. no really it does!

I
quit my job in March,
moved out of down town in June,
rented a big old house a week later,
became a foster parent in July,
and enrolled as a fake student at PSU.

it slices, it dices. well no, actually it only slices.

I had a poor experience with a cloud dashboard company and built autosnappy.com as a response. It makes snapshots for AWS volumes on a schedule. So simple. Happy dance. It broke even almost immediately, and although it doesn’t make tons of money, there is a lot of room here for growth. A customer is asking to pay for me to develop new features for it currently so I may revisit it and roll out new things. I wanted to build this as compartmentalized as possible. One of my design goals was that the front end know as little as possible about the backing AWS services as it could. So it talks exclusively to the middle python tier, even though it is powerful enough to accomplish both functions. This means that if I needed to I could scale those components separately, and keep the user facing process in a separate linux user/group as the process that talks to AWS and has the security keys.

Technologies used: Java Magnolia and Railo templating front end, python web service backend,  AWS SimpleDb storage. Scaleable! Stateless! Cloud!

on being a fake student

Being a fake student is the best! When you take a one credit class at PSU you have access to super high speed internet, a research library, and a place to go work. I spent a large part of my year hacking code in Food For Thought cafe, next to artists, musicians, and hippies. But that’s not all. I also have access to the new(ish) Rec Center which has a pool, spa, rock wall and all the normal workout stuff.

Oh, and my one credit class? Yoga. So stressful.

Next post: Basic Tools of Science and Flying Helicopters Upside Down

About a week before it closed I decided to start working on the Darpa Shredder Challenge. This challenge is to reassemble shredded documents using only the shredded pieces. Here is a quick video depicting an early part of the algorithm that I worked on.

Algorithm Notes

This video is a bunch of visualizations put back to back of the process described below. The algorithm is run against piece 1 of puzzle 1 of the challenge, shown below. You’ll note in my visualization it is upside down, this is simply due to the start of x,y in the graphing application is at the lower left, while the image library gave me x and y starting from upper left.

Edging

Since I had very little time, I wanted to use as much existing code as possible. To find edges I used gimp/python-fu to posterize each piece 5 times. Each time that ran, it simplified the colors of the image. I then saved out the green channel. This made the edge finding as simple as finding the uniform green edge pieces and rolling around each piece. The edge I found is shown in the right hand side as black pixels.

Segmentation

Because each piece could match on only part of another piece, I wanted to find good matches based on local shape part matching. I chose to segment the piece by rolling through each location with surrounding bits by distance. The red X represents the current location, surrounded on either side by an arbitrary sized window shown as blue highlight.

Rotation Invariance

Each piece is given to you in a random rotation. So the segments between pieces needed to be fitted together in such a way as to ignore which way they were originally rotated. So next I found a ‘landscape horizon’ using a linear average of the segment. I chose PCA because of the x/y arbitraryness in this problem. This is because ordinary least squares only penalizes on the y term, and I needed total least squares so it would penalize on both terms equally. This is shown as a red line.

Rotate!

After finding the horizon, this becomes the new relative x axis for that segment. I use a linear transformation to rotate to a normalized inward facing view, and then transpose it to be centered at zero zero. This is the lower left portion of the video. Most flat sides are nearly equal to the x axis at that point, with variation easily seen. If you watch this window alone as you see the video it is as if a camera is going around the piece looking in on it with smooth transitions.

Representing Shape

Next I built a representation of surrounding shape at every edge location based on the new normalized x,y from the previous step. Surrounding shape should include other markings page lines and pen marks, but I haven’t included them yet. I converted the rotated Cartesian coordinates to a log polar system, and then used binning to produce a log polar 2d histogram of the surrounding shape. This is the upper right picture. I adjusted my binning to be more sensitive to changes in y at the horizon rather than in the middle because most shapes are fairly close to the x axis after they have been rotated. Each histogram results in 100 total bins, each with a count of the items in the segment. The degrees are left to right, while distance is the vertical dimension. Inspiration for this shape representation came from Shape Matching and Object Recognition.

Cluster

Using the 100 integer histogram arrays of shape I then clustered them using K means so that I could simplify the comparisons between groups of ordered segments. The result is that I can classify a segment’s shape as a single cluster membership, and an array of segments (a side of a piece) as an ordered list of those cluster assignments. (Not Shown)

Fuzzy Segment-Cluster-Array Matching and Reassembly

The next task is to match segment cluster arrays with others that are closest, indicating a general shape match. (Not Shown) This produces lists of candidate matches to aid in a manual or automatic assembly.

Running Out Of Time

At about this point the contest ended. I am still working on getting the first puzzle completely automatically assembled. I learned a lot, and had a good time but the primary takeaway was this: visualize, visualize and then when in doubt, visualize. Visualization costs a lot, and building pretty graphs for yourself is a complete waste of time if you are a perfect coder. Countless times throughout this process I had implemented a part of it, and moved on but only later came back and visualized what I had actually done. Often that immediately showed what went wrong. One mistake that it helped me find was that the rotation and transposition that I had originally done were rotated correctly but transposed based on my original center, not the rotated center, so the histograms were erratic. Once I could see that the middle was not 0,0 (in the lower left picture) I immediately knew what the issue was. Slowly I came to learn that every minute spent visualizing a problem like this pays off immediately.

For the impatient: treat it like linux.

1. Don’t run IIS, run Apache or generically “Don’t run the GUI one, run the daemon one.” Version control your configuration. Version controlling your config in a usable way is way more difficult than it has to be or impossible using point and click tools. You’re running a server, not a candy store. It doesn’t need to look good, it needs to be reproducible. Ask someone who uses IIS to show you the changes that have been made in the last couple weeks, and why each one was made; they can’t. If you version control your config, thats all in your commit history. Love those log files.
2. Don’t run FTP, run Rsync, and use that to transfer files to and from your server over the internet. But don’t install the windows rsync server because thats not secure. More on point 3.
3. Install ICW Copssh instead, it is an openssh installation for windows packaged by ITeF!x Consulting. Next you install the rsync package on top and then you can ssh/sftp/scp/rsync to the box in the normal linux fashion complete with public key authentication. Use the same rsync client package that you installed on top of ICW on the other end to connect to your server from another windows box.
4. Never install server applications that are cross platform (aka not built only for windows) in the default location. Programs that are developed to work on multiple platforms will often miss conventions that apply only to windows paths. Recent default paths for 32 bit programs on newer Windows contain not only spaces, but parenthesis as well a.k.a C:\Program Files (x86)\My Program Blah\blah. Use your own convention, lower case, and keep it simple: D:\apps\mysql5\ for example.
5. Install the excellent sysinternals suite. Microsoft bought it, but this labor of love by Mark Russinovich was developed long before they got their hands on it. Process explorer in tree view is an excellent alternative to task manager. Junction is a great way to get simlink behavior.
6. Install notepad++ or use textpad. Notepad ++ is an excellent notepad alternative because it replaces notepad inline so whenever you would have seen notepad, you’ll just get the better one.
7. Use volume shadow copy to snapshot your data. Its nice. Good job Microsoft.
8. Put both the ICW and Sysinternals binary directories on your system path. This will give you many useful tools to use on the windows command line. If for example you type ls on accident, you will actually get a file listing back just as you had expected. “tail -f _filename_” will just work. Junctioning (symbolic linking) will be nearby.
9. Script administrative tasks, and version control your scripts. I don’t have experience with it, but the Windows Power Shell allows you to script many things, and if you’re serious about windows it is the strongest system platform. Good old bat/cmd is also available and still useful for many things.
10. Last one is a double bonus! Shut down services you aren’t using. The indexing service isn’t helpful if you aren’t searching for example. Run an anti virus, love those Microsoft system updates and lock down a firewall to only the ports you need; 22 for ssh, 80 for a website plus a few others.

You know what this picture tells me?

It tells me that BAC, KEY, MI, RF, SNV, and STI are related. They’re all acronyms, yes, but more than that. They are all banking stock symbols. The node that the arrow points to contains these values. All banks. The nodes on either side contain exclusively real estate holding companies and home builders respectively.

This puts a huge grin on my face.

If you’ve talked to me in the last several months, I’ll have probably mentioned at some point that I’ve been learning some machine learning concepts. I’ve been watching and working through the examples in the Stanford machine learning course by Andrew Ng. On the one hand, the course is excellent. Andrew clearly knows his stuff and teaches toward underlying theory and principle. His goals are exactly how I like to approach a new area of study; always asking why not how. The concepts are compelling but on the other hand the math is difficult for me and it is lacking the kind of “proof is in the pudding” mentality that I’m used to as a programmer. I decided that I also need to approach this topic from the practical side. I’ve just discovered RapidMiner and have been playing with it recently.

My chosen problem set is stock data. I love the uncertainty inherent in the market, its a mass of data, action and reaction. One problem (an easy one to start with) that I’ve always wanted to work on is stock correlation. Simply put, stocks that are similar move together. If you have two businesses that are similar in industry and size, they will likely move together as they have similar economic environment. News that affects one is much more likely to affect the other than a third unrelated company in another industry. This relationship can be coaxed out of the data. For each stock, you should be able to calculate a web of close “neighbors” that move similarly, and moving out from there you may approach another “neighborhood” of related stocks. In machine learning this problem could be approached as a time series or as clustering. Since we don’t know the labels (names of the clusters so to speak) it’s not classification.

After many false starts, I grabbed some stock data and loaded it into RM with the built in jdbc tools they have. I then selected stocks for the last two years without missing data points on days where volume was greater than zero and pulled the set into a hierarchical clustering algorithm. The hierarchical clusterer uses an internal simpler one level cluster (kmeans) and applies it it recursively and in parallel. I also had some promising results with a correlation matrix which showed for example that INTC (Intel) and MU (Micron) are related much more closely than MU and KFT (Kraft).

Its great to be able to test out some of the things that I’ve been learning about in an environment that lets me try a lot of things in a relatively short amount of time. Up next: Better clustering algorithms, and using class labels from my clustering to train a model for prediction.

Did you ever wonder what is shared between all those CF frameworks? I put together a copy paste detector implementation of the interface for cfm and cfc files, as part of the PMD project. I also plugged it into hudson via the DRY visualizer of the output, but that is just candy on top.

Basically it works like this:

1. It goes through your code line by line, character by character and builds up overlapping hashes of the content.
2. Based on your tokenizer, it will ignore certain parts of the code (like whitespace) and so the hashes will be able to handle your 2 tabs vs my 3 spaces and shows that two sections of code are ‘the same’. The tokenizer I am using is the ‘anyTokenizer’, not optimized for CF at all.
3. Based on a threshold of how big a token range the hashes should cover, it builds a report of all the code you give it and outputs it in a structured format.

Now presenting, all of the code duplication between the following ColdFusion projects, run with a token threshold of 200:

• ColdBricks
• Coldbox
• ModelGlue
• coldmock
• mura
• MangoBlog
• coldspring
• mxunit
• machii
• farcry
• fusebox5

Well… I’m not going to put them all here. Its a large set.

Here are a few samples:

Farcry and mura both use cfformprotect. Cool.

51,515: <file line=”288″ path=”/home/jfrank/temp/codeprojects/farcry/core/webtop/cffp/cfformprotect/cffpVerify.cfc”/>
51,516: <file line=”287″ path=”/home/jfrank/temp/codeprojects/mura-5.2.2709/www/requirements/cfformprotect/cffpVerify.cfc”/>

BlogCFC and MangoBlog both share xmlrpc bits. Nice.

53,956: <file line=”141″ path=”/home/jfrank/temp/codeprojects/BlogCFC5/client/xmlrpc/xmlrpc.cfc”/>
53,957: <file line=”122″ path=”/home/jfrank/temp/codeprojects/MangoBlog_1.5/api/xmlrpc.cfc”/>

The full results are not quite valid xml, because of encrypted cfms. They also contain a lot of boilerplate licenses as you can imagine. It would take a minor amount of cleanup of this output to make it parseable, and removing the licenses would make it much more compact. Also some of the duplication shown may be intentional due to code generation/plugin dependency domains.

Want to run it yourself?

Get the current PMD jar and grab my cfm-cpd.jar and this build file if you’d like to run it with ant.

Point Ant at a lib directory with those two jars, and run the build against your own code. Don’t forget the Dry Plugin if you run Hudson!

(Note, the cfm-cpd jar contains a couple overlapping classes with PMD, and relies on the fact that jars are loaded alphabetically and so it will win. This is lame, but so is PMD for making me hard code things in the ant task!)

I was pushing out the magnolia-railo-demo and was looking at the site performance. It was amazing by itself, sending back magnolia-railo built pages in 100ms including wire time from Portland to Dallas raw from Tomcat. On this box I have one IP though and I wanted to use a subdomain for the demo which means proxying vhosts.

When I proxied it through Apache’s httpd it took about twice as long for the page to load! (roughly 130-150ms for the base page) It seemed totally ridiculous… Now in all fairness I’m no Apache expert, but I decided to drop in nginx as a proxy on the front end and its cost as a proxy is nearly unnoticeable. I wouldn’t be able to tell it apart from hitting tomcat directly as far as speed is concerned.

I’ll have to give this a closer look for my other projects…

One slightly tricky thing is that if you need Apache’s ProxyPreserveHost in Nginx you need to use proxy_set_header Host like this:

location / {
   root   /usr/share/nginx/html;
   index  index.html index.htm;
   proxy_set_header Host $host;
   proxy_pass http://127.0.0.1:80;
}

Sometimes I wonder what it will be like when apache’s gone. I will probably find myself cding into /etc/httpd/logs and cat access_log over and over… just to remember old times.