Cloudera Developer Blog · Cloudera Life Posts
The guest post below is from Wei Yan, a 2013 summer intern at Cloudera. In this post, he helpfully describes his personal projects from this summer. Thanks for your contributions, Wei!
As a Ph.D. student at Vanderbilt University, I work on the Apache Hadoop MapReduce framework, with a focus on optimizing data intensive computing tasks. Although I’m very familiar with MapReduce itself, my curiosity about the use cases for MapReduce and where it generally fits in the Big Data are drew me to Cloudera for the summer of 2013.
At Cloudera, I mainly worked on two projects: a Hadoop YARN Scheduler Load Simulator and a testing framework called Hadoop MiniKDC. In the remainder of this post, I’ll describe my experiences with each and what I learned from them.
Predicting YARN Scheduler Performance
Catherine Ray, a Summer Intern at Cloudera this year, was kind enough to summarize her experiences for you below. Best of luck in your new field, Catherine!
I’m currently 16 and a rising senior at George Mason University, majoring in Computational Physics. (The full title is Computational and Data Sciences with a concentration in Physics.).
I had a wonderful time working on my project. In short, I worked on an Apache Hadoop-based downloads tracking system. In this system, raw downloads logs are ingested via Apache Flume into HDFS, then parsed with an MR Job into a Cloudera Impala-friendly format. I had the opportunity to collaborate with one of our teams in New York to pull the whole system together. To fully utilize the data contained in the logs, I created a Java library that finds the organizational information associated with a given IP address. I also helped to create dashboards that use queries against the collected data to analyze it and produce sales leads.
Every growing, dynamic engineering culture needs a hackathon every once in a while.
Earlier this week, Cloudera put that thought into action with a two-day, around-the-clock “What the Hack!” internal hackathon in our Palo Alto offices, with our friends from Accel Partners underwriting the omnipresent food and beverage (thanks!). The carrot: “Fun surprise awards, and most important, the rights to brag about your cool hacking ideas.”
The morning began with a warm and festive welcome: