Cloudera Developer Blog · Hadoop Posts
Among Cloudera’s engineer-presenters at Strata + Hadoop World 2013 this week, Philip Zeyliger (“Tricks for Distributed System Debugging and Diagnosis“) was particularly fortunate to have been interviewed by O’Reilly Media editor Meghan Blanchette on camera.
In the following 8-minute interview, Philip offers an overview of common pain points and failures when debugging distributed systems:
We are just a weekend away from the Biggest. Strata + Hadoop World. Ever.
Not to make non-attendees feel bad — the show sold out a couple weeks ago — but some great things are in store for technical end-users (as well as everyone else):
There are a number of special “users” with roles to play in the Apache Hadoop environment. For your reference, we have summarized them below as of CDH 4.4. Kerberos principals (used for authentication in a secure cluster) are not covered here.
The specific user IDs listed are the ones created by default on installation but they are configurable unless otherwise indicated.
The release of Apache Hadoop 2, as announced today by the Apache Software Foundation, is an exciting one for the entire Hadoop ecosystem.
Cloudera engineers have been working hard for many months with the rest of the vast Hadoop community to ensure that Hadoop 2 is the best it can possibly be, for the users of Cloudera’s platform as well as all Hadoop users generally. Hadoop 2 contains many major advances, including (but not limited to):
In a fast-moving project like Apache Hadoop, there are always exciting new features introduced in each release. While it is tempting to make the most of these new features by upgrading to the latest release, users are often concerned about their code continuing to run.
In this post, you’ll get an overview of the the Hadoop API annotations and compatibility policies. Hadoop annotates specific APIs to be safe for use by end-users. By using these APIs, users can ensure their code works across a set of releases and be aware of what releases it might not work against.
Hadoop Releases and Compatibility
Before discussing the API-specific aspects, let us quickly look at the types of releases and the kinds of changes allowed in them. Hadoop releases (CDH or stock Apache Hadoop) belong to one of three categories, and adopt an
x.y.z form of release numbering. The changes in a release determine the release type. The releases can be roughly characterized as follows (see Hadoop Roadmap for details):
Customer Spotlight: Learn How Edo Closes the Advertising Loop with Hadoop at Cloudera Sessions Milwaukee
The Cloudera Sessions fall series is coming to a close next week, but first we’ll make a final stop in Milwaukee, Wisconsin (on Oct. 17), where attendees will hear about edo — a company that is revolutionizing the advertising space by closing the loop between promotions and point-of-sale transactions.
In Milwaukee, edo CTO Jeff Sippel will engage in a fireside chat with Cloudera’s VP of marketing, Alan Saldich. At edo, Jeff is responsible for the strategy, planning, and execution for the systems — including Apache Hadoop — that power the edo offer platforms.
edo is a venture-backed startup that sits at the intersection of payments and advertising. It analyzes credit-card data from banking transactions to produce targeted offers for card-linked loyalty programs that are relevant and incredibly easy to redeem. Offers are preloaded on your credit card so savings happen instantly without a coupon or separate loyalty card.
Below please find our regularly scheduled quarterly update about where to find tech talks by Cloudera employees this year – this time, for October through December 2013. Note that this list will be continually curated during the period; complete logistical information may not be available yet.
As always, we’re standing by to assist your meetup by providing speakers, sponsorships, and schwag!
|Oct. 1||Aarhus, Denmark||GOTO Aarhus||Eva Andreeason on Hadoop use cases|
|Oct. 8||Sunnyvale, Calif.||Hadoop Happy Hour||Kathleen Ting and Jarek Cecho sign books!|
|Oct. 9||Santa Clara, Calif.||IEEE BigData Conference||Amr Awadallah on Hadoop use cases|
|Oct. 9||San Francisco||SF Hadoop Users||Eric Sammer on Hadoop app development (panelist)|
|Oct. 10||Sydney||DataCon||Sean Owen on data science|
|Oct. 15||Durham, NC||TriHUG||Mark Miller on Solr+Hadoop|
|Oct. 15||Mountain View, Calif.||Oracle NoSQL & Big Data Meetup||Mike Olson on virtues of key-value stores|
|Oct. 15-17||Burlingame, Calif.||Big Data TechCon||Apache Hive workshop with Mark Grover|
|Doug Cutting on the Hadoop revolution|
|Hadoop app development (CDK) workshop with Ryan Blue|
|Jonathan Seidman on extending data infrastructure with Hadoop|
|Jonathan Seidman on the Hadoop ecosystem|
|Himanshu Vashishtha on HBase use cases|
|Kate Ting on Apache ZooKeeper|
|Kate Ting on 7 Deadly Hadoop Misconfigurations|
|Oct. 16||Dallas, Tex.||DFW Big Data||John Ringhofer on Impala|
|Oct. 17||Milwaukee, Wis.||Cloudera Sessions||Hadoop app development lab (on CDK) with Ryan Blue|
|Oct. 17||St. Louis, Mo.||St. Louis HUG||Tom Wheeler on Parquet|
|Oct. 18||Munich||HUG Munich||Lars George on Impala|
|Oct. 22||London||UK HUG||Sean Owen on Scalable Big learning|
|Oct. 23||Seattle||Seattle Scalability Meetup||Ronan Stokes on Cloudera Search|
|Oct. 24||Palo Alto, Calif.||Bay Area HBase User Group||Michael Stack on HBase 0.96|
|Oct. 24||Raleigh, NC||All Things Open||Josh Wills on open source innovation|
|Oct. 28-30||New York||Strata Conference + Hadoop World 2013||Mike Olson on Hadoop’s impact on data management|
|Doug Cutting on the future of Hadoop|
|Henry Robinson on workload diversity in Hadoop|
|Hadoop app development (CDK) workshop with Eric Sammer|
|Matt Brandwein on leveraging mainframe data with Hadoop|
|Aaron T. Myers and Shreepadma Venugopalan on Hadoop security|
|Jayant Shekar on machine data analytics|
|Amandeep Khurana on Monsanto’s use case for Hadoop & HBase|
|Philip Zeyliger on debugging distributed systems|
|Greg Rahn on Impala performance tuning|
|Jon Hsieh on HBase roadmap|
|Oct. 28||New York||NYC HUG||Arvind Prabhakar on Apache Sentry (incubating)|
|Oct. 28||New York||Sqoop User Meetup||Abe Elmahrek on the Sqoop2 app for Hue|
|Oct. 29||New York||Impala + Parquet Meetup||Greg Rahn on Impala+Parquet performance tuning|
|Oct. 29||New York||Cloudera Manager Meetup||Aditya Achara on Cloudera Manager success stories|
|Oct. 30||New York||Apache Sentry User Meetup||Arvind Prabhakar and Shreepadma Venugopalan with a Sentry overview|
|Oct. 30||Philadelphia||Chariot Data IO Conference||Lars George on HBase sizing as well as on Parquet|
|Nov. 6||Chantilly, Va.||Open Source Search Conference||Alex Moundalexis on Search+Hadoop|
|Nov. 6||Munich||JAX Munich||Lars George on HBase and Impala|
|Nov. 7||Tokyo||Cloudera World Tokyo||Kiyoshi Mizumaru on CDH|
|Sho Shimauchi on Cloudera Manager|
|Tatsuo Kawasaki witha Hadoop 101|
|Daisuke Kobayashi on Hadoop ops|
|Nov. 11||London||UK HUG||Marcel Kornacker on Impala|
|Nov. 12-13||London||Strata London||Sean Owen on Scalable Big Learning; Tom White on Hadoop app development with CDK|
|Nov. 12||San Francisco||QCon SF||Josh Wills on machine learning|
|Nov. 13||Washington DC||LISA 2013||John Ridley on Hadoop 101 for sysadmins|
|Nov. 14||Seoul||Tech Planet Korea||Michael Stack on HBase roadmap|
|Nov. 14||Tokyo||Cloudera Manager Meetup||Sho Shimauchi, Kiyoshi Mizumaru: What is Cloudera Manager?|
|Nov. 14||Antwerp||Devoxx Belgium||Tom White on building Hadoop apps with CDK|
|Nov. 16||Los Angeles||Big Data Camp LA||Alex Behm on Impala|
|Nov. 20||Boulder, Colo.||Boulder/Denver Big Data Meetup||John Darrah on Hadoop 101|
|Dec. 2||Tokyo||Cloudera Manager Meetup||Sho Shimauchi, Kiyoshi Mizumaru: What is Cloudera Manager?|
History teaches us that ecosystem growth is fueled by enthusiasm, tools (including frameworks and APIs), and knowledge in roughly equal measures. To this point, the Apache Hadoop ecosystem has been blessed with the first two ingredients – thanks to the magic of open source – but in the third category, there is still plenty of work to be done.
For Cloudera, our Academic Partnership program is a major part of that effort. Through that program, accredited nonprofit universities around the world get access to Cloudera’s own Hadoop curriculum for their computer science departments, in addition to discounted training and certification for students and instructors. Thus far, Cloudera Academic Partners include (but are not limited to) San Jose State University, DePaul University, Fordham University, Vanderbilt University, Technische Universität Berlin, and Rensselaer Polytechnic Institute.
Welcome to our third edition of “This Month in the Ecosystem,” a digest of highlights from September 2013 (never intended to be comprehensive; for completeness, see Hadoop Weekly).
Note: there were a few other interesting developments this week, but out of respect for the calendar, I’ll address them next month.
It’s common to hear people describe themselves as being “left-brained” or “right-brained” based on their tendency to be more logical and mathematically driven (left-brained), or, conversely, to be intuitive and creatively driven (right-brained). For example, people who prefer math over art are often considered left-brained. People who get a higher verbal score on their SATs than for math are often considered right-brained.
In general, language and creative writing are considered right-brained exercises. Many people also associate marketing and advertising as a right-brained function, whereas engineering is considered very left-brained.
But Big Data is changing this. Many companies are applying math and engineering to creative writing and marketing in order to optimize marketing campaigns’ results. Persado has actually built its business around this idea.