Cloudera Developer Blog
Big Data best practices, how-to's, and internals from Cloudera Engineering and the community
There are a number of special “users” with roles to play in the Apache Hadoop environment. For your reference, we have summarized them below as of CDH 4.4. Kerberos principals (used for authentication in a secure cluster) are not covered here.
The specific user IDs listed are the ones created by default on installation but they are configurable unless otherwise indicated.
There’s good news for users of Hue, the open source web UI that makes Apache Hadoop easier to use: A new SAML 2.0-compliant backend, which is scheduled to ship in the next release of the Cloudera platform, will provide a better authentication experience for users as well as IT.
With this new feature, single sign-on (SSO) authentication can be achieved instead of using Hue credentials – thus, user credentials can be managed centrally (a big benefit for IT), and users needn’t log in to Hue if they have already logged in to another Web application sharing the SSO (a big benefit for users).
Here’s a demo that shows this experience:
The release of Apache Hadoop 2, as announced today by the Apache Software Foundation, is an exciting one for the entire Hadoop ecosystem.
Cloudera engineers have been working hard for many months with the rest of the vast Hadoop community to ensure that Hadoop 2 is the best it can possibly be, for the users of Cloudera’s platform as well as all Hadoop users generally. Hadoop 2 contains many major advances, including (but not limited to):
In a fast-moving project like Apache Hadoop, there are always exciting new features introduced in each release. While it is tempting to make the most of these new features by upgrading to the latest release, users are often concerned about their code continuing to run.
In this post, you’ll get an overview of the the Hadoop API annotations and compatibility policies. Hadoop annotates specific APIs to be safe for use by end-users. By using these APIs, users can ensure their code works across a set of releases and be aware of what releases it might not work against.
Hadoop Releases and Compatibility
Before discussing the API-specific aspects, let us quickly look at the types of releases and the kinds of changes allowed in them. Hadoop releases (CDH or stock Apache Hadoop) belong to one of three categories, and adopt an
x.y.z form of release numbering. The changes in a release determine the release type. The releases can be roughly characterized as follows (see Hadoop Roadmap for details):
The following guest post is provided by Artur Barseghyan, a web developer currently employed by Goldmund, Wyldebeast & Wunderliebe in The Netherlands.
Python is my personal (and primary) programming language of choice and also happens to be the primary programming language at my company. So, when starting to work with a new technology, I prefer to use a clean and easy (Pythonic!) API.
After studying tons of articles on the web, reading (and writing) white papers, and doing basic performance tests (sometimes hard if you’re on a tight schedule), my company recently selected Cloudera for our Big Data platform (including using Apache HBase as our data store for Apache Hadoop), with Cloudera Manager serving a role as “one console to rule them all.”
The following post was originally published by the Hue Team at the Hue blog in a slightly different form.
Hue, the open source web GUI that makes Apache Hadoop easy to use, has supported Cloudera Impala since its inception to enable fast, interactive SQL queries from within your browser. In this post, you’ll see a demo of Hue’s Impala app in action and explore its impressive query speed for yourself.
Impala App Demo
The demo below compares some queries across Hue’s Apache Hive and Impala applications. (Impala supports a broad range of SQL and HiveQL commands.) Although this comparison is not scientific, it does reflect general user experience across common cases.
Customer Spotlight: Learn How Edo Closes the Advertising Loop with Hadoop at Cloudera Sessions Milwaukee
The Cloudera Sessions fall series is coming to a close next week, but first we’ll make a final stop in Milwaukee, Wisconsin (on Oct. 17), where attendees will hear about edo — a company that is revolutionizing the advertising space by closing the loop between promotions and point-of-sale transactions.
In Milwaukee, edo CTO Jeff Sippel will engage in a fireside chat with Cloudera’s VP of marketing, Alan Saldich. At edo, Jeff is responsible for the strategy, planning, and execution for the systems — including Apache Hadoop — that power the edo offer platforms.
edo is a venture-backed startup that sits at the intersection of payments and advertising. It analyzes credit-card data from banking transactions to produce targeted offers for card-linked loyalty programs that are relevant and incredibly easy to redeem. Offers are preloaded on your credit card so savings happen instantly without a coupon or separate loyalty card.
Below please find our regularly scheduled quarterly update about where to find tech talks by Cloudera employees this year – this time, for October through December 2013. Note that this list will be continually curated during the period; complete logistical information may not be available yet.
As always, we’re standing by to assist your meetup by providing speakers, sponsorships, and schwag!
|Oct. 1||Aarhus, Denmark||GOTO Aarhus||Eva Andreeason on Hadoop use cases|
|Oct. 8||Sunnyvale, Calif.||Hadoop Happy Hour||Kathleen Ting and Jarek Cecho sign books!|
|Oct. 9||Santa Clara, Calif.||IEEE BigData Conference||Amr Awadallah on Hadoop use cases|
|Oct. 9||San Francisco||SF Hadoop Users||Eric Sammer on Hadoop app development (panelist)|
|Oct. 10||Sydney||DataCon||Sean Owen on data science|
|Oct. 15||Durham, NC||TriHUG||Mark Miller on Solr+Hadoop|
|Oct. 15||Mountain View, Calif.||Oracle NoSQL & Big Data Meetup||Mike Olson on virtues of key-value stores|
|Oct. 15-17||Burlingame, Calif.||Big Data TechCon||Apache Hive workshop with Mark Grover|
|Doug Cutting on the Hadoop revolution|
|Hadoop app development (CDK) workshop with Ryan Blue|
|Jonathan Seidman on extending data infrastructure with Hadoop|
|Jonathan Seidman on the Hadoop ecosystem|
|Himanshu Vashishtha on HBase use cases|
|Kate Ting on Apache ZooKeeper|
|Kate Ting on 7 Deadly Hadoop Misconfigurations|
|Oct. 16||Dallas, Tex.||DFW Big Data||John Ringhofer on Impala|
|Oct. 17||Milwaukee, Wis.||Cloudera Sessions||Hadoop app development lab (on CDK) with Ryan Blue|
|Oct. 17||St. Louis, Mo.||St. Louis HUG||Tom Wheeler on Parquet|
|Oct. 18||Munich||HUG Munich||Lars George on Impala|
|Oct. 22||London||UK HUG||Sean Owen on Scalable Big learning|
|Oct. 23||Seattle||Seattle Scalability Meetup||Ronan Stokes on Cloudera Search|
|Oct. 24||Palo Alto, Calif.||Bay Area HBase User Group||Michael Stack on HBase 0.96|
|Oct. 24||Raleigh, NC||All Things Open||Josh Wills on open source innovation|
|Oct. 28-30||New York||Strata Conference + Hadoop World 2013||Mike Olson on Hadoop’s impact on data management|
|Doug Cutting on the future of Hadoop|
|Henry Robinson on workload diversity in Hadoop|
|Hadoop app development (CDK) workshop with Eric Sammer|
|Matt Brandwein on leveraging mainframe data with Hadoop|
|Aaron T. Myers and Shreepadma Venugopalan on Hadoop security|
|Jayant Shekar on machine data analytics|
|Amandeep Khurana on Monsanto’s use case for Hadoop & HBase|
|Philip Zeyliger on debugging distributed systems|
|Greg Rahn on Impala performance tuning|
|Jon Hsieh on HBase roadmap|
|Oct. 28||New York||NYC HUG||Arvind Prabhakar on Apache Sentry (incubating)|
|Oct. 28||New York||Sqoop User Meetup||Abe Elmahrek on the Sqoop2 app for Hue|
|Oct. 29||New York||Impala + Parquet Meetup||Greg Rahn on Impala+Parquet performance tuning|
|Oct. 29||New York||Cloudera Manager Meetup||Aditya Achara on Cloudera Manager success stories|
|Oct. 30||New York||Apache Sentry User Meetup||Arvind Prabhakar and Shreepadma Venugopalan with a Sentry overview|
|Oct. 30||Philadelphia||Chariot Data IO Conference||Lars George on HBase sizing as well as on Parquet|
|Nov. 6||Chantilly, Va.||Open Source Search Conference||Alex Moundalexis on Search+Hadoop|
|Nov. 6||Munich||JAX Munich||Lars George on HBase and Impala|
|Nov. 7||Tokyo||Cloudera World Tokyo||Kiyoshi Mizumaru on CDH|
|Sho Shimauchi on Cloudera Manager|
|Tatsuo Kawasaki witha Hadoop 101|
|Daisuke Kobayashi on Hadoop ops|
|Nov. 11||London||UK HUG||Marcel Kornacker on Impala|
|Nov. 12-13||London||Strata London||Sean Owen on Scalable Big Learning; Tom White on Hadoop app development with CDK|
|Nov. 12||San Francisco||QCon SF||Josh Wills on machine learning|
|Nov. 13||Washington DC||LISA 2013||John Ridley on Hadoop 101 for sysadmins|
|Nov. 14||Seoul||Tech Planet Korea||Michael Stack on HBase roadmap|
|Nov. 14||Tokyo||Cloudera Manager Meetup||Sho Shimauchi, Kiyoshi Mizumaru: What is Cloudera Manager?|
|Nov. 14||Antwerp||Devoxx Belgium||Tom White on building Hadoop apps with CDK|
|Nov. 16||Los Angeles||Big Data Camp LA||Alex Behm on Impala|
|Nov. 20||Boulder, Colo.||Boulder/Denver Big Data Meetup||John Darrah on Hadoop 101|
|Dec. 2||Tokyo||Cloudera Manager Meetup||Sho Shimauchi, Kiyoshi Mizumaru: What is Cloudera Manager?|
History teaches us that ecosystem growth is fueled by enthusiasm, tools (including frameworks and APIs), and knowledge in roughly equal measures. To this point, the Apache Hadoop ecosystem has been blessed with the first two ingredients – thanks to the magic of open source – but in the third category, there is still plenty of work to be done.
For Cloudera, our Academic Partnership program is a major part of that effort. Through that program, accredited nonprofit universities around the world get access to Cloudera’s own Hadoop curriculum for their computer science departments, in addition to discounted training and certification for students and instructors. Thus far, Cloudera Academic Partners include (but are not limited to) San Jose State University, DePaul University, Fordham University, Vanderbilt University, Technische Universität Berlin, and Rensselaer Polytechnic Institute.
Welcome to our third edition of “This Month in the Ecosystem,” a digest of highlights from September 2013 (never intended to be comprehensive; for completeness, see Hadoop Weekly).
Note: there were a few other interesting developments this week, but out of respect for the calendar, I’ll address them next month.