To get the most out of any application, a graphical user interface improves your efficiency and data streaming without exception. A UI should help you through the steps of an often-complex flow as the visible layer between your problem and solution. Even the most hardcore back end enthusiasts will admit that its significance is undeniable for a complete product. It has to be well organized and easy to understand, yet be able to provide the right tools in the right place. It offers an opportunity to simplify your users’ workflow, provide guidance, and clarify what is possible. For Cloudera’s SQL Stream Builder (SSB), we did not miss this chance.
“SSB was built to give analysts the power of Flink in a no-code interface,” wrote a colleague as a great summary. But it can be read differently depending on where the emphasis is placed. “Power of Flink” sounds exciting enough by itself, so in the next couple of minutes let’s dive into the “no-code interface” part and see where we began one year ago, what we have accomplished, and what challenges we encountered along the way.
A well-built UI from a past era
You don’t have to be a SSB expert for this post. In a nutshell, our UI offers a feature-rich console with an SQL editor, display log messages, and execution results in real time, as well as providing an intuitive way to manage the following resources:
- SQL Jobs
- Virtual Tables
- User Defined Functions
- Data Sources
- Materialized Views
- API Keys
- Job Notificationshttps://docs.cloudera.com/csa/1.9.0/how-to-ssb/topics/csa-ssb-running-simple-job.html
- Connectors
- Data Formats
Although you could accomplish this with our original UI, it was lacking some important features that users might expect in the era of modern single-page applications. Things like maintaining user state for non-rendered elements, customizing layouts, and performant view transitions have all been attributes of a cutting-edge web application for some time now.
But when rebuilding an existing repository from scratch with a different tech stack, you should also focus on developer experience and long-term maintainability. It is obvious that making changes to a large JavaScript file takes more effort compared to a component-based approach where we only deal with smaller blocks. This is how we went from working with a mixture of vanilla JavaScript, Vue.js, and some helper libraries to unleashing the power of Angular and TypeScript.
The Angular way
Was this the perfect choice? If we remove all biases, no one knows for sure, but it’s more than enough to reach our goals. To mention a few benefits that we got out of the box:
- Well-structured repository
- Readable and testable code
- Fast development process
- Excellent component library.
In short, Angular is a full-fledged framework that provides solutions for server communication, routing within your application, and much more. TypeScript is the primary language for Angular application development, and also comes with many benefits in terms of language features, reference validation, project scalability, and code maintainability.
So we have our framework and language, but there are other things that have to be taken into consideration to prepare ourselves for the wilderness, to be able to deliver solutions quickly and with confidence.
Internal component library
Angular CLI offers many features including a built-in code generator to generate components and other building blocks based on a schematic, but the end result is just a blueprint. At the time of writing, Angular does not support generating magically finished components that just fit well into your application.
Since we have to code them, we want to reuse as much of them as possible. Even though SSB UI has a lot of custom ones, the idea is to create generic, reusable components, services, and style definitions, and put them in a separate library for the benefit of other applications.
This library is created by talented Clouderans and based on the popular Angular Material component library and CDK. It proved itself a helpful companion on our journey.
State management
When a web page is loaded, the browser creates a Document Object Model (DOM) of the document that represents it as nodes and objects.
To keep our application performant we want to avoid an excessive DOM size and get rid of unused elements, but at the same time we want to preserve the state of specific views. If the user has interacted with it, or if it’s already populated with items rendered from a back end response we don’t want to lose them.
Our previous UI had multiple pages that you could visit using the sidebar, but user states were not stored, so switching pages meant losing states.
The Redux inspired NgRx library is an obvious choice for Angular. Along with organizing the application state and making API interactions consistent, with NgRx we can also simplify the communication between multiple components by managing a common state that can be accessed from anywhere. This enables us to implement the following features in a couple of lines:
- Navigate away from the active job without losing its state.
- Send log entries to the log viewer from any component via the injected NgRx store.
- Load resources once and keep the results in state.
The last point was not that straightforward. While NgRx state is useful for storing changes made by the user it doesn’t support the easy management of large data sets. This is not a new problem and there are great solutions to it already. One of them can be found in our internal library. It’s a utility based on ngrx/data (but simpler to use) that loads a given collection and makes it available in the store. This made us ready to easily manage large data tables.
API generation
To load resources and manage our Flink jobs, the SSB front end calls back end endpoints. No surprise so far. But whenever our back end API changes, the front end needs to follow. Introducing API generation based on our back end’s Swagger JSON files made it possible to catch errors at build time. Angular’s http client doesn’t have to be imported anywhere as it’s only used in the generated services. We also don’t have to worry about DTO type safety as interfaces are also auto generated based on the back end code. Pretty cool compared to what we had before.
SQL editor
Another improvement was the replacement of one of the central components of our UI. Switching from CodeMirror to Monaco Editor (the code editor that powers VS Code) came with many features out of the box. It is worth checking its playground if you have a similar use case.
New possibilities bring new challenges
While the transition to our shiny new framework and libraries was taking place, SSB’s back end feature set kept evolving and its new capabilities required quick responses from the UI team.
After covering the architectural side of our new front end, let’s see how it supported the new UX that we were aiming for.
If you made it to this point, you deserve an image of the new, redesigned UI of SQL Stream Builder:
Restructured layout
As part of the recently introduced SDLC concept, our resources are now organized as projects that can be synchronized with a remote repository.
This required a complete rework of what we had before. We no longer have separate routes for resources. Once you select a project you will step into an integrated, IDE-like layout that has two resizable sections.
On the left side you can find an explorer tree along with other project related views. From the explorer you can easily access resources within a project’s scope. On the right there is a workspace section with tabs for each opened resource.
This structure allows you to set up custom layouts. You can either see everything at once or only focus on what matters.
Adding a new context to the existing NgRx state enabled us to maintain all project-related information including layout state and tab management.
SSB’s project explorer is a tree component that contains resources of the active project as well as external ones. Resources can be opened and managed directly from the tree.
Workspace
A tab group is the ultimate component for easy context switching. Tabs in SSB are loaded via URL, so reloading the page will load the same tabs.
Conclusion
We have covered the decisions that have been made to build a new UI that our product deserves. First we discussed the technical considerations, then some of the UX related improvements. Our API provides a lot of features already and the UI does not settle for less either. This makes Cloudera’s SQL Stream Builder one of the best options out there when it comes to data streaming with Flink.
Try it out yourself!
Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). The Community Edition makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.
Editor's Choice
