The Security Problem
Four Letter Words (acronym as 4lw) is a very popular feature of the Apache ZooKeeper project. In a nutshell, 4lw is a set of commands that you can use to interact with a ZooKeeper ensemble through a shell interface. Because it’s simple and easy to use, lots of ZooKeeper monitoring solutions are built on top of 4lw.
The simplicity of 4lw comes at a cost: the design did not originally consider security, there is no built in support for authentication and access control. Any user that has access to the ZooKeeper client port can send commands to the ensemble. The 4lw commands are read only commands: no actions can be performed. However, they can be computing intensive, and sending too many of them would effectively create a DOS attack that prevents the ensemble’s normal operation.
There are two approaches to address the lack of security in the original design of 4lw.
- Avoid a publicly accessible deployment of ZooKeeper. Both ZooKeeper clients and the ZooKeeper ensemble operate in a trusted environment where access to such environment is carefully controlled to eliminate the needs of authentication. Such a trusted environment is what the 4lw original design was based on. This is however not always possible, it depends on the use case. More importantly, the assumption of such a trusted environment runs contrary to the modern best practices.
- Build an external access control mechanism such that only selected users such as admins can access the 4lw port.
- The typical option is to use iptables but one issue is 4lw shares the same client port with the normal ZooKeeper clients.
- Another option is to run 4lw on top of a secured client port through SSL, but SSL is not available in 3.4 stable releases, and we need build authorization mechanism on server side to authorize specific users with permission on sending command.
There is no perfect existing solution for solving the security challenge exposed through 4lw. Instead of fixing 4lw, there are two better alternatives added to ZooKeeper over the years that both provide better functionality and also address the security concerns of 4lw.
Modern ZooKeeper Monitoring: JMX and AdminServer
JMX (Java Management Extensions) is a standard part of the Java Platform that provides provides a simple, standard way of managing resources such as applications, devices, and services. ZooKeeper also supports JMX to provide more powerful monitoring and management capabilities since 3.3.0. JMX has built in support for password authentication and SSL, so it’s secure by default. ZooKeeper also exposes these security configurations in its startup script zkServer.sh, where user can choose to customize and enable if needed.
ZooKeeper JMX monitoring is also battle tested in production environments. Cloudera’s Distribution including Apache Hadoop (CDH) has been using JMX for ZooKeeper monitoring since the inception of the product.
AdminServer was a new feature introduced in ZooKeeper 3.5.0 release with the intention to deprecate and replace 4lw. It addressed the shared port issue mentioned previously by moving the command port to a dedicated port, and having a dedicated Jetty server serving the commands while trying to maintain the semantics of the 4lw commands. It also provides enhanced functionalities of commands via extended command syntax as well as bug fixes.
Path on Deprecating Four Letter Words
ZooKeeper makes strong compatibility guarantees and because 4lw is a widely used feature, deprecating it will take multiple releases. To provide an upgrade path from 4lw to JMX or AdminServer, instead of completely removing the entire 4lw feature in a single release, ZooKeeper provides an option such that only a subset of the commands are enabled by default. This addressed the security concern that some computing intensive commands, when abused, will take down the entire ensemble, while still make 4lw as a feature usable.
This option is called “four letter words whitelist” and is available in latest stable 3.4.10 release and latest 3.5.3-beta release. With this feature, a 4lw command has to be put in the list in zoo.cfg as a new configuration option “4lw.commands.whitelist” otherwise ZooKeeper server will ignore the command. This option also provides an upgrade path for users to migrate from 4lw to JMX and AdminServer, where a selected subset of the commands can be replaced gradually.
The timeline on completely removing the 4lw feature from ZooKeeper is not decided yet. Want to contribute to the discussion? Our How To Contribute page is a great place to start if you’re interested in getting involved as a developer, or dive right into an open issue.
4lw is a great feature that reflects the design philosophy of ZooKeeper: keep it simple and solve a very specific problem. However the design tradeoff between simplicity and security was questionable; as ZooKeeper becomes more popular there are lots of emerging use cases that challenge the original lack of security design of 4lw feature. Today ZooKeeper provides JMX and AdminServer as alternatives for 4lw and it is recommended to migrate from 4lw to use these new tools for better security and new functionality.
Michael Han is a Software Engineer at Cloudera and a committer of Apache ZooKeeper project.