HiveMQ is thinking about how to scale the IoT

Getting from a few hundred sensors to hundreds of thousands or even millions is one of the top reasons IoT projects fail. Scale in the IoT is beyond human management and even comprehension for most of us. Much like the computing world has to embrace managing servers like cattle as opposed to pets, industrial and enterprise companies trying to use arrays of sensors in their efforts to modernize must take a few ideas from the cloud giants.

Managing IoT at scale requires new tools, new ways of thinking, and new processes. It will also require new technologies in the security, energy/battery, and management layers because it’s impossible to manage a million devices on a one-to-one or even 1,000-to-one-basis.

That’s why HiveMQ has released a new tool for developers called Swarm to help them test their MQTT messaging capabilities at scale. HiveMQ makes tools to manage MQTT messages between devices and between clouds. MQTT is a popular messaging protocol that’s widely used to send communications between connected devices, especially in commercial settings.

Dominik Obermaier, CTO and founder of Hive MQ, says that customers were trying to model how applications might use existing sensor deployments but found the current tools, such as Apache JMeter, too limiting. Indeed, many current developer tools can simulate and test a few thousand node systems passing messages, but not hundreds of thousands or millions of nodes sharing data.

Swarm lets developers simulate and plan their IoT networks so the developers can ensure things work, and also plan for capacity needs down the road. Moving software from development to production is always scary, but if your software is responsible for ensuring a crucial industrial process keeps working, it’s not okay to move fast and break things. Breaking things is not a badge of honor in the industrial world, which is why so many IoT deployments can get bogged down. But hopefully tools like Swarm will help developers move their IoT deployments from pilot to production with a little more confidence.

Obermaier also shared how his customers, which range from automotive manufacturers such as BMW to smart home device makers such as Awair and Flo, are building their IoT infrastructure. On the commercial side and for professional deployments by systems integrators, he sees a lot of containers, either Kubernetes or Red Hat’s OpenShift, being used to deploy applications. Containers are essential in the IoT because they help developers build applications that can run on a device and in the cloud. This makes it easier to build an app and deploy it widely at scale once it’s ready.

On the data pipeline side, where companies are ingesting lots of data for use immediately or later, he sees a lot of Apache Kafka, which may be in part because it works well with MQTT. Many of these customers are using services such as AWS’ Confluent cloud service.

However, for larger businesses or those with their own dedicated operations teams, Obermaier sees more open source adoption and work with providers such as Confluent. “The larger the company, the more hesitant the partners are to use those highly proprietary services,” Obermaier says. He adds that in the U.S., companies are more willing to work with cloud platforms, as opposed to Germany where auto manufacturing clients want to keep everything on-premise.