How I upgraded my water heater and discovered how bad smart home security can be - Ars Technica

Nov 05, 2024

Could you really control someone's hot water with just an email address?

The hot water took too long to come out of the tap. That is what I was trying to solve. I did not intend to discover that, for a while there, water heaters like mine may have been open to anybody. That, with some API tinkering and an email address, a bad actor could possibly set its temperature or make it run constantly. That’s just how it happened.

Let’s take a step back. My wife and I moved into a new home last year. It had a Rinnai tankless water heater tucked into a utility closet in the garage. The builder and home inspector didn't say much about it, just to run a yearly cleaning cycle on it.

Because it doesn’t keep a big tank of water heated and ready to be delivered to any house tap, tankless water heaters save energy—up to 34 percent, according to the Department of Energy. But they're also, by default, slower. Opening a tap triggers the exchanger, heats up the water (with natural gas, in my case), and the device has to push it through the line to where it's needed.

That led to me routinely holding my hand under cold water in the sink or shower, waiting longer than felt right for reasonably warm water to appear. I understood the water-for-energy trade-off I was making. But the setup wasted time, in addition to potable water, however plentiful and relatively cheap it was. It just irked me.

Little did I know the solution was just around the corner.

I mean that literally. When I went into the utility closet to shut off the hose bibbs for winter, I noticed a plastic bag magnetically stuck to the back side of the water heater. “Attention! The Control-R Wi-Fi Module must be installed for recirculation to operate,” read the intense yellow warning label. The water heater would not "recirculate" without it, it noted.

I found the manual, unplugged the water heater, and opened it up. The tone of the language inside (“DO NOT TOUCH,” unless you are “a properly trained technician”) did not match that of the can-do manual (“get the most from your new module”). But, having read the manual and slotted little beige nubs before, I felt trained and technical. I installed the device, went through the typical “Connect your phone to this weirdly named hotspot” process, and—it worked.

I now had an app that could start recirculation. I could get my shower hot while still in bed, or get started on the dinner dishes from the couch. And yet pulling out my phone whenever I wanted hot water felt like trading one inconvenience for another.

Being a home automation nerd, and thereby a Home Assistant enthusiast, I searched for a better way. I found an unofficial Rinnai component and installed it, and then I had real control. I could set recirculation to run on whatever schedule I wanted, triggered by anything, at any temperature. If I wanted to start hot water flowing on winter mornings as soon as the bedroom lights came on, but only if the moon was in Aquarius, I could do that (and I am not joking). The future felt warm, but not too warm, and on-demand.

If everything worked great, why am I still writing? Because I contacted the coders behind that integration, thinking I might write up my fun little hotter-water-quicker adventure. Maybe I'd learn more about what motivates the folks doing this useful work for free. This added a surprising second and surreal third part to my one-act play.

I emailed and spoke by phone with integration author Brad Barbour, who, like me, wanted to make his Rinnai-brand tankless water heater better, both for his code-savvy self and his water-using family. Barbour also lives where freezing temperatures are uncommon, but not impossible, so that most houses are not built for the cold. He wanted to automatically keep hot water circulating every so often whenever the temperature dropped into pipe-freezing range.

The first version of Rinnai's official app, “Control-R,” “left a big gap” in function and convenience, Barbour said. Only one account could be signed in. A Google integration “basically never really worked,” he said, and you could only set up timed schedules, not automate. Barbour, wanting to craft his own solution, started watching the app’s network traffic to see if he could do better.

The calls Control-R made to Rinnai’s servers were “very basic,” Barbour said. Digging into the undocumented API calls, Barbour saw something he didn’t think was real: You needed only a registered email address to retrieve information, or change settings, on a connected water heater.

“I thought this was crazy until another GitHub user reached out and we started collaborating and came to the same conclusion. You could control any Rinnai water heater that was connected, as long as you knew the registered account's email address,” Barbour wrote me.

Daniel Dulitz, the other GitHub user, had hashed things out with Barbour in a public GitHub issue on the project, “how does authentication work?” On June 29, 2021, Dulitz asked the core question about Rinnai’s app:

So it appears that this is an unauthenticated endpoint, and absolutely anyone on the Internet can read all the information about me and my water heater, and also set new temperatures for me at any time, without needing to know my password, just the API_KEY which is in this codebase (and is the same for everyone).

Please confirm or refute my observation.

The two prepared a security advisory for Rinnai, which I have seen. It notes that, while the technologies the system uses are "appropriate and defensible"—AWS Cognito, App Sync/Graph QL, and CloudFront—they were seemingly configured to allow reading and setting without requiring access tokens or keys. “Knowing only your email address, I can set your water heater’s temperature to very cold or scaldingly hot. I can put it into recirculation mode continuously so that it uses lots of gas… I can see your home street address that you have entered into the Control-R app when you registered your water heater,” it read.

The “scaldingly hot” water point is somewhat moot, as the maximum temperature on most residential Rinnai units is capped at 120°F, unless you unlock higher temps on the board. Had Control-R allowed for turning off the unit entirely, that could have been very bad for a gas-powered appliance; my heater manual specifically warned me against keeping the unit off for more than two weeks due to hydrogen gas build-up. In any case, people messing with your water heater from outside your house with just your email and a guess at your water heater brand is a generally bad thing.

Barbour and Dulitz didn’t actually send the notice; both had other things going on at the time, and it wasn't clear how to securely disclose it to the company, they said. Rinnai eventually updated to a new authentication system and a new app entirely, called “Rinnai Central.”

Another coder, Dustin D. Clark, who developed a Rinnai controller for the HomeBridge platform, arrived independently at the same conclusions and concerns as Barbour and Dulitz.

"I can tell you with 100% percent certainty that the API for setting state… did not require an auth token until sometime around August 23, 2022," Clark wrote me by email, noting a commit he made to his plug-in to start properly handing over a token. "Rinnai closed that gap, which necessitated this fix." (Clark added that there was a chance you needed an authentication token to look up a user's unique ID through their email address, though he can't say for certain now that it's changed).

I emailed Rinnai to ask about their former API, the changes to it, and a few other questions. I received back a response that was far more complicated than the company's initial API.

After trading a few emails with a PR firm that works with Rinnai, I heard back from a representative, who said that the Rinnai firm “would like to pass on the interview opportunity to be included in your article” (I did not request or offer an interview). Rinnai did, however, “appreciate that you are a new homeowner and current customer.” So answers to my questions were provided, but they were “written from a customer service perspective for your personal knowledge.”

I have never been told that only Kevin Purdy, water heater owner, had access to provided text, not Kevin Purdy, journalist. The answers were sent before any agreement had been reached about being on the record or not, on background, or other standard agreements between news gatherers and sources. I emailed the rep again to clarify:

Am I to understand that the answers… are not to be construed as a response from the company regarding my questions? That for my purposes, the answers to all the questions are a declination to comment?

I would generally quote customer service interactions if I was writing about a product. But your wording makes it seem like officially, they have no comment to the question of, for example, whether a professional should install the device?

“Yes, that’s correct,” the rep responded. Rinnai was declining to comment for the article. If I had further questions, the PR firm said, I could … reach out to customer service. I decided to skip the step of interviewing myself and close the loop.

Regardless, there wasn’t much to the “customer service” answers. The company said the Wi-Fi module was meant to be installed by a professional. It denied the claim that heater controls could previously be reached with just an email address. Notably, Rinnai has not issued any security advisories or Common Vulnerabilities and Exposures (CVE) notices regarding its API, past or present.

"Access control issues are by far the most significant challenge in API security," said Erez Yalon, vice president of security research at Checkmarx, and API Security Project Leader at the Open Worldwide Application Security Project (OWASP), an open-source-focused foundation. Four of OWASP's top 10 API security risks relate to authorization and authentication, Yalon noted. "This is no different, and sometimes even worse, for IoT and smart home appliances."

Mark Ostrowski, head of engineering at Check Point Software, noted that appliance vendors, working with open source operating systems, can design, build, and ship devices that have vulnerabilities by the time they arrive in a customer's home. Patching and updating those systems is often difficult, and typically left to the customer to initiate. "The increase in IoT security needs to be adopted in the home as we become more reliant on smart devices."

Unauthorized access through a product's app or smart features is something researchers at Consumer Reports have seen, "but rarely," according to a representative there. It's typically the domain of "startups or smaller white-label brands" seeking to have a seemingly high-tech product on the market, but without robust quality assurance for testing implementations.

There are privacy laws in 17 states (with Maryland just added) that demand companies safeguard buyers' personal information. The Federal Trade Commission and state attorneys general also have existing consumer protection laws to enforce security. Consumer Reports forwarded its findings about the Glow pregnancy app's lax security to the California Attorney General's office, which obtained a settlement in late 2020. And the FCC's proposed Cyber Trust Mark could further incentivize companies to keep with best security practices.

"But in general, we need more cops on the beat and greater consequences for companies when they break the law by using weak security protocols," Consumer Reports researchers said through a spokesperson.

I told Barbour, Dulitz, and Clark I was hesitant to share their story on a news site. They had made something supremely useful for a subset of people who were dealing with "smart" devices tied to not-entirely-smart phone apps. The potential rewards for their work had always been self-satisfaction, a few tip-jar-sized donations, and maybe some nice emails. The potential risks could now involve DMCA notices, threats of legal action, having their code taken down, or some combination of these.

Garage door opener conglomerate Chamberlain last year made a point of decrying “unauthorized usage” of its openers' API by third-party apps like Home Assistant, after it deliberately sabotaged the related extension. Paulus Schoutsen, founder of Home Assistant, wrote at Home Assistant’s blog that Chamberlain demanded payments from “authorized partners” to integrate with its myQ openers, which the open source, not-for-profit project could not offer. Owners could work around the block with a little ratgdo device and some wiring to restore access to their own devices.

In another case, a useful, unofficial Home Assistant tool for Mazda vehicle owners, developed from traffic watching, was wiped out in 2023 when its mostly sole developer received legal notices and DMCA takedown requests from Mazda. The coder told me by email at the time that while he believed his project was “morally correct and legally protected,” he couldn’t afford to take on the financial risk of legal action “for something that I do in my spare time to help others.”

That's why finding and writing about this kind of DIY tech thing—so specific, so helpful—makes me nervous. It feels like I discovered an underground rock club, but my dad's tight with the local troopers, and he put a GPS tracer on my used Accord.

All three coders said they were fine with Rinnai knowing about their work and the problems they saw early on. Barbour is working on a local network connection to the Wi-Fi module, to avoid Rinnai's servers and a potential Mazda/Chamberlain scenario.

Schoutsen said that Home Assistant reaches out to contributors accused of wrongdoing by companies, along with the companies. But even with a foundation behind it now, Home Assistant "doesn't have the resources to bring these cases to court." Nor is it feasible to really win. "Winning an individual case will not change anything; companies will always find new ways to keep their users out. That game of cat and mouse doesn’t stop until a law is established that forces them to allow their users access to their own data," Schoutsen wrote.

For now, my open source workaround really works, and it's far better than the app. It responds instantly, provides clear feedback on whether it's working or not, and surfaces lots of usable data. It also gave me my favorite thing: another home automation project.

Rinnai sells (through licensed installers) wireless push buttons and motion sensors that can start circulation. From what I can tell through a little chip ID, these are white-labeled Zigbee devices that connect to the Control-R module. What's their range? Will they repeat each other, as you'd expect from Zigbee devices? How long will they be supported? It's unclear.

Instead, using Home Assistant, I was able to trigger water recirculation through Ikea’s Tradfri buttons, which are far cheaper and nicer looking. After some Zigbee pairing and firmware updating, I can now press a button to send hot water my way. Plus, I can assign the Ikea button's other triggers, like double-click or long-press, to do other things, like dimming the lights for a bath.

The hot water now comes out of the tap when I need it. It just took a weird, circuitous path to get there.

Listing image: Getty Images