EasyIO or EasyI…uh Oh?
EasyIO or EasyI…uh Oh?
Part 1 – First sniff
After hearing about inputs failing on the EasyIO FG-32 as a common fault, I decided to take a peek for myself. Thanks to a member in the community, I secured a used unit with several bad inputs for a reasonable price. Got the unit a week later and cracking it open, the first fault was staring at me. One of the supercapacitors had leaked over a PCB.
This is a common failure mode for electrolytic capacitors. Heat is the enemy of electrolytic capacitors. Long life high temp capacitors can be had, but at an added expense. When they fail this way, they lose all their capacitance and spill electrolyte on the PCB. The electrolyte is corrosive and has little issue eating thru the solder mask and any metal it contacts given enough time. The FG-32 has a conformal coating on it which is an added layer of protection. Also didn’t help here.
It should be noted that this device was manufactured late in 2014 judging by the date codes. Its common for ICs and PCBs to have a 4-digit date code on them that is formed from two digits for the week of the year and last two digits of the year. These date codes are just that. The date the IC / PCB was manufactured. An OEM could purchase many key parts and used them over serval years or a run of a common PCB that is used over a long period of time. Not a perfect measure of Inservice use, but with JIT manufacturing all the rage these are a good indicator of device age.
This unit had ICs and PCB dates all around 44th week of 2014. The person I purchased this from also confirmed it was installed in 2015. Likely installed early 2015 and early 2021 in my hands with 5 dead inputs. Six years to the point so many inputs failed it was replaced in this case. They also indicated the first few inputs became a problem a year earlier.
Diving deeper into the FG-32…
Removing the first supercap, cleaning and getting a clear look at the damaged.
Inputs 1-4 have traces that run under the large supercap that blead out. Two had failed with a third taking on damage. A larger trace next to these seems to supply power to the op-amps buffering the signal to the final ADC that reads the inputs. If that fails, it could cause failure of all the inputs in one shot.
Reading the datasheet and the serial console, this supercap is the one responsible for its orderly shutdown on power loss. Once this fails, that goes out the window. After replacing this cap and looking at the console during shutdown it appears to check the supercap health and if that passes copies a RAM disk to flash. Failure of this cap may lead to firmware / filesystem corruption or loss of data. The real time clock also uses supercaps to keep the clock running during power loss, so that may be lost as well. Would like to hear if anyone has had any issues with this. Also wonder if this supercap test was in the original firmware or added after complaints piled up.
Each input has a TVS diode across it which is the main protection on the inputs. If polarity is reversed it will cap reverse voltage downstream to ~-0.65v. In the correct polarity, it will start shorting at ~12v.
Reversed input schematic, all inputs are identical.
Looking around the PCB for test points, found a Linux console port near the SD card. Looks like instant root access into the device. Interface is 3.3v @ 115,200 baud. Sample boot up output.
Looks like this device also has a populated JTAG header near the super capacitors.
In the process of getting CPT and loading a small I/O test program, of course one must run a quick port scan. Besides what the documentation (EasyIO FG Series FAQ v1.3) lists as ports/services running, it also has telnet. This also seems to drop you into a shell. Not sure yet if this can be disabled or at minimum the user/pw changed from default. Seems like a train wreck to have telnet enabled, default creds and undocumented.
Kind of a scatter shot first look but that’s how things goes. Looks like poor quality supercaps from HCCCap are the first issue. Basic cybers smells as well. Next steps will be looking closer at the input protection, poking more at its cybers and whatever else crops up along the way. If someone has another failed unit or anything else in the EasyIO line that has failed, I may be interested in purchasing it. PM me.
After hearing about inputs failing on the EasyIO FG-32 as a common fault, I decided to take a peek for myself. Thanks to a member in the community, I secured a used unit with several bad inputs for a reasonable price. Got the unit a week later and cracking it open, the first fault was staring at me. One of the supercapacitors had leaked over a PCB.
This is a common failure mode for electrolytic capacitors. Heat is the enemy of electrolytic capacitors. Long life high temp capacitors can be had, but at an added expense. When they fail this way, they lose all their capacitance and spill electrolyte on the PCB. The electrolyte is corrosive and has little issue eating thru the solder mask and any metal it contacts given enough time. The FG-32 has a conformal coating on it which is an added layer of protection. Also didn’t help here.
It should be noted that this device was manufactured late in 2014 judging by the date codes. Its common for ICs and PCBs to have a 4-digit date code on them that is formed from two digits for the week of the year and last two digits of the year. These date codes are just that. The date the IC / PCB was manufactured. An OEM could purchase many key parts and used them over serval years or a run of a common PCB that is used over a long period of time. Not a perfect measure of Inservice use, but with JIT manufacturing all the rage these are a good indicator of device age.
This unit had ICs and PCB dates all around 44th week of 2014. The person I purchased this from also confirmed it was installed in 2015. Likely installed early 2015 and early 2021 in my hands with 5 dead inputs. Six years to the point so many inputs failed it was replaced in this case. They also indicated the first few inputs became a problem a year earlier.
Diving deeper into the FG-32…
Removing the first supercap, cleaning and getting a clear look at the damaged.
Inputs 1-4 have traces that run under the large supercap that blead out. Two had failed with a third taking on damage. A larger trace next to these seems to supply power to the op-amps buffering the signal to the final ADC that reads the inputs. If that fails, it could cause failure of all the inputs in one shot.
Reading the datasheet and the serial console, this supercap is the one responsible for its orderly shutdown on power loss. Once this fails, that goes out the window. After replacing this cap and looking at the console during shutdown it appears to check the supercap health and if that passes copies a RAM disk to flash. Failure of this cap may lead to firmware / filesystem corruption or loss of data. The real time clock also uses supercaps to keep the clock running during power loss, so that may be lost as well. Would like to hear if anyone has had any issues with this. Also wonder if this supercap test was in the original firmware or added after complaints piled up.
Each input has a TVS diode across it which is the main protection on the inputs. If polarity is reversed it will cap reverse voltage downstream to ~-0.65v. In the correct polarity, it will start shorting at ~12v.
Reversed input schematic, all inputs are identical.
Looking around the PCB for test points, found a Linux console port near the SD card. Looks like instant root access into the device. Interface is 3.3v @ 115,200 baud. Sample boot up output.
Looks like this device also has a populated JTAG header near the super capacitors.
In the process of getting CPT and loading a small I/O test program, of course one must run a quick port scan. Besides what the documentation (EasyIO FG Series FAQ v1.3) lists as ports/services running, it also has telnet. This also seems to drop you into a shell. Not sure yet if this can be disabled or at minimum the user/pw changed from default. Seems like a train wreck to have telnet enabled, default creds and undocumented.
Kind of a scatter shot first look but that’s how things goes. Looks like poor quality supercaps from HCCCap are the first issue. Basic cybers smells as well. Next steps will be looking closer at the input protection, poking more at its cybers and whatever else crops up along the way. If someone has another failed unit or anything else in the EasyIO line that has failed, I may be interested in purchasing it. PM me.
Re: EasyIO or EasyI…uh Oh?
I shared this stuff with my contacts in JCI channel manager and Broudy a while back. Not expecting anything to come back to me about this but EasyIO has known that there's awareness of this in the community for about three weeks now.
Re: EasyIO or EasyI…uh Oh?
My two pennies..
This is not the first they have heard of it regardless what they say. This kind of damage takes time typically and is not a old unit by BMS terms. Installed in a hot location brings this up much quicker with crap parts. Good spec stuff might last decades in the same conditions. I'm aiming for the current FS to take a peek and see if anything has really improved. Not really impressed with this guy. Seems engineered to fail, outside of warranty of course... Maybe dumb luck design, but its likely a very common fault with the first four inputs. The other input failures are the TVS diodes shorting out. That failure mode could just as easily affect the first four inputs, but I assume its further down the road given the history I have on this unit. Still nothing to write home about.
This is not the first they have heard of it regardless what they say. This kind of damage takes time typically and is not a old unit by BMS terms. Installed in a hot location brings this up much quicker with crap parts. Good spec stuff might last decades in the same conditions. I'm aiming for the current FS to take a peek and see if anything has really improved. Not really impressed with this guy. Seems engineered to fail, outside of warranty of course... Maybe dumb luck design, but its likely a very common fault with the first four inputs. The other input failures are the TVS diodes shorting out. That failure mode could just as easily affect the first four inputs, but I assume its further down the road given the history I have on this unit. Still nothing to write home about.
Re: EasyIO or EasyI…uh Oh?
I shared it in the other place but I've got a FW series that was installed in a hot location for about two years and is leaking too. It was a beta test...
Re: EasyIO or EasyI…uh Oh?
Two years? That's just garbage.
About to pull the trigger on a FS-32. Kinda a pricey for just a tear down though.
About to pull the trigger on a FS-32. Kinda a pricey for just a tear down though.
Re: EasyIO or EasyI…uh Oh?
Scored a FS-32 for a tear down. Looks to be a NIB unit from 2017.
Uses pretty much the same HCCCap supercaps. The bigger has molded plastic outer covering now. It may contain its blood better when it fails. The smaller one looks identical. Will dig up the datasheets and post. The location of the bigger one has moved so maybe its in a better location.
Same Linux serial shell on it. This time its at least password protected, though its one of the common default creds. Bootup capture attached.
Uses pretty much the same HCCCap supercaps. The bigger has molded plastic outer covering now. It may contain its blood better when it fails. The smaller one looks identical. Will dig up the datasheets and post. The location of the bigger one has moved so maybe its in a better location.
Same Linux serial shell on it. This time its at least password protected, though its one of the common default creds. Bootup capture attached.
- Attachments
-
- FS Bootup.pdf
- (52.52 KiB) Downloaded 5561 times
Re: EasyIO or EasyI…uh Oh?
Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
Re: EasyIO or EasyI…uh Oh?
Comparing things, ALC latest release drvier reports this in logs; Linux version 3.12.10-alc (buildcomp@rdvm-yocto1404) (gcc version 4.7.3 20130226 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) ) #1 PREEMPT Tue Jun 11 19:25:09 EDT 2019
The latest beta I'm aware of reports this in the modstat, can't find it in the device logs.
Operating System: Linux - 4.9.59-alc
Java Version: 11.0.4+11 by AdoptOpenJDK
Processor Architecture: arm Cores: 1
Memory: 80MB Used, 27MB Free, 80MB Total
The latest beta I'm aware of reports this in the modstat, can't find it in the device logs.
Operating System: Linux - 4.9.59-alc
Java Version: 11.0.4+11 by AdoptOpenJDK
Processor Architecture: arm Cores: 1
Memory: 80MB Used, 27MB Free, 80MB Total
Re: EasyIO or EasyI…uh Oh?
Did update the firmware on this guy which seems to have been a huge leap forward. OpenVPN support with user supplied config file was one of the big adds it didn't have. Come to think of it, didn't support DHCP originally either. Will have to get a new bootup capture to see if the OS was also updated. Suspect so.Maxburn wrote: ↑Sat May 08, 2021 1:40 pm Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
Need to poke at the user accounts but haven't had the time yet. Thinking some of these cannot be modified by the user which could be a big fail imo. Also wondering if it has any cloudy type access that might be something to poke at. Had a fair bit of screwing around getting cpt connected to it and getting current firmware & cpt version which didn't leave much time for exploring.
Re: EasyIO or EasyI…uh Oh?
This seems to be the super cap on the FS32.
Not a great choice for your typical hot BMS locations. Down south, even northern installed in a typical RTU electrical cabinet or steam plant? Huston we have a problem...
Molded plastic enclosure, epoxy to seal the bottom, maybe better than the FG series. The epoxy doesn't seem to bond with the plastic well. Would be surprised if this contains liquids much better than without. Especially under any pressure.
That cap is providing the power for an orderly shutdown. Without that cap, she gets a bit unhappy after only a handful of power cycles. Boot up capture new firmware after a handful of power cycles with supercap removed.
Took some time to come around with the red error led flashing while it looked for the root FS.
After putting the cap back in, powering up letting it boot, this is the shutdown output with a good cap.
So its still appears to make a decision based on the backup power during a power loss. That of course only happens if it has some amount of onboard backup power. If not, its instantly dead in the water.
Booting it back up after the initial FS recovery with a good cap seems to have restored it back to normal. Boots up quick without any fuss now, never lost the control program in through this. I did have an SD card installed with a backup on it.
They did spend a bit more time on the 485 ports in the FS vs the FG.
Not a great choice for your typical hot BMS locations. Down south, even northern installed in a typical RTU electrical cabinet or steam plant? Huston we have a problem...
Molded plastic enclosure, epoxy to seal the bottom, maybe better than the FG series. The epoxy doesn't seem to bond with the plastic well. Would be surprised if this contains liquids much better than without. Especially under any pressure.
That cap is providing the power for an orderly shutdown. Without that cap, she gets a bit unhappy after only a handful of power cycles. Boot up capture new firmware after a handful of power cycles with supercap removed.
Took some time to come around with the red error led flashing while it looked for the root FS.
After putting the cap back in, powering up letting it boot, this is the shutdown output with a good cap.
So its still appears to make a decision based on the backup power during a power loss. That of course only happens if it has some amount of onboard backup power. If not, its instantly dead in the water.
Booting it back up after the initial FS recovery with a good cap seems to have restored it back to normal. Boots up quick without any fuss now, never lost the control program in through this. I did have an SD card installed with a backup on it.
They did spend a bit more time on the 485 ports in the FS vs the FG.
Re: EasyIO or EasyI…uh Oh?
Still says Linux-3.4.39. Will say most of the CVE issues will be resolved by not having those functions in use on this. Anything in the IP stack though...
Re: EasyIO or EasyI…uh Oh?
If the hardware build is subpar, wouldn't expect the software side to be much better.
There was an option on the FS after the firmware update to change the "OS password" which I would assume changes the root account. FG had telnet running, not sure if the FS does. Haven't had the time to poke at software much. Working on RE and drawing up the 485 / input circuits on the FG/FS to see what has changed, time permitting.
There was an option on the FS after the firmware update to change the "OS password" which I would assume changes the root account. FG had telnet running, not sure if the FS does. Haven't had the time to poke at software much. Working on RE and drawing up the 485 / input circuits on the FG/FS to see what has changed, time permitting.
- black_syphilis
- Posts: 22
- Joined: Tue May 11, 2021 12:35 pm
- Location: Montreal
Re: EasyIO or EasyI…uh Oh?
Really exciting investigation orion
Re: EasyIO or EasyI…uh Oh?
Didn't seem to be the case for the unit I had. For starters the serial console port drops you right into a shell with root access. No logon required. Would have to look at my notes, but pretty sure telnet was using one of the default creds.
- black_syphilis
- Posts: 22
- Joined: Tue May 11, 2021 12:35 pm
- Location: Montreal
Re: EasyIO or EasyI…uh Oh?
FS Bootup files? I didn't see a shell with root access.
By the way I'm checking FW and I think I need 2 hours to hack it and have a root access
By the way I'm checking FW and I think I need 2 hours to hack it and have a root access
Re: EasyIO or EasyI…uh Oh?
older FG not the FS
Re: EasyIO or EasyI…uh Oh?
Both FG & FS the serial console is 3.3v @ 115,200 baud.
FG the pinout is clearly marked in the solder mask. Its the same on the FS.
FG-32 FS-32 These seem to drop you into a shell on the main CPU. There is a second CPU handling I/O on both, doesn't appear to be the same easy access to those.
Again these are 3.3v interfaces, so don't screw around without that in mind. For a cheap USB to 3.3v serial google TTL-232R-3V3-WE.
The FG-32 also has JTAG headers on it, so that could be another avenue to dive in.
FG the pinout is clearly marked in the solder mask. Its the same on the FS.
FG-32 FS-32 These seem to drop you into a shell on the main CPU. There is a second CPU handling I/O on both, doesn't appear to be the same easy access to those.
Again these are 3.3v interfaces, so don't screw around without that in mind. For a cheap USB to 3.3v serial google TTL-232R-3V3-WE.
The FG-32 also has JTAG headers on it, so that could be another avenue to dive in.
- black_syphilis
- Posts: 22
- Joined: Tue May 11, 2021 12:35 pm
- Location: Montreal
Re: EasyIO or EasyI…uh Oh?
FG v1.5b50
ooopps!
ooopps!