EasyIO or EasyI…uh Oh?

If your vendor isn't listed discussion for that goes here. If we get a lot of discussion on it we will create a sub and move posts.
Post Reply
User avatar
orion242
Posts: 165
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

EasyIO or EasyI…uh Oh?

Post by orion242 »

Part 1 – First sniff

After hearing about inputs failing on the EasyIO FG-32 as a common fault, I decided to take a peek for myself. Thanks to a member in the community, I secured a used unit with several bad inputs for a reasonable price. Got the unit a week later and cracking it open, the first fault was staring at me. One of the supercapacitors had leaked over a PCB.
RIP SC.jpg
RIP SC.jpg (452.33 KiB) Viewed 296 times
This is a common failure mode for electrolytic capacitors. Heat is the enemy of electrolytic capacitors. Long life high temp capacitors can be had, but at an added expense. When they fail this way, they lose all their capacitance and spill electrolyte on the PCB. The electrolyte is corrosive and has little issue eating thru the solder mask and any metal it contacts given enough time. The FG-32 has a conformal coating on it which is an added layer of protection. Also didn’t help here.

It should be noted that this device was manufactured late in 2014 judging by the date codes. Its common for ICs and PCBs to have a 4-digit date code on them that is formed from two digits for the week of the year and last two digits of the year. These date codes are just that. The date the IC / PCB was manufactured. An OEM could purchase many key parts and used them over serval years or a run of a common PCB that is used over a long period of time. Not a perfect measure of Inservice use, but with JIT manufacturing all the rage these are a good indicator of device age.

This unit had ICs and PCB dates all around 44th week of 2014. The person I purchased this from also confirmed it was installed in 2015. Likely installed early 2015 and early 2021 in my hands with 5 dead inputs. Six years to the point so many inputs failed it was replaced in this case. They also indicated the first few inputs became a problem a year earlier.

Diving deeper into the FG-32…

Removing the first supercap, cleaning and getting a clear look at the damaged.
Broken traces under C18.jpg
Broken traces under C18.jpg (465.77 KiB) Viewed 296 times
Inputs 1-4 have traces that run under the large supercap that blead out. Two had failed with a third taking on damage. A larger trace next to these seems to supply power to the op-amps buffering the signal to the final ADC that reads the inputs. If that fails, it could cause failure of all the inputs in one shot.

Reading the datasheet and the serial console, this supercap is the one responsible for its orderly shutdown on power loss. Once this fails, that goes out the window. After replacing this cap and looking at the console during shutdown it appears to check the supercap health and if that passes copies a RAM disk to flash. Failure of this cap may lead to firmware / filesystem corruption or loss of data. The real time clock also uses supercaps to keep the clock running during power loss, so that may be lost as well. Would like to hear if anyone has had any issues with this. Also wonder if this supercap test was in the original firmware or added after complaints piled up.

Each input has a TVS diode across it which is the main protection on the inputs. If polarity is reversed it will cap reverse voltage downstream to ~-0.65v. In the correct polarity, it will start shorting at ~12v.

Reversed input schematic, all inputs are identical.
Input Circuit.png
Input Circuit.png (18.29 KiB) Viewed 283 times
Looking around the PCB for test points, found a Linux console port near the SD card. Looks like instant root access into the device. Interface is 3.3v @ 115,200 baud. Sample boot up output.
FG console bootup.pdf
(34.46 KiB) Downloaded 19 times
Looks like this device also has a populated JTAG header near the super capacitors.

In the process of getting CPT and loading a small I/O test program, of course one must run a quick port scan. Besides what the documentation (EasyIO FG Series FAQ v1.3) lists as ports/services running, it also has telnet. This also seems to drop you into a shell. Not sure yet if this can be disabled or at minimum the user/pw changed from default. Seems like a train wreck to have telnet enabled, default creds and undocumented.

Kind of a scatter shot first look but that’s how things goes. Looks like poor quality supercaps from HCCCap are the first issue. Basic cybers smells as well. Next steps will be looking closer at the input protection, poking more at its cybers and whatever else crops up along the way. If someone has another failed unit or anything else in the EasyIO line that has failed, I may be interested in purchasing it. PM me.
User avatar
Maxburn
Posts: 58
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

I shared this stuff with my contacts in JCI channel manager and Broudy a while back. Not expecting anything to come back to me about this but EasyIO has known that there's awareness of this in the community for about three weeks now.
User avatar
orion242
Posts: 165
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

My two pennies..

This is not the first they have heard of it regardless what they say. This kind of damage takes time typically and is not a old unit by BMS terms. Installed in a hot location brings this up much quicker with crap parts. Good spec stuff might last decades in the same conditions. I'm aiming for the current FS to take a peek and see if anything has really improved. Not really impressed with this guy. Seems engineered to fail, outside of warranty of course... Maybe dumb luck design, but its likely a very common fault with the first four inputs. The other input failures are the TVS diodes shorting out. That failure mode could just as easily affect the first four inputs, but I assume its further down the road given the history I have on this unit. Still nothing to write home about.
User avatar
Maxburn
Posts: 58
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

I shared it in the other place but I've got a FW series that was installed in a hot location for about two years and is leaking too. It was a beta test...
User avatar
orion242
Posts: 165
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Two years? That's just garbage.

About to pull the trigger on a FS-32. Kinda a pricey for just a tear down though.
User avatar
orion242
Posts: 165
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Scored a FS-32 for a tear down. Looks to be a NIB unit from 2017.

Uses pretty much the same HCCCap supercaps. The bigger has molded plastic outer covering now. It may contain its blood better when it fails. The smaller one looks identical. Will dig up the datasheets and post. The location of the bigger one has moved so maybe its in a better location.

Same Linux serial shell on it. This time its at least password protected, though its one of the common default creds. Bootup capture attached.
Attachments
FS Bootup.pdf
(52.52 KiB) Downloaded 3 times
User avatar
Maxburn
Posts: 58
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
User avatar
Maxburn
Posts: 58
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

Comparing things, ALC latest release drvier reports this in logs; Linux version 3.12.10-alc (buildcomp@rdvm-yocto1404) (gcc version 4.7.3 20130226 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) ) #1 PREEMPT Tue Jun 11 19:25:09 EDT 2019

The latest beta I'm aware of reports this in the modstat, can't find it in the device logs.
Operating System: Linux - 4.9.59-alc
Java Version: 11.0.4+11 by AdoptOpenJDK
Processor Architecture: arm Cores: 1
Memory: 80MB Used, 27MB Free, 80MB Total
User avatar
orion242
Posts: 165
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Maxburn wrote: Sat May 08, 2021 1:40 pm Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
Did update the firmware on this guy which seems to have been a huge leap forward. OpenVPN support with user supplied config file was one of the big adds it didn't have. Come to think of it, didn't support DHCP originally either. Will have to get a new bootup capture to see if the OS was also updated. Suspect so.

Need to poke at the user accounts but haven't had the time yet. Thinking some of these cannot be modified by the user which could be a big fail imo. Also wondering if it has any cloudy type access that might be something to poke at. Had a fair bit of screwing around getting cpt connected to it and getting current firmware & cpt version which didn't leave much time for exploring.
Post Reply