Page 1 of 1

EasyIO or EasyI…uh Oh?

Posted: Fri Mar 19, 2021 12:58 am
by orion242
Part 1 – First sniff

After hearing about inputs failing on the EasyIO FG-32 as a common fault, I decided to take a peek for myself. Thanks to a member in the community, I secured a used unit with several bad inputs for a reasonable price. Got the unit a week later and cracking it open, the first fault was staring at me. One of the supercapacitors had leaked over a PCB.
RIP SC.jpg
RIP SC.jpg (452.33 KiB) Viewed 129645 times
This is a common failure mode for electrolytic capacitors. Heat is the enemy of electrolytic capacitors. Long life high temp capacitors can be had, but at an added expense. When they fail this way, they lose all their capacitance and spill electrolyte on the PCB. The electrolyte is corrosive and has little issue eating thru the solder mask and any metal it contacts given enough time. The FG-32 has a conformal coating on it which is an added layer of protection. Also didn’t help here.

It should be noted that this device was manufactured late in 2014 judging by the date codes. Its common for ICs and PCBs to have a 4-digit date code on them that is formed from two digits for the week of the year and last two digits of the year. These date codes are just that. The date the IC / PCB was manufactured. An OEM could purchase many key parts and used them over serval years or a run of a common PCB that is used over a long period of time. Not a perfect measure of Inservice use, but with JIT manufacturing all the rage these are a good indicator of device age.

This unit had ICs and PCB dates all around 44th week of 2014. The person I purchased this from also confirmed it was installed in 2015. Likely installed early 2015 and early 2021 in my hands with 5 dead inputs. Six years to the point so many inputs failed it was replaced in this case. They also indicated the first few inputs became a problem a year earlier.

Diving deeper into the FG-32…

Removing the first supercap, cleaning and getting a clear look at the damaged.
Broken traces under C18.jpg
Broken traces under C18.jpg (465.77 KiB) Viewed 129645 times
Inputs 1-4 have traces that run under the large supercap that blead out. Two had failed with a third taking on damage. A larger trace next to these seems to supply power to the op-amps buffering the signal to the final ADC that reads the inputs. If that fails, it could cause failure of all the inputs in one shot.

Reading the datasheet and the serial console, this supercap is the one responsible for its orderly shutdown on power loss. Once this fails, that goes out the window. After replacing this cap and looking at the console during shutdown it appears to check the supercap health and if that passes copies a RAM disk to flash. Failure of this cap may lead to firmware / filesystem corruption or loss of data. The real time clock also uses supercaps to keep the clock running during power loss, so that may be lost as well. Would like to hear if anyone has had any issues with this. Also wonder if this supercap test was in the original firmware or added after complaints piled up.

Each input has a TVS diode across it which is the main protection on the inputs. If polarity is reversed it will cap reverse voltage downstream to ~-0.65v. In the correct polarity, it will start shorting at ~12v.

Reversed input schematic, all inputs are identical.
Input Circuit.png
Input Circuit.png (18.29 KiB) Viewed 129632 times
Looking around the PCB for test points, found a Linux console port near the SD card. Looks like instant root access into the device. Interface is 3.3v @ 115,200 baud. Sample boot up output.
FG console bootup.pdf
(34.46 KiB) Downloaded 8022 times
Looks like this device also has a populated JTAG header near the super capacitors.

In the process of getting CPT and loading a small I/O test program, of course one must run a quick port scan. Besides what the documentation (EasyIO FG Series FAQ v1.3) lists as ports/services running, it also has telnet. This also seems to drop you into a shell. Not sure yet if this can be disabled or at minimum the user/pw changed from default. Seems like a train wreck to have telnet enabled, default creds and undocumented.

Kind of a scatter shot first look but that’s how things goes. Looks like poor quality supercaps from HCCCap are the first issue. Basic cybers smells as well. Next steps will be looking closer at the input protection, poking more at its cybers and whatever else crops up along the way. If someone has another failed unit or anything else in the EasyIO line that has failed, I may be interested in purchasing it. PM me.

Re: EasyIO or EasyI…uh Oh?

Posted: Wed Mar 31, 2021 1:07 pm
by Maxburn
I shared this stuff with my contacts in JCI channel manager and Broudy a while back. Not expecting anything to come back to me about this but EasyIO has known that there's awareness of this in the community for about three weeks now.

Re: EasyIO or EasyI…uh Oh?

Posted: Fri Apr 02, 2021 12:34 am
by orion242
My two pennies..

This is not the first they have heard of it regardless what they say. This kind of damage takes time typically and is not a old unit by BMS terms. Installed in a hot location brings this up much quicker with crap parts. Good spec stuff might last decades in the same conditions. I'm aiming for the current FS to take a peek and see if anything has really improved. Not really impressed with this guy. Seems engineered to fail, outside of warranty of course... Maybe dumb luck design, but its likely a very common fault with the first four inputs. The other input failures are the TVS diodes shorting out. That failure mode could just as easily affect the first four inputs, but I assume its further down the road given the history I have on this unit. Still nothing to write home about.

Re: EasyIO or EasyI…uh Oh?

Posted: Mon Apr 19, 2021 11:11 pm
by Maxburn
I shared it in the other place but I've got a FW series that was installed in a hot location for about two years and is leaking too. It was a beta test...

Re: EasyIO or EasyI…uh Oh?

Posted: Thu Apr 22, 2021 10:54 pm
by orion242
Two years? That's just garbage.

About to pull the trigger on a FS-32. Kinda a pricey for just a tear down though.

Re: EasyIO or EasyI…uh Oh?

Posted: Fri May 07, 2021 12:44 am
by orion242
Scored a FS-32 for a tear down. Looks to be a NIB unit from 2017.

Uses pretty much the same HCCCap supercaps. The bigger has molded plastic outer covering now. It may contain its blood better when it fails. The smaller one looks identical. Will dig up the datasheets and post. The location of the bigger one has moved so maybe its in a better location.

Same Linux serial shell on it. This time its at least password protected, though its one of the common default creds. Bootup capture attached.

Re: EasyIO or EasyI…uh Oh?

Posted: Sat May 08, 2021 1:40 pm
by Maxburn
Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6

Re: EasyIO or EasyI…uh Oh?

Posted: Sun May 09, 2021 1:39 pm
by Maxburn
Comparing things, ALC latest release drvier reports this in logs; Linux version 3.12.10-alc (buildcomp@rdvm-yocto1404) (gcc version 4.7.3 20130226 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) ) #1 PREEMPT Tue Jun 11 19:25:09 EDT 2019

The latest beta I'm aware of reports this in the modstat, can't find it in the device logs.
Operating System: Linux - 4.9.59-alc
Java Version: 11.0.4+11 by AdoptOpenJDK
Processor Architecture: arm Cores: 1
Memory: 80MB Used, 27MB Free, 80MB Total

Re: EasyIO or EasyI…uh Oh?

Posted: Sun May 09, 2021 11:41 pm
by orion242
Maxburn wrote: Sat May 08, 2021 1:40 pm Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
Did update the firmware on this guy which seems to have been a huge leap forward. OpenVPN support with user supplied config file was one of the big adds it didn't have. Come to think of it, didn't support DHCP originally either. Will have to get a new bootup capture to see if the OS was also updated. Suspect so.

Need to poke at the user accounts but haven't had the time yet. Thinking some of these cannot be modified by the user which could be a big fail imo. Also wondering if it has any cloudy type access that might be something to poke at. Had a fair bit of screwing around getting cpt connected to it and getting current firmware & cpt version which didn't leave much time for exploring.

Re: EasyIO or EasyI…uh Oh?

Posted: Sat May 22, 2021 12:25 am
by orion242
This seems to be the super cap on the FS32.
HCAP-D 5R5 255-F.pdf
(423.59 KiB) Downloaded 6272 times
Not a great choice for your typical hot BMS locations. Down south, even northern installed in a typical RTU electrical cabinet or steam plant? Huston we have a problem...

Molded plastic enclosure, epoxy to seal the bottom, maybe better than the FG series. The epoxy doesn't seem to bond with the plastic well. Would be surprised if this contains liquids much better than without. Especially under any pressure.

That cap is providing the power for an orderly shutdown. Without that cap, she gets a bit unhappy after only a handful of power cycles. Boot up capture new firmware after a handful of power cycles with supercap removed.
FS Bootup New FW.pdf
(62.43 KiB) Downloaded 6274 times
Took some time to come around with the red error led flashing while it looked for the root FS.

After putting the cap back in, powering up letting it boot, this is the shutdown output with a good cap.
FS Power down With Cap.pdf
(52.22 KiB) Downloaded 6233 times
So its still appears to make a decision based on the backup power during a power loss. That of course only happens if it has some amount of onboard backup power. If not, its instantly dead in the water.

Booting it back up after the initial FS recovery with a good cap seems to have restored it back to normal. Boots up quick without any fuss now, never lost the control program in through this. I did have an SD card installed with a backup on it.

They did spend a bit more time on the 485 ports in the FS vs the FG.

Re: EasyIO or EasyI…uh Oh?

Posted: Tue May 25, 2021 1:56 pm
by Maxburn
Still says Linux-3.4.39. Will say most of the CVE issues will be resolved by not having those functions in use on this. Anything in the IP stack though...

Re: EasyIO or EasyI…uh Oh?

Posted: Thu May 27, 2021 12:47 am
by orion242
If the hardware build is subpar, wouldn't expect the software side to be much better.

There was an option on the FS after the firmware update to change the "OS password" which I would assume changes the root account. FG had telnet running, not sure if the FS does. Haven't had the time to poke at software much. Working on RE and drawing up the 485 / input circuits on the FG/FS to see what has changed, time permitting.

Re: EasyIO or EasyI…uh Oh?

Posted: Fri May 28, 2021 7:25 pm
by black_syphilis
Really exciting investigation orion :)

Re: EasyIO or EasyI…uh Oh?

Posted: Fri May 28, 2021 8:18 pm
by orion242
black_syphilis wrote: Fri May 28, 2021 7:41 pmall FG have an unique root password
Didn't seem to be the case for the unit I had. For starters the serial console port drops you right into a shell with root access. No logon required. Would have to look at my notes, but pretty sure telnet was using one of the default creds.

Re: EasyIO or EasyI…uh Oh?

Posted: Fri May 28, 2021 8:30 pm
by black_syphilis
FS Bootup files? I didn't see a shell with root access.

By the way I'm checking FW and I think I need 2 hours to hack it and have a root access :twisted:

Re: EasyIO or EasyI…uh Oh?

Posted: Fri May 28, 2021 8:38 pm
by orion242
older FG not the FS

Re: EasyIO or EasyI…uh Oh?

Posted: Sat May 29, 2021 2:45 am
by orion242
Both FG & FS the serial console is 3.3v @ 115,200 baud.

FG the pinout is clearly marked in the solder mask. Its the same on the FS.

FG-32
FG Serial Console.jpg
FG Serial Console.jpg (1.35 MiB) Viewed 129134 times
FS-32
FS Serial Console.jpg
FS Serial Console.jpg (968.04 KiB) Viewed 129134 times
These seem to drop you into a shell on the main CPU. There is a second CPU handling I/O on both, doesn't appear to be the same easy access to those.

Again these are 3.3v interfaces, so don't screw around without that in mind. For a cheap USB to 3.3v serial google TTL-232R-3V3-WE.

The FG-32 also has JTAG headers on it, so that could be another avenue to dive in.

Re: EasyIO or EasyI…uh Oh?

Posted: Sat May 29, 2021 7:26 pm
by black_syphilis
FG v1.5b50
ooopps!
Image

Re: EasyIO or EasyI…uh Oh?

Posted: Mon May 31, 2021 2:13 am
by orion242
easy.jpg
easy.jpg (8.1 KiB) Viewed 129083 times