EasyIO or EasyI…uh Oh?

If your vendor isn't listed discussion for that goes here. If we get a lot of discussion on it we will create a sub and move posts.
Post Reply
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

EasyIO or EasyI…uh Oh?

Post by orion242 »

Part 1 – First sniff

After hearing about inputs failing on the EasyIO FG-32 as a common fault, I decided to take a peek for myself. Thanks to a member in the community, I secured a used unit with several bad inputs for a reasonable price. Got the unit a week later and cracking it open, the first fault was staring at me. One of the supercapacitors had leaked over a PCB.
RIP SC.jpg
RIP SC.jpg (452.33 KiB) Viewed 122959 times
This is a common failure mode for electrolytic capacitors. Heat is the enemy of electrolytic capacitors. Long life high temp capacitors can be had, but at an added expense. When they fail this way, they lose all their capacitance and spill electrolyte on the PCB. The electrolyte is corrosive and has little issue eating thru the solder mask and any metal it contacts given enough time. The FG-32 has a conformal coating on it which is an added layer of protection. Also didn’t help here.

It should be noted that this device was manufactured late in 2014 judging by the date codes. Its common for ICs and PCBs to have a 4-digit date code on them that is formed from two digits for the week of the year and last two digits of the year. These date codes are just that. The date the IC / PCB was manufactured. An OEM could purchase many key parts and used them over serval years or a run of a common PCB that is used over a long period of time. Not a perfect measure of Inservice use, but with JIT manufacturing all the rage these are a good indicator of device age.

This unit had ICs and PCB dates all around 44th week of 2014. The person I purchased this from also confirmed it was installed in 2015. Likely installed early 2015 and early 2021 in my hands with 5 dead inputs. Six years to the point so many inputs failed it was replaced in this case. They also indicated the first few inputs became a problem a year earlier.

Diving deeper into the FG-32…

Removing the first supercap, cleaning and getting a clear look at the damaged.
Broken traces under C18.jpg
Broken traces under C18.jpg (465.77 KiB) Viewed 122959 times
Inputs 1-4 have traces that run under the large supercap that blead out. Two had failed with a third taking on damage. A larger trace next to these seems to supply power to the op-amps buffering the signal to the final ADC that reads the inputs. If that fails, it could cause failure of all the inputs in one shot.

Reading the datasheet and the serial console, this supercap is the one responsible for its orderly shutdown on power loss. Once this fails, that goes out the window. After replacing this cap and looking at the console during shutdown it appears to check the supercap health and if that passes copies a RAM disk to flash. Failure of this cap may lead to firmware / filesystem corruption or loss of data. The real time clock also uses supercaps to keep the clock running during power loss, so that may be lost as well. Would like to hear if anyone has had any issues with this. Also wonder if this supercap test was in the original firmware or added after complaints piled up.

Each input has a TVS diode across it which is the main protection on the inputs. If polarity is reversed it will cap reverse voltage downstream to ~-0.65v. In the correct polarity, it will start shorting at ~12v.

Reversed input schematic, all inputs are identical.
Input Circuit.png
Input Circuit.png (18.29 KiB) Viewed 122946 times
Looking around the PCB for test points, found a Linux console port near the SD card. Looks like instant root access into the device. Interface is 3.3v @ 115,200 baud. Sample boot up output.
FG console bootup.pdf
(34.46 KiB) Downloaded 7857 times
Looks like this device also has a populated JTAG header near the super capacitors.

In the process of getting CPT and loading a small I/O test program, of course one must run a quick port scan. Besides what the documentation (EasyIO FG Series FAQ v1.3) lists as ports/services running, it also has telnet. This also seems to drop you into a shell. Not sure yet if this can be disabled or at minimum the user/pw changed from default. Seems like a train wreck to have telnet enabled, default creds and undocumented.

Kind of a scatter shot first look but that’s how things goes. Looks like poor quality supercaps from HCCCap are the first issue. Basic cybers smells as well. Next steps will be looking closer at the input protection, poking more at its cybers and whatever else crops up along the way. If someone has another failed unit or anything else in the EasyIO line that has failed, I may be interested in purchasing it. PM me.
User avatar
Maxburn
Posts: 101
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

I shared this stuff with my contacts in JCI channel manager and Broudy a while back. Not expecting anything to come back to me about this but EasyIO has known that there's awareness of this in the community for about three weeks now.
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

My two pennies..

This is not the first they have heard of it regardless what they say. This kind of damage takes time typically and is not a old unit by BMS terms. Installed in a hot location brings this up much quicker with crap parts. Good spec stuff might last decades in the same conditions. I'm aiming for the current FS to take a peek and see if anything has really improved. Not really impressed with this guy. Seems engineered to fail, outside of warranty of course... Maybe dumb luck design, but its likely a very common fault with the first four inputs. The other input failures are the TVS diodes shorting out. That failure mode could just as easily affect the first four inputs, but I assume its further down the road given the history I have on this unit. Still nothing to write home about.
User avatar
Maxburn
Posts: 101
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

I shared it in the other place but I've got a FW series that was installed in a hot location for about two years and is leaking too. It was a beta test...
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Two years? That's just garbage.

About to pull the trigger on a FS-32. Kinda a pricey for just a tear down though.
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Scored a FS-32 for a tear down. Looks to be a NIB unit from 2017.

Uses pretty much the same HCCCap supercaps. The bigger has molded plastic outer covering now. It may contain its blood better when it fails. The smaller one looks identical. Will dig up the datasheets and post. The location of the bigger one has moved so maybe its in a better location.

Same Linux serial shell on it. This time its at least password protected, though its one of the common default creds. Bootup capture attached.
Attachments
FS Bootup.pdf
(52.52 KiB) Downloaded 5962 times
User avatar
Maxburn
Posts: 101
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
User avatar
Maxburn
Posts: 101
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

Comparing things, ALC latest release drvier reports this in logs; Linux version 3.12.10-alc (buildcomp@rdvm-yocto1404) (gcc version 4.7.3 20130226 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.03-20130313 - Linaro GCC 2013.03) ) #1 PREEMPT Tue Jun 11 19:25:09 EDT 2019

The latest beta I'm aware of reports this in the modstat, can't find it in the device logs.
Operating System: Linux - 4.9.59-alc
Java Version: 11.0.4+11 by AdoptOpenJDK
Processor Architecture: arm Cores: 1
Memory: 80MB Used, 27MB Free, 80MB Total
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Maxburn wrote: Sat May 08, 2021 1:40 pm Only thing that jumps out to me is inux-3.4.39 is about 2013 vintage, Very old. This page suggests there might be 269 CVSS on it but I can't get it to display. https://www.cvedetails.com/vulnerabilit ... ea319d63b6
Did update the firmware on this guy which seems to have been a huge leap forward. OpenVPN support with user supplied config file was one of the big adds it didn't have. Come to think of it, didn't support DHCP originally either. Will have to get a new bootup capture to see if the OS was also updated. Suspect so.

Need to poke at the user accounts but haven't had the time yet. Thinking some of these cannot be modified by the user which could be a big fail imo. Also wondering if it has any cloudy type access that might be something to poke at. Had a fair bit of screwing around getting cpt connected to it and getting current firmware & cpt version which didn't leave much time for exploring.
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

This seems to be the super cap on the FS32.
HCAP-D 5R5 255-F.pdf
(423.59 KiB) Downloaded 6097 times
Not a great choice for your typical hot BMS locations. Down south, even northern installed in a typical RTU electrical cabinet or steam plant? Huston we have a problem...

Molded plastic enclosure, epoxy to seal the bottom, maybe better than the FG series. The epoxy doesn't seem to bond with the plastic well. Would be surprised if this contains liquids much better than without. Especially under any pressure.

That cap is providing the power for an orderly shutdown. Without that cap, she gets a bit unhappy after only a handful of power cycles. Boot up capture new firmware after a handful of power cycles with supercap removed.
FS Bootup New FW.pdf
(62.43 KiB) Downloaded 6104 times
Took some time to come around with the red error led flashing while it looked for the root FS.

After putting the cap back in, powering up letting it boot, this is the shutdown output with a good cap.
FS Power down With Cap.pdf
(52.22 KiB) Downloaded 6066 times
So its still appears to make a decision based on the backup power during a power loss. That of course only happens if it has some amount of onboard backup power. If not, its instantly dead in the water.

Booting it back up after the initial FS recovery with a good cap seems to have restored it back to normal. Boots up quick without any fuss now, never lost the control program in through this. I did have an SD card installed with a backup on it.

They did spend a bit more time on the 485 ports in the FS vs the FG.
User avatar
Maxburn
Posts: 101
Joined: Wed Mar 04, 2020 12:51 am

Re: EasyIO or EasyI…uh Oh?

Post by Maxburn »

Still says Linux-3.4.39. Will say most of the CVE issues will be resolved by not having those functions in use on this. Anything in the IP stack though...
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

If the hardware build is subpar, wouldn't expect the software side to be much better.

There was an option on the FS after the firmware update to change the "OS password" which I would assume changes the root account. FG had telnet running, not sure if the FS does. Haven't had the time to poke at software much. Working on RE and drawing up the 485 / input circuits on the FG/FS to see what has changed, time permitting.
User avatar
black_syphilis
Posts: 22
Joined: Tue May 11, 2021 12:35 pm
Location: Montreal

Re: EasyIO or EasyI…uh Oh?

Post by black_syphilis »

Really exciting investigation orion :)
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

black_syphilis wrote: Fri May 28, 2021 7:41 pmall FG have an unique root password
Didn't seem to be the case for the unit I had. For starters the serial console port drops you right into a shell with root access. No logon required. Would have to look at my notes, but pretty sure telnet was using one of the default creds.
User avatar
black_syphilis
Posts: 22
Joined: Tue May 11, 2021 12:35 pm
Location: Montreal

Re: EasyIO or EasyI…uh Oh?

Post by black_syphilis »

FS Bootup files? I didn't see a shell with root access.

By the way I'm checking FW and I think I need 2 hours to hack it and have a root access :twisted:
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

older FG not the FS
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

Both FG & FS the serial console is 3.3v @ 115,200 baud.

FG the pinout is clearly marked in the solder mask. Its the same on the FS.

FG-32
FG Serial Console.jpg
FG Serial Console.jpg (1.35 MiB) Viewed 122448 times
FS-32
FS Serial Console.jpg
FS Serial Console.jpg (968.04 KiB) Viewed 122448 times
These seem to drop you into a shell on the main CPU. There is a second CPU handling I/O on both, doesn't appear to be the same easy access to those.

Again these are 3.3v interfaces, so don't screw around without that in mind. For a cheap USB to 3.3v serial google TTL-232R-3V3-WE.

The FG-32 also has JTAG headers on it, so that could be another avenue to dive in.
User avatar
black_syphilis
Posts: 22
Joined: Tue May 11, 2021 12:35 pm
Location: Montreal

Re: EasyIO or EasyI…uh Oh?

Post by black_syphilis »

FG v1.5b50
ooopps!
Image
User avatar
orion242
Posts: 232
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: EasyIO or EasyI…uh Oh?

Post by orion242 »

easy.jpg
easy.jpg (8.1 KiB) Viewed 122397 times
Post Reply