ACC2 irrigation panel via a Chipkin/MSA/Fieldserver gateway

[Maxburn]

This is my experience with integrating to a Hunter ACC2 irrigation panel via a Chipkin/MSA/Fieldserver gateway. This won't make me any friends but this is a true account and I think it needs to be told.

All of this is in 2022, so far.

Early April;
inquiry about irrigation system integration into our front end. Presitige item? Local office found this; https://store.chipkin.com/products/prot ... r-gateways
This is the panel in question; https://www.hunterindustries.com/irriga ... llers/acc2

Based on the point list complexity, my prior experience with these products, and industry professionals experience with these products I recommended that we NOT pursue this job. The best companies are defined by what they say no to, right?

Late April
So of course we are going ahead with this. Initial quote from Chipkin and investigating local purchase of A2C-LAN so the panel can communicate on IP wired network.

April-May:
11 email exchanges and two web meetings with support on how to decipher the points list. Pay particular attention to these items and think about how you would do the programming in your panel to deal with this.
https://cdn.chipkin.com/assets/uploads/ ... 50-16.xlsx
'0x1E – SET FIELD CONTROLLER PROGRAM HEADER DATA (v2.13.000 and Later)
'0x3E – REPORT FIELD CONTROLLER PROGRAM HEADER DATA (v2.13.000 and Later)
Later on in the meetings someone named Peter Chipkin entered the chat. Web search on him turns this up; https://www.linkedin.com/in/peter-chipk ... bdomain=ca
Top level help on this project.

May-July:
Several more meetings on implementation (documentation is not complete) and it was revealed that the gateway they are supplying is capable of running a web app which supplies a web page that duplicates all the controls the local hunter panel can do. Chipkin was offering this as a programing example of what I can do to translate the data received in our BMS. Reverse engineer a web app and figure out how to put that into ladder logic? Nope. BUT I can put a iframe in our system and present that to the customer inside their network. Then I only need to integrate alarms/status info for display in the BMS system so I decided to go ahead with that.

Customer site delays; need LAN drops and IP's

[Maxburn]

September 14
First site visit; Hunter panel sees LAN option as WiFi, and fails firmware update. Supplied SD card issues? Tried several different formats (not mentioned in hunter docs). Reached out to local Hunter support.

September 19
Met hunter rep late in afternoon, they did firmware update with their SD card no problem, now sees LAN adapter.

September 22
First attempt to get chipkin gateway connected to hunter panel. Local network on my switch connect hunter panel and chipkin gateway. Receiving data, many errors. Bulk of the results in BACnet and chipkin data previews is 0. Calls in to chipkin for troubleshooting didn't really get anywhere and they want to get the hunter panel attached to the hunter cloud management Centralus so they can see what's going on and compare the two systems side by side.

September 28
Return to site with wired LTE modem so we can have internet and connect to Centralis. Will not connect to centralus, kick off of involving hunter support to troubleshoot that. Solid ~20 emails and Hunter concludes my LTE modem is blocking comm, even though I can demonstrate my laptop can get there.
Meanwhile Chipkin tries several different firmwares and config CSV files. After doing a data capture on board their controller they conclude that they need a couple more days to develop a driver.

[Maxburn]

October 6
(in office) Set up web server in digital ocean on the two TCP ports hunter says they need to talk to for AWS. Demonstrate that my laptop behind the LTE modem can get those pages so not only can I resolve DNS for the specific IoT AWS thing they need to reach but my modem is not blocking that sort of traffic. Hunter support now willing to escalate to their cloud guy because I'm talking about some technical things and proving it's not the modem. A bunch more emails and several calls and we have a new theory; is chipkin protonode blocking the AWS connection somehow?

[Maxburn]

October 13
Back at site with LTE modem and my own network. Set everything up and this time leave the chipkin gateway off my local network. Hunter panel connects to Centralus no problem at all, explore around and everything looks good. It offered to upload the local panel which worked. Connect chipkin gateway to network, wait a few minutes, hit refresh in Centralus; unable to connect. Further call with Hunter AWS guy and he concludes that yes the chipkin gateway is capturing that traffic somehow and acting as a local "cloud" centralus. It is not possible to have both the Centralus and the Chipkin gateway working at the same time. Hunter support mentioned some of these gateways working fine in Europe, they don't understand the problem.

Meeting in the afternoon with chipkin. They load a new version of the web app. Clicking around all the data looks empty, zeros, and conditions don't reflect what is actually going on. There's javascript messages popping up with contents AAAAA. Chipkin tech "OK, you are all set." Me; "Uh, no I don't think so, it really looks like the back end is not gathering data from the panel." Show him in teamviewer on my phone the current irrigation panel alarms, water percentage, enumerated program names and station times, etc. Try and trace data down in chipkin diagnostic data registers and looks like some data might be there but nothing coming out bacnet, all zeros. Still not showing on web app either. Focus in on the "master off" alarm, not one alarm showing for the panel in BACnet but does show at local display and in Centralus. Focus in on one piece of data for Time of last data, is not seconds past midnight as documented. It is Unix time code so documentation is wrong, but I can deal with that. Chipkin support took some screen shots comparing what their web app is doing vs what centralus is doing. The two web pages look VERY similar. Chipkin is going to think about this for a bit.

[Maxburn]

November 1 (Tuesday)
Please be patient until next week. We hope to be done by Monday. I will test it and let you know.

November 14 (two Mondays after, but they didn't specify which Monday)
New firmware and web app files sent.

November 29
Back at site, uploaded firmware and web app files. Web app and BACnet appear to have the same information and symptoms as last time. Several alarms active in the panel, all information out BACnet is zeros for all alarm registers. Web app will not show anything but busy / refreshing data when viewing alarms no matter how long you leave the page up. Still throwing popups with contents "AAAAA". Web app showing irrigation 300% when panel is 100%. Performed diagnostic dump on Chipkin gateway, sent that to support. During troubleshooting chipkin tech confirmed what model of hardware I have here (the big armv7 one) and showed me where the reset to defaults button is on it, had me hold that for 10 seconds. Nothing happened after four attempts at resetting with hardware button, it's like it ignores it. Chipkin is going to have their developers work on this some more.

December 5
Talked it over with sales, I don't think we can pull this one off. No other integration options I'm aware of for this irrigation panel. Recommend we give the centralus front end credentials to customer and not pursue the integration option.


To be continued?

[orion242]

So what is chipkin saying on this?

I wouldn't use any fieldserver product anymore even if it was the last thing on the planet. Their junk has wasted countless hours, its blacklisted in my book.

[Maxburn]

Following up on this, I think I spent two days out there in 2023 so far but we put off testing until I could see irrigation running. The web page they are sending is steadily improving and I seem to have valid data out of some arrays like alarms and zones working on the BACnet side of things.

• Irrigation zone active report is supposed to be triggered according to the points list but it looks like something is triggering it for me, maybe the web app? Question for later...
• Report time is documented as seconds past midnight but what they actually send is a Unix epoch time code; anyone know how to convert that to something I can use in logic for ALC??

Bonus; went out yesterday to install the latest web app. Set up early and their appointment starts at 11AM. They joined and we worked through loading the new web app. After noticing no real changes from last time we realized I missed a new download link in a email from a couple months ago. The download link is now expired and they don't have the file on hand. They reached out to the "web developer" for the file and that person can't find it either. As I was walking out the door at 3:30 PM they sent me a download link but I'm not sure what it contains yet.

Bonus2; while I was futzing around in the BACnet side of this making programs (trying to not waste time on site) I took a look at the web app. I realize it is loaded in two different folders. Apparently there is a webroot/[apphere] folder and a webroot/files/[apphere] folder. Instructions mention uploading to webroot/files/ folder but they don't mention what I'm uploading like the app folder they packed it in or index.htm and all the other files and folders at that level. Result? Both of these URL's show schedule page
http://10.20.193.51/ChipkinFS/get/files/schedule.html
http://10.20.193.51/ChipkinFS/get/schedule.html

So now I'm wondering how much time we were wasting debugging the wrong web app, as I specifically noticed and commented on their tech uploading the webapp to webroot/[apphere] instead of the files sub folder as documented. What I do know is I wasn't able to get in with SFTP, webdav or a couple other things. Didn't think of plain SSH while I was there. Their tech says he can fix that but he was busy all day looking for files.

[Maxburn]

Reply on cleaning up the file mess, must format the device. Which means you have to reapply the firmware(?) and config, and of course put the IP address back in. Actually not a bad day if I'm on site but a big nope to fix remotely. Did I mention this site is a 2 hour drive?

[orion242]

At this point, seems it would have been easier to just replace the irrigation system with BMS controllers.

This just reinforces my black list of any fieldserver product.

Is the customer paying you T&M to follow this rabbit hole to the other side of the planet?

[Maxburn]

Is the customer paying you T&M to follow this rabbit hole to the other side of the planet?

No.

But I have gained a lot of traction towards black listing these MSA, Fieldserver, protonode, Safety Company, Chipkin devices. We are on notice now and any failures of these devices to provide valid data we are not going to help them out for free any more. The worst part is they are in a LOT of equipment.

Most recent problem is a generator fuel cleaner. Device fieldserver is supposed to supply BACnet but doesn't, again looks like modbus on the other side but this time the register list we scan from the PLC in it doesn't show a bit change when the E-Stop is pushed which is a pretty basic alarm. So of course deeper problems we can fortunately dump on the vendor, and by the say fix the fieldserver while you are at it.

[orion242]

Feel your pain. It would be an extreme case to deal with them on any project at this point. I'll have a mile of cable pulled and sort out insane modbus nonsense before I will use their stuff. Even after bid day, its elimination at all cost.

Facking miserable things have wasted countless hours of my time.

[Maxburn]

At this point, seems it would have been easier to just replace the irrigation system with BMS controllers.

We would have to trench the site to do it. Each zone valve is digitally addressed. The only wiring is a combination power & comm cable, 2 wire.

[orion242]

We would have to trench the site to do it. Each zone valve is digitally addressed. The only wiring is a combination power & comm cable, 2 wire.

Oh FFS. Doesn't get any more miserable than that.