PSA; ALC optiflex under AAR; very long download times.

All things ALC
Post Reply
User avatar
Maxburn
Posts: 84
Joined: Wed Mar 04, 2020 12:51 am

PSA; ALC optiflex under AAR; very long download times.

Post by Maxburn »

First of all this doesn't currently work unless you change system.properties file setting bacnet.segmentation.on = false. Doesn't seem to be recommended to leave it there either but it will allow the driver download to complete.

My conclusion is if you have one of these under an AAR and need to do significant driver updates do it locally via USB or network. Even Rnet is going to take too long if you have that available. Time to complete depends heavily on how much is different in the driver from the version it has to the one you want to go to. Could be as short as fifteen minutes or as long as three or four hours.

Remember you can update the driver near instantly over direct ethernet or USB. Ethernet will require reconfiguration of the modules router settings page to update the home network and network numbers, you don't have to have the exact right job file, just get the driver in it and fix the router setting page home network so you can reach it through the customer server again.


-----------------------------------------------------------------------------------------
---There are no programs in use, no other MAC address taking time on the token, and straight ethernet connection; these will represent best case results. Field times will be longer than these.
-----------------------------------------------------------------------------------------
---Testing between the two latest released drivers;
Results G5RE on Ethernet
unknown to drv_fwex_103-06-2055; 11:45
drv_fwex_103-06-2055 -> drv_fwex_103-06-2045: 2:10
drv_fwex_103-06-2045 -> drv_fwex_103-06-2055: 1:30
drv_fwex_103-06-2055 -> drv_fwex_103-06-2045: 1:20

Results G5RE on ARCnet under LGR
drv_fwex_103-06-2055 -> drv_fwex_103-06-2045: 5:26
drv_fwex_103-06-2045 -> drv_fwex_103-06-2055: 5:25
drv_fwex_103-06-2055 -> drv_fwex_103-06-2045: 5:26

Results G5RE on ARCnet under AAR and LGR
All downloads failed with default system.properties settings
results with; bacnet.segmentation.on = false
drv_fwex_103-06-2055 -> drv_fwex_103-06-2045: 16:40
drv_fwex_103-06-2045 -> drv_fwex_103-06-2055: 16:27

Results using Rnet cable
drv_fwex_103-06-2045 -> drv_fwex_103-06-2055: 17:00


-----------------------------------------------------------------------------------------
---Second round add testing results for older drv_fwex_101-00-2056 - drv_fwex_103-06-2055
Results G5RE on Ethernet
drv_fwex_103-06-2045 -> drv_fwex_101-00-2056: 6:18
drv_fwex_101-00-2056 -> drv_fwex_103-06-2055: 6:26

Results G5RE on ARCnet under LGR
drv_fwex_103-06-2055 -> drv_fwex_101-00-2056: 54:12 (almost one hour under main ARCnet)
drv_fwex_101-00-2056 -> drv_fwex_103-06-2055: 56.15

Results G5RE on ARCnet under AAR and LGR
drv_fwex_103-06-2055 -> drv_fwex_101-00-2056: 3:02:25 (three hours two minutes)
drv_fwex_101-00-2056 -> drv_fwex_103-06-2055: Cancelled at 4:32:20, needed to go home. Was at 71% done.

Results using Rnet cable
Unable to test, drv_fwex_101-00-2056 not allowed for OFBBC
drv_fwex_104-00-2130 -> drv_fwex_103-06-2055: 2:14:00 (two hours, fourteen minutes)
-----------------------------------------------------------------------------------------
User avatar
orion242
Posts: 208
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: PSA; ALC optiflex under AAR; very long download times.

Post by orion242 »

Willing to bet they have some sort of issue lying in there somewhere. When transferring files takes more than 3x what they should, I start asking questions depending on how often I get stuck watching the progress bar.

Keep in mind the VPN performance on the ERX isn't anything to write home about. Been a while since I looked at it, but off mem 1-5Mbps is about all she's got and that depends on what else its doing. Not an issue for typical BMS use, but start transferring loads of data its going to get annoying. Was one of the main reasons I put a few more bucks down when I replaced my home router for an ER8.

Hmm maybe I was seeing 10Mbs

https://community.ui.com/questions/Test ... e428661d1a

I think it really variable on the crypto in use, what else you have setup, etc. I had two ERXs setup with the VPN between them on WAN ports and then NAT PCs on LAN side and it wasn't fast enough to max out my internet connection. Didn't think it was even close, so crossed that off and went bigger but that was some time ago.
User avatar
Maxburn
Posts: 84
Joined: Wed Mar 04, 2020 12:51 am

Re: PSA; ALC optiflex under AAR; very long download times.

Post by Maxburn »

First off on this thread all those test times are with a laptop straight to the module, no VPN.

VPN results
-----------------------------------------------------------------------------------------
Server in Azure, OpenVPN connection to site and G5RE on ethernet.
drv_fwex_101-00-2056 -> drv_fwex_103-06-2055: 1:51:00 (almost 2 hours)
-----------------------------------------------------------------------------------------
The OpenVPN config I have is rather horrible bandwidth wise. iperf worst results for the packet size BACnet uses looks something like 1.05mbps. But still with those numbers a 35 MB file at that speed should only be 4 Minutes and 53 Seconds. Not almost two hours.

It must be EXTREMELY latency sensitive if that's the answer as latency for this is perfectly fine 48ms in worst case which seems to be good and trending down to 23ms great for smaller traffic. Still significant compared to 1ms locally though.

I've got half a mind to set up a UDP tunnel for this situation to see if that makes a difference with speed in this case. I chose TCP tunnel for reliability reasons on trends/alarms not always getting to the server.
User avatar
orion242
Posts: 208
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: PSA; ALC optiflex under AAR; very long download times.

Post by orion242 »

Maxburn wrote: Wed Apr 15, 2020 3:21 pmFirst off on this thread all those test times are with a laptop straight to the module, no VPN.
IMO you know its an issue then with that setup.

Another thing to think about, just because there is an Ethernet port doesn't mean the host processor can handle anything close to what the limits are on that interface. Almost all controllers I look at the actual hardware level are using some interface IC for IP connectivity. So it might be 10/100M support but there is no way in hell your getting anything close to that because the host processor simply cannot swallow this. Would love to test this on controllers but its nearly impossible to test in some common fashion if even at all. TLDR - if the host processor doesn't support this natively, there is a snowballs chance can deal with theoretical limits.

Still don't really have my head around your setup and my ALC knowledge is near zero. Something still stinks if its 3x slower than the weakest link should be able to handle.
User avatar
Maxburn
Posts: 84
Joined: Wed Mar 04, 2020 12:51 am

Re: PSA; ALC optiflex under AAR; very long download times.

Post by Maxburn »

orion242 wrote: Mon Apr 20, 2020 12:38 am
Maxburn wrote: Wed Apr 15, 2020 3:21 pmFirst off on this thread all those test times are with a laptop straight to the module, no VPN.
IMO you know its an issue then with that setup.
You mean the VPN tunnel is creating the problem? I'm putting together an article with my thoughts on this but I don't see how that's the case.

That's an excellent point on the processor data handling speed. All these new modules support a gigabit link speed and nothing I've seen says they can take data at that rate. I look at it as more of a compatibility situation. We have plenty of very old 10 baseT half duplex controllers which is a little more realistic for what they can handle but it's becoming a pain as newer switches being installed can't link to that low speed. I was looking at one recently and every single port on their new switches could handle 10GB but not 10M/half. So the new modules are compatible with new switches but no, they can't process data at that rate.

What I'm thinking is that ALC painting themselves into a corner to create this situation, but I'm not sure if this is a bad thing. In the tridium world the new edge controllers are using ethernet and fox connections. These edge controllers are very roughly equivalent to the ALC optiflex line of controllers, BUT they only communicate to the supervisor or workbench via ethernet. IF the edge line was able to be stuck under MS/TP and talk to super/WB I think we would see the same problem, it would take forever to send modules and OS upgrade to it. So ALC has compatibility with enabling some situations the edge can't do but it is going to create some support troubles. Not really a bad thing but worthy of a PSA to make people aware of what they are getting in to the next time they hit update and that OR AHU is down for hours.
User avatar
orion242
Posts: 208
Joined: Fri Feb 21, 2020 12:55 am
Location: New England
Contact:

Re: PSA; ALC optiflex under AAR; very long download times.

Post by orion242 »

Maxburn wrote: Mon Apr 20, 2020 1:50 pmYou mean the VPN tunnel is creating the problem? I'm putting together an article with my thoughts on this but I don't see how that's the case.
If your getting slow results in your testing directly connected, can't be the VPN, IP network, etc. Can only assume its some oddity either in the way its passing through routers or maybe something with the ATF not properly dealing with latency. Latency might cause the issue, though its just exposing a problem elsewhere. Maybe the ATF has some really tight timing on things and that's causing a ton of re-transmits on the wire if data is getting to it fast enough for example. Wouldn't think designing it such you need <10ms latency for reasonable transfer times is a great idea. The token passing shouldn't have a huge impact either. As the network get busy, token passing, PFM, etc all start to become insignificant portion of the traffic. Would expect this to be the case when doing file transfer over token networks. Sending and receiving devices will send more packets each token pass, which slows down how often it passes and quickly becomes a minor portion of the traffic. Same thing happens with PFM traffic. Only PFM every 50 tokens, and if devices are sending more packets per token...it also gets out of the way and becomes less of the overall packets.

What's the default max info frames on these guys? Have to tried playing with that? Doubt its going to cause the slowdowns your seeing even if its 1.
Maxburn wrote: Mon Apr 20, 2020 1:50 pmI look at it as more of a compatibility situation.
Exactly. Its one thing that drives me a bit batty when the reps start talking about how fast things are going to be on their IP line, yet almost none I suspect could even use 10% of the available bandwidth.
Maxburn wrote: Mon Apr 20, 2020 1:50 pm PSA to make people aware of what they are getting in to the next time they hit update and that OR AHU is down for hours.
Yea that's going to be a real treat.
User avatar
Maxburn
Posts: 84
Joined: Wed Mar 04, 2020 12:51 am

Re: PSA; ALC optiflex under AAR; very long download times.

Post by Maxburn »

Yeah, seems to be hyper sensitive to latency and/or BACnet atomic file might be a bad file transfer method. I've started discussions internally at ALC and others are experiencing it. I have not played with max master polling or any timing or sizing, ALC sets some defaults in these areas and with all ALC equipment if something needs to change they really should be the one to do it, they are out on their own with the special case ARCnet anyway.

I spent plenty of time looking at this and the VPN tunnel and did a little writeup on it. Mainly I was concerned the VPN was causing problems but I can't find anything wrong. https://blog.jalbert.me/why-did-i-do-that/
Post Reply