Vol 2 - No 1

Troubleshooting Complex Systems

Written by Joe Sass

Created on Monday, 30 January 2012 19:48

A 769Kb PDF of this article as it appeared in the magazine—complete with images—is available by clicking HERE

Our lives are being saturated with technology. Cell phones have become mini-computers, iPODs are replacing radios, Kindles are replacing books and hotspot, http and mpeg have become common terms. Technology breeds complexity. The instruction manual for using your television remote control is larger than the remote control itself. In the world of machine control, these complexities are in the form of integrated systems. From a GNSS (Global Navigation Satellite System) perspective, a receiver must be able to communicate through serial ports, cell phones and Ethernet connections. Radios must communicate with other radios, tilt sensors provide inputs, hydraulics may be actuated; software is controlling all of the different signals and providing visual updates to the operator. It's a beautiful thing.... when it's all working. But what do we do when things go wrong? Regardless of the system being discussed, there are some basic principles of troubleshooting that can be applied across the spectrum of integrated technologies. This article will elaborate on some of these techniques as it pertains to complex systems.

It has been said that the best offense is a strong defense. Understanding the system is integral to keeping it operational. Each piece of the configuration has a role in providing the desired outcome. Understanding how each piece contributes to the final answer will help an operator understand what is happening when the system stops working. Some of the knowledge can only be gained through experience. When a system is working as it should, there is a certain rhythm to the operations that becomes familiar to the user. When the rhythm changes even slightly, the user has a sense that something is not right. Listen to that voice and do not discard the ideas that flash through. Often these initial reactions point directly to the problem but are discarded with scurrilous thoughts such as "I already checked that" or "This can't be the problem."

Another defensive posture is to be prepared. Have you had training on the system? The documentation that accompanies most integrated systems will often have a section on troubleshooting. This information should be familiar to the operator. Where are the reference manuals and Help menus for the system you're operating? Have you browsed through them to become familiar with chapter titles? It is not necessary to know the details verbatim, but it is important to understand the contribution each piece makes to the overall solution. For example, with machine control GNSS, the basic configuration is an antenna with a full view of the sky providing satellites signals to a receiver that processes the information and then feeds that data to a communication device or application software. Within these basics, there are port settings, communication parameters and other details that can be referenced when needed, so digging into the minutia is not as important as holistically understanding how the system works and being prepared with all of the available resources in advance of needing them. Many companies host some type of user forum or post FAQ's and application notes describing various conditions. It is likely that other users have experienced and solved the same problem you are currently facing.

When a system stops working as expected, the reasons may be obvious or subtle. If a piece of hardware gets crushed by a vehicle rolling over it, troubleshooting the problem is quite easy. Look for the obvious. Are the cables in tact? Are all of the component pieces still attached? Are there cracks in the hardware? Is smoke coming out of one of the pieces? More often, the problem is subtle and only a portion of the system stops working correctly. Symptoms may be intermittent and what seems like the obvious source of failure is really a secondary problem to something else that has failed.

A good first step for most troubleshooting efforts is to completely power cycle the equipment. Depending upon the difficulty in accomplishing this, a power cycle may mean turning the system OFF and then ON or it could mean completely removing all power sources including batteries for up to an hour before restarting the system. Some equipment has a button or a routine to reset itself. Many times, this simple step will fix a problem that was caused by unknown demons. Nearly all electronics possess capacitors which can hold onto an electric charge for some period of time after a regular power source has been removed. This is the reason many support documentation will suggest leaving everything disconnected for longer than just a few seconds to allow these capacitors throughout the system to discharge as this could be causing the problem.

Most electronic systems are accompanied by indicator lights? We have them all around us. When my cell phone's LED blinks green, I'm in a coverage area. When it blinks red, I've just gotten an email. These lights are for the sole purpose of aiding the user in the operation and diagnosis of the equipment. But do we understand what they're telling us in the sophisticated systems that can be found on a construction site? Some lights convey simple messages such as whether the power is ON of OFF. But other patterns can be more complex. Consider the GNSS receiver that blinks green seven times, then blinks orange once, blinks green five times and then blinks red before repeating the cycle. For the operator that has browsed through the reference manual or received some training, these lights indicate that the receiver is tracking seven GPS and five Glonass satellites. Is this good or bad? Is this the average in normal operations? What if this same indicator was slowly blinking red indicating a complete loss of satellite tracking? Would the operator notice this and understand the seriousness of its meaning?

Buttons and displays are normally associated with each other. The buttons change the display or settings and allow a user to work their way through some type of interface. The architecture of these interfaces varies a lot so understanding the basic concept and workflows are important for each component in the configuration. Just as with indicator lights, buttons and displays are for the sole purpose of aiding the user in the operation and diagnosis of the equipment. Whereas indicator lights show status, buttons and displays usually allow for configurations. Be careful! I'm sure that many of us have had the experience of changing a setting without realizing it. The system becomes completely inoperable and it takes hours to get it restored. A working knowledge of how the buttons control the display, what the display is informing and being able to negotiate the menu tree without molesting the settings along the way is well worth the learning effort. This will increase efficiency in troubleshooting a situation while decreasing the amount of time and frustration that is often accompanied with these efforts.

Is the equipment and software up to date? From an end user perspective, upgrades often appear to be a vehicle from the manufacturer to coax more money from their wallet. This may happen at times, but I think the majority of upgrades provide feature enhancements and fix bugs. The feature enhancements may or may not be important to the end user, but the bug fixes can be critical. Why spend time troubleshooting a problem that has already been fixed by newer software? Further, will access to support services be allowed without the equipment being up to date? Compare the cost of having your system disabled against the cost and effort of keeping the equipment up to date. It is my belief that the costs and efforts are well justified.

What are the system requirements and limitations? If the system worked on one day and not the next, what changed? Again, these changes can be obvious or subtle. For instance, temperature and humidity are obvious indicators of atmospheric conditions. But less obvious are tropospheric perturbations, radio frequency jamming and solar flares. Depending upon the system in question, any one of these conditions could change enough from one day to the next and noticeably affect or completely disable operations. Knowing the system limitations is a big advantage when troubleshooting. Understanding the potential weaknesses of a system and being armed with the tools to mitigate these weaknesses conserves effort, facilitates scheduling and saves money.

Having spare components available is the best method for troubleshooting hardware failures. Individual pieces can be replaced one at a time until the system returns to being functional again. Every system has vulnerable components that will completely incapacitate a system and that are more prone to failure. Ensure that there are enough of these critical spare parts available within an acceptable timeframe. It is not a matter of IF a system will fail, but WHEN a system will fail. Having spare parts on hand will provide the fastest track back to productivity.

Troubleshooting a complex system is rarely easy. However, the fundamentals are mostly the same regardless of the system being discussed. A broad understanding of how the components work together to generate a solution is critical to fixing a broken system. Being prepared with the latest resources will facilitate finding a solution. The ideas highlighted in this article are not new. They have been validated through many, many tense hours. I want to extend a personal thank you to my colleague, Art Sauer who helped crystallize these thoughts into an article. Between Art and me, we have over thirty years of experience supporting a variety of complex systems. MachineControlOnline.com is committed to parlaying this experience into information that can be used in the dirt. Come back next time when we'll explore another topic of interest.

Joe Sass has more than 14 years experience in GNSS and is Technical Support Manager for Spectra Precision/Nikon/Ashtech in charge of technical support and training throughout the Americas.

A 769Kb PDF of this article as it appeared in the magazine—complete with images—is available by clicking HERE

[ Back ]