- About Us
Folks, if you are attending Field Service Medical 2014, let’s get together and help you carve out your M2M Strategy. Simply put — don’t create a “needle in a haystack” problem by spending all those big bucks collecting tons of data in the hope that it may be useful to somebody somewhere? Who is going to ever analyze it? Don’t collect a haystack – collect only the needles, and deliver smarter service! Check out our M2M demo for more information!
Folks, I want to share with you this interesting article titled “10 Field Service Predictions for 2014” by Joyce Tam of Trimble’s Field Service Management. All I can say is — its about time!
Talk to us, we can help you with most of these things.
When you are putting together your field service strategy, ask yourself this — does your expert technician use luck or logic to solve the most challenging troubleshooting problems? If it is logic, shouldn’t a computer be able to do it better?
When you are building your remote monitoring and M2M strategy, ask yourself this — are you creating a “needle in a haystack” problem by spending all those big bucks collecting tons of data in the hope that it may be useful to somebody somewhere? Who is going to ever analyze it? Don’t collect a haystack – collect only the needles, and deliver smarter service! Check out our M2M demo for more information!
I am continually amazed at how modeling is made into a task much more complex than it needs to be! Modelers often get enamored by the power of the TEAMS analysis tool, or, obsessed with the intricacies of their equipment, and get lost in the details. You do not need to be Einstein to model, just follow his words — “Make things as simple as possible, but not simpler.”
In this blog, I want to walk you through a simple modeling approach, where you match the depth of the model to the quality and quantity of information available, and the results you want to get out of it.
This model was being developed for the field service organization entrusted with the support and maintenance of a sophisticated Bio-Medical instrument. So, the purpose of this model is to capture enough information to enable the reasoner to effectively utilize all available observables (error codes, status LEDs) to narrow down the possible list of suspects, and then add just enough manual troubleshooting steps to determine the root cause so that service agents can fix the problem right the first time.
We start by identifying the error codes from the service manuals of the system. Why start with these? Because they are continuously monitoring and can automatically detect any malfunctions. Many instruments are capable of uploading these error codes back to field service center, or, often the operator will call in with the error codes displayed on the instrument; so these are usually the first available information from the instruments.
For the reasoner to be able to interpret the error code data, we need to provide it with the list of possible causes (or faults) that would trigger each of these error codes. Many OEMs document these relationships rather well in their service manuals; while others may need to dig deeper into engineering knowledge to identify them. Once the list is available, it is a simple task in TEAMS to lay in the FRUs and their interconnections, identify the Error Codes, and capture their relationships.
Most error codes are broad functional tests, and therefore, each error code can be triggered by multiple faults in multiple FRUs. Traditionally, field service organizations have disregarded such Error codes as they do not provide a specific root cause. However, the reasoner in TEAMS can look across all Error Codes, and use the presence and absence of Error Codes to rule in or rule out possible faults, to significantly reduce the number of possible causes. In fact, we can run an analysis in TEAMS to quantify how often the reasoner will be able to identify the root cause using just the Error codes! For those situations, no further manual troubleshooting will be necessary – the reasoner will be able to pin-point the root cause — and therefore no further manual test procedures need to be modeled.
However, more than likely, there will be a significant percentage of cases where additional information is required to identify the root cause. So now we look for additional observables that are also low effort and readily available. An example would be the status LEDs which may provide additional information not provided by the Error Codes. These status LEDs can also be modeled similar to the Error Codes, and the model updated with these new sources of observation.
Next, we run the TEAMS analysis again, and identify the remaining faults that cannot be isolated uniquely. TEAMS identifies these groups of faults, called the ambiguity groups, where each member of the group cause the same exact combination of Error codes and status LED states, and are therefore indistinguishable from each other. A field service agent would need to troubleshoot further to identify the root cause from among this ambiguity group. The modeling effort, from this point on, only needs to identify the manual procedures that will help differentiate between the faults of an ambiguity group — if a group has N members, one would need to model anywhere from (N – 1) to ln (N) troubleshooting steps to achieve full fault isolation. This focused approach to capturing manual procedures that are only necessary for troubleshooting after the reasoner has utilized all available information to limit the possible set of suspects to minimum, not only reduces the effort in modeling, but also helps the subject matter experts helping in the modeling to only what is important to improve the serviceability of the instrument.
Once the desired level of fault isolation capability is achieved (remember, sometimes it is cheaper to replace a couple of FRUs than to research which might be the real culprit), we can augment the model with better instructions and illustrations on some of the more complex troubleshooting steps. We can link reference material, images, repair instructions, etc. to the manual procedures to help less experienced service agents perform complex tasks. We can identify the skill or certification required for some of the steps, and the reasoner can determine to what degree a problem can be resolved based on the service agent’s skills, and when to transfer the problem to an agent with higher level of certifications. The important thing is to do all of those only for the procedures that are actually necessary for troubleshooting, and not get bogged down in the tedious and long troubleshooting sequences service agents have to undertake because they lack the ability to extract most information from Error codes and status LEDs.
The best model for the job is the simplest model that gets the job done!
If you have any questions or need further clarification, please send me your comments and I will be happy to respond.
We have been building TEAMS-Designer and TEAMS-RDS in both 32bit and 64bit versions. However, for most users, we recommend the 32bit version. This is based on the following considerations:
See the following table for our performance test results for different models of different sizes.
|Model Size||Testability Analysis Memory Usage||GUI/Reachability Memory Usage||Testability Analysis Runtime|
|10,000||293MB||288MB||1 min, 44s|
|14,000||280MB||301MB||4 min, 4s|
|20,000||1.88GB||311MB||8 min, 11s|
|39,000||1.79GB||1.29GB||19 min, 1s|
QSI has been featured as a NASA Spinoff by the Office of Chief Technologist, NASA, for its TEAMS Toolsets to Maintain Health of Complex Systems.
A NASA spinoff is a technology, originally developed to meet NASA mission needs, that has been transferred to the public and now provides benefits for the Nation and world as a commercial product or service. NASA spinoffs enhance many aspects of daily life, including health and medicine, transportation, public safety, consumer goods, energy and environment, information technology, and industrial productivity. These spinoffs are transferred to the public through various NASA partnerships including licensing, funding agreements, assistance from NASA experts, the use of NASA facilities, and other collaborations between the Agency, private industry, other government agencies, and academia.
QSI was selected for application of its toolset in Detection and Isolation of Problems on the International Space Station and Ares I-X programs. Click on the following for the listing and the full article from the NASA Spinoff website.
A key benefit of the TEAMS solution is that it helps Field Service Organizations improve the Quality of Service (QoS) while lowering the Cost of Service (CoS).
Without TEAMS, organizations rely on the intelligence of the individual Field Service Engineers (FSEs) to solve complex troubleshooting problems. However, not all the FSEs are equally smart, leading to inconsistent performance that no amount of training can overcome.
The consistent QoS delivered by the TEAMS solution is therefore a key motivation for adopting the TEAMS solution. But, how do you measure and demonstrate a consistent performance when there are so many variables that affect the field service process? More to the point, while variability is to be expected, how much variability is cause for concern?
Fortunately, quality measurement is a mature science, and we need to look no further than “Control Charts for attributes“, originally conceptualized in 1924 by Walter Shewart and extended by Edward Demming.
The underlying theory is very simple.
Supposing you want a consistent success rate of S%, where success implies the Guided Troubleshooting Solution enabled the FSE to identify the root-cause of the failure and apply the appropriate corrective action. How would you periodically verify that you are still achieving S% success rate?
If periodically you sampled N cases, the standard deviation, s, associated with the estimate of S from N measurements is inversely proportional to the sqrt(N). This makes intuitive sense; the more data you have the more precise your estimates of S will be.
Assuming the cases you sampled are independent of each other, the expected number of successes from N trials will be between (S – 3s) and (S + 3s) with 99% confidence. This is basic property of normal distribution, and we will assume that it is applicable to S.
So, what does this all mean? Let’s plugin some values to make sense.
Let’s assume your goal is to have a success rate S = 80%. So, when you have a large enough data set, say thousands of troubleshooting test cases, you will get approximately 80% successful troubleshooting outcomes.
But how many successes should you expect if you just sampled 30, 100, 300 or 1000 test cases? The following table gives you the answer:
|Number of Cases||Minimum Number of Successes||Expected Number of Successes||Maximum Number of Successes|
You could monitor the accuracy of your Guided Troubleshooting Solution by periodically sampling N test cases and counting how many of those are successful. How to interpret these results?
Hope this helps you in measuring success in your Field Service Organization. Let me know how it works out.
I am often asked by a prospective customer this simple question – do you do Prognosis?
If the customer is in DoD/Aerospace world, or has an established R&D and Health Management program, the short answer is yes!
However, customers asking this question often are field service organizations that maintain expensive assets and are looking for an alternative to the traditional break-fix service model. The term Prognosis has become quite popular over the years, thanks to millions of dollars invested by the US Department of Defense in programs such as the Joint Strike Fighter. The promise of Prognostics is simple — wouldn’t it be nice if you could predict how much life each component has left, so that you could replace them just before they failed? But is Prognosis the right tool for you?
First, how good is your Diagnosis? If you are struggling with faults that have already happened, chances are you won’t do any better with faults that have not happened yet! Think of Prognosis as something that involves Diagnosing impending failures and predicting when they will develop into full blown faults. So, Diagnosis is the foundation for Prognosis, and the need for predictive capability makes Prognosis a powerful but expensive technique that should be used wisely only where it is necessary.
This brings us the second question: do you really need Prognosis? For example, your car has two headlights. While it is no fun driving in the rain with one headlight (have you noticed how headlights always seem to fail on rainy days?), having two headlights means that you can still get back home when one has failed. So, redundancy and fault-management are effective ways of reducing unscheduled downtime. For your critical components, evaluate what is the most cost-effective method to avoid disruption in service, and choose wisely.
And now, the all important third question: why do you want Prognosis?
Some people will answer I want to use less maintenance. For example, you may be used to changing the oil in your car every 3 months (schedule-based maintenance) or 3000 miles (usage-based maintenance), but by monitoring the condition of the oil and the filter, you could change the oil only when needed (condition-based maintenance or CBM). This is a valid application of condition monitoring, although strictly speaking, this is not Prognosis. Also, keep in mind that you may not be able to extend maintenance interval for safety critical components without exposing yourself to more liability.
However, most of our customers answer they need prognosis to reduce unscheduled downtime by doing preemptive repairs. Here too, Prognosis is not the only answer.
Let’s take an example — supposing you want to avoid being stranded on the highway due to tire failures. You could add sophisticated techniques that monitor the tread of the tires and how it is wearing out, how the underlying structure of the tire is holding up, the stress on the tire, etc, and predict when failure is imminent. You could develop such Prognosis at significant R&D expense, or, you could simply replace the tires when they look worn (CBM) (e.g., cracks on sidewall and/or tread-depth of 1/32nd inch or less). The second method may cause you to use at most one extra set of tires over the life of the car since you will throwaway tires with still some useful life left on it, but newer tires also improves your safety and performances, which has its own reward. Best of all, you can use the second technique on your current installed base without having to develop new technology.
Let’s also not forget, Prognosis or CBM does not completely prevent unscheduled downtime. You could still hit a pothole and get a flat tire – no matter how new your tire is!
To sum up, there is more than one way to reduce unscheduled downtime: Prognosis, Condition Based Maintenance, redundancy and fault tolerance and fault management. QSI can help you implement all of the techniques discussed here in a balanced health management solution. Health Management is not just Diagnosis or Prognosis, but an effectively engineered delivery of uptime at a reasonable cost.
Let us help you find the right balance of techniques to achieve your objectives.
Are you finding it cumbersome to create a test that only detects a single failure mode or failure modes on the same hierarchy level? Try using a Direct Test. A Direct Test has all the functionality of a regular test without the complexity. The complexity is caused by having to create a new unique function and attach it to the tests and applicable failure modes. With Direct Tests all you need to do is specify which failure modes are detected by the test, eliminating the need to create and attach a new function to the failure modes and test. Try it!
NASA formed the Constellation Program specifically to establish a base there, and to lay the foundation for eventual explorations of Mars and the solar system before the end of the first half of this century. Much has already been learned from NASA’s previous spaceflight programs, beginning with Mercury and Gemini, and continuing through current projects with the Space Shuttle and the International Space Station. The Constellation Program, however, focuses on a new generation of spacecraft for human spaceflight, consisting primarily of the Ares I and Ares V launch vehicles, the Orion crew capsule, the Earth Departure Stage, and the Altair lunar lander. These spacecraft will be capable of performing a variety of sophisticated missions ranging from Space Station resupply to lunar landings.
Qualtech Systems, Inc., (QSI), a technology company based in East Hartford, Connecticut, has been intimately involved with the NASA Space Program since the mid-1990s. QSI’s team has worked closely with engineers and scientists at NASA’s Ames Research Center on designs for the program’s evolving launch vehicles, ranging from reusable launch vehicles (RLVs) in the mid-90s to the current Ares launch vehicle, as well as the International Space Station. QSI develops software that captures the knowledge of how a system fails and how the failures are detected, then uses that knowledge to guide engineers to make troubleshooting and real-time diagnosis more efficient, capabilities essential to the design of any crew launch vehicle. QSI’s software suite includes three tools used by NASA specifically for the Ares launch vehicles and Orion crew modules. These tools support systems engineering; systems design and testability; automated diagnostics and troubleshooting; and system autonomy. The first, TEAMS-Designer® (Testability Engineering and Maintenance System), is a tool used in design/analysis phases of complex systems. TEAMS-RT© is a real-time diagnostic engine that provides diagnostic functionality for integrated vehicle health systems on board a flight vehicle or embedded into a run-time architecture. Finally, TEAMS-RDS™ (Remote Diagnosis Server) is an application that can support multiple simultaneous diagnostic sessions from a variety of remote systems. It was for this software suite, developed during a seven-year collaboration with ARC researchers, that QSI was selected for NASA’s prestigious Space Act Award in 2002.
This exceptional level of technical expertise, combined with the company’s responsiveness and adaptability to NASA’s needs, are among the many reasons why Dr. Ann Patterson-Hine, Tech Area Lead for Discovery and System Health, and the technical contact for the projects QSI has done with NASA since ’93-’94, enjoys working with the QSI team. She notes that the NASA and QSI teams have developed a genuine collaboration that often alters the outcome of a project and that Qualtech’s technology sometimes materially changes the way NASA proceeds with its designs. NASA’s approach isn’t simply to purchase a software product and apply it to their designs: they like the ability to test, analyze and modify as they go along. NASA quickly realized that they could get analysis results out of QSI’s software, (modifying, for example, the placement of sensors based on TEAMS® analysis) and thereby use the tools to improve their designs.
QSI’s willingness to adapt to NASA’s requirements has resulted in some exciting developments in critical systems engineering efforts in the design phases of the Ares launch vehicle project. Working with Crew Launch Vehicle Fault Detection, Diagnostics and Response (FDDR) team members at Ames Research Center and Marshall Space Flight Center, QSI began to enhance their original diagnostic software, actually building NASA’s requirements into their toolset. According to Dr. Patterson-Hine, “The responsiveness of the QSI team enabled us to define new features that we needed in the TEAMS-Designer® tool to support development of launch abort algorithms and QSI rapidly implemented the new functionality to support our project schedule.”
As a result, the systems engineering analysis capability has been significantly improved. Initially, FDDR engineers were building functional fault analysis models of vehicle subsystems that would ultimately support the design of the abort logic for the new launch vehicle. As details were added and reviews were held, the subsystems engineers began asking for QSI’s tools so that they could utilize the various analysis features of TEAMS-Designer® themselves in the design of the subsystems. As timing analysis and effects analysis were added, a TEAMS-NASA version emerged. Due to the popularity and widespread benefits of these capabilities, QSI merged TEAMS-NASA with their commercial product and have recently released TEAMS-Designer® 11.0. According to Dr. Somnath Deb, President and Chief Technical Officer for QSI, “QSI draws on its customers’ needs to select the feature sets for its tools. It is a win-win situation: the customer gets features that are important, and QSI gets to refine the new concept with a real-life application. This new functionality forms the FMECA-module of TEAMS-Designer® 11.0, and other Aerospace customers, such as Gulfstream, are already benefiting from these.”
Now, multiple centers are using the QSI technology. Ames Research Center, Marshall Space Flight Center, and Johnson Space Center all have TEAMS-Designer® site licenses and several other centers are currently purchasing single-user licenses to support their Constellation tasks. The innovative modeling and analysis capabilities provided by the QSI tool suite enable design information to be utilized during later system lifecycle phases, such as in ground processing at the launch site and on board the vehicle itself. QSI’s Remote Diagnostic Server is slated for use in the Engineering Technology Development Program (ETDP) ground systems automation project which is a joint effort between Ames Research Center and other key NASA centers. In addition, Honeywell, the Orion avionics subcontractor, has selected TEAMS-RT® for on-board Vehicle Health Determination.
Another aspect that Dr. Patterson-Hine likes about working with QSI is that their methodology fits right in with NASA’s goals. While they are using this technology specifically for Project Constellation, NASA is also developing a process that future project managers can use to ensure that there is greater consistency throughout the project lifecycle. According to Dr. Patterson-Hine, Qualtech’s methodologies are part of this “blueprint”. “NASA continually works to improve its systems engineering practices and the QSI approach, starting with design assessments and utilizing the same basic model throughout the development and implementation of run-time diagnostic systems and into operations and maintenance phases, enables system health management practices to be integrated seamlessly with conventional systems engineering processes.”
According to Dr. Deb, many organizations, not just NASA, are embracing a systems-oriented approach to the design process. He notes that, “Tools that can reduce unnecessary tests provide cost effective testing procedures, and which are built to work seamlessly in an integrated design environment can yield substantial cost savings over the system’s lifetime. Markets for such design tools include large prime contractors and government organizations that must integrate numerous subsystems and study testability at the system level.” He adds, “The TEAMS® tools can be used as training aids for field maintenance engineers and/or diagnosticians. Most significantly, Qualtech’s “Remote Diagnostic Server (RDS)” or TEAMS-RDS® provides these capabilities from a central server computer to thin clients (PC with a web browser) over the Internet, anywhere in the world. The e-commerce potential of this solution has very large (and rapidly expanding) commercial potential. The Qualtech Integrated Toolset is unique in that it provides a comprehensive solution for cradle-to-grave supportability of complex systems whether they are new development or legacy platforms.”
Qualtech was just recently selected a second time for NASA’s prestigious Space Act Award, which highlights “significant and technical contributions to aeronautical, commercialization, technology transfer and space goals” and is given for mature technologies that are proven in NASA applications. The 2008 award was announced in early May, citing QSI’s technologies used by Project Constellation, specifically TEAMS®RT and its relation to real-time diagnosis of dynamic systems.
The beauty of the TEAMS® software is that while it can be used in the complex applications of NASA’s research projects, it functions in essentially the same way for the simpler applications required by other industries and companies, especially those who, like QSI, work closely with NASA. The idea is to help companies capture information about how their equipment might fail so they can keep it working. The software improves fault isolation, eliminates shotgun maintenance, and increases equipment availability. What works so well for NASA can work equally well, even at a less complex level, for field service organizations with high value business critical assets, where downtime is expensive. Orbotech, for example, a company that designs, develops, manufactures, markets and services automated optical inspection (AOI) systems and imaging solutions for PCB production, has deployed the TEAMS® solution worldwide for their entire field service workforce to minimize troubleshooting time and eliminate false pulls. The advanced reasoning of the software enables every technician to perform like an expert, and the solution has paid for itself many times over in the last few years. KLA-Tencor, a leading supplier of yield enhancement equipment to IC manufacturing lines, acquired the TEAMS® solution after an extensive evaluation showed that a novice technician equipped with the TEAMS® guided troubleshooting solution (GTS) could routinely outperform an expert in troubleshooting a complex system. Their entire field service workforce (about 1000 technicians) is now using the TEAMS GTS. Other companies that utilize Qualtech software include Lockheed Martin, Honeywell, BAE and General Motors, making it an integral part of a number of critical, innovative projects that receive international attention.
As Dr. Deb likes to point out, these companies, and the customers they serve, don’t always stop to consider how the collaboration between NASA and QSI benefits them in terms of efficiencies and cost savings. “The technology that works so effectively in a space station environment can be adapted to work just as well on an oil rig or even in a kitchen. Software has to be of the highest quality and innovation to meet NASA’s needs. Historically, technology developed for NASA requirements has been prohibitively expensive for any down-to-earth application. In this case, we have taken a commercial off-the-shelf technology, the TEAMS® toolset, and raised it to NASA standards, without raising the price. The net result is that we now have a very sophisticated software toolset that companies can utilize to save money.” He adds, “The same technology that allows NASA to make complex systems easy to maintain, can now help any technician troubleshoot like an expert and fix problems right the first time – be it business jets or automobiles or submarines; and from million dollar semi-conductor fab equipment to affordable kitchen appliances – if it is hard to troubleshoot, our TEAMS® toolset can help service organizations to save money!”
For more information visit http://www.teamqsi.com