.png)
Thoughts on Improving DND Maintenance Data
We are in an information era that increasingly uses Artificial Intelligence (AI) and machine learning to improve support solutions (to better understand our concept of support solutions, see https://www.teamilcs.com/support-solutions) by analyzing Enterprise Resource Planning (ERP) / Maintenance Management Information System (MMIS) transaction data to improve maintenance planning and support services.
While such use of AI for maintenance improvement has delivered success in industry, the reporting cultures and habits of military maintainers present a particular challenge. In this article, I’ll offer some of my own thoughts on how we might better use AI to help itself become a better tool for continuous improvement of equipment maintenance and its associated support solution. These are based on my experiences and will focus on the significant challenge in applying AI to improve maintenance that is facing defense departments, specifically the Canadian Department of National Defence (DND).
I am far from an expert on AI (though CoPilot and Google may eventually better inform me), so I welcome your comments and corrections to this article. Hopefully, the idea is of some value to you and your equipment support teams.
About the Author – Pat Read has over 55 years experience in supporting equipment systems, both as a Canadian Army RCEME officer and as Pennant Canada’s lead consultant. He has participated in international ILS/IPS standards committees (SAE and ISO:PLCS). He and his teams have modelled processes and developed policies/guidance for ILS/IPS and defence equipment management within the Canadian Department of National Defence (DND). He recently retired from the role of Pennant Canada’s Operations Manager and has now founded Team ILCS to continue to mentor and guide ILCS practitioners and managers.
NOTE: We will use the term Integrated Life Cycle Support (ILCS) in this article to represent more common ILS/IPS terminology and concepts. For more information on why we like the ILCS name/acronym, see https://www.teamilcs.com/why-we-like-ilcs
The Concept – Using Maintenance Data to Truly Reflect & Improve Support
For new defense equipment systems, initial maintenance programs (and their enabling support solution resources) are usually created from an in-depth Product Support Analysis/Logistics Support Analysis (PSA/LSA) effort which is based on predictions of how the equipment will be used; how it will fail; and how it will be repaired. Like most plans, once the equipment enters military service, things are rarely as predicted:
-
The use of the equipment varies from predictions
-
Predicted failures do not occur, or occur at rates different than forecasted
-
New unforeseen failures occur
-
Repairs are delayed, adapted and improved in an ad-hoc fashion by military maintainers
-
Reliability-Centered Maintenance (RCM) preventive tasks are delayed or subsumed into next cycles, often with no apparent or immediate impact
-
These factors suggest periodic reviews and updates to maintenance programs should be conducted to reset the maintenance program to better meet actual experiences and needs, and to make most efficient use of scarce resources.
Maintenance Effectiveness Reviews (MERs)
In the early 2000s, the United States Navy (USN) and other US services introduced the idea of a periodic review of maintenance programs using actual data, which they called Maintenance Effectiveness Reviews (MERs). The MER process is described here…
https://reliabilityweb.com/articles/entry/the_maintenance_effectiveness_review
MERs can be very effective at streamlining in-service maintenance programs to best meet actual needs. At the 2003 DoD Maintenance Symposium, Dr Kenneth Jacobs presented his SurfMER (for surface ships) process, and showed the results that reduced shipboard maintenance workloads by over 40%, primarily by reducing excess preventive maintenance tasks in programs developed by industry (who, in many cases, avoided risk by overloading military maintainers with just-in-case preventive maintenance tasks). MERs are usually focused on the most critical or maintenance intensive systems and sub-systems.
For DND, a team at Pennant Canada adapted the USN SurfMER concept and used it to adapt maintenance programs for some equipment installed on Royal Canadian Navy (RCN) frigates.
ASD/AIA S4000P
S4000P is “the international specification for developing and continuously improving preventive maintenance”. S4000P describes an in-service preventive maintenance optimization process based on analysis of usage and maintenance data, much like the MER process.
See… https://www.s-series.org/s4000p/ for more info.
The Challenge – DND Maintenance Data Quality
For DND (and many defense departments worldwide), the primary obstacle in conducting MER-like reviews remains the poor quality of data reporting by maintainers.
For most industries, the maintenance work order is the justification and documentation of work done to get PAID. For military maintainers, however, reporting the work is more a distraction from their focus on getting the job done and putting the equipment back in the hands of the operators. As such, the quality of maintenance data captured in military ERP/MMIS systems is a major concern.
The reporting culture of the services, and of individual technicians, varies. Generally, Royal Canadian Air Force (RCAF) reporting is of better quality due to involvement of immediate supervisors in the job reporting; and to the overarching concerns around safety and airworthiness. The work reporting by the Army and the RCN is usually less accurate.
The following are some examples of past DND efforts to try and improve maintenance reporting and associated analysis.
Canadian Army LOMMIS Error Reporting
In the late 1980s, the Canadian Army used a Land Ordnance Maintenance Management Information System (LOMMIS) to report maintenance activities. A central cell in National Defence Headquarters (NDHQ) analyzed all work orders entered in LOMMIS, and – monthly – sent LOMMIS error reports to each workshop supporting land equipment (over 100 maintenance facilities of up to 250 technicians each). These reports drove corrections to the data that allowed the database to be used for some analytics, such as supporting a staffing estimation guide for new deployed operations workshops; and for justifying the staffing of existing workshops. It was based on calculated workloads by type of equipment/technician/line-of-maintenance responsibility, using the improved reported data.
CAE DMS CF188 Data Cleanup
CAE supports the Weapons System Manager’s (WSM’s) team managing the RCAF CF188 fleet. In the early 2000s, CF188 maintenance was reported with a system called DMS (sorry, I can’t locate what that acronym is). CAE had access to a copy of the raw DMS data as reported, and their ILS team reviewed it on a continuous basis, to then create a corrected DMS dataset which they could use for analysis.
Our RCN MER Experience
Our experience in applying the MER concept to RCN frigate sub-systems was that almost every maintenance work order (or related repair parts orders) needed to be examined by reliability engineers to try to deduce – often years after the job was completed – what were the failures, the maintenance actions, and the parts used. As well, system usage information and failure effects had to dug out or surmised from labour-intensive analysis of legacy data. Much of the data used was not stored as defined data elements, but rather as long text.
AI-Enabled Improvement ??
Current and future versions of the SAP ERP used in DND to report maintenance have significant business intelligence (BI) and new AI capabilities to analyze maintenance data.
The Concept/Capability
Here is a link to an overview of some these BI/AI capabilities applied to maintenance…
It seems an impressive capability is being developed to integrate data from sensors and merge it with maintenance order reports to enable AI prediction of failures and even scheduling of new jobs.
The Fundamental Disease – Bad Data
Sadly, it seems any AI-generated analysis of DND data remains seriously infected with a virus, called bad data reported on the maintenance notification/order sets filed in the ERP. The notification/order data is the heart of maintenance data in the ERP, and if it is incorrect, inappropriate AI analysis may result.
Perhaps – Doctor AI Can Help Heal Itself!!
But, perhaps (1) we are not terminally ill and (2) “Dr AI” may be able to help heal us.
Capabilities of DND’s SAP
The ERP can have a standard notification triggering the work order with some required data elements.
Then, the ERP has the capacity to hold three related versions of the work order:
-
A standard (“flat rate”) job estimate order (General Maintenance Task Lists in SAP ERP) used by the maintenance planner to plan the job, which can identify the resources, parts and tools needed to complete the work and aligned with/connecting to content in maintenance manuals
-
A job plan order created by the maintenance planner for the specific work order
-
The actual order reporting what was done
The ERP can also have Failure Catalogues to support failure identification within the notification.
What if AI Helped Fix Reporting in Near Real Time?
We would generally expect the notification to provide usage and failure effect data, and the three “orders” to resemble the standard job estimate. RCAF processes for authorizing non-standard work reinforce this linkage. If we wish to improve data, it’s important to ensure the notification/order set captures an accurate data set. If there are any errors or omissions (or even improvement lessons learned) these need to be identified and addressed soon after the job is completed.
So, could AI, in near-real-time during or after on job completion:
IMPROVE ACCURATE USAGE AND FAILURE REPORTING IN NOTIFICATIONS?
-
Usage levels reported in notifications could be confirmed against prior reported levels and data from equipment counters, and drive corrections in near-real-time
-
Failure modes could be confirmed against failure catalogues, and new values added to update the catalogues to reflect observed failure modes.
COMPARE ORDER ESTIMATE-PLAN-ACTUALS? -
AI could use the standard job template to compare with the plan and the actual work order.
-
Variances, errors or omissions could be identified and corrected.
-
By triggering data confirmation in near-real time, data could be verified and improved while the job is fresh in the minds of the stakeholders.
ATTRIBUTE DOWNTIME?
A key metric to analyze maintenance services is the attribution of equipment downtime to causes. Using the timestamping of transaction actions related to the work order, AI could assist in more accurate attribution of downtime to dominant causes for future analysis.
CAPTURE NEW AND GOOD IDEAS?
AI-supported feedback could help capture new methods and ideas from stakeholders to improve the support solution and maintenance execution. This could include feedback on any issues with support resources, such as technical publication errors.
ATTRIBUTE WORK TO CORRECT SYSTEMS?
In poor reporting of maintenance, work can often be attributed to higher level systems or platforms, rather than the specific failed system. AI could analyze the parts used (and their relationship to specific systems) to ensure the work was attributed properly.
What if AI Used Related Data to Fix the Order??
Often other ERP transactions provide information that can improve maintenance reporting. For example, supply transactions showing issue of an assembly to a work order (and return of an unserviceable one) signal that a remove/replace task for that assembly has occurred. Could AI ensure the related maintenance work order was fully completed and linked to the parts usage?
What if AI Incorporated Sensor Data??
More and more, sensors on our equipment provide useful timestamped data on equipment condition, both to predict condition-based maintenance work orders, and to report equipment status (like usage, when failed, when repaired, etc). It seems AI capabilities – as described in the SAP link provided – can be used to integrate sensor data into maintenance orders and perhaps reduce data entry errors by technicians.
Conclusion
In my opinion, the weakness of AI in analysis of defense department maintenance programs is it often seems to assume entered data is good data. For the Canadian Department of National Defence (DND) and other military maintenance organizations, this is often far from the truth. However, perhaps Dr Maintenance AI can fix – or at least improve – itself, by automating analysis and correction of maintenance-related transaction data in near real-time.
Such AI-enabled actions would replace labour-intensive and long delayed corrections. It could ensure the right and correct maintenance-related transaction data was held, keeping the data needed for analysis current, fresh and healthy.
If so, we could have a key AI-enabler for responsive and sound continuous improvement of maintenance programs, with resulting improvements in the support we provide.
Feedback is expected and welcomed.
References
Primary references used to prepare this paper are my own experiences, supported by:
• Dr Kenneth Jacobs SurfMER presentation – 2003 DoD Maintenance Sysposium
• Draft RCN Naval Maintenance Effectiveness Review Guide
• Maintenance Effectiveness Reviews
https://reliabilityweb.com/articles/entry/the_maintenance_effectiveness_review
• ASD/AIA S4000P
https://www.s-series.org/s4000p/
• SAP Community AI for Predictive Maintenance
