Why do we need AI to be explainable? And what challenges should we prepare for in a major upgrade of legacy platforms? Moshe Kranc shares insights into today’s technological challenges and solutions.
As a Technical Director, I have the opportunity to meet with many customers across various sectors, including retail, finance, education, and healthcare. This unique perspective allows me to understand the business and technology challenges these companies face and how they are addressing them. In this article, I’ll discuss common themes I’ve encountered in companies across different industries.
Explainable Artificial Intelligence
Artificial Intelligence (AI) has become a solution to many of today’s business challenges. However, humans—not algorithms—are ultimately responsible for the decisions made. Relying solely on AI recommendations without understanding the reasoning behind them isn’t a practical approach, especially when things go wrong.
For businesses to base critical decisions on AI, they need to understand why an algorithm recommended a particular action and the rationale behind it. This not only builds trust in the system but also helps identify inaccurate or problematic recommendations.
Many AI algorithms, especially deep learning models, base their recommendations on patterns recognized in large volumes of training data. However, the data often relies on statistical models rather than human-understandable logic, which may introduce hidden biases or distortions.
For example, if a company uses historical wage data to determine salary proposals for new employees, the AI may replicate historical wage disparities due to biases against women or minorities.
To mitigate this, any AI-based system we use must be explainable. This means that any recommendation made by the AI can be justified to humans. Techniques such as LIME (Local Interpretable Model-Agnostic Explanations) are helping us understand otherwise opaque deep learning algorithms.
Explainable AI is essential in scenarios where biases can impact people’s lives, from university admissions to credit card approvals or insurance decisions. These areas must be carefully managed to avoid perpetuating bias and ensure fairness.
Modernization with Respect
Some of our customers are undertaking significant upgrades, migrating from legacy mainframe systems to more modern, containerized, cloud-based architectures. Such decisions often follow major events like the end-of-life announcement for their mainframe systems, or the need to boost agility and reduce the time required to implement new business functions.
Whatever the motivation, the transition must be managed carefully to ensure business continuity and prepare the company for the successful adoption of the new system at the end of the process.
Veterans vs. Newbies
A common pitfall in these modernization projects is the divide between “veterans,” who are familiar with existing business processes and technologies, and “newbies,” who are experts in the latest technologies and methodologies. This cultural divide can lead to friction, with veterans feeling excluded from the process and the modernization team lacking the deep domain knowledge needed to upgrade the system effectively.
One solution is to blend these two groups. For example, using experienced product managers from the legacy system to manage the upgrade requirements can bridge this gap. It’s also crucial to foster respect for the legacy systems, which, while outdated, were once state-of-the-art and instrumental in funding today’s modernization.
Over-Optimistic Planning
Overly optimistic planning is another pitfall. Common challenges include:
- Availability of subject matter experts to capture requirements
- Evolving requirements after users see the first version
- Time needed to address non-functional requests, such as parallel operation and mainframe retirement
- The complexities of creating a functional development environment
We’ve learned that the “big bang” approach—deploying a product all at once at the end of development—is doomed to fail. Business decision-makers will lose patience long before the system is fully ready.
Recognizing the Right Approach
We’ve also learned to identify “shortcuts” that allow parts of a new system to be deployed in a matter of months, delivering tangible business benefits that encourage continued progress.
It can be tempting to pursue other goals while modernizing, such as improving business processes or transforming from a waterfall model to an agile development approach. However, in our experience, this is not advisable. Tackling too many objectives at once can open too many battlefronts. For example, how can we be sure the new processes defined by the modernization team will be better than the existing ones? Or that users will prefer frequent releases enabled by agile development?
Modernization is a significant challenge on its own and should be handled as a standalone project, with its own dedicated resources and funding.