85% of AI projects fail! To avoid failure, clear goals and high-quality data are essential! The path of an AI creator News #AI development #Data analysis #MLOps
Video explanation
How to prevent your AI project from "going wrong"? Tips for success that developers should know!
Hello and welcome to my blog, where I explain AI technology in an easy-to-understand way for beginners! I'm your friend, John.
Recently, the term "AI (Artificial Intelligence)" has been appearing a lot in the news and on the Internet. You may even hear people say, "We're going to introduce AI to our company!" But did you know that even the most glamorous AI projects often end up failing to go well?
You may be wondering, "Oh, really? Well, then, how can you succeed?" Today, I will explain in an easy-to-understand way, as always, with reference to the original article, why AI projects tend to fail and what developers in particular can do to make their projects successful!
Is that AI really necessary? ~A starting point that is surprisingly often overlooked~
First of all, when you hear the word AI, it sounds like it can do amazing things, and you might be inclined to think, "That can be done with AI! That can be done with AI!" But wait a minute!Not every problem requires AIHm.
For example, when it comes to tasks such as organizing customer data or making decisions based on specific rules, in many cases AI is not necessary; simpler analytical tools or old-fashioned programs (known as rule-based systems) are sufficient.
AI expert Santiago Valdarama also advises, "Start with simple rules and heuristics (methods that generally work well, learned from past experience)." This will help you better understand what problem you really want to solve, and even if you end up using AI in the future, you'll have a benchmark to compare how useful it is.
So before you start using AI development tools (such as TensorFlow or PyTorch, which are like a set of programming components for creating AI), take a step back and think about it.
- "What problem do we really want to solve?"
- "Is AI the best way to solve that problem?"
Sometimes a simple formula or a spreadsheet like Excel is all you need.
"Garbage in, garbage out" - The magic of data that makes AI smarter
Now, even if you find a problem that is perfect for using AI, the next big hurdle awaits you: data.
In order for AI to become smarter, it needs to learn a lot of data. But what if the data it is being trained on is not very good? There is a famous AI quote that goes, "Garbage in, Garbage Out" This means, "If you put in garbage data, you'll get garbage results."
For example, if you want to teach an AI that "this is an apple, this is a mandarin orange," but you only show it pictures of rotten apples or pictures of only half a mandarin orange, the AI will never be able to properly tell the difference.
In fact, according to one study,Approximately 85% of AI projects fail due to poor data quality or a lack of necessary data in the first place.Yes, it's surprising!
Within a company, data is often stored separately by department (this is called "data siloing" - information is isolated), full of errors, or unrelated to the problem being solved. Even if you train an AI with such data, it may not be useful at all when you try to use it in production.
So, to make your AI project successful, you must firstData Preparation"It's extremely important to do the "data preparation" process properly (the process of collecting the right data, cleaning it up, and labeling it so that the AI can easily understand it in order to teach it). The "data engineering" that supports this data preparation (creating a system for collecting, storing, and processing data so that it is easy for the AI to use) is the "unsung hero" of AI projects.
It is important for developers to confirm what kind of data the AI model needs and whether the existing data really fits that purpose. For example, if you want to create an AI that can predict when customers will cancel their service, do you have the latest usage data on customers? Without it, no matter how amazing the AI mechanism (such as a neural network or an AI mechanism that mimics the neural circuits of the human brain) is, it will not work.
We understand the desire to use AI, but you must not neglect the painstaking data cleaning (ETL: the process of extracting, converting, and loading data; data cleaning: correcting and deleting errors and unnecessary parts of data; feature engineering: modifying the features of data to make it easier for AI to learn).
What is "success"? ~ Setting goals saves your project ~
Another big reason why AI projects end before we even know what's going on is thatThe definition of success is unclear"That is what I mean.
If you start a project with just a vague expectation of "I want to create some kind of great value using AI," then everyone will have different understandings of what constitutes "success."
For example, let's say a store introduces AI that displays recommended products tailored to each customer. But what if there are no specific goals, such as "Is it a success if more people click on the recommended products?", "Is it a success if sales per customer increase?", or "Is it a success if more customers like the store?" Even if the AI works very well technically, you'll end up thinking, "Hmm, is this really a success?"
Especially in the recently popular field of "generative AI" (AI that automatically creates text, images, etc.), many teams create and release models, but do not properly decide how to evaluate their quality. ML engineer Shreya Shankar points out that "Most people do not systematically evaluate something before releasing it to the world. Therefore, expectations are determined purely by the 'vibe'." The "vibe" may be good in a demo, but you may be disappointed when you actually use it.
So what should you do? The answer is simple.Before you start your project, decide on specific success metrics (KPIs: Key Performance Indicators - specific numerical targets to measure whether or not your goals are achieved)!
For example, if you're creating an AI system to detect fraud, you might say something like, "We want to reduce the number of cases where we mistakenly identify fraud (false positives) by X%, while finding Y% more actual fraud." By setting one or two specific goals like this, everyone on the team can work hard in the same direction, and you can also answer questions like, "Is this AI really useful?"
Developers and data scientists (those who specialize in AI and data) should strongly seek to clarify this "definition of success." Rather than explaining "this AI is actually so amazing!" with numbers that cherry-pick the best parts, it will lead to much better results if you have a thorough discussion from the beginning about "what it would take to be successful."
AI also grows! ~It's not over once you've created it, it's important to keep nurturing it~
"Okay, we've created the first version of our AI model! Let's release it!"... You think that's the end? Actually, this is just the beginning.
One of the main reasons why AI projects falter isNo plan for continuous learning and improvement" Unlike ordinary software, the performance of AI models can change over time. For example, data trends in the world can change, or users can have unexpected reactions to the results produced by the AI. In other words, even the shiny new AI we've created will have to deal with the rough waves of the real world.
If you ignore feedback from your AI (information about how it works when used) and don’t have a plan for continually adjusting your AI model, your AI project will quickly become an “outdated experiment.”
The real secret to AI success isConstantly tweaking the modelIn the excitement of having a new AI, this is something that is surprisingly easy to forget, but it is very important.
Specifically, the MLOps (MLOPs: a series of initiatives and mechanisms for developing and improving machine learning models) team said,Data Flywheel"The idea is to put this idea into practice.
- Monitor the results of your AI model
- Collect new data on areas where the AI makes mistakes or seems unsure
- Retrain and improve AI models with collected data
- Re-release the improved AI model
It's about keeping this cycle going. Shankar warns that "teams often expect too much accuracy right after their AI applications are released, and don't build the infrastructure to continually inspect the data, incorporate new tests, and improve the whole system." The release of an AI model is not the finish line, it's just the start of a long marathon!
The trap of "AI for now" - Don't just stop at prototyping
And finally, a common pitfall many organizations face is:The prototype is great, but it doesn't go any further than that.The problem is that even if they create an AI that makes people go "Wow!" in a demo, they neglect the effort to develop it into a solid system that many people can actually use with confidence.
Why does this happen so often? One reason is the excessive expectations caused by the "AI boom" that I mentioned earlier. When companies are pressured by their CEOs or executives to "don't miss the AI train," they become impatient and think, "we have to get something done quickly!" and end up being satisfied with superficial progress without any real substance.
the other one is,"Pilot RengokuThis is a state called "trial run." In this case, many "trial projects (pilot projects)" are launched to test what AI can do, but the budget and manpower are minimal, and they are often separated from existing proper systems. Even if these projects are not technically unsuccessful, they are often lost before you know it because they were not created with the intention of being used in production. Repeating these half-baked experiments costs money and reduces everyone's motivation. One survey showed that 88% of AI pilot projects never make it into production.
According to experts from research firm IDC, many failed generative AI projects tend to start not for solid business reasons, but as executive-level orders, a kind of "top-down economics" that says, "Let's just try AI."
To avoid this trap, it is important to set aside time and resources to polish your prototype so that it can be used in production.
- Make it work with real data
- Create a system to collect user feedback
- Be able to handle edge cases that rarely occur but are problematic if they do occur.
- Filtering out strange prompts and installing safety devices (called guardrails, mechanisms to prevent AI from going out of control) so that difficult decisions are made by humans instead
There are so many things to do. Ultimately, the success of an AI project depends on the developers.
Calling all developers, it's your turn! ~A guide to successful AI projects~
When you hear that AI projects have a high failure rate, you might be a little pessimistic. But even among the piles of failed projects, there are plenty of great examples of AI that have been successful. And when you look closely at those success stories, you'll often find that the developers have a mindset of "valuing content over appearance."
The good news is that the power to prevent these failures lies largely in the hands of us, developers, data scientists, and tech leaders!
We can:
- When the project's objectives or success indicators are unclear, speak up to clarify "why do we do this?"
- I want to spread the word to those around me about the importance of ensuring data quality, which may seem mundane but is extremely important, and the importance of a system for continuously developing AI (MLOps), so that they understand that "AI is not magic, it is proper technology (engineering)."
- When working on an AI solution, we don't just do a demo, but plan the entire process (product lifecycle) from the beginning to the end of the product.
The key to making AI projects successful is for us developers to believe in the potential of AI while taking a down-to-earth approach.
A word from John
Wow, AI is really deep, isn't it? While writing this blog, I once again felt that "basics are important." No matter how amazing a technology is, if the people who use it don't have a clear purpose and don't carefully nurture it, it will go to waste. If you have the opportunity to be involved in an AI project, please remember what I said today!
This article is based on the following original articles and is summarized from the author's perspective:
Why AI projects fail, and how developers can help them
succeed