Skip to content

The era of agent AI has arrived! Is your data infrastructure ready?

Agentic AI: Is Your Data Infrastructure Ready for the Real-Time Revolution?

The Path of an AI Creator News Real-time response is important for data infrastructure! How to prepare for the era of agent AI? #Agent AI #Data infrastructure #AI infrastructure

Video explanation

Will we see an era when AI can think and act on its own? A story about "agent AI" and data

Hello, I'm John, here to help you understand the world of AI technology!
Recently, the word "AI" has become commonplace. However, the world of AI is also evolving rapidly, and it is said that the next big wave is coming."Agent AI"That's it.

You may be wondering, "What is an agent?" Simply put, it is "AI that can think and act autonomously." Just like the smart robots in science fiction movies, AI may be able to judge situations by itself and carry out various tasks on our behalf.

In fact, this is a big change somewhat similar to when "cloud computing" (a system for using various IT services via the Internet) emerged about 10 years ago. At that time, open source (technology with open blueprints that anyone can use freely) such as Docker and Kubernetes appeared, and the speed and flexibility of software development increased dramatically. However, it took time for companies to get used to it.

Now, this "agent AI" is about to bring about another major wave of change.Real-time data exchangeFor AI to operate intelligently, it needs to constantly process new information at an incredible speed of "milliseconds" (thousandths of a second!) rather than "minutes". If you miss this change, it may be difficult for your business to survive in the new AI era.

So, how should we prepare for this era of agent AI? How we handle "data" seems to be the key. Let's take a look!

Are conventional data storage sites no good enough? A new data infrastructure for the AI ​​era

Traditional company data storage (called data platforms) was mainly created for experts who analyze numbers (SQL analysts) and technicians who organize data (data engineers). However, in today's AI era, that's not enough.

Going forward, machine learning engineers (the people who create AI), developers, product planning teams, and most importantly the AI ​​agents themselves, will need access to data in real time, using a variety of programming languages ​​and tools, including Python, Java, and SQL.

What is being noted here is that"Apache Iceberg"This open source technology is expected to become the foundation of a new data infrastructure in the AI ​​era, just as Docker and Kubernetes did in the cloud era.

Here's what's great about Iceberg:

  • It's okay if the data shape changes along the way (this is called schema evolution)
  • Time travel function allows you to look back at past data and ask yourself, "What was the data like at that time?"
  • No confusion even when many people and systems use the data at the same time (high concurrency access)

In addition, when combined with a powerful data platform that doesn't require you to worry about managing servers (a serverless data platform), we can achieve the ultra-fast data flows required by AI agents with unpredictable behavior.

With all these technologies in place, people and systems with different roles can work together smoothly, and smart AI agents can go beyond simply looking at data to "act" safely and quickly in a changing data environment.

The most difficult part is what happens after implementation

The most difficult part of creating a data infrastructure for agent AI is not actually choosing the technology. After deciding, "Okay, let's use this technology!", it's actuallyKeeping it running stably, efficiently, and safely ("Day two" operation, i.e. daily operation after installation)But that is the biggest challenge.

No matter how great your data formats and processing tools are, they are meaningless if you cannot use them reliably, affordably, and safely in a world where AI is constantly interacting with data and taking action at unpredictable times.

Specifically, the following problems tend to occur:

  • Data Tracking and Compliance: Keeping track of where your data comes from, what happens to it, and deleting it to comply with laws like GDPR (Europe's strict data protection rules) is very complex and important.
  • Efficient use of resourcesIf not used wisely, the cost (and electricity!) of components such as GPUs and TPUs required for AI calculations can add up quickly. Some cloud services make it easier to manage these computing resources.
  • Access Management and Security: If you make a mistake in the settings of who can access what data, it can be a big problem. If you allow access to a wide range of people, there is a risk that important information will be leaked.
  • Data Discovery and BackgroundEven with useful tools like Iceberg, it's surprisingly difficult to find information (called metadata) about what the data represents so that AI can quickly use it when needed.
  • ease of use: While modern data tools are multifunctional, they can also be complicated to use. If you don't simplify the workflow for developers, analysts, and the AI ​​agents themselves, productivity will decrease.

Without a solid operational system, no matter how impressive your data infrastructure is, it won't be able to keep up with the agent AI's quick decision-making and action loop.

The best of both worlds: "Collaboration" and "Leaving it to the professionals"

Modern IT complexity, especially data-related ones, has been driven by open source innovation, where open source communities (where developers congregate) often come up with cutting-edge solutions for use cases that are often beyond the reach of corporate data teams.

However, when it comes to actually using open source tools to import large amounts of data, process that data in real time, and quickly prepare computing resources when needed, problems can arise. Many companies find that the flow of data processing becomes unstable, costs are rising, and old systems cannot keep up with the real-time requirements of agent AI.

In this situation, we can rely onCompanies that provide cloud services (cloud providers)They have a lot of know-how in operating large-scale systems stably.

The goal is to use open standard technologies (technology that is not tied to a specific company, which avoids "vendor lock-in" where you can only use one company's products) and combine them with cloud infrastructure to automate tedious tasks such as tracking data and preparing resources. The smart approach is to work with a cloud partner that actively contributes to open source technologies and provides services that support reliable operations. You should be able to achieve results much faster and more reliably than trying to create an unstable system on your own, or relying on a proprietary platform whose contents you don't fully understand.

For example, Google Cloud's BigQuery service works with Apache Iceberg to handle large amounts of data in real time using open data formats. It also automates table management, improves performance, and provides integration with an AI development platform called Vertex AI, helping with the development of agent AI applications.

Ultimately, the goal is to reduce the risks of managing complex data infrastructure on our own while accelerating new technological innovations.

The labor shortage in the age of AI is also a real problem

In fact, even large companies are facing a shortage of people who can design and safely operate AI-enabled data platforms. What is particularly serious is the lack of people who are not just knowledgeable about data, but alsoEngineers who can handle large-scale systems that operate in real timeWhat's lacking is...

Agent AI will further raise the bar on operational requirements and accelerate the pace of change. That's why we need a platform that allows flexible collaboration, solid management, and instantaneous communication. And these systems must simplify operations without compromising reliability.

Summary: Now is the time to start preparing for the future!

The new market centered on "agent AI" may bring about bigger changes to society than the advent of the Internet. If your company's data infrastructure is not yet real-time, open, and scalable...The time to act is now!

The future in which AI becomes more familiar and helps us in our daily lives and work may just be upon us.

(From the author)
Wow, agent AI is kind of exciting! It's like the world of science fiction becoming reality. But behind the scenes, a major overhaul of the data infrastructure is needed, which is both a test of the engineers' skills and a headache... (laughs). But it's interesting to ride this wave of change!

This article is based on the following original articles and is summarized from the author's perspective:
Agentic AI won't wait for your data architecture to catch
up

Related posts

tag:

Leave a comment

There is no sure that your email address is published. Required fields are marked