AI Creator's Path News: Accelerate data analysis! Introducing Databricks Lakeflow Designer! Reduce the burden on data engineers and streamline AI projects! #Databricks #AIDevelopment #DataAnalysis
Video explanation
Is AI development becoming more accessible? What is Databricks' new weapon, Lakeflow Designer?
Hello everyone! I'm John, a blog writer specializing in AI technology. Recently, the word "AI" has become more common in the news and in smartphone apps. I'm sure there are many people who think, "It seems kind of difficult..." But in fact, new technologies are constantly being developed to make the world of AI easier to use and more familiar to us!
Today, I would like to introduce a new tool that Databricks has announced that is sure to become a hot topic in the AI development field.Lakeflow DesignerI'll explain it in an easy-to-understand way so that even those who are new to AI can say, "I see!"
The "invisible hero" of AI development: the difficulty of data preparation
First of all, what do you think is necessary for AI to operate intelligently?Lots of high-quality dataFor example, if you want an AI to distinguish between photos of cats, you need to train it with lots of photos of different kinds of cats.
However, the data collected cannot often be used as is. The data is often messy, or contains unnecessary information. Therefore, it is essential to organize the data so that it is easy for AI to learn. This process is called "organizing the data."ETL" This refers to the process of collecting data (Extract), cleaning it up so that it is easy to use (Transform), and then saving it somewhere where it can be used (Load).
This ETL work is actually the most time-consuming and labor-intensive part of AI development, and is usually done byData Engineer" are specialized technicians in charge of data engineering. However, these data engineers are very popular and always busy! As a result, AI development projects are not progressing smoothly...ボトルネック(the cause of things not going smoothly)" tended to occur.
A savior has arrived? "Lakeflow Designer" makes data work easy!
That's where the new "Lakeflow Designer" The biggest feature of this tool is "No code" It can be used.
You may be wondering, "What is no-code?" This is a system that allows you to create apps and systems by simply tapping on the screen, as if you were combining blocks, without any complex programming knowledge. With Lakeflow Designer, more people will be able to do some of the ETL work that has previously been left to data engineers.
In particular, the company is focusing on "data analyst" This is great news for people like you! Here are some of the main benefits:
- AI to help you:The tool itself is equipped with an AI assistant to help you with your work.
- Don't worry if you're not an expert:The data pathwayData PipelineYou can intuitively create a series of steps (from when data is created, to when it is processed so that AI can use it, and finally when it is delivered to the AI model).
- Speed up your AI projects:If the time required for data preparation can be reduced, the speed of AI development as a whole will also increase!
What's great about Lakeflow Designer
For those of you who are thinking, "Hmm, that sounds useful, but what exactly is good about it?" let's take a closer look at the key features of Lakeflow Designer.
- Easy to use, just like Canva:
One expert describes Lakeflow Designer as the "Canva of the ETL world." Canva is popular as a tool that allows people with no design knowledge to easily create stylish flyers and presentation materials. In the same way, Lakeflow Designer also allows you to design data pipelines in an easy-to-understand and intuitive way. - Gentle on the outside, powerful on the inside:
Just because it is easy to use does not mean that it has inferior functionality.Spark SQL" is an amazing engine that can process huge amounts of data extremely quickly.Unity CatalogThis system ensures data security and management, so you can rest assured. This is like a data manager, and it properly manages where things are and who can use them. - Easier team collaboration:
Data engineers can easily view, correct, and improve the data pipelines created by data analysts as needed. This makes it easier for each team to share their work, improving the efficiency of the entire team. - Secure and safe with proper management:
Previously, simple no-code tools had issues such as "Can you manage it according to the rules? (Governance)?" and "Can it be managed if many people use it at the same time or the amount of data handled increases? (Scalability)?" Lakeflow Designer seems to be able to address these issues well. In addition, it has the "Git", a system that records file change history,CI/CD Pipeline(A system that automatically tests what you create and makes it available for immediate use if there are no problems) is also supported.Lineage" and decide who can access what data "Access control" and records of who did what and when "AuditabilityIt also has all the important functions for corporate use, such as:
Of course, if extremely complex data integration or processing is required, there will still be cases where the expertise of a data engineer is needed, but it looks like it will be extremely useful for relatively simple cases that arise in daily work, such as "I want to summarize sales by region" or "I want to create data for compliance reporting."
There are competitors, but what makes you different?
In fact, Snowflake, a rival company of Databricks, has also released a similar tool called "Openflow." Although the purpose of both is to "make data work easier for the AI era," there seems to be a slight difference in their approach.
- Databricks (Lakeflow Designer):This platform emphasizes flexibility and openness. It is designed to be easily combined with various tools while taking advantage of its powerful data processing engine (Spark).
- Snowflake (Openflow):This one emphasizes "integration" and "simplicity." It has an image of being neatly organized so that everything can be completed within Snowflake's services.
In addition, Lakeflow Designer is an evolution of the proven technologies that Databricks has previously provided (such as the data import function "Arcion," the data conversion function "Delta Live Tables," and the job management function "Databricks Workflows"), so it seems that another key point is that it has a high level of maturity in terms of functionality.
Databricks' Big Strategy
What's interesting is that Databricks is also developing a full-fledged development tool for data engineers (technical term:IDE: Integrated Development EnvironmentThe company also announced a new workspace where all the tools needed for development are gathered in one place.
This seems to be an indication of Databricks' larger strategy to meet a wide range of needs: "Let beginners get started on development quickly with no-code tools, and then when more advanced development or large-scale operations are required, we'll provide expert tools to help you."
A word from John
Well, when you hear the word AI, you might think, "You have to be an expert to do this, right?" But with the emergence of tools like Lakeflow Designer, it seems like more people will be able to easily try their hand at AI development and data utilization! It's sometimes said that "data is the new oil," but it would be exciting to imagine a future where that precious data can be more easily "refined" and used by anyone. I'm really looking forward to seeing what new ideas and services will emerge from this!
This article is based on the following original articles and is summarized from the author's perspective:
Databricks targets AI bottlenecks with Lakeflow
Designer