No. ETL is not dead. Not in the slightest. AI can not build ETL.
AI can copy data from one place to another. But then again, so can I using my ETL software.
Using my ETL Software I can create an extraction process and staging area for a 50,000 field large operational system in a week. So how does AI help me do that? If it can do it in a day? So what?
Taking data from large operational systems and putting it into a dimensional data warehouse requires that a data warehouse architect look into what needs to be done and builds what needs to be done.
Currently, we are at rates of 12K-15K fields mapped in a 220 hour work month for my paid for ETL and 6K-8K per work month for my free version.
Mapping 6K-8K fields from a large operational system across to a data warehouse per 220 months work month makes that task essentially free when compared to all the other tasks.
I have heard people talking about "data pipelines" for some time now. The demos I see and the discussions I hear are all about moving similar, and even the SAME data more than once.
People are very, very, VERY secretive about these "data pipelines" they are building and I presume the reason is that they are very bad. I have seen demos of dbt and similar products and they are a joke.
BTW I have invented the future of data warehousing and the future is "Mega Models". I have written posts about it.
But no. AI is not doing anything useful in the ETL area. The most useful thing I have found for AI to do is to do voicer overs for my posts in a womans voice because men will listen to a womans voice longer than they will listen to a mans voice.
You can get my ETL software, for free, from the freebies link in my profile. Those people who have used it saves a lot of money, even if they went into production with Informatica or DataStage.
Ok?
If the men who call themselves "data engineers" who are building star schema data warehouses want to improve their standing with the business side of the house? They can save their companies a whole lot of money by using my free ETL and free data models.
Any man who wants to run his own consulting company building star schema data warehouses can win more business by having greater pricing flexibility by using my free software.
"The medallion architecture — Bronze to Silver to Gold — is an assembly line designed for human inspection at each station."
Medallion Architecture is not an architecture. It's a marketing slogan.
No one who knows how to build a data warehouse takes "Medallion Architecture" seriously. And we don't build anything that looks like "Medallion Architecture".
We build the following.
1. Landing Area.
2. Staging Area.
3. Archival Layer,
4. Base Layer.
5. Multi-Level Summary Layer.
Since 2002 the archive layer and base layer have become relatively collapsed. They exist but in 2002 I invented the last piece of the puzzle to be able to archive data in a dimensional model on a daily basis without bankrupting the company on processing power needed.
Bill Inmon brought out his Data Warehouse 2.0 book in 2008 to deal with textual style data and he has since brought out his tool called Textual ETL.
There are far too many articles like yours being written as "advertising" when men like you would do far better to actually use my software and learn how to do data warehousing properly.
Please don’t ever say “xyz” is dead. It makes you look like an idiot. I mean, really, really stupid and/or ignorant. Or a bot.
/unsubscribe
What part of the article you disagree with?
Good Article.
May many more such insights come through.
good article!!! a great read! thank you!
Great insightful article. Can you recommend any book related to this topic with the changing things. Thank you.
Hello Ananth.
No. ETL is not dead. Not in the slightest. AI can not build ETL.
AI can copy data from one place to another. But then again, so can I using my ETL software.
Using my ETL Software I can create an extraction process and staging area for a 50,000 field large operational system in a week. So how does AI help me do that? If it can do it in a day? So what?
Taking data from large operational systems and putting it into a dimensional data warehouse requires that a data warehouse architect look into what needs to be done and builds what needs to be done.
Currently, we are at rates of 12K-15K fields mapped in a 220 hour work month for my paid for ETL and 6K-8K per work month for my free version.
Mapping 6K-8K fields from a large operational system across to a data warehouse per 220 months work month makes that task essentially free when compared to all the other tasks.
I have heard people talking about "data pipelines" for some time now. The demos I see and the discussions I hear are all about moving similar, and even the SAME data more than once.
People are very, very, VERY secretive about these "data pipelines" they are building and I presume the reason is that they are very bad. I have seen demos of dbt and similar products and they are a joke.
BTW I have invented the future of data warehousing and the future is "Mega Models". I have written posts about it.
But no. AI is not doing anything useful in the ETL area. The most useful thing I have found for AI to do is to do voicer overs for my posts in a womans voice because men will listen to a womans voice longer than they will listen to a mans voice.
You can get my ETL software, for free, from the freebies link in my profile. Those people who have used it saves a lot of money, even if they went into production with Informatica or DataStage.
Ok?
If the men who call themselves "data engineers" who are building star schema data warehouses want to improve their standing with the business side of the house? They can save their companies a whole lot of money by using my free ETL and free data models.
Any man who wants to run his own consulting company building star schema data warehouses can win more business by having greater pricing flexibility by using my free software.
Ok?
Hello Ananth,
"The medallion architecture — Bronze to Silver to Gold — is an assembly line designed for human inspection at each station."
Medallion Architecture is not an architecture. It's a marketing slogan.
No one who knows how to build a data warehouse takes "Medallion Architecture" seriously. And we don't build anything that looks like "Medallion Architecture".
We build the following.
1. Landing Area.
2. Staging Area.
3. Archival Layer,
4. Base Layer.
5. Multi-Level Summary Layer.
Since 2002 the archive layer and base layer have become relatively collapsed. They exist but in 2002 I invented the last piece of the puzzle to be able to archive data in a dimensional model on a daily basis without bankrupting the company on processing power needed.
Bill Inmon brought out his Data Warehouse 2.0 book in 2008 to deal with textual style data and he has since brought out his tool called Textual ETL.
There are far too many articles like yours being written as "advertising" when men like you would do far better to actually use my software and learn how to do data warehousing properly.
Ok?
Long live ETL.