This article is for you if you considering to use Data Warehouse Automation (DWA) and asking yourself why you should use Data Warehouse Automation tools what does it do for you. After I explained in my previous blog Why Data Warehouse Automation is not more popular, you will find the why and what of Data Warehouse Automation in this second post of the series.
Why automate your Data Warehouse?
Every industry has used automation to increase productivity, reduce manual effort, improve quality and consistency, and speed delivery. Henry Ford introduced the assembly to produce automobiles, and today Uber and countless other startups use the Internet and digital processing to reduce friction in commercial transactions. Thus, the time has come to introduce automation to data warehousing. Pointed out by Eckerson Group.
I would say it like this. In a society where time flys remarkably fast and data became the new gold, it’s crucial to have proper analyses or even data predictive trends to navigate the company in the best direction. In order to get there, I believe automation is needed more than ever before. It’s not the time to do recurring tasks manually over again and producing additional errors. DWA could deliver you the first KPI‘s within weeks based on your data foundation you already have. Why then waste another month, or two to figure out technical details instead of focusing on analysing the data and the business requirements?
Some key reason why I would want to automate:
- DWA enables you to concentrate on examining the data not hunting and building them
- Changes the way of working from a laboratory to an agile way
- Iterating fast lets you economically test an idea and then change it until you find the best way to achieve your desired business outcomes
- Automation provides almost immediate gratification per se with the actual source data
- ETL code is automated, implicates consistency and proven code which leads to higher quality solutions
- Scheduling of initial or incremental loads are simple
- ETL development is based on templates which save repetitive work
- Testing can be automated (which is almost impossible doing it in the traditional way)
- Documentation is produced with the Data Warehouse automatically, each time you generate
- Using your valuable IT resources more efficiently lets you innovate faster and get far more done on a limited budget
What does Data Warehouse Automation for us?
Now we know why we want to automate, but what is it actually that DWA is doing? Mainly the following:
- Agile design & build
- Standardize development
- Operation of Data Warehouses (e.g. loading, design validation, scheduling, error tracking and performance monitoring)
- Quick and easy change management processes
- Easy roll back to the latest fully functional release as everything is version controlled
But probably the most important point here to highlight is repetition. We do the same thing over and over again in Data Warehousing. Is it adding staging tables or to any layer, changing the structure of an attribute, creating schema (star, snowflake, third normal form, or data vault), adding foreign keys to the fact tables, creating the view layers, adding slowly changing dimension, create default indexes, …, the list goes on and on.
Same thing said through a video made by Variance and their brand new DWA framework BimlFlex:
Conclusion
It is actually hard to find a reason not to automate. As highlighted firmly above it improves quality, increased agility, provides you with the ability to iterate fast and reduces cost with better sustainability, maintainability and operability. At the end of the day, you cut development time and effort to reduce project costs and increase return on investment. Don’t waste more time and work smart at the speed of business with DWA!