Today, there are 6,500 people on LinkedIn who call themselves data engineers according to stitchdata.com. In San Francisco alone, there are 6,600 job listings for this same title. The number of data engineers has doubled in the past year, but engineering leaders still find themselves faced with a significant shortage of data engineering talent. So is it really the future of data warehousing? What is data engineering? These questions and much more I want to answer in this blog post.
In unicorn companies like Facebook, Google, Apple where data is the fuel for the company, mostly in America, is where data engineers are largely used. In Europe, the job title does not completely exist besides the startup mecca Berlin, Munich, etc. They are called or included in jobs like software engineer, big data engineer, business analyst, data analyst, data scientist and also the business intelligence engineer. Myself, I started as a…
There is a bit of a confusion between Data Warehouse vs Data Lake or ETL vs ELT. I hear that Data Warehouses are not used anymore, that they are replaced by Data Lakes altogether, but is that true? And why do we need Data Warehouses anyway? I will go into that as well as the definitions of both pluses explain the differences between them.
Data Warehouse vs Data Lake
Data Warehouse definition
A Data Warehouse, in short DWH and also known as an Enterprise Data Warehouse (EDW), is the traditional way of collecting data as we do since 31 years. The DWH serves the purpose of being the data integration from many different sources, the single point of truth and the data management meaning cleaning, historize and data joined together. It provides greater executive insight into corporate performance with management Dashboards, Reports or Ad-Hoc Analyses.
Various types of business data are analysed with Data Warehouses. The need for it often…
As promised in part I here is the part II of tools I use and in this part, I will entirely focus on one of my most used and favourite tool called Microsoft OneNote. I use it for almost everything, you may ask “Why?! What is the big deal, I can use Microsoft Word, Google Keep, a paper block or anything else, why MS OneNote?” Yes, that is fully true but you are lacking fundamental structure and essential features that you won’t have in these tools.
Maybe the biggest feature itself is to organisation and structure inside OneNote, it keeps your work, university, private material perfectly organized, and you can get things done. If your project is growing bigger than expected, you are able to quickly restructure your notes by increasing an additional section and turn the existing section into a section group which allows you to split notes into different segments like releases, parts,…
The 5 top most searched Data Warehouse Automation tools on the market compared with GoogleTrends is telling you that WhereScape is first before TimeXtender and BiReady (new Attunity Compose) over the last year. See the picture in full size or go directly to GoogleTrend comparison and change to your own needs.
Although the analysis is not representative, it still gives some insights and a good overview to size and presumably usage compared to each other, worldwide. Please consider that WhereScape and TimeXtender have more search results as the company name is the same as their product, meaning some of them are dedicated to the company name rather the Data Warehouse Automation (DWA) tool itself. And BimlFlex just published their first release and biGENiUS is rather new to market their product actively, they will probably increase slightly in the soon future.
Data Warehouse Automation Tools on the market
As you can imagine, there are plenty of…
This article is for you if you considering to use Data Warehouse Automation (DWA) and asking yourself why you should use Data Warehouse Automation tools what does it do for you. After I explained in my previous blog Why Data Warehouse Automation is not more popular, you will find the why and what of Data Warehouse Automation in this second post of the series.
Why automate your Data Warehouse?
Every industry has used automation to increase productivity, reduce manual effort, improve quality and consistency, and speed delivery. Henry Ford introduced the assembly to produce automobiles, and today Uber and countless other startups use the Internet and digital processing to reduce friction in commercial transactions. Thus, the time has come to introduce automation to data warehousing.
Pointed out by Eckerson Group.
I would say it like this. In a society where time flys remarkably fast and data became the new gold, it’s crucial to have proper analyses…
I was working with a Data Warehouse Automation (DWA) tool for a little more than a year, and I have to say I loved it. As a BI developer you could focus on the challenges you had in dimensional modelling, what granularity should you have the fact tables and going crazy with the business requirements and everything fast, consistent and tested!
But why is Data Warehouse Automation not used more often and more popular? I’m asking that myself more and more. That’s why I’m writing a series of blog posts all about DWA. In this first blog, I’m trying to find possible reasons behind and also argue for DWA, and why we should use it more often.
Every one needs to make data driven decision faster, why not use a generator which gives you answers in days instead of months..?
What do I mean by that? Many people and therefore many companies fear…
Very long time ago, 1986, 31 years ago to be precise, IBM in Europe created the very first architecture of a data warehouse. And it seems to be a masterpiece as it didn’t change much since. Though how can we improve or bring some innovation into Data Warehouse business in times everyone is talking about big data, data lakes, Internet of Things (IoT), predictive analytics, Data Vault, etc.?
No matter how we want to improve the architecture, it has to be automated as much as possible. Nowadays it became too slow for serving the business needs doing it the traditional way. However, I don’t think DWH will go away anytime soon (see more DWH vs Data Lake). I strongly believe that DWA tools are the future and will boost the Data Warehousing reputation back to earlier years.
That’s why I wrote this blog series all about DWA. You can start with the first blog…
As I’m using a lot of tools on a daily basis which makes my life easier, I’d like to share them with you. Also because lately, I recommending them more regularly to friends and co-workers. Hope you find them useful, please leave a comment if you’re using other cool tools or if you disagree with my opinion.
Some tools I got inspired by the author Tim Ferriss and his guests in his popular podcasts. Find the links at the end of this blog for more information.
Capture the time
Rescue Time – rescuetime.com
This little tool you install on each computer or smartphone you own, and it will keep track of how productive you are. It shows you what program you used, how long and you can compare days, weeks or years. If you like, you’re able to set goals for an application to not use more than X amount of hours.
I can say…
Quote by Leonard Nimoy.
I’ve experienced the truth of this a long time ago during my apprenticeship, without being aware of it. You probably remember those days when you had to prepare for an exam, but due to time constraints, you couldn’t look up everything in the books. So my friend and I compiled summaries of the most important things. Not surprisingly, these summaries quickly became very popular in class, and more people asked us if we could share it with them. I remember that back then I was sceptical about this and was asking myself “Why should I give it to them? We put a lot of work in it and they just want to copy that? No way!” At the end of the day, they got it anyway either through my friend or some other way. Eventually, I just said to myself “Oooookaay, give it to everyone who asks,”…
Continued from Migrate from Oracle to Microsoft (Views) – Part II here part III with the main focus on very interesting topic testing ;-).
When I implemented all about 20 views from Oracle to SQL Server, I had to test the performance and not least important, the content of the views if their still contain the same. And how do you do that the easiest way?
For measuring the performance you simply select alls Views with a SELECT COUNT(*) on both databases. Don’t forget to clear the cache before each run.
To generate all the COUNT-Statements I created a little script, just to make life easier right ;-):
--Generate Queries - Performance Test...
--stuff(c.list,1,1,'') as cols,
'PRINT ''' + v.Table_name + ''' ' +
'SELECT '''+v.Table_name+''' as Tbl_name, NULL as cnt UNION ALL ' +
'SELECT '''+v.Table_name+''' as Tbl_name, count(*) as cnt FROM [dbo].['+v.Table_name +'] '--WHERE 1=2 '
FROM INFORMATION_SCHEMA.VIEWS v
--where v.TABLE_NAME LIKE '%What you like%'
ORDER BY v.TABLE_NAME
--To the generated output you just add below in front and you're ready to go:
SET STATISTICS TIME ON
PRINT 'My_View' SELECT 'My_View' as Tbl_name, NULL as cnt UNION ALL SELECT 'My_View' as Tbl_name, count(*) as cnt FROM [dbo].[My_View]
First you create a database linked server on your SQL Server database. If you don’t know how to do this, please read the following How-to.
If you have only a few comparison you easily create EXCEPT-SELECT-Query and compare your Oracle Views over the database link (linked-Server)…