Data Engineering, the future of Data Warehousing?
Today, there are 6,500 people on LinkedIn who call themselves data engineers according to stitchdata.com. In San Francisco alone, there are 6,600 job listings for this same title. The number of data engineers has doubled in the past year, but engineering leaders still find themselves faced with a significant shortage of data engineering talent. So is it really the future of data warehousing? What is data engineering? These questions and much more I want to answer in this blog post.
In unicorn companies like Facebook, Google, Apple where data is the fuel for the company, mostly in America, is where data engineers are largely used. In Europe, the job title does not completely exist besides the startup mecca Berlin, Munich, etc. They are called or included in jobs like software engineer, big data engineer, business analyst, data analyst, data scientist and also the business intelligence engineer. Myself, I started as a…
Why automate? What does Data Warehouse Automation for us?
This article is for you if you considering to use Data Warehouse Automation (DWA) and asking yourself why you should use Data Warehouse Automation tools what does it do for you. After I explained in my previous blog Why Data Warehouse Automation is not more popular, you will find the why and what of Data Warehouse Automation in this second post of the series.
Why automate your Data Warehouse?
Every industry has used automation to increase productivity, reduce manual effort, improve quality and consistency, and speed delivery. Henry Ford introduced the assembly to produce automobiles, and today Uber and countless other startups use the Internet and digital processing to reduce friction in commercial transactions. Thus, the time has come to introduce automation to data warehousing.
Pointed out by Eckerson Group.
I would say it like this. In a society where time flys remarkably fast and data became the new gold, it’s crucial to have proper analyses…
Why Data Warehouse Automation is not more popular
I was working with a Data Warehouse Automation (DWA) tool for a little more than a year, and I have to say I loved it. As a BI developer you could focus on the challenges you had in dimensional modelling, what granularity should you have the fact tables and going crazy with the business requirements and everything fast, consistent and tested!
But why is Data Warehouse Automation not used more often and more popular? I’m asking that myself more and more. That’s why I’m writing a series of blog posts all about DWA. In this first blog, I’m trying to find possible reasons behind and also argue for DWA, and why we should use it more often.
Every one needs to make data driven decision faster, why not use a generator which gives you answers in days instead of months..?
Losing control
What do I mean by that? Many people and therefore many companies fear…
Migrate from Oracle to Microsoft (Views) – Part III
![Continued from Migrate from Oracle to Microsoft (Views) – Part II here part III with the main focus on very interesting topic testing ;-).
Testing
When I implemented all about 20 views from Oracle to SQL Server, I had to test the performance and not least important, the content of the views if their still contain the same. And how do you do that the easiest way?
Measure performance
For measuring the performance you simply select alls Views with a SELECT COUNT(*) on both databases. Don’t forget to clear the cache before each run.
To generate all the COUNT-Statements I created a little script, just to make life easier right ;-):
–Generate Queries – Performance Test…
SELECT v.TABLE_NAME,
–stuff(c.list,1,1,”) as cols,
‘PRINT ”’ + v.Table_name + ”’ ‘ +
‘SELECT ”’+v.Table_name+”’ as Tbl_name, NULL as cnt UNION ALL ‘ +
‘SELECT ”’+v.Table_name+”’ as Tbl_name, count(*) as cnt FROM [dbo].[‘+v.Table_name +’] ‘–WHERE 1=2 ‘
+ CHAR(13)+CHAR(10)
FROM INFORMATION_SCHEMA.VIEWS… Read more](https://www.sspaeti.com/blog/wp-content/uploads/2016/09/microsoft-sql-server-vs-oracle-700x523.jpg "microsoft-sql-server-vs-oracle")
Continued from Migrate from Oracle to Microsoft (Views) – Part II here part III with the main focus on very interesting topic testing ;-).
Testing
When I implemented all about 20 views from Oracle to SQL Server, I had to test the performance and not least important, the content of the views if their still contain the same. And how do you do that the easiest way?
Measure performance
For measuring the performance you simply select alls Views with a SELECT COUNT(*) on both databases. Don’t forget to clear the cache before each run.
To generate all the COUNT-Statements I created a little script, just to make life easier right ;-):
–Generate Queries – Performance Test…
SELECT v.TABLE_NAME,
–stuff(c.list,1,1,”) as cols,
‘PRINT ”’ + v.Table_name + ”’ ‘ +
‘SELECT ”’+v.Table_name+”’ as Tbl_name, NULL as cnt UNION ALL ‘ +
‘SELECT ”’+v.Table_name+”’ as Tbl_name, count(*) as cnt FROM [dbo].[‘+v.Table_name +’] ‘–WHERE 1=2 ‘
+ CHAR(13)+CHAR(10)
FROM INFORMATION_SCHEMA.VIEWS…
Migrate from Oracle to Microsoft (Views) – Part II
Continued from Migrate from Oracle to Microsoft (Views) – Part I here part II with more focus on the performance part.
Performance issues recursive queries
When it comes to performance I had one big issue with the built-in function SYS_CONNECT_BY_PATH which is used for rekursiv queries in Oracle. This function is well know by the Oracle optimizer which makes this function unbelievable fast and there is no similar function on SQL Server. Even more complex was, that this queries was used on nested views, so the SQL was quite complex.
I started to translate it to an CTE view which is the pendant in SQL server and the answer to recursive queries in SQL Server. But I ended up having big performance issues because the queries where to complex. I had good performance measures on an easy simple query but invinitive response with more complex.
So what’s the solution to that?
Inline User-Defined Functions. Of course it…
Migrate from Oracle to Microsoft (Views) – Part I
Sometimes you have to migrate from an Oracle database to a Microsoft SQL Server. I’m not going into the reason that could be behind. I had different projects where I had to achieve this goal. The issues that I faced is what I would like to share with you.
Different functions
When it comes to rewrite code from Ora-SQL to MS-SQL, in my case it were Oracle Views that had to be transfered to Microsoft Views, then the first thing you figure out is, that there’s not the same built in functions on both sides. For example the well know TO_NUMBER() or TO_CHAR() in Oracle dosen’t exist in Microsoft. So what you do?
Below you see a list of the common functions compared between the two technologies:
Description
Oracle
MS SQL Server
Smallest integer >= n
CEIL
CEILING
Modulus
MOD
%
Truncate number
TRUNC
<none>
Max or min number or string in list
GREATEST,LEAST
<none>
Translate NULL to n
NVL
ISNULL
Return NULL if two values are equal
DECODE
NULLIF
String concatenation
CONCAT(str1,str2)
str1 + str2
Convert ASCII…
SSAS Cubes – Dynamic generation of partition
![There is no easy way to generate partition for SSAS Cubes by default. So you have to do a SSIS-Package (here is a way you can do it dynamic-cube-partitioning-in-ssas-2008) or write a SQL Script which generates XMLA executables.
As shown in the Statement, you can dynamically generate month partitions. The trick is, to create a linked server, that you can execute XMLA statements with relational T-SQL, so i created a Linked Server first.
Thanks for any advice or improvements in the comment section:
EXEC master.dbo.sp_addlinkedserver
@server = ‘SSAS’
, @srvproduct = ”
, @provider = ‘MSOLAP’
, @datasrc = ‘[DB-NAME]’;
EXEC master.dbo.sp_serveroption @server=’SSAS’, @optname=’rpc out’, @optvalue=’true’;
CREATE SCHEMA ssas;
CREATE PROCEDURE ssas.usp_Execute (
@XMLA XML
)
AS
DECLARE… Read more](https://www.sspaeti.com/blog/wp-content/uploads/2015/10/cube-partitions-e1444391131116.jpg "cube partitions")
There is no easy way to generate partition for SSAS Cubes by default. So you have to do a SSIS-Package (here is a way you can do it dynamic-cube-partitioning-in-ssas-2008) or write a SQL Script which generates XMLA executables.
As shown in the Statement, you can dynamically generate month partitions. The trick is, to create a linked server, that you can execute XMLA statements with relational T-SQL, so i created a Linked Server first.
Thanks for any advice or improvements in the comment section:
EXEC master.dbo.sp_addlinkedserver
@server = ‘SSAS’
, @srvproduct = ”
, @provider = ‘MSOLAP’
, @datasrc = ‘[DB-NAME]’;
EXEC master.dbo.sp_serveroption @server=’SSAS’, @optname=’rpc out’, @optvalue=’true’;
CREATE SCHEMA ssas;
CREATE PROCEDURE ssas.usp_Execute (
@XMLA XML
)
AS
DECLARE…
ORDER BY Oracle vs. Microsoft SQL
Like you would think ORDER BY is the same in Oracle as in the Microsoft environment, it isn’t. It depends on the Collation of the database. In the following example you see, that two manually added text in Oracle and Microsoft don’t order the same. After I added the Collation for SQL_Latin1_General_CP850_BIN2, it ordered the same.
ORDER BY Oracle:
ORDER BY Microsoft – normal and with collation:
…