In my blog yesterday (Part 1) I suggested that we could evaluate RDBMS-Hadoop integration architecture using three criteria:
- How parallel are the pipes to move data between the RDBMS and the parallel file system;
- Is there intelligence to push down predicates; and
- Is there more intelligence to push down joins and other relational operators?
But Exadata is a split RDBMS with a parallel file system backing it… how does it measure up by these criteria?
There are effective parallel pipes between the Oracle RAC RDBMS and the Exadata Storage Subsystem… so Exadata passes the first test. Further, Exadata is smart about pushing scan and projection both down to the Storage layer.
Unfortunately there is a fairly severe imbalance between the number of nodes on the RAC side and the number of nodes on the Storage side and this creates a bottleneck. We cannot give Exadata full marks here… but as far as parallel pipes goes it stacks up pretty well.
The ability to push down predicates goes a long way towards solving this as the predicate push-down reduces the amount of data that has to move over the bottleneck. But in every data warehouse there will be queries that return lots of rows from the early execution steps… and Exadata cannot join data in the Storage Subsystem so it tries to pull data up sparingly and push down semi-joins whenever possible… it just cannot be done in every case (Note: in Exadata POCs Oracle will try to ensure that no queries are included that pull lots of data up to the RAC layer… and competitors will try to include queries that expose this weakness…).
So… Oracle also includes some intelligence to push some data down to reduce data movement. There is no way to choose to move data from the RAC layer to the Storage Subsystem and execute the query there… the Storage Subsystem can only scan and project… so again we cannot give Exadata full marks… but it is pretty smart as you will see when we start looking at alternative implementations.
Finally, Exadata cannot effectively split a single query plan across both layers… so no marks at all here.
So Exadata is pretty good… but it has weak spots that will be severe for an important set of DW queries in any implementation.
Now, on to the Hadoop implementations… to Part 3
Hi Rob,
I enjoy reading your blogs. I know that in your post you are referring to the specific three criteria that you are using to evaluate. However, there are a few other areas that are noteworthy to point out from an SAP on Exadata perspective.. Exadata for SAP does not support Hybrid Columnar Compression. As a result Oracle relies only upon index key compression and table level compression(Advanced Compression) as part of the 11gR2 Exadata architecture which does not stack up to the compression benefits of other alternative solutions. Also, the patching strategy for Exadata in an SAP environment is very complex. If you have databases in the same Exadata rack that you want to support SAP and NON SAP applications you have your work cut out for you from a patching and maintenance perspective. This is largely because you have different patches for Exadata for SAP environments than you do for a non SAP Exadata environment. The other element to this is deploying Oracle RAC for an SAP environment is no small task either. Again, different architecture and different rules to deploying RAC in an SAP environment than in a non SAP environment. Anyway, Exadata for SAP is a pretty good solution as you say and thought I would add in a little more color.
Derek Oats
IBM
Hi Derek,
My post is not about any DBMS and SAP at all… I am trying to suggest general architectural trade-offs in the analytic and EDW space. I apologize for the confusion… but expect that there will be a segment of readers who will find your comment of interest. So thanks…
Rob
My bad Rob. I guess I got out a little over my ski’s there.
Derek
Hi, Rob,
I am not sure I understand your point here, “there is a fairly severe imbalance between the number of nodes on the RAC side and the number of nodes on the Storage side and this creates a bottleneck.”
For Exadata, a quarter rack contains 2 db nodes and 3 storage nodes, a half rack contains 4 db nodes and 7 storage nodes, and 1 full rack contains 8 db nodes and 14 storage nodes. I guess your bottleneck refers fewer number of db nodes can not handle the data sent back from storage nodes. Well, I have been to many Exadata clients. Compared with clients’ non-Exadata (still Oracle) and Exadata environment, one of major differences is the load is significantly lower in Exadata environment, sometimes even with fewer db nodes used in RAC environment. For majority of the clients, the load is so low that I usually consider it as “no activity” on db nodes. Majority of activities happen on storage nodes because of Smart Scan or offloading. Another interesting aspect is some of my clients add storage nodes or Exadata Storage Expansion Rack instead of db nodes after some time of use.
Just some of my thoughts.
Thanks,
Weidong
Hi Weidong,
There was once a great post by Kevin Closson that showed very clearly the bottleneck… but it is no longer available. In the post he used a demo put up by Oracle to show the bottleneck I referred to would starve the DB nodes. In other words, the reason that you see no activity on the DB nodes is because the storage nodes cannot serve data to them fast enough to keep them busy.
Rob
Hi, Rob,
Let’s assume storage nodes can not serve data to db nodes fast enough. If this is the case, the active db session will spend more time in IO wait events instead of CPU. As more db sessions are waiting for IO, the number of active db sessions will increase. It will cause system load higher, so we will see more activities on both db and storage nodes. This is not the scenario I haven seen on my clients.
Let’s look at from another perspective. If storage node can not feed data fast enough to db nodes, it could be network bottleneck or IO bottleneck on storage nodes. As we know, Exadata is using 40 Gigabit InfiniBand links between DB nodes and storage nodes for their private network. It is very unlikely to see bottleneck here. For IO, if the number of IO request exceeds the Exadata storage IO capacity, sure, it will cause bottleneck in IO. But this will be true for any other databases. In fact, when this happens on Exadata, Exadata’s IO throttling will kick in. Oracle OEM will show Average Throttle Time for Disk I/Os jump from 0 to a non zero value on Exadata storage. For the clients I refers to with “no activities”, the throttle time for disk I/O is constantly zero almost all the time. From my observations, IO throttling on storage nodes will cause higher loads on db nodes as more db sessions are waiting for IO to complete on storage side. Can this happen on Exadata? Absolutely. Actually I have a blog post specifically about the IO throttling issue at one of my clients.
“AWR is not Enough to Track Down IO Problem on Exadata”
http://weidongzhou.wordpress.com/2013/08/23/awr-is-not-enough-to-track-down-io-problem-on-exadata/
But this is uncommon event on Exadata and I have only seen just a few times.
Weidong
Hiya Weidong,
I’m trying to look a little deeper to get you some details… but a couple of comments:
– I do not think that the db nodes show IO Wait when they are waiting for the storage nodes to serve data. I may be wrong about this… but it does not make sense to start a query, push the scan to the storage node, and show IO Wait while the storage node is processing.
– the aggregate IO bandwidth of the Infiniband network is 40 Gigabits… but each node has a limited number of network cards/connections and each of these can serve only a fraction of that 40 Gigabits. This is a bottleneck.
– In any architecture if you saw 8 nodes of a full rack system idle while the other 14 nodes were slammed you would rightfully claim there is an issue… a bottleneck. Think about it… 1/3 of the system is unused. Your examples prove that something is not right, I think.
Rob