UPDATE: I’ve made a few changes to how the component works - Read about it here. Recently I have been playing around with the Hortonworks Big Data sandbox, the tutorials were very good, and made it easy to get going with some sample data.
Given that I mainly use the Microsoft BI stack I was hoping for the ability to use SSIS to load my data, especially as it would be nice down the line to do this as part of an ETL process.
This post is a note to my future self on how to configure a TeamCity build to run tests that use Microsoft Fakes. If you haven’t ever come across Microsoft Fakes then take a look at this post - Testing the untestable with Microsoft Fakes (http://msmvps.com/blogs/bsonnino/archive/2013/08/11/testing-the-untestable-with-microsoft-fakes.aspx) for a good introduction.
Setting up the Build Agent You will need to install Visual Studio 2012 and make sure that Update 3 is applied, no additional installation is required as Microsoft Fakes comes bundled.
In my previous post, I made reference to the Twitter Big Data example for Microsoft StreamInsight (project page).
The sample collects tweets in real-time from Twitter then does a few things:
Displays current information about trends in real-time on a web dashboard Stores information about the tweets into a SQL Azure database Store the actual tweets into an Azure Blob Store Then there are some commands that you can use with Azure HDInsight to do some post-processing, this is great if you have access to the HDInsight Preview, but what if you are stuck on the waiting list?
Recently I have been looking into complex event processing, what the use cases for it are and the kind of insights you can get from it.
As I am a .Net developer, I thought it best to start with the Microsoft stack, so I have been experimenting with StreamInsight. There are a few examples floating around, but the one that really inspired me was the Big Data Twitter Demo (link).
In this post I’m going to take a break from the actual development of custom components for SSIS and show how the RabbitMQ Source that I have created so far could be used in an ETL package to load a data warehouse.
The Scenario The scenario for this post is a company that sells cool stuff over the internet, they have a website where people can signup and registered users can place orders.
A lot of the content in this article is similar to what I did in Part 2, so if you haven’t read it I suggest you go back and read it now :)
The next part of this series will take a look at implementing a custom UI for configuration of the RabbitMQ Source component. The main reference for this entry can be found on MSDN.
A shout out and my thanks goes to Tillmann Eitelberg (Web | Twitter) who pointed me in the right direction when I was trying to figure out how to get a list of all the connection managers in a package.
