Skip to content

Chris Koester

Modern data platform and engineering in Azure

Primary Navigation Menu
Menu
  • Home
  • About
  • Contact
  • Privacy Policy
Generate Big Datasets with Hive in HDInsight

Generate Big Datasets with Hive in HDInsight

2019-03-21
By: Chris Koester
On: 2019-03-21
In: Big Data
With: 0 Comments

Update 2020-05-29: Generate Big Datasets with Databricks I authored a post on the BlueGranite site on how to generate big datasets with Databricks. This is a more modern approach to generating the same TPC-DS benchmark datasets, and I’d recommend it over the HDInsight solution described below. This post describes how to generate big datasets with Hive in HDInsight, specifically TPC-DS benchmarking datasets. There are many tools for generating sample data, and this one is particularly nice due to its familiarity and ability to generate massive datasets up to 100 terabytes in size. The intended purpose of TPC data is for benchmarking purposes, but big sampleRead More →

Delete SSAS Tabular Partitions with C#

Delete SSAS Tabular Partitions with C#

2018-05-14
By: Chris Koester
On: 2018-05-14
In: SSAS Tabular
With: 1 Comment

This post shows how to delete SSAS Tabular partitions with C#. This is an improvement over a PowerShell script I previously wrote about, Delete SSAS Partitions with TMSL and PowerShell, because it is more dynamic and doesn’t require a manually-authored TMSL script. Specifically, the code below deletes all partitions that contain a specified suffix, which is useful for quickly deleting all partitions used for development purposes. I’ve worked with SSAS models that required several development partitions in order to prevent the workspace database from growing too large. Deleting them manually after a deployment to a server would be tedious, and it would be easy to forgetRead More →

Retrieve JSON Data from SQL Server using a Stored Procedure and C#

Retrieve JSON Data from SQL Server using a Stored Procedure and C#

2018-03-25
By: Chris Koester
On: 2018-03-25
In: Data Integration
With: 4 Comments

The previous post demonstrated how to load JSON into SQL Server using a stored procedure and C#. This post will examine the opposite – how to retrieve JSON data from SQL Server using a stored procedure and C#. Just like with the JSON loading technique, the new JSON functions in SQL Server 2016 allow for this retrieval technique. Here’s how it works in short: Data stored in a database table will be converted to JSON text in the stored procedure and returned to the C# client via an output parameter. SQL Server Table Let’s start with the SQL Server table, which will contain one rowRead More →

Load JSON Into SQL Server Using a Stored Procedure and C#

Load JSON into SQL Server Using a Stored Procedure and C#

2018-03-21
By: Chris Koester
On: 2018-03-21
In: Data Integration
With: 3 Comments

This post will demonstrate how to load JSON into SQL Server using a stored procedure and C#. This technique is possible with the new JSON functions starting in SQL Server 2016, and is nice for the following reasons: Simplifies application code. Just serialize the C# objects to JSON and send them to the database. Executing a stored procedure with C# is easily done with a small amount of code. JSON is widely used and familiar to developers. Structural changes to the data could potentially be done entirely on the database side, meaning that you could make changes without having to touch the application code. SQLRead More →

Push Performance Counter Data into a Power BI Streaming Dataset

Push Performance Counter Data into a Power BI Streaming Dataset

2017-12-27
By: Chris Koester
On: 2017-12-27
In: Power BI, SSAS Tabular
With: 0 Comments

In the previous post I showed how you can Push Data into Power BI Streaming Datasets with C#. That example used dummy data. In this post I’ll show how to push performance counter data into a Power BI Streaming Dataset as a real world example. This scenario allows for monitoring a computer or application in near real time in the browser. I won’t go through the steps of creating a Power BI Streaming Dataset. You can reference my previous post if you need instructions. I will note that the value names that you choose in the Streaming Dataset must match the C# property names for theRead More →

Power BI Streaming Datasets with C#

Push Data into Power BI Streaming Datasets with C#

2017-11-05
By: Chris Koester
On: 2017-11-05
In: Power BI
With: 5 Comments

This post will demonstrate how to push data into Power BI Streaming Datasets with C#. For demo purposes I normally use LINQPad to run the code, but you could also create a .Net or .Net Core console application. LINQPad is an excellent, lightweight scratchpad for C# and other .Net languages. Power BI Streaming Datasets are a very cool feature because dashboard tiles that use them update in real time. You don’t have to refresh the browser window to display new data. With this feature you can watch your data in near real-time. This could be compelling in scenarios involving sensors, IoT, website traffic, etc. Here isRead More →

Ludicrous Speed

Optimized Multi-value DAX Parameters in SSRS

2017-10-18
By: Chris Koester
On: 2017-10-18
In: Reporting Services, SSAS Tabular
With: 0 Comments

This post shows how you can generate optimized multi-value DAX parameters in SSRS and achieve greater performance compared to the DAX PathContains function. This will be a short post that provides the SSRS expression to convert multiple SSRS parameters into a double-pipe delimited string for use in a DAX query. In other words, the goal is to use the DAX OR operator (||) instead of the PathContains function. I’m assuming the reader has experience with SSRS, so not all steps will be shown. Imagine that you have multi-value parameter that allows users to select multiple month – year values such as Sep 2017, Oct 2017, etc. TheRead More →

Convert TMSL JSON to a Table with OPENJSON

2017-08-07
By: Chris Koester
On: 2017-08-07
In: SSAS Tabular
With: 1 Comment

The previous post looked at how to process SSAS Tabular models with TMSL. Since SQL Server adds new JSON capabilities in 2016, let’s look at how to convert TMSL JSON to a Table with OPENJSON. OPENJSON is a new function in SQL Server 2016 that, per Microsoft: OPENJSON is a table-valued function that parses JSON text and returns objects and properties from the JSON input as rows and columns. In short, OPENJSON converts JSON text to a table. Since TMSL is JSON, this function can convert a SSAS Tabular processing script into a table. This could be useful if you wanted to document a SSASRead More →

Process SSAS Tabular Tables and Partitions with TMSL

Process SSAS Tabular Tables and Partitions with TMSL

2017-06-21
By: Chris Koester
On: 2017-06-21
In: SSAS Tabular
With: 20 Comments

This post shows how to process SSAS Tabular tables and partitions with TMSL. TMSL stands for Tabular Model Scripting Language, and it is used for working with SSAS Tabular databases programmatically starting with compatibility level 1200 (SQL Server 2016). Multidimensional models and previous versions of SQL Server all use Analysis Services Scripting Language (ASSL for XMLA). A common use for TMSL is automating the processing of tabular models using the SQL Agent, SSIS, PowerShell, or some other application. One typical processing sequence is a series of Process Data commands followed by a Process Recalc at the end. Here’s what the TMSL code looks like in this scenario:

JavaScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
{
  "sequence": {
    "operations": [
      {
        "refresh": {
          "type": "dataOnly",
          "objects": [
            {
              "database": "DatabaseName",
              "table": "Table1Name",
              "partition": "Table1Partition"
            },
            {
              "database": "DatabaseName",
              "table": "Table2Name",
              "partition": "Table2Partition"
            },
            {
              "database": "DatabaseName",
              "table": "Table3Name"
            },
            {
              "database": "DatabaseName",
              "table": "Table4Name"
            }
          ]
        }
      },
      {
        "refresh": {
          "type": "calculate",
          "objects": [
            {
              "database": "DatabaseName"
            }
          ]
        }
      }
    ]
  }
}

Read More →

Download JSON Data with SSIS

Download JSON Data with SSIS

2017-06-06
By: Chris Koester
On: 2017-06-06
In: Data Integration
With: 15 Comments

In a post on the Allegient blog, I showed how you can download JSON data with PowerShell (Link no longer available due to company purchase and reorganization). Now we’ll take a look at how to download JSON data with SSIS. As a dedicated data integration tool, SSIS has many advantages over PowerShell for this type of work. SSIS does not have a built in source component for JSON data, so you’ll have to use the Script Component. When you drag the Script Component into a data flow, you’ll be asked how you want to use it. In this case you’ll use it as a source. DefineRead More →

Posts navigation

1 2 Next

Follow Me

  • Twitter

Topics

Analysis Services API Azure Azure Blob Storage Azure Data Lake Store Azure Functions Azure Storage big data C# code CSV Data integration DAX Excel HDInsight Hive JSON M MDX OPENJSON ORC Parameters Power BI Power Map Power Query PowerShell REGEX Reporting Services REST SQL Server SSAS SSAS Tabular SSIS SSRS Stored Procedure Streaming text TMSL TOM TPC TPC-DS Twitter usgs VB xml

Certifications

MCSA: SQL 2016 Business Intelligence Development

Recent Posts

  • Generate Big Datasets with Hive in HDInsight
  • Delete SSAS Tabular Partitions with C#
  • Retrieve JSON Data from SQL Server using a Stored Procedure and C#
  • Load JSON into SQL Server Using a Stored Procedure and C#
  • Push Performance Counter Data into a Power BI Streaming Dataset

Categories

  • Big Data
  • Data Integration
  • Power BI
  • Power Query
  • PowerShell
  • Reporting Services
  • SSAS Tabular

Archives

  • March 2019
  • May 2018
  • March 2018
  • December 2017
  • November 2017
  • October 2017
  • August 2017
  • June 2017
  • March 2017
  • January 2017
  • May 2016
  • April 2016
  • December 2015
  • July 2015
  • August 2014

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Privacy Policy Designed using Responsive Brix. Powered by WordPress.