Azure

Azure Information Lake Storage Gen2 Question Acceleration

Introduction

 

These days, I used to be working to improve the structure of my IoT-based software that was primarily based on IoT Hub and Occasion Hub. I used to be dumping the machine’s telemetry knowledge into the on-premises Postgres database however the knowledge is rising extra quickly day-to-day subsequently I made a decision to maneuver my all outdated knowledge to the Azure knowledge lake Storage Gen2. However I additionally wanted the outdated knowledge in my software for some causes. So, I used to be trying to find one of the best resolution for my software to entry the precise knowledge from the Azure knowledge lake then I discovered an optimum resolution for querying particular knowledge from my information.

 

Azure Information Lake Storage Question Acceleration

 

Question acceleration is used to retrieve a subset of knowledge out of your storage account. Question acceleration helps CSV and JSON formatted knowledge as enter to every request. Question acceleration permits purposes and analytics frameworks to dramatically optimize knowledge processing by retrieving solely the info that they require to carry out a given operation.

 

Information stream

 

The next diagram illustrates how a typical software makes use of question acceleration to course of knowledge.

 

 

Conditions

  1. Create a general-purpose v2 storage account
  2. Allow question acceleration
  3. Azure Storage .Internet SDK

Highway Map

  • Azure Login
  • Allow Question Acceleration
  • Create an Azure Useful resource Group
  • Create Azure Information Lake Storage Gen2 Account
  • Create Azure Information Lake Container
  • Add JSON File
  • Create a brand new Console App in Visual Studio 2019
  • Add References to NuGet Packages
  • Question Acceleration Principal Logic

Step 1 – Login Utilizing Azure Portal OR Azure CLI

 

Log in to the Azure portal or log in via Azure CLI. I’ll present you each methods. Open your command immediate and use the next command as a way to login to Azure. Be sure you have already got put in the Azure CLI in your native.

Build CI/CD Pipeline For Azure Container Instances

 

After logging in, you may see your all lively subscriptions within the output and you want to set the subscription for this present context. To take action use the next command. I’ve a number of subscriptions and I’ll use my Azure move sponsorship for this context.

  1. az account set –subscription “Visual Studio Enterprise Subscription”  //Write your subscription title
Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Step 2 – Allow Question Acceleration

  1. az function register —namespace Microsoft.Storage –name BlobQuery

Step 3 – Create an Azure Useful resource Group Utilizing Portal and CLI

 

As you realize, we already logged in utilizing CLI. Now we’ll create a useful resource group for the Azure knowledge lake storage gen2. We are going to hold our all sources on this useful resource group that we’re creating. Use the next command as a way to create the useful resource group.

  1. az group create –name “datalake-rg” –location “centralus”
Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Step 4 – Create Azure Information Lake Storage Gen2 Account

 

Click on on create a brand new useful resource and search storage account and select the storage account from the consequence. It’s also possible to use the next instructions as a way to create the storage account utilizing Azure CLI.

  1. az storage account create –name <STORAGEACCOUNTNAME>
  2. –resource-group <RESOURCEGROUPNAME>
  3. –location eastus –sku Standard_LRS
  4. –kind StorageV2 –hierarchical-namespace true
Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Select your subscription, useful resource group, and storage account title and click on Subsequent: Networking >

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

The networking, tab these are the default chosen choices and possibly you wish to use one other one however for this demo please use the default chosen choices and hits Subsequent: Information Safety >

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

The info safety, tab these are the optionally available choice you wish to use however for this demo please additionally use the default chosen choices and hits Subsequent: Information Safety >

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

The advance, tab is the default chosen choice and possibly you wish to use one other one however for this demo please use the default chosen choices and select the allow towards the Information Lake Storage Gen2 choice, and hits Subsequent: Tags > and subsequent assessment + create.

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

After creating the azure knowledge lake gen2 account, open the newly created azure knowledge lake storage gen2 account.

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Step 5 – Create Azure Information Lake Container

 

Choose the container choice from the left navigation. Click on on the + Container button and create a brand new container.

 

 

Step 6 – Add Json File

 

Open the newly created container and add a JSON file contained in the container. I additionally hooked up the pattern Json file on the highest. You possibly can obtain and use it in your studying.

 

Step 7 – Create a brand new Console App in Visual Studio 2019

 

Open the visible studio and create a brand new venture and add a console app (.Internet Core) with C#.

 

Azure Data Lake Storage Gen2 Reading Avro Files Using .Net Core

 

Step 8 – Add References to NuGet Packages

 

Initially, in References, add the references to Azure.Storage.Blob and Newtonsoft.Json utilizing NuGet Bundle Supervisor, as proven beneath.

 

 

Step 9 – Question Acceleration Principal Logic

 

Open the Program class and exchange the next code with an acceptable namespace. Please seize your connection string out of your storage account and exchange it within the code and likewise use your container title contained in the code.

  1. utilizing Azure.Storage.Blobs;  
  2. utilizing Azure.Storage.Blobs.Fashions;  
  3. utilizing Azure.Storage.Blobs.Specialised;  
  4. utilizing Newtonsoft.Json;  
  5. utilizing System;  
  6. utilizing System.Collections.Generic;  
  7. utilizing System.Diagnostics;  
  8. utilizing System.IO;  
  9. utilizing System.Threading.Duties;  
  10.   
  11. namespace AzureDataLake.QueryAcceleration  
  12. {  
  13.     class Program  
  14.     {  
  15.         static void Principal(string[] args)  
  16.         {  
  17.             MainAsync().Wait();  
  18.         }  
  19.         non-public static async Activity MainAsync()  
  20.         {  
  21.             var connectionString = “DefaultEndpointsProtocol=https;AccountName=ahsan;AccountKey=51sarUyYuhNLCv/+w9LjcbT7914Q==;EndpointSuffix=core.home windows.internet”;  
  22.             var blobServiceClient = new BlobServiceClient(connectionString);  
  23.             var containerClient = blobServiceClient.GetBlobContainerClient(“ContainerName”);  
  24.   
  25.             await foreach (var blobItem in containerClient.GetBlobsAsync(BlobTraits.Metadata,BlobStates.None, “File Prefix”))  
  26.             {  
  27.                     var blobClient = containerClient.GetBlockBlobClient(blobItem.Title);  
  28.   
  29.                     var choices = new BlobQueryOptions  
  30.                     {  
  31.                         InputTextConfiguration = new BlobQueryJsonTextOptions(),  
  32.                         OutputTextConfiguration = new BlobQueryJsonTextOptions()  
  33.                     };  
  34.   
  35.                     var consequence = await blobClient.QueryAsync(@“SELECT * FROM BlobStorage WHERE measuringpointid = 547”, choices);  
  36.   
  37.                     var jsonString = await new StreamReader(consequence.Worth.Content material).ReadToEndAsync();  
  38.   
  39.                     Console.WriteLine(jsonString);  
  40.                     Console.ReadLine();  
  41.                 }  
  42.             }  
  43.         }  
  44.     }  

Run your console app after which you will notice the next output within the console.

 

 

Tags
Show More

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Close