Azure

Write And Learn Pandas Dataframe And CSV To And From Azure Storage Desk

Introduction

 

This text is about how you can learn and write Pandas DataFrame and CSV to and from Azure Storage Tables. The Pandas DataFrames are utilized in many Knowledge Analytics functions. Subsequently, storing it in a cloud is a repetitive activity in lots of instances. Right here we are able to see how we are able to do the identical. First, we see how you can save information in CSV file to Azure Desk Storage after which we’ll see how you can take care of the identical scenario with Pandas DataFrame.

 

Conditions

  • Azure Account : (If not you may get a free account with ₹13,300 price of credit from right here. If you’re a scholar you possibly can confirm scholar standing to get it with out coming into bank card particulars else bank card particulars are necessary)
  • Azure Storage Account: To know how you can create a storage account comply with the official documentation.

Saving a CSV file to Azure Storage Desk

 

We’d like a CSV module and Azure cosmosdb desk for a similar so set up the identical, It’s possible you’ll use pip set up CSV and pip set up azure-cosmosdb-table for the identical.

 

Learn CSV File

 

The beneath perform makes use of a CSV module to learn a CSV file at a specified location.

  1. import csv  
  2. def readCSV(location):  
  3.    with open(location) as csv_file:  
  4.        csv_reader = csv.reader(csv_file, delimiter=‘,’)  
  5.        tab=[]  
  6.        for row in csv_reader:  
  7.            tab.append(row)  
  8.        return tab  

Initialize Desk Service

 

To initialize desk service we’d like credentials of an lively Azure Storage Account. If you do not know how you can create and get the credentials, comply with official documentation. Substitute the values of account_key and account_name.
  1. from azure.cosmosdb.desk.tableservice import TableService  
  2. from azure.cosmosdb.desk.fashions import Entity  
  3. table_service = TableService(account_name=‘xxxx’, account_key=‘xxxxxxxx’)  

Create a Azure Storage Desk

 

It’s possible you’ll create tables by way of completely different strategies like by utilizing Azure Net Portal, Azure CLI, and many others,… Beneath is a solution to create a desk utilizing python script. Substitute the table-name together with your most well-liked desk title.

  1. table_service.create_table(table-name)  

Save desk to Azure

 

To save lots of a desk (listing of listing) to Azure we’d like Partition Key and Row Key. We’re simply updating the identical with incremental values since it isn’t issues right here. Right here is the snippet which does the identical.

  1. def setTable(tablename,desk):  
  2.    table_service = TableService(account_name=‘xxxx’, account_key=xxxx’)  
  3.    index=0  
  4.    for row in desk:  
  5.        activity = {‘PartitionKey’“P”+str(index), ‘RowKey’:  “R”+str(index+1)}  
  6.        index=index+1  
  7.        for ele in row:  
  8.           activity[“Row”+str(row.index(ele))]=ele  
  9.        table_service.insert_entity(tablename, activity)  
  10.    return True   

Remaining Code

  1. import csv  
  2. from azure.cosmosdb.desk.tableservice import TableService  
  3. from azure.cosmosdb.desk.fashions import Entity  
  4. def readCSV(location):  
  5.    with open(location) as csv_file:  
  6.        csv_reader = csv.reader(csv_file, delimiter=‘,’)  
  7.        tab=[]  
  8.        for row in csv_reader:  
  9.            tab.append(row)  
  10.        return tab  
  11. def setTable(tablename,desk,table_service):  
  12.    index=0  
  13.    for row in desk:  
  14.        activity = {‘PartitionKey’“P”+str(index), ‘RowKey’:  “R”+str(index+1)}  
  15.        index=index+1  
  16.        for ele in row:  
  17.           activity[“Row”+str(row.index(ele))]=ele  
  18.        table_service.insert_entity(tablename, activity)  
  19.    return True  
  20. table_service = TableService(account_name=‘xxxx’, account_key=‘xxxx’)  
  21. table_service.create_table(table-name)  
  22. tab=readCSV(“<location>”)  
  23. res = setTable(table-name, tab,table_service)  

Retrieving information from Azure Storage Desk and saving to CSV File

 

Retrieving Knowledge from Azure Storage Desk

 

The beneath code snippet lets you retrieve the Desk:

  1. def getTab(tableName):  
  2.    table_service = TableService(account_name=‘xxx’, account_key=xxxx’)  
  3.    duties = table_service.query_entities(tableName)  
  4.    tab=[]  
  5.    newrow=[]  
  6.    for row in duties:  
  7.        for ele in row:  
  8.            newrow.append(row[ele])  
  9.        tab.append(newrow)  
  10.        newrow=[]  
  11.    return tab   

Saving the desk as CSV File

 

To save lots of the desk (listing of listing) as csv file we are able to use the csv module. The beneath snippet lets you obtain the identical.

  1. def saveToCSV(tab, fileName):  
  2.    with open( fileName +“Output.csv”‘w+’, newline=) as file:  
  3.        author = csv.author(file)  
  4.        author.writerows(tab)   

Remaining Code

  1. import csv  
  2. from azure.cosmosdb.desk.tableservice import TableService  
  3. from azure.cosmosdb.desk.fashions import Entity  
  4. def getTab(tableName):  
  5.    table_service = TableService(account_name=‘xxx’, account_key=xxxx’)  
  6.    duties = table_service.query_entities(tableName)  
  7.    tab=[]  
  8.    newrow=[]  
  9.    for row in duties:  
  10.        for ele in row:  
  11.            newrow.append(row[ele])  
  12.        tab.append(newrow)  
  13.        newrow=[]  
  14.    return tab  
  15. def saveToCSV(tab, fileName):  
  16.    with open( fileName +“Output.csv”‘w+’, newline=) as file:  
  17.        author = csv.author(file)  
  18.        author.writerows(tab)  
  19. table_service = TableService(account_name=‘xxxx’, account_key=‘xxxx’)  
  20. desk=getTab(table-name)  
  21. saveToCSV(tab, fileName)   

Pandas DataFrame to and from Azure Storage Desk

 

We all know Pandas DataFrames could be transformed to the desk (listing of listing) instantly by df.values.tolist(). We have now already mentioned how you can retailer the listing of lists to Azure Storage Desk. A pattern of the primary perform is given beneath:

  1. import pandas as pd  
  2. from azure.cosmosdb.desk.tableservice import TableService  
  3. from azure.cosmosdb.desk.fashions import Entity  
  4. ​  
  5. l = [[1 , 2 , 3],[4,5,6] , [8 , 7 , 9]]  
  6. df = pd.DataFrame (l)  
  7. list_of_lists = df.values.tolist()  
  8. ​  
  9. table_service = TableService(account_name=‘xxxx’, account_key=‘xxxxx’)  
  10. table_service.create_table(table-name)  
  11. res = setTable(table-name, list_of_lists)   

Conclusion

 

Saving information to Azure Cloud from CSV file and pandas dataframe is mentioned on this article. That is considered one of many strategies to attain the identical. You may as well save the CSV file as such in an Azure Blob additionally. I hope you discovered this text helpful.

Show More

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button