Introduction
This text is about how you can learn and write Pandas DataFrame and CSV to and from Azure Storage Tables. The Pandas DataFrames are utilized in many Knowledge Analytics functions. Subsequently, storing it in a cloud is a repetitive activity in lots of instances. Right here we are able to see how we are able to do the identical. First, we see how you can save information in CSV file to Azure Desk Storage after which we’ll see how you can take care of the identical scenario with Pandas DataFrame.
Conditions
- Azure Account : (If not you may get a free account with ₹13,300 price of credit from right here. If you’re a scholar you possibly can confirm scholar standing to get it with out coming into bank card particulars else bank card particulars are necessary)
- Azure Storage Account: To know how you can create a storage account comply with the official documentation.
Saving a CSV file to Azure Storage Desk
We’d like a CSV module and Azure cosmosdb desk for a similar so set up the identical, It’s possible you’ll use pip set up CSV and pip set up azure-cosmosdb-table for the identical.
Learn CSV File
The beneath perform makes use of a CSV module to learn a CSV file at a specified location.
- import csv
- def readCSV(location):
- with open(location) as csv_file:
- csv_reader = csv.reader(csv_file, delimiter=‘,’)
- tab=[]
- for row in csv_reader:
- tab.append(row)
- return tab
Initialize Desk Service
- from azure.cosmosdb.desk.tableservice import TableService
- from azure.cosmosdb.desk.fashions import Entity
- table_service = TableService(account_name=‘xxxx’, account_key=‘xxxxxxxx’)
Create a Azure Storage Desk
It’s possible you’ll create tables by way of completely different strategies like by utilizing Azure Net Portal, Azure CLI, and many others,… Beneath is a solution to create a desk utilizing python script. Substitute the table-name together with your most well-liked desk title.
- table_service.create_table(table-name)
Save desk to Azure
To save lots of a desk (listing of listing) to Azure we’d like Partition Key and Row Key. We’re simply updating the identical with incremental values since it isn’t issues right here. Right here is the snippet which does the identical.
- def setTable(tablename,desk):
- table_service = TableService(account_name=‘xxxx’, account_key=xxxx’)
- index=0
- for row in desk:
- activity = {‘PartitionKey’: “P”+str(index), ‘RowKey’: “R”+str(index+1)}
- index=index+1
- for ele in row:
- activity[“Row”+str(row.index(ele))]=ele
- table_service.insert_entity(tablename, activity)
- return True
Remaining Code
- import csv
- from azure.cosmosdb.desk.tableservice import TableService
- from azure.cosmosdb.desk.fashions import Entity
- def readCSV(location):
- with open(location) as csv_file:
- csv_reader = csv.reader(csv_file, delimiter=‘,’)
- tab=[]
- for row in csv_reader:
- tab.append(row)
- return tab
- def setTable(tablename,desk,table_service):
- index=0
- for row in desk:
- activity = {‘PartitionKey’: “P”+str(index), ‘RowKey’: “R”+str(index+1)}
- index=index+1
- for ele in row:
- activity[“Row”+str(row.index(ele))]=ele
- table_service.insert_entity(tablename, activity)
- return True
- table_service = TableService(account_name=‘xxxx’, account_key=‘xxxx’)
- table_service.create_table(table-name)
- tab=readCSV(“<location>”)
- res = setTable(table-name, tab,table_service)
Retrieving information from Azure Storage Desk and saving to CSV File
Retrieving Knowledge from Azure Storage Desk
The beneath code snippet lets you retrieve the Desk:
- def getTab(tableName):
- table_service = TableService(account_name=‘xxx’, account_key=xxxx’)
- duties = table_service.query_entities(tableName)
- tab=[]
- newrow=[]
- for row in duties:
- for ele in row:
- newrow.append(row[ele])
- tab.append(newrow)
- newrow=[]
- return tab
Saving the desk as CSV File
To save lots of the desk (listing of listing) as csv file we are able to use the csv module. The beneath snippet lets you obtain the identical.
- def saveToCSV(tab, fileName):
- with open( fileName +“Output.csv”, ‘w+’, newline=”) as file:
- author = csv.author(file)
- author.writerows(tab)
Remaining Code
- import csv
- from azure.cosmosdb.desk.tableservice import TableService
- from azure.cosmosdb.desk.fashions import Entity
- def getTab(tableName):
- table_service = TableService(account_name=‘xxx’, account_key=xxxx’)
- duties = table_service.query_entities(tableName)
- tab=[]
- newrow=[]
- for row in duties:
- for ele in row:
- newrow.append(row[ele])
- tab.append(newrow)
- newrow=[]
- return tab
- def saveToCSV(tab, fileName):
- with open( fileName +“Output.csv”, ‘w+’, newline=”) as file:
- author = csv.author(file)
- author.writerows(tab)
- table_service = TableService(account_name=‘xxxx’, account_key=‘xxxx’)
- desk=getTab(table-name)
- saveToCSV(tab, fileName)
Pandas DataFrame to and from Azure Storage Desk
We all know Pandas DataFrames could be transformed to the desk (listing of listing) instantly by df.values.tolist(). We have now already mentioned how you can retailer the listing of lists to Azure Storage Desk. A pattern of the primary perform is given beneath:
- import pandas as pd
- from azure.cosmosdb.desk.tableservice import TableService
- from azure.cosmosdb.desk.fashions import Entity
-
- l = [[1 , 2 , 3],[4,5,6] , [8 , 7 , 9]]
- df = pd.DataFrame (l)
- list_of_lists = df.values.tolist()
-
- table_service = TableService(account_name=‘xxxx’, account_key=‘xxxxx’)
- table_service.create_table(table-name)
- res = setTable(table-name, list_of_lists)
Conclusion
Saving information to Azure Cloud from CSV file and pandas dataframe is mentioned on this article. That is considered one of many strategies to attain the identical. You may as well save the CSV file as such in an Azure Blob additionally. I hope you discovered this text helpful.
