dask : How to read CSV files into a DataFrame from Microsoft Azure Blob

Question

S3Fs is a Pythonic file interface to S3, does DASK have any Pythonic interface to Azure Storage Blob. Python SDK's for Azure Storage Blob provide ways to read and write to blob, but the interface requires the file to be downloaded to the local machine from cloud. I am looking for solutions that which read the blob to support DASK parallel read as either stream or string without persisting to local disk.

Answer 1

I have newly pushed code here: https://github.com/dask/dask-adlfs

You may pip-install from that location, although you may be best served by conda-installing the requirements (dask, cffi, oauthlib) beforehand. In a python session, doing import dask_adlfs will be enough to register the backend with Dask, such that thereafter you can use azure URLs with dask functions like:

import dask.dataframe as dd
df = dd.read_csv('adl://mystore/path/to/*.csv', storage_options={
    tenant_id='mytenant', client_id='myclient', 
    client_secret='mysecret'})

Since this code is totally brand new and untested, expect rough edges. With luck, you can help iron out those edges.