Azure Data Lake Gen 1 vs Gen 2 Azure Data Lake Gen 1 vs Gen 2 azure azure

Azure Data Lake Gen 1 vs Gen 2


Basically, think of gen2 as a superset of gen1 plus all of the best parts of blob storage: tiers, HDFS and object store API's and presumably the ability to efficiently handle the management of over 35K files and efficiently dealing with many small sizes and more trickle write type operations.. plus its cheaper.

I'm trying to get some clarity on a few specifics but not finding much in the meantime try these links:

https://azure.microsoft.com/en-us/blog/a-closer-look-at-azure-data-lake-storage-gen2/

https://docs.microsoft.com/en-us/azure/storage/data-lake-storage/introduction


Azure data lake storage Gen2 is a super set of Azure data lake Gen 1. It also called as a "no-compromise data lake" by Microsoft. Gen 2 extends Azure blob storage capabilities and it is best optimized for analytics workloads. It can store data once and access via existing blob storage and HDFS-compliant file system interfaces with no programming changes or data copying when doing database operations since it supports atomic file and folder operations.
At present, it is only available in West US 2 and West Central US data centers. But it will be expanded into other data centers in the near future according to Microsoft.


There is a Microsoft doc that talks about the the differences. For Example:

Data Organization:

Gen1

  • Hierarchical namespace, File and folder support.

Gen2

  • Hierarchical namespace, container, file and folder support

Geo-redundancy:

Gen1

  • LRS.

Gen2

  • LRS, ZRS, GRS, RA-GRS.

Ecosystem:

Gen1

  • HDInsight (3.6), Azure Databricks (3.1 and above), SQL DW, ADF

Gen2

  • HDInsight (3.6, 4.0), Azure Databricks (5.1 and above), SQL DW, ADF