Dajbych.net

Azure Stor­age of­fers many use­ful fea­tures and ser­vices which makes in­te­gra­tion into ex­ist­ing sys­tems eas­ier. New so­lu­tions can take ad­van­tages of those fea­tures on ar­chi­tec­tural level. Know­ing which com­mon pat­terns are na­tively sup­ported by Azure Stor­age can ac­cel­er­ate de­vel­op­ment rad­i­cally. Here is an overview of them.

Blob Storage / Blob service / Containers

Shared access tokens

Con­tainer can del­e­gate cer­tain per­mis­sions via SAS (shared ac­cess sig­na­ture) to­ken to JavaScript code (for ex­am­ple). Ev­ery to­ken is signed by a pri­vate key. There are three meth­ods of gen­er­at­ing the to­ken, each of them pro­vides per­mis­sions by a dif­fer­ent way:

• Role-based per­mis­sions of a user au­then­ti­cated by Azure Ac­tive Di­rec­tory.

• In­di­vid­ual per­mis­sions del­e­gated by user of Azure Ac­tive Di­rec­tory.

• In­di­vid­ual per­mis­sions based on one of two keys of stor­age ac­count, which can be man­u­ally re­gen­er­ated.

These to­kens can be eas­ily gen­er­ated dy­nam­i­cally by code, so it makes sense to limit their du­ra­tion for a spe­cific time. When the to­ken some­how leaks, at­tacker has a lim­ited time to per­form a ma­li­cious ac­tion.

Immutable Storage

Many pa­pers must be archived due to le­gal rea­sons. Im­mutable Stor­age is a dig­i­tal equi­va­lent of a doc­u­ment archive. The con­tainer can be locked with an Im­mutable Blob Stor­age pol­icy. Locked blob can­not be deleted, mod­i­fied, or moved. There are two kinds of locks – Time-based re­ten­tion and Le­gal hold. Time-based re­ten­tion holds the lock for a spec­i­fied pe­riod. Le­gal hold is an as­signed tag which locks con­tainer of blob.

Metadata

A blob can hold ad­di­tional key-value pairs of data. Typ­i­cal ex­am­ple is the Con­tent-Type, which is served as an HTTP header. How­ever even cus­tom meta­data are con­tained in HTTP headers with a X pre­fix. It is ex­tremely help­ful in plenty sce­nar­ios be­cause JavaScript have a ca­pa­bil­ity to read this metadata.

Index tags

Blob in­dex tags pro­vides a built-in ca­pa­bil­ity to list blobs by cus­tom at­tributes. Blob’s tag can be set dur­ing or af­ter upload. Each blob can have up to 10 in­dex tags. Ad­di­tional pric­ing is based on the monthly av­er­age num­ber of in­dex tags in the stor­age ac­count.

Hierarchical namespace

Ev­ery (gen­eral pur­pose v2) stor­age can be up­graded to Data Lake Gen2 stor­age. This mi­gra­tion al­lows us to take ad­van­tage of hi­er­ar­chi­cal names­pace. More specif­i­cally it brings us:

• Ef­fi­cient query of sub­fold­ers.

• Much faster renaming or moving blobs.

• Atomic op­er­a­tions with ABFS driver over DFS end­point.

• Gran­u­lar POSIX-com­pli­ant se­cu­rity.

Soft delete

Con­tain­ers or blobs don’t have to be al­ways per­ma­nently deleted. It is pos­si­ble to set a pe­riod which de­lays ac­tual dele­tion. Dur­ing this time those items are hid­den and can be re­stored. When this pe­riod ends, per­ma­nent delete oc­curs au­to­mat­i­cally.

Access tier

To achieve cost sav­ings, data can be dis­tributed among dif­fer­ent stor­age ac­counts with a spe­cific ac­cess tier that fits best to data na­ture. Azure cur­rently of­fers three kinds:

• Hot tier – high­est stor­age costs, low­est ac­cess cost.

• Cool tier – about 30 % less ex­pen­sive than Hot tier but write or read op­er­a­tions are more ex­pen­sive. If you delete a blob be­fore it be­comes 30 days old, you must pay Early dele­tion fee.

• Archive tier – about 95 % less ex­pen­sive than Hot tier but the de­lay be­tween data re­quest and de­liv­ery is in hours. Early dele­tion fee is ap­plied if you delete a blob be­fore it be­comes 180 days old. Re­trieve an archived blob can take up to 15 hours. If you pay a lit­tle bit of money for the pri­or­ity op­er­a­tion, it can take less than 1 hour.

Blobfuse

Blob­fuse al­lows to ac­cess block blob data in your stor­age ac­count through the Linux file sys­tem. It is a virtual file sys­tem driver for Ubuntu, De­bian, SUSE, Cen­tOS, Or­a­cle Linux and RHEL dis­tri­bu­tions.

Inventory

In­ven­tory re­ports is a tool to get an overview of all your data within a stor­age ac­count. Re­ports are cre­ated pe­ri­od­i­cally – daily or weekly. They have CSV for­mat and are au­to­mat­i­cally stored to a spe­cific con­tainer.

Snapshots

A snap­shot is a read-only copy of a blob taken at cer­tain point in time. Snap­shots, un­like ver­sions, are cre­ated man­u­ally. Snap­shots of blobs in the Archive tier are not sup­ported.

Versions

Azure Stor­age can au­to­mat­i­cally save a pre­vi­ous ver­sion ev­ery time a blob is mod­i­fied (or deleted). Pre­vi­ous ver­sions can be listed via SDK (or Azure Por­tal). Older ver­sions can be stored in dif­fer­ent ac­cess tier than cur­rent (prop­a­gated) ver­sion.

Task: Delete old blobs

This fea­ture, cur­rently in pre­view, can sim­plify many cloud so­lu­tions and save many lines of code. It deletes all blobs in a spe­cific con­tainer older than a given pe­riod.

Lifecycle management

Blobs which haven’t been mod­i­fied for a spe­cific pe­riod can be au­to­mat­i­cally deleted or moved to cool stor­age or archive stor­age. The rule ap­plies to whole stor­age or to a spe­cific sub­set (ex­clud­ing ap­pend blobs) based on blob’s name or meta­data.

Table Storage / Table service

Cosmos DB Table API

Azure Cos­mos DB is ac­ces­si­ble is the same way as tra­di­tional Ta­ble Stor­age (with newer Azure Ta­ble SDK). An en­tity in Azure Stor­age can be up to 1 MB in size. An en­tity in Azure Cos­mos DB can be up to 2 MB in size.

Queue Storage

Infinite TTL interval

The max­i­mum mes­sage life­time was al­ways 7 days. It is now pos­si­ble to opt-in for im­mor­tal mes­sage which never ex­pires.

File Shares / File Service / Azure Files

Large file shares

Max­i­mum file size 5 TB. If you ac­ti­vate Large file shares op­tion, this limit grows to 100 TB. How­ever, this ac­tion is ir­re­v­ersible. A mi­nor side ef­fect is that this stor­age can­not be geo-re­dun­dant, so it is lim­ited to a sin­gle re­gion.

Soft delete

If a mapped net­work drive is con­nected via SMB pro­to­col, deleted files can be re­stored via Azure Por­tal (or Pow­er­Shell script). Max­i­mum re­tain­ment pe­riod is 365 days. Soft delete for NFS or SFTP is sup­ported by Azure Data Lake Stor­age.

Premium performance

Ba­sic stor­age ac­counts are phys­i­cally on HDDs. Pre­mium ac­counts are lo­cated on SSDs which pro­vides much higher IOPS and much lower la­tency. Pre­mium stor­age ac­count can host pre­mium tier file share. It has cheaper trans­ac­tion cost com­pared to stan­dard tier. The IOPS and through­put is based on the pro­vi­sioned size. On the other hand, pre­mium file share does not sup­port any form of geo-re­dun­dancy.

Storage access policies

If you authorize ac­cess to Stor­age, Ta­ble or Queue via Azure Ac­tive Di­rec­tory, you can as­sign cer­tain roles to se­cu­rity prin­ci­pals (users, groups, or ap­pli­ca­tion ser­vices). A role per­mits or de­nies spe­cific ac­tions:

• Stor­age – read, add, cre­ate, write, delete, list

• Ta­ble – read, add, up­date, delete

• Queue – read, add, up­date, pro­cess