Versions Compared

Version Old Version 7 New Version 8
Changes made by

thierry sinassamy

thierry sinassamy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


Choisir la technologie de transfert de données vers Azure ou à partir de Azure

  1. Criteria

  • Need to transfer large amounts of data with a time-consuming Internet connection: physical transfer (Azure Data Box).

  • Need to transfer large amounts of data with a moderate or fast Internet connection:

    • if moderate Internet speed: physical transfer (Azure Data Box).

    • if high-speed (from 1 Gbps): AzCopy, Azure Data Box (Virtual Version), Azure Data Factory, Azure Storage REST Apis (SDKs).

  • Need to orchestrate any quantity: Azure Data Factory.

  • Need to log whatever the quantity: Azure Data Factory, Azure Storage REST APIs.

  • Need to transfer small amounts of data with a slow to moderate Internet connection:

    • Without programming: GUI (Azure Storage Explorer Utility), Azure Portal, SFTP Client.

    • With programming: AzCopy/PowerShell, Azure CLI, Azure Storage REST APIs (Azure Functions, Applications).

  • Regular or continuous transfer requirements:

    • Regular interval: AzCopy, Azure Storage APIs.

    • Continuous interval: Azure Data Factory, Azure Data Box for online transfer

...

Technology

Limitations

SFTP Client

Authentication & Authorization

  • Microsoft Entra ID (Azure Active Directory Identity) not supported for SFTP endpoint.

  • ACLs (Access Control lists) not supported for SFTP endpoint.

  • Local user is the only form of identity management supported.
    Network

  • Network must open port 22

  • Static IP addresses are not supported.

  • Must use Microsoft Internet Routing as Internet routing is not supported.

  • 2-minute timeout for standby or inactive connections.
    Performance

  • Performance degradation over time, impact of network latency.
    Other

  • Constraint on use of account hierarchical namespace functionality (must be enabled, requires Azure Data Lake Gen2 capabilities).

  • Maximum file size is 100 GB.

  • To launch storage redundancy/replication, SFTP must be deactivated.

AzCopy Utility

  • Synchronous operation only, no asynchronous copying.

  • Potential timeout, dependency on on-premise infrastructure, dependency on our network.

  • Impact of log generation on performance.

  • Unexpected behavior: if file size reaches 200 GB, AzCopy splits the exported file.

  • No traceability, just a technical log.

Azure Data Factory

  • Resource limitations - but not really, as it's designed for large quantities of data.

  • If we develop pipelines and there is a lot of transformation that is done outside ADF's native activities, this will make it more difficult to estimate the cost of the operation at the end.

  • Long-term operations are more expensive than on-premise solutions.

Azure Function

  • Not suitable for long, resource-intensive runs.

  • Cold-start latency means you have to pay more for the plan.

  • Despite stateless architecture, if data persistence is required, then the architecture becomes more complex as data is fetched across services.

  • Limited debugging

Initial/subsequent document loading process

Option A : Azure Data Factory with Copy Data Tool (built-in service)

...

1- Configure the self-hosted integration runtime with the on-premise server containing the documents.

2- Azure Data Factory invokes its native copy task with the Copy Data Tool.

3 & 4- Azure Data Factory extracts and copies the documents into the Storage Account's Blob Storage.

5- From the Azure Storage Explorer application interface, visual validation is possible.