CloudStorage Tasks

Properties   Configuration Settings  

The CloudStorage Task allows easy uploading and downloading to major cloud service providers.

Remarks

The CloudStorage Task provides a simple way to upload and download files from a variety of cloud storage providers. Supported services are:

  • Amazon S3
  • Azure Blob
  • Azure File
  • Backblaze B2
  • Box.com
  • Digital Ocean Spaces
  • Dropbox
  • Google Cloud Storage
  • Google Drive
  • Hadoop DFS
  • IBM Cloud Object Storage
  • Linode Object Storage
  • Microsoft OneDrive
  • Wasabi
To begin, the adapter must first be configured to allow access to the specified ServiceProvider. Depending on the provider specified the credentials may either be specified or obtained using OAuth.

Authorization

The following providers use OAuth 2.0 for authorization:

  • Box.com
  • Dropbox
  • Google Cloud Storage*
  • Google Drive
  • Microsoft OneDrive

To perform OAuth authorization first open the OAuth Authorization dialog by clicking the ellipses (...) button of the OAuthAuthorization property at design time. This will open a dialog. Within the dialog specify the ClientId, ClientSecret, and CallbackURL values for the registered application.

After specifying the necessary values press the Authorize button to authorize the application. This will launch a browser and prompt the user to grant access. After access is granted the task will populate the following values:

All necessary values will be saved in the CacheLocation for use in subsequent requests by the task. The task will use the cached values and will automatically refresh the token if needed. In this way any user interaction is only required at design time, not at runtime.

To authenticate with the the other providers, set the AccountInfo property values shown in the table below:

Provider Relevant fields
Amazon S3 AccessKey and SecretKey
Azure Blob Account and AccessKey
Azure File Account and AccessKey
Backblaze B2 ApplicationKeyId and ApplicationKey
Digital Ocean Spaces AccessKey and SecretKey
Google Cloud Storage* ProjectId (if using OAuth authentication); or, AccessKey and SecretKey
Hadoop DFS Multiple HTTP authentication schemes are supported; refer to AuthMechanism for more information. HadoopURL must always be set.
IBM Cloud Object Storage Account and AccessKey
Linode Object Storage Account and AccessKey
Wasabi AccessKey and SecretKey

*Google Cloud Storage supports two authentication methods. To authenticate with OAuth 2.0 (preferred) follow the instructions for OAuth, above, and set the ProjectId field of the AccountInfo property. To authenticate with an access and secret key, set the AccessKey and SecretKey fields of the AccountInfo property instead.

CloudStorage Receive Task

At runtime the task will connect to the cloud storage provider and list the directory specified by RemotePath. All files that match the FileMask will be downloaded and submitted to the SSIS Package as individual messages.

In order to guarantee that files are not downloaded multiple times, you may set the DeleteMode to remove files from the server that have been downloaded. Alternatively DownloadCacheFile may be used to keep a list of files which have already been downloaded.

CloudStorage Send Task

The task will connect to the cloud storage provider whenever it has a message to send. The task will attempt to upload to the specified RemotePath. If Overwrite is set to True, the task will delete or overwrite remote files as necessary to complete the operation.

Receiver Property List


The following is the full list of the properties of the receiver task with short descriptions. Click on the links for further details.

AccountInfoAccount credentials.
DirListString denoting the name of the variable which contains the last directory listing entries.
EncryptionAlgorithmThe encryption algorithm.
EncryptionPasswordThe encryption password.
FileMaskIndicates the type of files to download from the server.
FirewallA set of properties related to firewall access.
HTTPProxyA set of properties related to proxy access.
LocalDirectoryThe directory on the local machine to write downloaded files to.
LogFileThe file to write logging information to at runtime.
LogModeWhat information gets logged during task execution.
OAuthAuthorizationOAuth Authorization Information.
OperationThe operation which will be executed by the task.
OtherDefines a set of configuration settings to be used by the task.
OverwriteWhether or not the task should overwrite files during transfer.
RemotePathThe current path on the server.
RuntimeLicenseSpecifies the task runtime license key.
ServiceProviderThe cloud storage service provider.
SSLAcceptServerCertInstructs the task to unconditionally accept the server certificate that matches the supplied certificate.
SSLAcceptServerCertAcceptAnyTells the task to accept any server certificate.
SSLCertThe certificate to use for client authentication during the SSL handshake.
TimeoutA timeout for the task.

Sender Property List


The following is the full list of the properties of the sender task with short descriptions. Click on the links for further details.

AccountInfoAccount credentials.
EncryptionAlgorithmThe encryption algorithm.
EncryptionPasswordThe encryption password.
FirewallA set of properties related to firewall access.
HTTPProxyA set of properties related to proxy access.
LocalFileThe path to a local file for upload.
LogFileThe file to write logging information to at runtime.
LogModeWhat information gets logged during task execution.
OAuthAuthorizationOAuth Authorization Information.
OtherDefines a set of configuration settings to be used by the task.
OverwriteWhether or not the task should overwrite files during transfer.
RemoteFileThe name of the remote file.
RemotePathThe current path on the server.
RuntimeLicenseSpecifies the task runtime license key.
ServiceProviderThe cloud storage service provider.
SSLAcceptServerCertInstructs the task to unconditionally accept the server certificate that matches the supplied certificate.
SSLAcceptServerCertAcceptAnyTells the task to accept any server certificate.
SSLCertThe certificate to use for client authentication during the SSL handshake.
TimeoutA timeout for the task.

Configuration Settings


The following is a list of configuration settings for the task with short descriptions. Click on the links for further details.

DateFormatThe date format of the file time.
DocsExportFormatThe file format to export Google Docs files as when downloading.
DownloadCacheFileWhen set, only new files will be downloaded from the server.
DownloadCacheFileDurationThe number of minutes that a file name will remain in the download cache.
DrawingsExportFormatThe file format to export Google Drawings files as when downloading.
EnableFileDetailsComparisonWhether the receive task should consider metadata differences when deciding whether to download a file again.
LocalFileThe name of the LocalFile to which the remote file will be saved.
RegionThe region the to make requests against when using an S3-like service provider.
SheetsExportFormatThe file format to export Google Sheets files as when downloading.
SlidesExportFormatThe file format to export Google Slides files as when downloading.
UseVirtualHostingThe URL style to use when making requests with an S3-like service provider.
ReuseSSLSessionDetermines if the SSL session is reused.
SSLCipherStrengthThe minimum cipher strength used for bulk encryption.
SSLEnabledCipherSuitesThe cipher suite to be used in an SSL negotiation.
SSLEnabledProtocolsUsed to enable/disable the supported security protocols.
SSLIncludeCertChainWhether the entire certificate chain is included in the SSLServerAuthentication event.
SSLSecurityFlagsFlags that control certificate verification.
TLS12SignatureAlgorithmsDefines the allowed TLS 1.2 signature algorithms when UseInternalSecurityAPI is True.
TLS13SignatureAlgorithmsThe allowed certificate signature algorithms.
TLSNamedGroupsThe supported (EC)DHE groups.

 
 
Copyright (c) 2021 /n software inc. - All rights reserved.
/n software Tasks for SSIS - Version 20.0 [Build 7723]