CloudStorage Task

Properties   Config Settings  

The CloudStorage Component allows easy uploading and downloading to major cloud service providers.

Remarks

The CloudStorage Task provides a simple way to upload and download files from a variety of cloud storage providers. Supported services are:

• Amazon S3
• Azure Blob
• Azure File
• Backblaze B2
• Box.com
• Digital Ocean Spaces
• Dropbox
• Google Cloud Storage
• Google Drive
• Hadoop DFS
• IBM Cloud Object Storage
• Linode Object Storage
• Microsoft OneDrive
• Wasabi

To begin, the adapter must first be configured to allow access to the specified ServiceProvider. Depending on the provider specified the credentials may either be specified or obtained using OAuth.

Authorization

The following providers use OAuth 2.0 for authorization:

• Box.com
• Dropbox
• Google Cloud Storage*
• Google Drive
• Microsoft OneDrive

To perform OAuth authorization first open the OAuth Authorization dialog by clicking the ellipses (...) button of the OAuthAuthorization property at design time. This will open a dialog. Within the dialog specify the ClientId, ClientSecret, and CallbackURL values for the registered application.

After specifying the necessary values press the Authorize button to authorize the application. This will launch a browser and prompt the user to grant access. After access is granted the task will populate the following values:

• AuthorizationString
• RefreshToken
• ExpiresIn
• TimeStamp

All necessary values will be saved in the CacheLocation for use in subsequent requests by the task. The task will use the cached values and will automatically refresh the token if needed. In this way any user interaction is only required at design time, not at runtime.

To authenticate with the the other providers, set the AccountInfo property values shown in the table below:

*Google Cloud Storage supports two authentication methods. To authenticate with OAuth 2.0 (preferred) follow the instructions for OAuth, above, and set the ProjectId field of the AccountInfo property. To authenticate with an access and secret key, set the AccessKey and SecretKey fields of the AccountInfo property instead.

CloudStorage Receive Task

At runtime the task will connect to the cloud storage provider and list the directory specified by RemotePath. All files that match the FileMask will be downloaded and submitted to the SSIS Package as individual messages.

In order to guarantee that files are not downloaded multiple times, you may set the DeleteMode to remove files from the server that have been downloaded. Alternatively DownloadCacheFile may be used to keep a list of files which have already been downloaded.

CloudStorage Send Task

The task will connect to the cloud storage provider whenever it has a message to send. The task will attempt to upload to the specified RemotePath. If Overwrite is set to True, the task will delete or overwrite remote files as necessary to complete the operation.

Receiver Property List

---

The following is the full list of the properties of the receiver task with short descriptions. Click on the links for further details.

AccountInfo	Account credentials.
DirList	String denoting the name of the variable which contains the last directory listing entries.
EncryptionAlgorithm	The encryption algorithm.
EncryptionPassword	The encryption password.
FileMask	Indicates the type of files to download from the server.
Firewall	A set of properties related to firewall access.
HTTPProxy	A set of properties related to proxy access.
LocalDirectory	The directory on the local machine to write downloaded files to.
LogFile	The file to write logging information to at runtime.
LogMode	What information gets logged during component execution.
OAuthAuthorization	OAuth Authorization Information.
Operation	The operation which will be executed by the component.
Other	Defines a set of configuration settings to be used by the component.
Overwrite	Whether or not the component should overwrite files during transfer.
RemotePath	The current path on the server.
RuntimeLicense	Specifies the component runtime license key.
ServiceProvider	The cloud storage service provider.
SSLAcceptServerCert	Instructs the component to unconditionally accept the server certificate that matches the supplied certificate.
SSLAcceptServerCertAcceptAny	Tells the component to accept any server certificate.
SSLCert	The certificate to use for client authentication during the SSL handshake.
Timeout	A timeout for the component.

Sender Property List

---

The following is the full list of the properties of the sender task with short descriptions. Click on the links for further details.

AccountInfo	Account credentials.
EncryptionAlgorithm	The encryption algorithm.
EncryptionPassword	The encryption password.
Firewall	A set of properties related to firewall access.
HTTPProxy	A set of properties related to proxy access.
LocalFile	The path to a local file for upload.
LogFile	The file to write logging information to at runtime.
LogMode	What information gets logged during component execution.
OAuthAuthorization	OAuth Authorization Information.
Other	Defines a set of configuration settings to be used by the component.
Overwrite	Whether or not the component should overwrite files during transfer.
RemoteFile	The name of the remote file.
RemotePath	The current path on the server.
RuntimeLicense	Specifies the component runtime license key.
ServiceProvider	The cloud storage service provider.
SSLAcceptServerCert	Instructs the component to unconditionally accept the server certificate that matches the supplied certificate.
SSLAcceptServerCertAcceptAny	Tells the component to accept any server certificate.
SSLCert	The certificate to use for client authentication during the SSL handshake.
Timeout	A timeout for the component.

Config Settings

---

The following is a list of config settings for the task with short descriptions. Click on the links for further details.

DateFormat	The date format of the file time.
DocsExportFormat	The file format to export Google Docs files as when downloading.
DownloadCacheFile	When set, only new files will be downloaded from the server.
DownloadCacheFileDuration	The number of minutes that a file name will remain in the download cache.
DrawingsExportFormat	The file format to export Google Drawings files as when downloading.
EnableFileDetailsComparison	Whether the receive component should consider metadata differences when deciding whether to download a file again.
IgnoreEmptyFiles	Whether to ignore empty files when downloading.
LocalFile	The name of the LocalFile to which the remote file will be saved.
Region	The region the to make requests against when using an S3-like service provider.
SheetsExportFormat	The file format to export Google Sheets files as when downloading.
SlidesExportFormat	The file format to export Google Slides files as when downloading.
UseVirtualHosting	The URL style to use when making requests with an S3-like service provider.
ReuseSSLSession	Determines if the SSL session is reused.
SSLCipherStrength	The minimum cipher strength used for bulk encryption.
SSLEnabledCipherSuites	The cipher suite to be used in an SSL negotiation.
SSLEnabledProtocols	Used to enable/disable the supported security protocols.
SSLIncludeCertChain	Whether the entire certificate chain is included in the SSLServerAuthentication event.
SSLSecurityFlags	Flags that control certificate verification.
TLS12SignatureAlgorithms	Defines the allowed TLS 1.2 signature algorithms when UseInternalSecurityAPI is True.
TLS12SupportedGroups	The supported groups for ECC.
TLS13KeyShareGroups	The groups for which to pregenerate key shares.
TLS13SignatureAlgorithms	The allowed certificate signature algorithms.
TLS13SupportedGroups	The supported groups for (EC)DHE key exchange.

AccountInfo Property (CloudStorage Task)

Account credentials.

Data Type

CloudFilesAccount

Remarks

This property holds account credentials for all cloud storage service providers that do not use OAuth for authentication (or that do use OAuth, but require additional configuration). This includes the following fields:

• AccessKey
• Account
• ApplicationKey
• ApplicationKeyId
• AuthMechanism
• HadoopURL
• Password
• ProjectId
• SecretKey
• User

DirList Property (CloudStorage Task)

String denoting the name of the variable which contains the last directory listing entries.

Data Type

String

Default Value

"<none>"

Remarks

This property denotes a variable name which contains the collection of directory listings returned from the server when the Operation property has been set to opListDirectory. The collection is made up of entries for each listing in the current directory, specified by the RemotePath property.

The variable is in the form of a datatable. The table's rows represent a directory listing, and the columns represent a different property of the listing. The columns of the datatable are listed below:

NOTE: If this property is set to a valid ADO object when the Operation property is set to ReceiveFiles, the task will populate this table with a directory listing of all the files matching the FileMask on the server. The columns of this datatable are listed below:

This property is read-only.

This property is not available in the Sender.

EncryptionAlgorithm Property (CloudStorage Task)

The encryption algorithm.

Data Type

Enumeration

Possible Values

AES (0)
Blowfish (1)
CAST (2)
DES (3)
IDEA (4)
RC2 (5)
RC4 (6)
TEA (7)
Triple DES (8)
Twofish (9)
Rijndael (10)
Cha Cha (11)
XSalsa 20 (12)

Default Value

0

Remarks

This property specifies the encryption algorithm to be used. The maximum allowable key size is automatically used for the selected algorithm. Possible values are:

EncryptionPassword Property (CloudStorage Task)

The encryption password.

Data Type

Password

Default Value

""

Remarks

When this property is specified the task will attempt to encrypt or decrypt the file when uploading or downloading.

The task uses the value specified here to generate the necessary encryption Key and IV values using the PKCS5 password digest algorithm. This provides a simpler alternative to creating and managed Key and IV values directly.

FileMask Property (CloudStorage Task)

Indicates the type of files to download from the server.

Data Type

String

Default Value

"*.*"

Remarks

This property will limit what kinds of files the task will download from the server. Only files matching the pattern specified in FileMask will be retrieved.

The following special characters are supported for pattern matching:

If these characters need to be used as a literal in a pattern, then they must be escaped by surrounding them with brackets []. Note: "]" and "-" do not need to be escaped. See below for the escape sequences:

For example, to match the value [Something].txt, specify the pattern [[]Something].txt.

This property is not available in the Sender.

Firewall Property (CloudStorage Task)

A set of properties related to firewall access.

Data Type

Firewall

Remarks

This is a Firewall type property which contains fields describing the firewall through which the task will attempt to connect.

HTTPProxy Property (CloudStorage Task)

A set of properties related to proxy access.

Data Type

Proxy

Remarks

This is a Proxy type property which contains fields describing the proxy through which the task will attempt to connect.

LocalDirectory Property (CloudStorage Task)

The directory on the local machine to write downloaded files to.

Data Type

String

Default Value

""

Remarks

This property tells the task where to write all downloaded files to on the local machine. The downloaded files will be written to the remote file name in the specified directory.

This property is not available in the Sender.

LocalFile Property (CloudStorage Task)

The path to a local file for upload.

Data Type

String

Default Value

""

Remarks

The LocalFile property is the name of the file or a file mask on the local host to be uploaded to the FTP server. For instance "C:\files\*.txt".

This property is not available in the Receiver.

LogFile Property (CloudStorage Task)

The file to write logging information to at runtime.

Data Type

String

Default Value

""

Remarks

To write logging information to a file instead of using the task's logging API, set this property to a valid file on disk.

When set, any logging information is appended to the specified file and not written to the task's logging API.

LogMode Property (CloudStorage Task)

What information gets logged during component execution.

Data Type

Enumeration

Possible Values

Verbose (0)
Info (1)
Warning (2)
Error (3)
Fatal (4)

Default Value

3

Remarks

This property controls what information the task logs. The possible values have the following affect on the task's behavior:

OAuthAuthorization Property (CloudStorage Task)

OAuth Authorization Information.

Data Type

OAuthAuthorizationParam

Remarks

This property holds details about the OAuth authorization. This includes the following fields:

• AuthorizationScopes
• AuthorizationString
• CacheLocation
• CallbackURL
• ClientId
• ClientSecret
• ExpiresIn
• RefreshToken
• ServerAuthURL
• ServerTokenURL
• ServiceProvider
• TimeStamp

Operation Property (CloudStorage Task)

The operation which will be executed by the component.

Data Type

Enumeration

Possible Values

Receive Files (0)
List Directory (1)
Delete Files (2)

Default Value

0

Remarks

This property specifies the operation to be performed. Possible values are:

• opReceiveFiles (default)
• opListDirectory
• opDeleteFiles

When set to opListDirectory DirList defines the package variable which will be populated with the results.

This property is not available in the Sender.

Other Property (CloudStorage Task)

Defines a set of configuration settings to be used by the component.

Data Type

String

Default Value

""

Remarks

The task accepts one or more configuration settings. Configuration settings are similar in functionality to properties, but they are rarely used. In order to avoid "polluting" the property namespace of the task, access to these internal properties is provided through the Other property.

The Other property may be set to one or more configuration settings (name/value pairs). Set one setting per line. For example: configname1=value1 configname2=value2

Overwrite Property (CloudStorage Task)

Whether or not the component should overwrite files during transfer.

Data Type

Boolean

Default Value

false

Remarks

This property is a value indicating whether or not the task should overwrite files on the server or local system if they already exist. If Overwrite is false, an error will be thrown whenever RemoteFile exists before an upload operation, or the file exists in LocalDirectory before a download operation.

RemoteFile Property (CloudStorage Task)

The name of the remote file.

Data Type

String

Default Value

"%SourceFileName%"

Remarks

This property specifies the remote filename. This is either an absolute path, or a relative path based on RemotePath. For instance:

Filename only: file.txt

Relative Path: subfolder/file.txt

Absolute Path: /MyFolder/subfolder/file.txt

This property is not available in the Receiver.

RemotePath Property (CloudStorage Task)

The current path on the server.

Data Type

String

Default Value

""

Remarks

This property specifies the path on the remote server. This may be set to an absolute or relative path.

If the path begins with a "/" it is considered an absolute path.

If the first two characters of the path are ".." this indicates a change to the parent directory.

RuntimeLicense Property (CloudStorage Task)

Specifies the component runtime license key.

Data Type

String

Default Value

""

Remarks

You can use the RuntimeLicense property to set the runtime key for the task license.

ServiceProvider Property (CloudStorage Task)

The cloud storage service provider.

Data Type

Enumeration

Possible Values

Amazon S3 (0)
Box (1)
Dropbox (2)
Google Drive (3)
One Drive (4)
Azure Blob (6)
Wasabi (7)
Digital Ocean (8)
Google Storage (9)
Hadoop DFS (10)
Azure File (11)
Backblaze B2 (12)
IBMStorage (13)
Linode (14)

Default Value

0

Remarks

This property specifies the cloud storage provider to which requests will be made. Possible values are:

• spAmazonS3 (0) (default): Amazon S3
• spBox (1): Box.com
• spDropbox (2): Dropbox
• spGoogleDrive (3): Google Drive
• spOneDrive (4): Microsoft OneDrive
• spAzureBlob (6): Azure Blob
• spWasabi (7): Wasabi
• spDigitalOcean (8): Digital Ocean Spaces
• spGoogleStorage (9): Google Cloud Storage
• spHadoopDFS (10): Hadoop DFS
• spAzureFile (11): Azure File
• spBackblazeB2 (12): Backblaze B2
• spIBMStorage (13): IBM Cloud Object Storage
• spLinode (14): Linode Object Storage

SSLAcceptServerCert Property (CloudStorage Task)

Instructs the component to unconditionally accept the server certificate that matches the supplied certificate.

Data Type

Certificate

Remarks

If it finds any issues with the certificate presented by the server, the task will normally terminate the connection with an error. You may override this behavior by supplying a value for SSLAcceptServerCert. If the certificate supplied in SSLAcceptServerCert is the same as the certificate presented by the server, then the server certificate is accepted unconditionally, and the connection will continue normally.

This property is used to set a Public Key Certificate.

Ordinarily, the system will attempt to locate the public key in trusted certificate stores in the system registry. If a match is found, the certificate is trusted, and the process proceeds without error. If the certificate is not trusted, the task will report an error.

You may explicitly set a public key, either to accept a key that is not installed on the system, or to ensure that a specific key is presented. Public key certificates may be loaded from the machine registry or from file. If you click on the ellipses, a certificate selection dialog will open. To select a public key from the system registry, select the System Store tab or the User Store tab, and highlight the appropriate registry store. The list of available certificates will be shown below.

Alternatively, you may specify a public key on file. Select the PEM or Certificate File tabs to load a public key from file. You can use the browse button to examine the file system for the certificate that you wish to use. Public keys typically are stored in .CER, .DER, or .PEM files, but other formats may be used. You can click on the Open button to examine the contents of the certificate. Or, if you have a certificate stored in memory, you can write the contents of the certificate into the text box provided. Note that if you are loading a certificate from memory, the certificate must be Base-64 encoded, or the string representation of the certificate may become corrupted and uninterpretable.

Regardless of how you selected the certificate, once you hit the OK button, the task will attempt to verify that certificate selection. If successful, the subject of the certificate will be displayed in the property field. If the task was not able to verify the selection, a dialog box will appear instead detailing the verification error.

Note: You may also set SSLAcceptServerCertAcceptAny to True without opening the certificate selection dialog to force the task to unilaterally authenticate any server during the security handshake. It is strongly recommended that you use this only for testing purposes.

SSLAcceptServerCertAcceptAny Property (CloudStorage Task)

Tells the component to accept any server certificate.

Data Type

Boolean

Default Value

false

Remarks

When set to true, the task will unilaterally authenticate any server during the security handshake. It is strongly recommended that you use this only for testing purposes.

SSLCert Property (CloudStorage Task)

The certificate to use for client authentication during the SSL handshake.

Data Type

Certificate

Remarks

This property is used to assign a specific certificate for SSL client authentication.

This field is used to set a Private Key Certificate.

Private key certificates may be loaded from the registry, from files in PKCS#12 format, or from a PEM file format. If you click on the ellipses, a certificate selection dialog will open. To select a private key from the system registry, select the System Store tab or the User Store tab, and highlight the appropriate registry store. The list of certificates which have private keys will be shown below.

To select a private key certificate from a file in PKCS#12 format, select the PFX Store tab. The browse button can be used to examine the file system for PKCS#12 (.pfx or .p12) certificates. If you wish to examine the certificate, provide the password in the field provided and click on the Open button to examine the file store for certificates present. To load a certificate from PEM files, select the PEM tab. Like the PFX file selection, you can specify the password and click the Open button to examine the certificates in the PEM store, or you may paste any PEM data stored in memory.

Regardless of how you selected the certificate, once you hit the OK button, the task will attempt to verify that certificate selection. If successful, the subject of the certificate will be displayed in the property field. If the task was not able to verify the selection, a dialog box will appear instead detailing the verification error.

Timeout Property (CloudStorage Task)

A timeout for the component.

Data Type

Integer

Default Value

60

Remarks

If the Timeout property is set to 0, all operations will run uninterrupted until successful completion, or an error condition is encountered.

If Timeout is set to a positive value, the task will wait for the operation to complete before returning control.

If Timeout expires, and the operation is not yet complete, the task fails with an error.

Please note that by default, all timeouts are inactivity timeouts, i.e. the timeout period is extended by Timeout seconds when data is successfully sent or received.

Optionally, the behavior of the task may be changed to absolute timeouts, i.e. the task will wait for a maximum of Timeout seconds since the beginning of the operation, without extending the timeout period during communications.

This behavior is controlled by the AbsoluteTimeout configuration setting.

The default value for the Timeout property is 60 (seconds).

Certificate Type

The digital certificate being used.

Remarks

This type describes the current digital certificate. The certificate may be a public or private key. The fields are used to identify or select certificates.

Fields

CloudFilesAccount Type

Cloud storage service provider account credentials.

Remarks

This type is used to hold account credentials for cloud storage service providers which do not use OAuth for authentication.

Fields

Firewall Type

The firewall the component will connect through.

Remarks

When connecting through a firewall, this type is used to specify different properties of the firewall such as the firewall Host and the FirewallType.

Fields

OAuthAuthorizationParam Type

This type holds details of the OAuth authorization.

Remarks

This type holds details of the OAuth authorization.

Fields

Proxy Type

The proxy the component will connect to.

Remarks

When connecting through a proxy, this type is used to specify different properties of the proxy such as the Server and the AuthScheme.

Fields

Config Settings (CloudStorage Task)

The task accepts one or more of the following configuration settings. Configuration settings are similar in functionality to properties, but they are rarely used. In order to avoid "polluting" the property namespace of the task, access to these internal properties is provided through the Other property.

CloudStorage Config Settings

SSL Config Settings