HadoopDFS Class

Properties   Methods   Events   Config Settings   Errors  

The HadoopDFS class provides easy access to files stored in HDFS clusters.

Syntax

class cloudstorage.HadoopDFS

Remarks

The HadoopDFS class offers an easy-to-use API compatible with any Hadoop distributed file system (HDFS) cluster that exposes Hadoop's standard WebHDFS REST API. Capabilities include uploading and downloading files, strong encryption support, creating folders, file manipulation and organization, and more.

Authentication

First, set the url property to the base WebHDFS URL of the server (see url for more details).

Depending on how the server is configured, there are a few different authentication mechanisms that might be used; or, the server might not require authentication at all). Refer to the auth_mechanism property for more information about configuring the class to authenticate correctly.

Addressing Resources

HDFS addresses resources (files, directories, and symlinks) using Linux-style absolute paths. Unless otherwise specified, the class always works in terms of absolute paths, and will always prepend a forward slash (/) to any path passed to it that does not already start with one.

Listing Directory Contents

list_resources lists resources (files, directories, and symlinks) within the specified directory. Calling this method will fire the on_resource_list event once for each resource, and will also populate the Resource* properties. // ResourceList event handler. hdfs.OnResourceList += (s, e) => { Console.WriteLine(e.Name); }; hdfs.ListResources("/work_files/serious_business/cats"); for (int i = 0; i < hdfs.Resources.Count; i++) { // Process resources here. }

Downloading Files

The download_file method downloads files.

If local_file is set, the file will be saved to the specified location; otherwise, the file data will be held by resource_data.

To download and decrypt an encrypted file, set encryption_algorithm and encryption_password before calling this method.

Download Notes

In the simplest use-case, downloading a file looks like this: hdfs.LocalFile = "../MyFile.zip"; hdfs.DownloadFile(hdfs.Resources[0].Path);

Resuming Downloads

The class also supports resuming failed downloads by using the start_byte property. If a download is interrupted, set start_byte to the appropriate offset before calling this method to resume the download. string downloadFile = "../MyFile.zip"; hdfs.LocalFile = downloadFile; hdfs.DownloadFile(hdfs.Resources[0].Path); //The transfer is interrupted and DownloadFile() above fails. Later, resume the download: //Get the size of the partially downloaded file hdfs.StartByte = new FileInfo(downloadFile).Length; hdfs.DownloadFile(hdfs.Resources[0].Path);

Resuming Encrypted File Downloads

Resuming encrypted file downloads is only supported when local_file was set in the initial download attempt.

If local_file is set when beginning an encrypted download, the class creates a temporary file in TempPath to hold the encrypted data until the download is complete. If the download is interrupted, DownloadTempFile will be populated with the path of the temporary file that holds the partial data.

To resume, DownloadTempFile must be populated, along with start_byte, to allow the remainder of the encrypted data to be downloaded. Once the encrypted data is downloaded it will be decrypted and written to local_file. hdfs.LocalFile = "../MyFile.zip"; hdfs.EncryptionPassword = "password"; hdfs.DownloadFile(hdfs.Resources[0].Path); //The transfer is interrupted and DownloadFile() above fails. Later, resume the download: //Get the size of the partially download temp file hdfs.StartByte = new FileInfo(hdfs.Config("DownloadTempFile")).Length; hdfs.DownloadFile(hdfs.Resources[0].Path);

Uploading Files

The upload_file method uploads new files.

If local_file is set the file will be uploaded from the specified path. If local_file is not set the data in resource_data will be used.

To encrypt the file before uploading it, set encryption_algorithm and encryption_password. hdfs.LocalFile = "../MyFile.zip"; hdfs.UploadFile("/MyFile.zip");

Additional Functionality

The HadoopDFS class offers advanced functionality beyond simple uploads and downloads. For instance:

• Encrypt and decrypt files using the encryption_algorithm and encryption_password properties.
• Basic file and folder manipulation and organization using methods such as append_file, delete_resource, make_directory, move_resource, and truncate_file.
• Advanced file and directory manipulation with set_file_replication, set_owner, set_permission, and set_times.
• Retrieval of both general file/directory information, as well as directory quota information, using get_resource_info and get_dir_summary.
• Execute any arbitrary WebHDFS operation with ease using the do_custom_op method.
• And more!

Property List

The following is the full list of the properties of the class with short descriptions. Click on the links for further details.

Method List

The following is the full list of the methods of the class with short descriptions. Click on the links for further details.

Event List

The following is the full list of the events fired by the class with short descriptions. Click on the links for further details.

Config Settings

The following is a list of config settings for the class with short descriptions. Click on the links for further details.

auth_mechanism Property

The authentication mechanism to use when connecting to the server.

Syntax

def get_auth_mechanism() -> int: ...
def set_auth_mechanism(value: int) -> None: ...

auth_mechanism = property(get_auth_mechanism, set_auth_mechanism)

Default Value

0

Remarks

This property controls what authentication mechanism the class should use when connecting to the server.

Possible values are:

When set to 1 (amSimple), the value of the user property is automatically sent in every request using the user.name query parameter.

When set to 2 (amBasic), 3 (amNTLM), or 4 (amNegotiate), the values held by the user and password properties will be used to perform Basic, NTLM, or Negotiate (e.g., Kerberos SPNEGO) authentication.

When set to 5 (amOAuth), the value of the authorization property is automatically sent in every request using the Authorization HTTP header.

authorization Property

OAuth 2.0 Authorization Token.

Syntax

def get_authorization() -> str: ...
def set_authorization(value: str) -> None: ...

authorization = property(get_authorization, set_authorization)

Default Value

""

Remarks

This class supports authentication via OAuth 2.0. First, perform OAuth authentication using the OAuth* properties, using the OAuth class or a separate process. If using the OAuth* properties, then the authorization property will not be used.

Bearer ACCESS_TOKEN

Refer to auth_mechanism for more information.

dir_summary_dir_count Property

The number of subdirectories within the directory.

Syntax

def get_dir_summary_dir_count() -> int: ...

dir_summary_dir_count = property(get_dir_summary_dir_count, None)

Default Value

0

Remarks

The number of subdirectories within the directory.

This property reflects the number of subdirectories contained within the directory, calculated recursively. Note that this count will always include the directory itself; i.e., this property would return 1 for a directory with no subfolders.

This property is read-only.

dir_summary_file_count Property

The number of files within the directory.

Syntax

def get_dir_summary_file_count() -> int: ...

dir_summary_file_count = property(get_dir_summary_file_count, None)

Default Value

0

Remarks

The number of files within the directory.

This property reflects the number of files contained within the directory, calculated recursively.

This property is read-only.

dir_summary_name_quota Property

The name quota imposed on the directory.

Syntax

def get_dir_summary_name_quota() -> int: ...

dir_summary_name_quota = property(get_dir_summary_name_quota, None)

Default Value

-1

Remarks

The name quota imposed on the directory.

This property reflects the name quota imposed on the directory, or -1 if the directory doesn't have a name quota set.

A name quota limits the number of files and directories that can be created within a directory (calculated recursively). Note that a directory's own name is counted against its own quota, so the minimum name quota that may be applied to a directory is 1 (which will force the directory to stay empty).

This property is read-only.

dir_summary_size Property

The total size of the directory contents, excluding file replicas.

Syntax

def get_dir_summary_size() -> int: ...

dir_summary_size = property(get_dir_summary_size, None)

Default Value

0

Remarks

The total size of the directory contents, excluding file replicas.

This property reflects the total size (in bytes) of the directory's contents, calculated recursively.

Unlike dir_summary_space_used, this property's value does not take file replicas into account, and thus should not be interpreted as the actual number of bytes the directory's contents use on disk.

This property is read-only.

dir_summary_space_quota Property

The space quota imposed on the directory.

Syntax

def get_dir_summary_space_quota() -> int: ...

dir_summary_space_quota = property(get_dir_summary_space_quota, None)

Default Value

-1

Remarks

The space quota imposed on the directory.

This property reflects the overall space quota (in bytes) imposed on the directory, or -1 if the directory doesn't have a space quota set.

A space quota limits the total number of bytes the files within a directory (or any of its subdirectories) may consume across all storage mediums. Space quotas are tracked separately from each directory's various dir_summary_storage_quotas, and only one space quota may be applied to any given directory.

This property is read-only.

dir_summary_space_used Property

The total amount of space the directory consumes on disk.

Syntax

def get_dir_summary_space_used() -> int: ...

dir_summary_space_used = property(get_dir_summary_space_used, None)

Default Value

0

Remarks

The total amount of space the directory consumes on disk.

This property reflects the total amount of space (in bytes) that the directory's contents, calculated recursively, consume on disk.

Unlike dir_summary_size, this property's value includes space consumed by file replicas.

This property is read-only.

dir_summary_storage_quota Property

The storage type quota imposed on the directory.

Syntax

def get_dir_summary_storage_quota() -> int: ...

dir_summary_storage_quota = property(get_dir_summary_storage_quota, None)

Default Value

-1

Remarks

The storage type quota imposed on the directory.

This property reflects the storage type quota (in bytes) imposed on the directory for the dir_summary_storage_quota_type currently selected by dir_summary_storage_quota_index.

Storage type quotas limit the number of bytes that files within a directory (or any of its subdirectories) may consume on specific types of storage mediums. Multiple storage type quotas, each associated with a different type of storage medium, may be imposed on a directory simultaneously. Storage type quotas are tracked separately from a directory's overall dir_summary_space_quota.

Use dir_summary_storage_quota_count to determine how many storage type quotas are imposed on a directory, and dir_summary_storage_quota_index to select which storage type quota's information to reflect in the dir_summary_storage_quota, dir_summary_storage_quota_type, and dir_summary_storage_quota_used properties.

Note that it is possible for a storage type quota to be imposed on a directory without actually restricting any space usage, in which case this property's value will be -1.

This property is read-only.

dir_summary_storage_quota_count Property

The number of storage type quotas associated with the directory.

Syntax

def get_dir_summary_storage_quota_count() -> int: ...

dir_summary_storage_quota_count = property(get_dir_summary_storage_quota_count, None)

Default Value

0

Remarks

The number of storage type quotas associated with the directory.

This property reflects the number of storage type quotas associated with the directory. Use the dir_summary_storage_quota_index property to select which storage type quota's information to reflect in the dir_summary_storage_quota, dir_summary_storage_quota_type, and dir_summary_storage_quota_used properties.

This property is read-only.

dir_summary_storage_quota_index Property

Selects the storage type quota to show information for.

Syntax

def get_dir_summary_storage_quota_index() -> int: ...

dir_summary_storage_quota_index = property(get_dir_summary_storage_quota_index, None)

Default Value

-1

Remarks

Selects the storage type quota to show information for.

This property selects which storage type quota's information to reflect in the dir_summary_storage_quota, dir_summary_storage_quota_type, and dir_summary_storage_quota_used properties; those properties are re-populated when this property's value is changed.

Valid values for this property are -1 to (dir_summary_storage_quota_count - 1); invalid indices are ignored. The default value is 0 if dir_summary_storage_quota_count is greater than 0, and -1 otherwise.

This property is read-only.

dir_summary_storage_quota_type Property

The storage type associated with the storage type quota.

Syntax

def get_dir_summary_storage_quota_type() -> str: ...

dir_summary_storage_quota_type = property(get_dir_summary_storage_quota_type, None)

Default Value

""

Remarks

The storage type associated with the storage type quota.

This property reflects the storage type associated with the storage type quota currently selected by dir_summary_storage_quota_index

This property is read-only.

dir_summary_storage_quota_used Property

The number of bytes consumed for the storage type quota.

Syntax

def get_dir_summary_storage_quota_used() -> int: ...

dir_summary_storage_quota_used = property(get_dir_summary_storage_quota_used, None)

Default Value

0

Remarks

The number of bytes consumed for the storage type quota.

This property reflects the number of bytes that have been consumed for the storage type quota currently selected by dir_summary_storage_quota_index

This property is read-only.

encryption_algorithm Property

The encryption algorithm.

Syntax

def get_encryption_algorithm() -> int: ...
def set_encryption_algorithm(value: int) -> None: ...

encryption_algorithm = property(get_encryption_algorithm, set_encryption_algorithm)

Default Value

0

Remarks

This property specifies the encryption algorithm to be used. The maximum allowable key size is automatically used for the selected algorithm. Possible values are:

encryption_password Property

The encryption password.

Syntax

def get_encryption_password() -> str: ...
def set_encryption_password(value: str) -> None: ...

encryption_password = property(get_encryption_password, set_encryption_password)

Default Value

""

Remarks

If this property is populated when upload_file or download_file is called, the class will attempt to encrypt or decrypt the data before uploading or after downloading it.

The class uses the value specified here to generate the necessary encryption Key and IV values using the PKCS5 password digest algorithm. This provides a simpler alternative to creating and managing Key and IV values directly.

However, it is also possible to explicitly specify the Key and IV values to use by setting the EncryptionKey and EncryptionIV configuration settings. This may be necessary if, e.g., the data needs to be encrypted/decrypted by another utility which generates Key and IV values differently.

firewall_auto_detect Property

This property tells the class whether or not to automatically detect and use firewall system settings, if available.

Syntax

def get_firewall_auto_detect() -> bool: ...
def set_firewall_auto_detect(value: bool) -> None: ...

firewall_auto_detect = property(get_firewall_auto_detect, set_firewall_auto_detect)

Default Value

FALSE

Remarks

This property tells the class whether or not to automatically detect and use firewall system settings, if available.

firewall_type Property

This property determines the type of firewall to connect through.

Syntax

def get_firewall_type() -> int: ...
def set_firewall_type(value: int) -> None: ...

firewall_type = property(get_firewall_type, set_firewall_type)

Default Value

0

Remarks

This property determines the type of firewall to connect through. The applicable values are as follows:

firewall_host Property

This property contains the name or IP address of firewall (optional).

Syntax

def get_firewall_host() -> str: ...
def set_firewall_host(value: str) -> None: ...

firewall_host = property(get_firewall_host, set_firewall_host)

Default Value

""

Remarks

This property contains the name or IP address of firewall (optional). If a firewall_host is given, the requested connections will be authenticated through the specified firewall when connecting.

If this property is set to a Domain Name, a DNS request is initiated. Upon successful termination of the request, this property is set to the corresponding address. If the search is not successful, the class fails with an error.

firewall_password Property

This property contains a password if authentication is to be used when connecting through the firewall.

Syntax

def get_firewall_password() -> str: ...
def set_firewall_password(value: str) -> None: ...

firewall_password = property(get_firewall_password, set_firewall_password)

Default Value

""

Remarks

This property contains a password if authentication is to be used when connecting through the firewall. If firewall_host is specified, the firewall_user and firewall_password properties are used to connect and authenticate to the given firewall. If the authentication fails, the class fails with an error.

firewall_port Property

This property contains the transmission control protocol (TCP) port for the firewall Host .

Syntax

def get_firewall_port() -> int: ...
def set_firewall_port(value: int) -> None: ...

firewall_port = property(get_firewall_port, set_firewall_port)

Default Value

0

Remarks

This property contains the transmission control protocol (TCP) port for the firewall firewall_host. See the description of the firewall_host property for details.

Note: This property is set automatically when firewall_type is set to a valid value. See the description of the firewall_type property for details.

firewall_user Property

This property contains a user name if authentication is to be used connecting through a firewall.

Syntax

def get_firewall_user() -> str: ...
def set_firewall_user(value: str) -> None: ...

firewall_user = property(get_firewall_user, set_firewall_user)

Default Value

""

Remarks

This property contains a user name if authentication is to be used connecting through a firewall. If the firewall_host is specified, this property and firewall_password properties are used to connect and authenticate to the given firewall. If the authentication fails, the class fails with an error.

idle Property

The current status of the class.

Syntax

def get_idle() -> bool: ...

idle = property(get_idle, None)

Default Value

TRUE

Remarks

idle will be False if the component is currently busy (communicating and/or waiting for an answer), and True at all other times.

This property is read-only.

local_file Property

The location of the local file.

Syntax

def get_local_file() -> str: ...
def set_local_file(value: str) -> None: ...

local_file = property(get_local_file, set_local_file)

Default Value

""

Remarks

This property specifies the location of a file on disk. This is used as the source file when calling upload_file or append_file, and as the destination file when calling download_file or do_custom_op.

local_host Property

The name of the local host or user-assigned IP interface through which connections are initiated or accepted.

Syntax

def get_local_host() -> str: ...
def set_local_host(value: str) -> None: ...

local_host = property(get_local_host, set_local_host)

Default Value

""

Remarks

The local_host property contains the name of the local host as obtained by the gethostname() system call, or if the user has assigned an IP address, the value of that address.

In multi-homed hosts (machines with more than one IP interface) setting LocalHost to the value of an interface will make the class initiate connections (or accept in the case of server classs) only through that interface.

If the class is connected, the local_host property shows the IP address of the interface through which the connection is made in internet dotted format (aaa.bbb.ccc.ddd). In most cases, this is the address of the local host, except for multi-homed hosts (machines with more than one IP interface).

NOTE: local_host is not persistent. You must always set it in code, and never in the property window.

other_headers Property

This property includes other headers as determined by the user (optional).

Syntax

def get_other_headers() -> str: ...
def set_other_headers(value: str) -> None: ...

other_headers = property(get_other_headers, set_other_headers)

Default Value

""

Remarks

This property can be set to a string of headers to be appended to the HTTP request headers created from other properties like content_type and from_.

The headers must follow the format Header: Value as described in the HTTP specifications. Header lines should be separated by CRLF ("\r\n") .

Use this property with caution. If this property contains invalid headers, HTTP requests may fail.

This property is useful for extending the functionality of the class beyond what is provided.

overwrite Property

Whether to overwrite the local or remote file.

Syntax

def get_overwrite() -> bool: ...
def set_overwrite(value: bool) -> None: ...

overwrite = property(get_overwrite, set_overwrite)

Default Value

FALSE

Remarks

When calling download_file, this property determines if local_file should be overwritten if it already exists.

When calling upload_file, this property determines if the remote file should be overwritten if it already exists.

parsed_header_count Property

The number of records in the ParsedHeader arrays.

Syntax

def get_parsed_header_count() -> int: ...

parsed_header_count = property(get_parsed_header_count, None)

Default Value

0

Remarks

This property controls the size of the following arrays:

• parsed_header_field
• parsed_header_value

The array indices start at 0 and end at parsed_header_count - 1.

This property is read-only.

parsed_header_field Property

This property contains the name of the HTTP header (this is the same case as it is delivered).

Syntax

def get_parsed_header_field(parsed_header_index: int) -> str: ...

Default Value

""

Remarks

This property contains the name of the HTTP Header (this is the same case as it is delivered).

The parsed_header_index parameter specifies the index of the item in the array. The size of the array is controlled by the parsed_header_count property.

This property is read-only.

parsed_header_value Property

This property contains the header contents.

Syntax

def get_parsed_header_value(parsed_header_index: int) -> str: ...

Default Value

""

Remarks

This property contains the Header contents.

The parsed_header_index parameter specifies the index of the item in the array. The size of the array is controlled by the parsed_header_count property.

This property is read-only.

password Property

The password to use for authentication.

Syntax

def get_password() -> str: ...
def set_password(value: str) -> None: ...

password = property(get_password, set_password)

Default Value

""

Remarks

This property specifies the password to user for authentication.

Refer to auth_mechanism for more information.

proxy_auth_scheme Property

This property is used to tell the class which type of authorization to perform when connecting to the proxy.

Syntax

def get_proxy_auth_scheme() -> int: ...
def set_proxy_auth_scheme(value: int) -> None: ...

proxy_auth_scheme = property(get_proxy_auth_scheme, set_proxy_auth_scheme)

Default Value

0

Remarks

This property is used to tell the class which type of authorization to perform when connecting to the proxy. This is used only when the proxy_user and proxy_password properties are set.

proxy_auth_scheme should be set to authNone (3) when no authentication is expected.

By default, proxy_auth_scheme is authBasic (0), and if the proxy_user and proxy_password properties are set, the component will attempt basic authentication.

If proxy_auth_scheme is set to authDigest (1), digest authentication will be attempted instead.

If proxy_auth_scheme is set to authProprietary (2), then the authorization token will not be generated by the class. Look at the configuration file for the class being used to find more information about manually setting this token.

If proxy_auth_scheme is set to authNtlm (4), NTLM authentication will be used.

For security reasons, setting this property will clear the values of proxy_user and proxy_password.

proxy_auto_detect Property

This property tells the class whether or not to automatically detect and use proxy system settings, if available.

Syntax

def get_proxy_auto_detect() -> bool: ...
def set_proxy_auto_detect(value: bool) -> None: ...

proxy_auto_detect = property(get_proxy_auto_detect, set_proxy_auto_detect)

Default Value

FALSE

Remarks

This property tells the class whether or not to automatically detect and use proxy system settings, if available. The default value is False.

proxy_password Property

This property contains a password if authentication is to be used for the proxy.

Syntax

def get_proxy_password() -> str: ...
def set_proxy_password(value: str) -> None: ...

proxy_password = property(get_proxy_password, set_proxy_password)

Default Value

""

Remarks

This property contains a password if authentication is to be used for the proxy.

If proxy_auth_scheme is set to Basic Authentication, the proxy_user and proxy_password are Base64 encoded and the proxy authentication token will be generated in the form Basic [encoded-user-password].

If proxy_auth_scheme is set to Digest Authentication, the proxy_user and proxy_password properties are used to respond to the Digest Authentication challenge from the server.

If proxy_auth_scheme is set to NTLM Authentication, the proxy_user and proxy_password properties are used to authenticate through NTLM negotiation.

proxy_port Property

This property contains the Transmission Control Protocol (TCP) port for the proxy Server (default 80).

Syntax

def get_proxy_port() -> int: ...
def set_proxy_port(value: int) -> None: ...

proxy_port = property(get_proxy_port, set_proxy_port)

Default Value

80

Remarks

This property contains the Transmission Control Protocol (TCP) port for the proxy proxy_server (default 80). See the description of the proxy_server property for details.

proxy_server Property

If a proxy Server is given, then the HTTP request is sent to the proxy instead of the server otherwise specified.

Syntax

def get_proxy_server() -> str: ...
def set_proxy_server(value: str) -> None: ...

proxy_server = property(get_proxy_server, set_proxy_server)

Default Value

""

Remarks

If a proxy proxy_server is given, then the HTTP request is sent to the proxy instead of the server otherwise specified.

If the proxy_server property is set to a domain name, a DNS request is initiated. Upon successful termination of the request, the proxy_server property is set to the corresponding address. If the search is not successful, an error is returned.

proxy_ssl Property

This property determines when to use a Secure Sockets Layer (SSL) for the connection to the proxy.

Syntax

def get_proxy_ssl() -> int: ...
def set_proxy_ssl(value: int) -> None: ...

proxy_ssl = property(get_proxy_ssl, set_proxy_ssl)

Default Value

0

Remarks

This property determines when to use a Secure Sockets Layer (SSL) for the connection to the proxy. The applicable values are as follows:

proxy_user Property

This property contains a username if authentication is to be used for the proxy.

Syntax

def get_proxy_user() -> str: ...
def set_proxy_user(value: str) -> None: ...

proxy_user = property(get_proxy_user, set_proxy_user)

Default Value

""

Remarks

This property contains a username if authentication is to be used for the proxy.

If proxy_auth_scheme is set to Basic Authentication, the proxy_user and proxy_password properties are Base64 encoded and the proxy authentication token will be generated in the form Basic [encoded-user-password].

If proxy_auth_scheme is set to Digest Authentication, the proxy_user and proxy_password properties are used to respond to the Digest Authentication challenge from the server.

If proxy_auth_scheme is set to NTLM Authentication, the proxy_user and proxy_password properties are used to authenticate through NTLM negotiation.

query_param_count Property

The number of records in the QueryParam arrays.

Syntax

def get_query_param_count() -> int: ...
def set_query_param_count(value: int) -> None: ...

query_param_count = property(get_query_param_count, set_query_param_count)

Default Value

0

Remarks

This property controls the size of the following arrays:

• query_param_name
• query_param_value

The array indices start at 0 and end at query_param_count - 1.

query_param_name Property

The name of the query parameter.

Syntax

def get_query_param_name(query_param_index: int) -> str: ...
def set_query_param_name(query_param_index: int, value: str) -> None: ...

Default Value

""

Remarks

The name of the query parameter.

This property specifies the name of the query parameter.

The query_param_index parameter specifies the index of the item in the array. The size of the array is controlled by the query_param_count property.

query_param_value Property

The value of the query parameter.

Syntax

def get_query_param_value(query_param_index: int) -> str: ...
def set_query_param_value(query_param_index: int, value: str) -> None: ...

Default Value

""

Remarks

The value of the query parameter.

This property specifies the value of the query parameter. The class will automatically URL-encode this value when sending the request.

The query_param_index parameter specifies the index of the item in the array. The size of the array is controlled by the query_param_count property.

read_bytes Property

The number of bytes to read when downloading a file.

Syntax

def get_read_bytes() -> int: ...
def set_read_bytes(value: int) -> None: ...

read_bytes = property(get_read_bytes, set_read_bytes)

Default Value

-1

Remarks

This property specifies how many bytes should be read when download_file is called. It can be used in tandem with start_byte to specify a specific range of the file to download.

If set to -1 (default), there is no limit on how many bytes will be read.

resource_data Property

The data that was downloaded, or that should be uploaded.

Syntax

def get_resource_data() -> bytes: ...
def set_resource_data(value: bytes) -> None: ...

resource_data = property(get_resource_data, set_resource_data)

Default Value

""

Remarks

This property is populated with file data after calling download_file if local_file is not set.

This property can also be set before calling upload_file; its data will be uploaded if local_file is not set.

resource_count Property

The number of records in the Resource arrays.

Syntax

def get_resource_count() -> int: ...

resource_count = property(get_resource_count, None)

Default Value

0

Remarks

This property controls the size of the following arrays:

• resource_access_time
• resource_block_size
• resource_child_count
• resource_group
• resource_modified_time
• resource_name
• resource_owner
• resource_path
• resource_permission
• resource_replication
• resource_size
• resource_symlink_target
• resource_type

The array indices start at 0 and end at resource_count - 1.

This property is read-only.

resource_access_time Property

The last access time of the resource.

Syntax

def get_resource_access_time(resource_index: int) -> int: ...

Default Value

0

Remarks

The last access time of the resource.

This property reflects the last access time of the resource, in milliseconds relative to the Unix epoch.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_block_size Property

The block size of the file.

Syntax

def get_resource_block_size(resource_index: int) -> int: ...

Default Value

0

Remarks

The block size of the file.

This property reflects the block size of the file, in bytes.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_child_count Property

The number of children in the directory.

Syntax

def get_resource_child_count(resource_index: int) -> int: ...

Default Value

0

Remarks

The number of children in the directory.

This property reflects the number of immediate children in the directory. Always 0 for files.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_group Property

The name of the resource's group.

Syntax

def get_resource_group(resource_index: int) -> str: ...

Default Value

""

Remarks

The name of the resource's group.

This property reflects the name of the resource's group.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_modified_time Property

The last modified time of the resource.

Syntax

def get_resource_modified_time(resource_index: int) -> int: ...

Default Value

0

Remarks

The last modified time of the resource.

This property reflects the last modified time of the resource, in milliseconds relative to the Unix epoch.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_name Property

The name of the resource.

Syntax

def get_resource_name(resource_index: int) -> str: ...

Default Value

""

Remarks

The name of the resource.

This property reflects the name of the resource.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_owner Property

The name of the resource's owner.

Syntax

def get_resource_owner(resource_index: int) -> str: ...

Default Value

""

Remarks

The name of the resource's owner.

This property reflects the name of the resource's owner.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_path Property

The full path of the resource.

Syntax

def get_resource_path(resource_index: int) -> str: ...

Default Value

""

Remarks

The full path of the resource.

This property reflects the full path of the resource. (Note that this property's value is not returned by the server, it is calculated by the class for convenience.)

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_permission Property

The resource's permission bits.

Syntax

def get_resource_permission(resource_index: int) -> str: ...

Default Value

""

Remarks

The resource's permission bits.

This property reflects the resource's permission bits, represented as an octal string (e.g., 755).

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_replication Property

The replication factor of the file.

Syntax

def get_resource_replication(resource_index: int) -> int: ...

Default Value

0

Remarks

The replication factor of the file.

This property reflects the replication factor of the file. Always 0 for directories.

A file's replication factor determines how many copies of the file's data ("replicas") are maintained by HDFS. For example, a replication factor of 3 means that HDFS will maintain 2 replicas in addition to the original file. Thus, the minimum replication factor a file can have is 1.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_size Property

The size of the file.

Syntax

def get_resource_size(resource_index: int) -> int: ...

Default Value

0

Remarks

The size of the file.

This property reflects the size of the file, in bytes. Always 0 for directories.

Note that the actual amount of space the file consumes will be greater than this property's value if resource_replication is greater than 1. In that case, multiply the values of this property and resource_replication to obtain the total number of bytes consumed by the file and its replicas.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_symlink_target Property

The full target path of the symlink.

Syntax

def get_resource_symlink_target(resource_index: int) -> str: ...

Default Value

""

Remarks

The full target path of the symlink.

This property reflects the full target path of the symlink, if resource_type is 2 (ftSymLink) and the server returns the target path information, or empty string otherwise.

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

resource_type Property

The resource type.

Syntax

def get_resource_type(resource_index: int) -> int: ...

Default Value

0

Remarks

The resource type.

This property reflects the resource's type. Possible values are:

The resource_index parameter specifies the index of the item in the array. The size of the array is controlled by the resource_count property.

This property is read-only.

ssl_accept_server_cert_encoded Property

This is the certificate (PEM/Base64 encoded).

Syntax

def get_ssl_accept_server_cert_encoded() -> bytes: ...
def set_ssl_accept_server_cert_encoded(value: bytes) -> None: ...

ssl_accept_server_cert_encoded = property(get_ssl_accept_server_cert_encoded, set_ssl_accept_server_cert_encoded)

Default Value

""

Remarks

This is the certificate (PEM/Base64 encoded). This property is used to assign a specific certificate. The ssl_accept_server_cert_store and ssl_accept_server_cert_subject properties also may be used to specify a certificate.

When ssl_accept_server_cert_encoded is set, a search is initiated in the current ssl_accept_server_cert_store for the private key of the certificate. If the key is found, ssl_accept_server_cert_subject is updated to reflect the full subject of the selected certificate; otherwise, ssl_accept_server_cert_subject is set to an empty string.

ssl_cert_encoded Property

This is the certificate (PEM/Base64 encoded).

Syntax

def get_ssl_cert_encoded() -> bytes: ...
def set_ssl_cert_encoded(value: bytes) -> None: ...

ssl_cert_encoded = property(get_ssl_cert_encoded, set_ssl_cert_encoded)

Default Value

""

Remarks

This is the certificate (PEM/Base64 encoded). This property is used to assign a specific certificate. The ssl_cert_store and ssl_cert_subject properties also may be used to specify a certificate.

When ssl_cert_encoded is set, a search is initiated in the current ssl_cert_store for the private key of the certificate. If the key is found, ssl_cert_subject is updated to reflect the full subject of the selected certificate; otherwise, ssl_cert_subject is set to an empty string.

ssl_cert_store Property

This is the name of the certificate store for the client certificate.

Syntax

def get_ssl_cert_store() -> bytes: ...
def set_ssl_cert_store(value: bytes) -> None: ...

ssl_cert_store = property(get_ssl_cert_store, set_ssl_cert_store)

Default Value

"MY"

Remarks

This is the name of the certificate store for the client certificate.

The ssl_cert_store_type property denotes the type of the certificate store specified by ssl_cert_store. If the store is password protected, specify the password in ssl_cert_store_password.

ssl_cert_store is used in conjunction with the ssl_cert_subject property to specify client certificates. If ssl_cert_store has a value, and ssl_cert_subject or ssl_cert_encoded is set, a search for a certificate is initiated. Please see the ssl_cert_subject property for details.

Designations of certificate stores are platform dependent.

The following designations are the most common User and Machine certificate stores in Windows:

When the certificate store type is PFXFile, this property must be set to the name of the file. When the type is PFXBlob, the property must be set to the binary contents of a PFX file (i.e., PKCS#12 certificate store).

ssl_cert_store_password Property

If the type of certificate store requires a password, this property is used to specify the password needed to open the certificate store.

Syntax

def get_ssl_cert_store_password() -> str: ...
def set_ssl_cert_store_password(value: str) -> None: ...

ssl_cert_store_password = property(get_ssl_cert_store_password, set_ssl_cert_store_password)

Default Value

""

Remarks

If the type of certificate store requires a password, this property is used to specify the password needed to open the certificate store.

ssl_cert_store_type Property

This is the type of certificate store for this certificate.

Syntax

def get_ssl_cert_store_type() -> int: ...
def set_ssl_cert_store_type(value: int) -> None: ...

ssl_cert_store_type = property(get_ssl_cert_store_type, set_ssl_cert_store_type)

Default Value

0

Remarks

This is the type of certificate store for this certificate.

The class supports both public and private keys in a variety of formats. When the cstAuto value is used, the class will automatically determine the type. This property can take one of the following values:

ssl_cert_subject Property

This is the subject of the certificate used for client authentication.

Syntax

def get_ssl_cert_subject() -> str: ...
def set_ssl_cert_subject(value: str) -> None: ...

ssl_cert_subject = property(get_ssl_cert_subject, set_ssl_cert_subject)

Default Value

""

Remarks

This is the subject of the certificate used for client authentication.

This property must be set after all other certificate properties are set. When this property is set, a search is performed in the current certificate store to locate a certificate with a matching subject.

If a matching certificate is found, the property is set to the full subject of the matching certificate.

If an exact match is not found, the store is searched for subjects containing the value of the property.

If a match is still not found, the property is set to an empty string, and no certificate is selected.

The special value "*" picks a random certificate in the certificate store.

The certificate subject is a comma-separated list of distinguished name fields and values. For instance, "CN=www.server.com, OU=test, C=US, E=support@nsoftware.com". Common fields and their meanings are as follows:

If a field value contains a comma, it must be quoted.

ssl_provider Property

This specifies the SSL/TLS implementation to use.

Syntax

def get_ssl_provider() -> int: ...
def set_ssl_provider(value: int) -> None: ...

ssl_provider = property(get_ssl_provider, set_ssl_provider)

Default Value

0

Remarks

This property specifies the SSL/TLS implementation to use. In most cases the default value of 0 (Automatic) is recommended and should not be changed. When set to 0 (Automatic) the class will select whether to use the platform implementation or the internal implementation depending on the operating system as well as the TLS version being used.

Possible values are:

In most cases using the default value (Automatic) is recommended. The class will select a provider depending on the current platform.

When Automatic is selected, on Windows the class will use the platform implementation. On Linux/macOS the class will use the internal implementation. When TLS 1.3 is enabled via SSLEnabledProtocols the internal implementation is used on all platforms.

ssl_server_cert_encoded Property

This is the certificate (PEM/Base64 encoded).

Syntax

def get_ssl_server_cert_encoded() -> bytes: ...

ssl_server_cert_encoded = property(get_ssl_server_cert_encoded, None)

Default Value

""

Remarks

This is the certificate (PEM/Base64 encoded). This property is used to assign a specific certificate. The ssl_server_cert_store and ssl_server_cert_subject properties also may be used to specify a certificate.

When ssl_server_cert_encoded is set, a search is initiated in the current ssl_server_cert_store for the private key of the certificate. If the key is found, ssl_server_cert_subject is updated to reflect the full subject of the selected certificate; otherwise, ssl_server_cert_subject is set to an empty string.

This property is read-only.

start_byte Property

The byte offset from which to start downloading a file.

Syntax

def get_start_byte() -> int: ...
def set_start_byte(value: int) -> None: ...

start_byte = property(get_start_byte, set_start_byte)

Default Value

0

Remarks

This property specifies an offset (in bytes) from which to start reading a file when download_file is called. It can be used in tandem with read_bytes to specify a specific range of the file to download.

timeout Property

A timeout for the class.

Syntax

def get_timeout() -> int: ...
def set_timeout(value: int) -> None: ...

timeout = property(get_timeout, set_timeout)

Default Value

60

Remarks

If the timeout property is set to 0, all operations will run uninterrupted until successful completion or an error condition is encountered.

If timeout is set to a positive value, the class will wait for the operation to complete before returning control.

The class will use do_events to enter an efficient wait loop during any potential waiting period, making sure that all system events are processed immediately as they arrive. This ensures that the host application does not "freeze" and remains responsive.

If timeout expires, and the operation is not yet complete, the class fails with an error.

Please note that by default, all timeouts are inactivity timeouts, i.e. the timeout period is extended by timeout seconds when any amount of data is successfully sent or received.

The default value for the timeout property is 60 seconds.

url Property

The URL of the Hadoop WebHDFS server.

Syntax

def get_url() -> str: ...
def set_url(value: str) -> None: ...

url = property(get_url, set_url)

Default Value

""

Remarks

This property specifies the URL of the Hadoop WebHDFS server to make requests against.

A full WebHDFS URL looks like http[s]://<HOST>:<PORT>/webhdfs/v1. Anytime this property is set to a non-empty string, the class will automatically append /webhdfs/v1 to its value (if necessary).

The class automatically detects whether to secure the connection using SSL/TLS based on whether the url begins with http (not secured) or https (secured).

user Property

The user name to use for authentication.

Syntax

def get_user() -> str: ...
def set_user(value: str) -> None: ...

user = property(get_user, set_user)

Default Value

""

Remarks

This property specifies the user name to use for authentication.

Refer to auth_mechanism for more information.

add_query_param Method

Adds a query parameter to the QueryParams properties.

Syntax

def add_query_param(name: str, value: str) -> None: ...

Remarks

This method is used to add a query parameter to the QueryaParam* properties. Name specifies the name of the parameter, and Value specifies the value of the parameter.

All specified Values will be URL encoded by the class automatically. Consult the service documentation for details on the available parameters.

append_file Method

Appends data to an existing file.

Syntax

def append_file(file_path: str) -> None: ...

Remarks

This method appends data to the existing file at FilePath.

If local_file is set data will be read from the file at the specified path. If local_file is not set the data in resource_data will be used.

calc_authorization Method

Calculates the Authorization header based on provided credentials.

Syntax

def calc_authorization() -> None: ...

Remarks

This method calculates the authorization value using the values provided in auth_scheme, user and password.

In most cases this method does not need to be called. The class will automatically calculate any required authorization values when a method is called such as get or post.

This method may be useful in cases where the authorization value needs to be calculated prior to sending a request.

config Method

Sets or retrieves a configuration setting.

Syntax

def config(configuration_string: str) -> str: ...

Remarks

config is a generic method available in every class. It is used to set and retrieve configuration settings for the class.

These settings are similar in functionality to properties, but they are rarely used. In order to avoid "polluting" the property namespace of the class, access to these internal properties is provided through the config method.

To set a configuration setting named PROPERTY, you must call Config("PROPERTY=VALUE"), where VALUE is the value of the setting expressed as a string. For boolean values, use the strings "True", "False", "0", "1", "Yes", or "No" (case does not matter).

To read (query) the value of a configuration setting, you must call Config("PROPERTY"). The value will be returned as a string.

delete_resource Method

Deletes a resource.

Syntax

def delete_resource(path: str) -> None: ...

Remarks

This method deletes the resource at Path.

The RecursiveDelete configuration setting can be used to control whether this method will recursively delete non-empty directories (it is enabled by default).

do_custom_op Method

Executes an arbitrary WebHDFS operation.

Syntax

def do_custom_op(http_method: str, request_path: str, op: str) -> None: ...

Remarks

This method can be used to execute any WebHDFS operation the class does not explicitly implement.

Valid values for HttpMethod are:

• GET (default if empty)
• POST
• PUT
• DELETE

RequestPath must either be a valid resource path, or empty string. Op must be a valid WebHDFS operation (refer to the Hadoop WebHDFS documentation for a full list of supported operations).

Usage

When this method is called, the class will do the following:

1. Build a request URL using url, RequestPath, Op, and the query parameters held by the QueryParam* properties.
2. Send the request using the given HttpMethod, the request URL built in step 1, the headers held by other_headers, and the currently-configured authentication (if any; refer to auth_mechanism for more information). The request is always sent with an empty body.
3. Store the response headers in the ParsedHeader* properties, and the response body in the specified local_file, or resource_data (using the same logic as download_file).

If the response body is JSON data, the XPath, XText, and other X* configuration settings can then be used to navigate and extract information from it.

download_file Method

Downloads a file.

Syntax

def download_file(file_path: str) -> None: ...

Remarks

This method downloads the file at FilePath.

If local_file is set, the file will be saved to the specified location; otherwise, the file data will be held by resource_data.

To download and decrypt an encrypted file, set encryption_algorithm and encryption_password before calling this method.

Download Notes

In the simplest use-case, downloading a file looks like this: hdfs.LocalFile = "../MyFile.zip"; hdfs.DownloadFile(hdfs.Resources[0].Path);

Resuming Downloads

The class also supports resuming failed downloads by using the start_byte property. If a download is interrupted, set start_byte to the appropriate offset before calling this method to resume the download. string downloadFile = "../MyFile.zip"; hdfs.LocalFile = downloadFile; hdfs.DownloadFile(hdfs.Resources[0].Path); //The transfer is interrupted and DownloadFile() above fails. Later, resume the download: //Get the size of the partially downloaded file hdfs.StartByte = new FileInfo(downloadFile).Length; hdfs.DownloadFile(hdfs.Resources[0].Path);

Resuming Encrypted File Downloads

Resuming encrypted file downloads is only supported when local_file was set in the initial download attempt.

If local_file is set when beginning an encrypted download, the class creates a temporary file in TempPath to hold the encrypted data until the download is complete. If the download is interrupted, DownloadTempFile will be populated with the path of the temporary file that holds the partial data.

To resume, DownloadTempFile must be populated, along with start_byte, to allow the remainder of the encrypted data to be downloaded. Once the encrypted data is downloaded it will be decrypted and written to local_file. hdfs.LocalFile = "../MyFile.zip"; hdfs.EncryptionPassword = "password"; hdfs.DownloadFile(hdfs.Resources[0].Path); //The transfer is interrupted and DownloadFile() above fails. Later, resume the download: //Get the size of the partially download temp file hdfs.StartByte = new FileInfo(hdfs.Config("DownloadTempFile")).Length; hdfs.DownloadFile(hdfs.Resources[0].Path);

get_dir_summary Method

Gets a content summary for a directory.

Syntax

def get_dir_summary(dir_path: str) -> None: ...

Remarks

This method gets a content summary for the directory at DirPath, populating the DirSummary* properties.

DirPath may be empty to get a content summary for the root directory (/).

get_resource_info Method

Gets information about a specific resource.

Syntax

def get_resource_info(path: str) -> None: ...

Remarks

This method gets information about the resource at Path. Path may be empty to get information about the root directory (/).

Calling this method will fire the on_resource_list event, and will populate the Resource* properties with a single item (clearing any previously-held items in the process).

interrupt Method

Interrupt the current method.

Syntax

def interrupt() -> None: ...

Remarks

If there is no method in progress, interrupt simply returns, doing nothing.

join_file_blocks Method

Joins multiple files' blocks together into one file.

Syntax

def join_file_blocks(target_file_path: str, source_file_paths: str) -> None: ...

Remarks

This method joins the blocks from the files at one or more SourceFilePaths onto the end of the file at TargetFilePath. (If this operations is successful, the original source files will no longer be accessible.)

The order of the source files' paths in SourceFilePaths determines the order in which their blocks are joined onto the file at TargetFilePath.

Usage Constraints

The server imposes a number of constraints that must be satisfied in order for the join operation to complete successfully.

TargetFilePath and SourceFilePaths must both be non-empty.

SourceFilePaths must be specified as a comma-separated list of source file paths, with no duplicates or paths that match TargetFilePath.

Additionally, all files referred to by both TargetFilePath and SourceFilePaths must satisfy the following constraints:

• All files must already exist.
• All files must be located in the exact same directory (i.e., all files must be siblings).
• The block size of all source files must be less than or equal to the block size of the target file.

If any of the above constraints are not satisfied, the server will return an error.

The class will take care of verifying that all parameters are non-empty and that all paths begin with a forward slash (/).

list_resources Method

Lists resources in a given directory.

Syntax

def list_resources(dir_path: str) -> None: ...

Remarks

This method lists resources within the directory at DirPath.

Calling this method will fire the on_resource_list event once for each resource, and will also populate the Resource* properties.

DirPath may be empty to list resources in the root directory (/). // ResourceList event handler. hdfs.OnResourceList += (s, e) => { Console.WriteLine(e.Name); }; hdfs.ListResources("/work_files/serious_business/cats"); for (int i = 0; i < hdfs.Resources.Count; i++) { // Process resources here. }

make_directory Method

Makes a directory.

Syntax

def make_directory(new_dir_path: str) -> str: ...

Remarks

This method makes a new directory at NewDirPath and returns the full path of the new directory. Any non-existent parent directories will also be created.

If the CreatePermission configuration setting is non-empty, the directory will be created with the permission it specifies. Otherwise, the server's default (755) will be used.

move_resource Method

Moves a resource.

Syntax

def move_resource(from_path: str, to_path: str) -> None: ...

Remarks

This method moves the resource at FromPath to ToPath.

reset Method

Resets the class to its initial state.

Syntax

def reset() -> None: ...

Remarks

This method resets the class to its initial state.

set_file_replication Method

Sets the replication factor for a file.

Syntax

def set_file_replication(file_path: str, replication: int) -> None: ...

Remarks

This method sets the replication factor for the file at FilePath to Replication.

Valid values for Replication are 1 through 32767 (inclusive); or 0 to reset the file's replication factor back to the server default.

set_owner Method

Sets a resource's owner and/or group.

Syntax

def set_owner(path: str, owner: str, group: str) -> None: ...

Remarks

This method set the owner and/or group of the resource at Path to the given values.

If non-empty, Owner must be a valid user name, and Group must be a valid group name. If either parameter is empty, the file's current value will remain unchanged. (If both parameters are empty, no request is sent.)

set_permission Method

Assigns the given permission to a resource.

Syntax

def set_permission(path: str, permission: str) -> None: ...

Remarks

This method assigns the given Permission to the resource at Path.

Permission must be formatted as an octal permission string. If Permission is empty string, the resource's permission will be reset to the server default (755).

set_times Method

Sets a resource's modification and/or access times.

Syntax

def set_times(path: str, modified_time: int, access_time: int) -> None: ...

Remarks

This method sets the modification and/or access times of the resource at Path to the given values.

Both ModifiedTime and AccessTime should be specified as a number of milliseconds relative to the Unix epoch. If either parameter is negative, the file's current value will remain unchanged. (If both parameters are negative, no request is sent.)

truncate_file Method

Truncates a file to a given size.

Syntax

def truncate_file(file_path: str, new_size: int) -> bool: ...

Remarks

This method truncates the file at FilePath to the given NewSize (specified in bytes). NewSize must not be less than 0 or greater than the file's current size.

This method returns True if the file has been truncated successfully and is immediately ready for further modifications.

This method returns False if the server is still in the process of truncating the file (this could happen, e.g., if the server needs to truncate multiple replicas of the file). In this case, the server will reject any further attempts to modify the file until it has finished truncating it. Monitor the file's size using get_resource_info to determine when the truncation process has finished.

upload_file Method

Uploads a file.

Syntax

def upload_file(new_file_path: str) -> str: ...

Remarks

This method uploads a new file to NewFilePath and returns the full path of the uploaded file. If a file already exists on the server at NewFilePath, the overwrite property controls whether the server will overwrite the file or return an error.

If the CreatePermission configuration setting is non-empty, the file will be created with the permission it specifies. Otherwise, the server's default (755) will be used.

If local_file is set the file will be uploaded from the specified path. If local_file is not set the data in resource_data will be used.

To encrypt the file before uploading it, set encryption_algorithm and encryption_password. hdfs.LocalFile = "../MyFile.zip"; hdfs.UploadFile("/MyFile.zip");

on_end_transfer Event

This event fires when a document finishes transferring.

Syntax

class HadoopDFSEndTransferEventParams(object):
  @property
  def direction() -> int: ...

# In class HadoopDFS:
@property
def on_end_transfer() -> Callable[[HadoopDFSEndTransferEventParams], None]: ...
@on_end_transfer.setter
def on_end_transfer(event_hook: Callable[[HadoopDFSEndTransferEventParams], None]) -> None: ...

Remarks

The on_end_transfer event is fired when the document text finishes transferring from the server to the local host.

The Direction parameter shows whether the client (0) or the server (1) is sending the data.

on_error Event

Fired when information is available about errors during data delivery.

Syntax

class HadoopDFSErrorEventParams(object):
  @property
  def error_code() -> int: ...

  @property
  def description() -> str: ...

# In class HadoopDFS:
@property
def on_error() -> Callable[[HadoopDFSErrorEventParams], None]: ...
@on_error.setter
def on_error(event_hook: Callable[[HadoopDFSErrorEventParams], None]) -> None: ...

Remarks

The on_error event is fired in case of exceptional conditions during message processing. Normally the class fails with an error.

The ErrorCode parameter contains an error code, and the Description parameter contains a textual description of the error. For a list of valid error codes and their descriptions, please refer to the Error Codes section.

on_header Event

This event is fired every time a header line comes in.

Syntax

class HadoopDFSHeaderEventParams(object):
  @property
  def field() -> str: ...

  @property
  def value() -> str: ...

# In class HadoopDFS:
@property
def on_header() -> Callable[[HadoopDFSHeaderEventParams], None]: ...
@on_header.setter
def on_header(event_hook: Callable[[HadoopDFSHeaderEventParams], None]) -> None: ...

Remarks

The Field parameter contains the name of the HTTP header (which is the same as it is delivered). The Value parameter contains the header contents.

If the header line being retrieved is a continuation header line, then the Field parameter contains "" (empty string).

on_log Event

This event fires once for each log message.

Syntax

class HadoopDFSLogEventParams(object):
  @property
  def log_level() -> int: ...

  @property
  def message() -> str: ...

  @property
  def log_type() -> str: ...

# In class HadoopDFS:
@property
def on_log() -> Callable[[HadoopDFSLogEventParams], None]: ...
@on_log.setter
def on_log(event_hook: Callable[[HadoopDFSLogEventParams], None]) -> None: ...

Remarks

This event fires once for each log message generated by the class. The verbosity is controlled by the LogLevel setting.

LogLevel indicates the level of message. Possible values are as follows:

The value 1 (Info) logs basic information, including the URL, HTTP version, and status details.

The value 2 (Verbose) logs additional information about the request and response.

The value 3 (Debug) logs the headers and body for both the request and response, as well as additional debug information (if any).

Message is the log entry.

LogType identifies the type of log entry. Possible values are as follows:

• "Info"
• "RequestHeaders"
• "ResponseHeaders"
• "RequestBody"
• "ResponseBody"
• "ProxyRequest"
• "ProxyResponse"
• "FirewallRequest"
• "FirewallResponse"

on_progress Event

Fires during an upload or download to indicate transfer progress.

Syntax

class HadoopDFSProgressEventParams(object):
  @property
  def direction() -> int: ...

  @property
  def bytes_transferred() -> int: ...

  @property
  def total_bytes() -> int: ...

  @property
  def percent_done() -> int: ...

# In class HadoopDFS:
@property
def on_progress() -> Callable[[HadoopDFSProgressEventParams], None]: ...
@on_progress.setter
def on_progress(event_hook: Callable[[HadoopDFSProgressEventParams], None]) -> None: ...

Remarks

This event fires during an upload or download to indicate the progress of the transfer of the entire request. By default, this event will fire each time PercentDone increases by one percent; the ProgressStep configuration setting can be used to alter this behavior.

Direction indicates whether the transfer is an upload (0) or a download (1).

BytesTransferred reflects the number of bytes that have been transferred so far, or 0 if the transfer is starting (however, see note below).

TotalBytes reflects the total number of bytes that are to be transferred, or -1 if the total is unknown. This amount includes the size of everything in the request like HTTP headers.

PercentDone reflects the overall progress of the transfer, or -1 if the progress cannot be calculated.

Note: By default, the class tracks transfer progress absolutely. If a transfer is interrupted and later resumed, the values reported by this event upon and after resumption will account for the data that was transferred before the interruption.

For example, if 10MB of data was successfully transferred before the interruption, then this event will fire with a BytesTransferred value of 10485760 (10MB) when the transfer is first resumed, and then continue to fire with successively greater values as usual.

This behavior can be changed by disabling the ProgressAbsolute configuration setting, in which case the class will treat resumed transfers as "new" transfers. In this case, the BytesTransferred parameter will always be 0 the first time this event fires, regardless of whether the transfer is new or being resumed.

on_resource_list Event

Fires once for each resource returned when listing resources.

Syntax

class HadoopDFSResourceListEventParams(object):
  @property
  def name() -> str: ...

  @property
  def path() -> str: ...

  @property
  def resource_type() -> int: ...

  @property
  def modified_time() -> int: ...

  @property
  def access_time() -> int: ...

  @property
  def size() -> int: ...

  @property
  def permission() -> str: ...

  @property
  def owner() -> str: ...

  @property
  def group() -> str: ...

  @property
  def replication() -> int: ...

# In class HadoopDFS:
@property
def on_resource_list() -> Callable[[HadoopDFSResourceListEventParams], None]: ...
@on_resource_list.setter
def on_resource_list(event_hook: Callable[[HadoopDFSResourceListEventParams], None]) -> None: ...

Remarks

This event fires once for each resource returned when list_resources or get_resource_info is called.

Name is the name of the resource.

Path is the full path of the resource.

ResourceType reflects the resource's type. Possible values are:

ModifiedTime and AccessTime reflect the resource's last modified and last access times, in milliseconds relative to the Unix epoch.

Size reflects the size of the file, in bytes. Always 0 for directories.

Permission reflects the resource's permission bits, represented as an octal string (e.g., 755).

Owner is the name of the resource's owner.

Group is the name of the resource's group.

Replication reflects the file's replication factor. Always 0 for directories.

on_ssl_server_authentication Event

Fired after the server presents its certificate to the client.

Syntax

class HadoopDFSSSLServerAuthenticationEventParams(object):
  @property
  def cert_encoded() -> bytes: ...

  @property
  def cert_subject() -> str: ...

  @property
  def cert_issuer() -> str: ...

  @property
  def status() -> str: ...

  @property
  def accept() -> bool: ...
  @accept.setter
  def accept(value) -> None: ...

# In class HadoopDFS:
@property
def on_ssl_server_authentication() -> Callable[[HadoopDFSSSLServerAuthenticationEventParams], None]: ...
@on_ssl_server_authentication.setter
def on_ssl_server_authentication(event_hook: Callable[[HadoopDFSSSLServerAuthenticationEventParams], None]) -> None: ...

Remarks

During this event, the client can decide whether or not to continue with the connection process. The Accept parameter is a recommendation on whether to continue or close the connection. This is just a suggestion: application software must use its own logic to determine whether or not to continue.

When Accept is False, Status shows why the verification failed (otherwise, Status contains the string OK). If it is decided to continue, you can override and accept the certificate by setting the Accept parameter to True.

on_ssl_status Event

Fired when secure connection progress messages are available.

Syntax

class HadoopDFSSSLStatusEventParams(object):
  @property
  def message() -> str: ...

# In class HadoopDFS:
@property
def on_ssl_status() -> Callable[[HadoopDFSSSLStatusEventParams], None]: ...
@on_ssl_status.setter
def on_ssl_status(event_hook: Callable[[HadoopDFSSSLStatusEventParams], None]) -> None: ...

Remarks

The event is fired for informational and logging purposes only. This event tracks the progress of the connection.

on_start_transfer Event

This event fires when a document starts transferring (after the headers).

Syntax

class HadoopDFSStartTransferEventParams(object):
  @property
  def direction() -> int: ...

# In class HadoopDFS:
@property
def on_start_transfer() -> Callable[[HadoopDFSStartTransferEventParams], None]: ...
@on_start_transfer.setter
def on_start_transfer(event_hook: Callable[[HadoopDFSStartTransferEventParams], None]) -> None: ...

Remarks

The on_start_transfer event is fired when the document text starts transferring from the server to the local host.

The Direction parameter shows whether the client (0) or the server (1) is sending the data.

on_transfer Event

This event is fired while a document transfers (delivers document).

Syntax

class HadoopDFSTransferEventParams(object):
  @property
  def direction() -> int: ...

  @property
  def bytes_transferred() -> int: ...

  @property
  def percent_done() -> int: ...

  @property
  def text() -> bytes: ...

# In class HadoopDFS:
@property
def on_transfer() -> Callable[[HadoopDFSTransferEventParams], None]: ...
@on_transfer.setter
def on_transfer(event_hook: Callable[[HadoopDFSTransferEventParams], None]) -> None: ...

Remarks

The Text parameter contains the portion of the document text being received. It is empty if data are being posted to the server.

The BytesTransferred parameter contains the number of bytes transferred in this Direction since the beginning of the document text (excluding HTTP response headers).

The Direction parameter shows whether the client (0) or the server (1) is sending the data.

The PercentDone parameter shows the progress of the transfer in the corresponding direction. If PercentDone can not be calculated the value will be -1.

Note: Events are not re-entrant. Performing time-consuming operations within this event will prevent it from firing again in a timely manner and may affect overall performance.

HadoopDFS Config Settings

HadoopDFS Config Settings

<firstlevel>
  <one>value</one>
  <two>
    <item>first</item>
    <item>second</item>
  </two>
  <three>value three</three>
</firstlevel>

{
  "firstlevel": {
    "one": "value",
    "two": ["first", "second"],
    "three": "value three"
  }
}

HTTP Config Settings

TCPClient Config Settings

SSL Config Settings

-----BEGIN CERTIFICATE-----
MIIEKzCCAxOgAwIBAgIRANTET4LIkxdH6P+CFIiHvTowDQYJKoZIhvcNAQELBQAw
...
eWHV5OW1K53o/atv59sOiW5K3crjFhsBOd5Q+cJJnU+SWinPKtANXMht+EDvYY2w
F0I1XhM+pKj7FjDr+XNj
-----END CERTIFICATE-----
\r \n
-----BEGIN CERTIFICATE-----
MIIEFjCCAv6gAwIBAgIQetu1SMxpnENAnnOz1P+PtTANBgkqhkiG9w0BAQUFADBp
..
d8q23djXZbVYiIfE9ebr4g3152BlVCHZ2GyPdjhIuLeH21VbT/dyEHHA
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIEKzCCAxOgAwIBAgIRANTET4LIkxdH6P+CFIiHvTowDQYJKoZIhvcNAQELBQAw
...
eWHV5OW1K53o/atv59sOiW5K3crjFhsBOd5Q+cJJnU+SWinPKtANXMht+EDvYY2w
F0I1XhM+pKj7FjDr+XNj
-----END CERTIFICATE-----
\r \n
-----BEGIN CERTIFICATE-----
MIIEFjCCAv6gAwIBAgIQetu1SMxpnENAnnOz1P+PtTANBgkqhkiG9w0BAQUFADBp
..
d8q23djXZbVYiIfE9ebr4g3152BlVCHZ2GyPdjhIuLeH21VbT/dyEHHA
-----END CERTIFICATE-----

Socket Config Settings

Base Config Settings

HadoopDFS Errors

Common Errors

The class may also return one of the following error codes, which are inherited from other classes.

HTTP Errors

The class may also return one of the following error codes, which are inherited from other classes.

TCPClient Errors

SSL Errors

TCP/IP Errors