Caching Information

All our drive products use various types of caching. Caching is a complex topic and the actual observed behavior can vary. Depending on the specifics of a situation, multiple different actors may be involved. In S3 Drive, caching will help to ensure file content is uploaded to the remote machine.

The S3 REST API does not support writing a few bytes to a specific offset on an existing file. To support randomly writing files and the propagating changes to S3, we created a file-based cache which holds pieces of files on disk and uploads them once the modifications are complete.

Cache Overview

The cache is a storage location for file content and metadata. When applications interact with files on the virtual drive, they might behave differently. The cache mediates access to allow complete files to be uploaded regardless of how they were written. It also helps reduce the overhead of transmitting all operations received from the OS.

S3 Drive will store cache files in the location specified by the Cache Directory setting in the advanced tab of the drive configuration page. The default location is %ProgramData%\S3Drive\<drive> .

File Read

When an application reads a file on the drive, we will start multiple download threads to download the requested byte range and subsequent chunks. If the file is small then the first thread will download the entire content. Once the file is closed, it is removed from the cache.

The CacheEnabledForRead setting in the registry controls whether the requested content from read operations will be stored in the S3 Drive file cache. The CacheConcurrentDownloadThreadsPerFile config defines how many concurrent download threads can be active at the same time (one thread per chunk).

If the size of the file being read is smaller than CachePreDownloadSmallFileSizeLimit, we will pre-download other small files in alphabetical order in the background. The maximum size for files that will be pre-downloaded is controlled by the CachePreDownloadSmallFileSizeLimit setting. The maximum count of small files that will be pre-downloaded is controlled by the CacheConcurrentDownloadFileCount setting.

File Write

Once the file system driver receives a request to write a file, a corresponding cache file is created with the same size as the original. The modified byte ranges are noted in the cache metadata.

If an application writes to a file on the drive and the file is smaller than CacheFileUploadDelayThreshold and the entire file content is modified, then the file will be uploaded as soon as the file is closed. If only a piece is modified, then after the file is closed the entire content will be downloaded and the modifications applied to it. Then the entire file will be uploaded.

If an application writes to a file on the drive and file is larger than the CacheFileUploadDelayThreshold, then the cache will first ensure the chunks of the file are ready to be uploaded. Any chunks of the file which are partially modified will be first downloaded then modifications are applied to them. All modified chunks are uploaded after the file has been closed and CacheFileUploadDelay seconds have passed without any additional operations on the file.

If the file has any chunks which have not been read or modified before, then S3 Drive uses the CopyParts S3 API call to fill in the content of the new file with existing file data on S3.

File Information

The OS will ask for file information frequently. S3 Drive will store file metadata such as size, creation time, and last modified time for a short time specified by the "Directory (Metadata) caching" setting which corresponds with the CacheInfoValidityTime configuration setting in the registry. If the OS sends subsequent requests for file information during this validity interval, the request will be serviced from the cache without provoking another round trip. This is primarily helpful at the S3-protocol level for deletion performance.

Separate from application-level caching, the filesystem driver will cache file information and service requests from the OS to improve the responsiveness of the virtual drive. This is controlled by the UseMetadataCache configuration setting.

Directory Information

S3 Drive will store the results of previous directory listing requests for a short time specified by the "Directory (Metadata) caching" setting which corresponds with the CacheInfoValidityTime configuration setting in the registry. If the OS sends subsequent requests during this validity interval, the request will be serviced from the cache without provoking another round trip.

S3 Drive 2020 Website Download Support Questions / Feedback?
	Caching Information All our drive products use various types of caching. Caching is a complex topic and the actual observed behavior can vary. Depending on the specifics of a situation, multiple different actors may be involved. In S3 Drive, caching will help to ensure file content is uploaded to the remote machine. The S3 REST API does not support writing a few bytes to a specific offset on an existing file. To support randomly writing files and the propagating changes to S3, we created a file-based cache which holds pieces of files on disk and uploads them once the modifications are complete. Cache Overview The cache is a storage location for file content and metadata. When applications interact with files on the virtual drive, they might behave differently. The cache mediates access to allow complete files to be uploaded regardless of how they were written. It also helps reduce the overhead of transmitting all operations received from the OS. S3 Drive will store cache files in the location specified by the Cache Directory setting in the advanced tab of the drive configuration page. The default location is `%ProgramData%\S3Drive\<drive>` . File Read When an application reads a file on the drive, we will start multiple download threads to download the requested byte range and subsequent chunks. If the file is small then the first thread will download the entire content. Once the file is closed, it is removed from the cache. The `CacheEnabledForRead` setting in the registry controls whether the requested content from read operations will be stored in the S3 Drive file cache. The `CacheConcurrentDownloadThreadsPerFile` config defines how many concurrent download threads can be active at the same time (one thread per chunk). If the size of the file being read is smaller than `CachePreDownloadSmallFileSizeLimit`, we will pre-download other small files in alphabetical order in the background. The maximum size for files that will be pre-downloaded is controlled by the `CachePreDownloadSmallFileSizeLimit` setting. The maximum count of small files that will be pre-downloaded is controlled by the `CacheConcurrentDownloadFileCount` setting. File Write Once the file system driver receives a request to write a file, a corresponding cache file is created with the same size as the original. The modified byte ranges are noted in the cache metadata. If an application writes to a file on the drive and the file is smaller than `CacheFileUploadDelayThreshold` and the entire file content is modified, then the file will be uploaded as soon as the file is closed. If only a piece is modified, then after the file is closed the entire content will be downloaded and the modifications applied to it. Then the entire file will be uploaded. If an application writes to a file on the drive and file is larger than the `CacheFileUploadDelayThreshold`, then the cache will first ensure the chunks of the file are ready to be uploaded. Any chunks of the file which are partially modified will be first downloaded then modifications are applied to them. All modified chunks are uploaded after the file has been closed and `CacheFileUploadDelay` seconds have passed without any additional operations on the file. If the file has any chunks which have not been read or modified before, then S3 Drive uses the CopyParts S3 API call to fill in the content of the new file with existing file data on S3. File Information The OS will ask for file information frequently. S3 Drive will store file metadata such as size, creation time, and last modified time for a short time specified by the "Directory (Metadata) caching" setting which corresponds with the `CacheInfoValidityTime` configuration setting in the registry. If the OS sends subsequent requests for file information during this validity interval, the request will be serviced from the cache without provoking another round trip. This is primarily helpful at the S3-protocol level for deletion performance. Separate from application-level caching, the filesystem driver will cache file information and service requests from the OS to improve the responsiveness of the virtual drive. This is controlled by the `UseMetadataCache` configuration setting. Directory Information S3 Drive will store the results of previous directory listing requests for a short time specified by the "Directory (Metadata) caching" setting which corresponds with the `CacheInfoValidityTime` configuration setting in the registry. If the OS sends subsequent requests during this validity interval, the request will be serviced from the cache without provoking another round trip.
Copyright (c) 2022 /n software inc. - All rights reserved. S3 Drive 2020 - Version 20.0 [Build 8318]

S3 Drive 2020

Caching Information