Microsoft Azure Blob

This section assumes that the FileCatalyst server will be installed on a VM in the Microsoft Azure cloud for the greatest performance. It is possible, though not recommended, to install the server outside of Azure. All that is required if installing outside Azure is that the machine have HTTP access to the public internet.

Version Support

The following table shows which versions of FileCatalyst officially support the Azure Blob EFS driver. Note that FileCatalyst introduced support for EFS in v3.5 (for Amazon S3). Subsequent FileCatalyst versions have added support for other drivers. Newer drivers are designed to be backwards compatible in principle, but are not tested or officially supported in earlier product versions unless specified in the table below.

Version Compatible Supported
3.5 Yes No
3.6+ Yes Yes

Prerequisites

  • The Account Key used to define an Azure Blob file system must have the ability to list containers. This allows the file system to be verified and for a container to be selected during setup.

  • Ensure your Azure VM has the appropriate Endpoints defined. TCP port 21, 8000-8100, UDP ports 8000-8100 are the minimum ports to transfer files. Optionally, port 990 (SSL) and port 22 (SFTP) may be opened if you have enabled them. If you require remote administration, TCP port 12400 should be open. Note that these are all default ports. The ports you open should reflect any changes you have made to these defaults.

NOTE: Azure limits the number of endpoints to 150 total. Because of this, the default port range in FileCatalyst Server should be reduced accordingly. By default we recommend ports 8000-8100.
NOTE: Only one endpoint may be added at a time through the Azure portal. Because of this limitation, in order to save time mapping ports you can install Azure PowerShell by following the instructions here.

  • Once PowerShell is installed, you can use the following script to open ports 8000-8100 TCP. You can then replace "TCP" with "UDP" and re-run the script:

Copy 
$i=8000
do
{ Get-AzureVM -ServiceName "[CLOUD_SERVICE_NAME]" -Name "[VM_NAME]" | Add-AzureEndpoint
   -LocalPort $i -PublicPort $i -Name TCP$i -Protocol TCP | Update-AzureVM; $i+=1}
until ($i -gt 8100)

  • Ensure the firewall on your Azure VM instance is either disabled, or configured to allow the same ports as your defined endpoints.

  • For best performance, run your Azure VM in the same region as your Azure Blob Storage.

Getting Started

  1. Install FileCatalyst Server 3.6 or higher.

  2. On first run, install the provided license key from your sales rep.

  3. Modify the Data Port Range (Under Advanced/Network) to match the endpoints you defined for you Azure VM (should be 8000-8100 if the pre-requisite steps were followed).

  4. Modify the IP masquerade address (Under Advanced/Network) to match your Azure VM Public IP address. Note that the Public IP address of your instance in located on the top right corner of the desktop.

  5. On File Systems tab, click "New."

  6. Select "Azure Blob" from the Type drop-down menu.

  7. Enter your Account Name, Account Key, and Endpoint (optional).

  8. Click the Test button to ensure that the credentials are entered correctly.

  9. Click the Save button.

  10. Make sure File System appears in the list.

  11. Create a new user or Edit an existing user. Under the Home Directory section (located on Account Information panel for New user, or Account Settings when editing an existing user) the File System type drop down should have an option "Azure Blob". Select it and click Browse.

  12. Select a home directory like you normally would. Note that the top level directories are the Azure Blob containers. User accounts will only have READ access at the top level, and will have access to all buckets if this is set as the home directory.

You have now successfully setup the FileCatalyst Server to connect to your Azure Blob storage.

Encryption at Rest

The Azure Blob EFS driver supports encryption at rest out of the box. It is automatically enabled on your Azure Blob storage and cannot be disabled. You may change the way that keys are managed for the encryption through the Azure portal. The supported options are Microsoft Managed encryption keys, or Customer Managed encryption keys.

For more information on encryption at rest on Azure, and how to manage encryption keys click here.

URL Composition

When manually defining an Azure Blob EFS URL (i.e. not using the Browse button, or when no static File System has previously been defined) as part of a User home directory or Virtual File/Folder path, the following URL composition should be followed.

Scheme: msab

Default Endpoint: core.windows.net

Full Syntax: msab://<account name>:<account key>@core.windows.net/<container>/<path>

NOTE: The account name and account key are only required when when no File System is defined for the file system you are attempting to connect to.

Limitations

  • The following features are NOT supported in FileCatalyst 3.6 client applications when connecting to a user account backed by Microsoft Azure Blob. Features marked with an asterisk (*) will be implemented in a future release.

  • Keep File Permissions (OSX/Linux Only).

  • Keep file modification timestamp.

  • Transfer in priority sequence - works only by creation date not modified date.

  • Creation/Modification time for directories always show (in listings) as current date since they are virtual objects.

  • Also see Common EFS Limitations.

Notes

Your FileCatalyst Server on your Azure VM sees Azure Blob Storage as a complete file system. However in the background, it is all using the Azure REST API, hence all communication is done with HTTP. This means reading and writing is not going to be as fast as a local hard drive. There are several performance limitations when dealing with Azure Blob. Here is an idea of what you can expect:

  1. Large individual file uploads - Uploads will be fast, up to 480 Mbps write speed for individual files. Files are uploaded in chunks (using parallel HTTP uploads) in the background, and then need to be assembled by Azure at completion. This can cause a pause at the end of each transferred file. The larger the file the longer the pause, though it should not be more than a few seconds for files under 2 GB

  2. Large individual file downloads - Downloads will be fast, up to 480 Mbps write speed for individual files. No pause after download like there is with uploads.

  3. Upload large groups of smaller files - Large groups of small files will upload more slowly than individual files, but multi-client compensates for this just like on local file systems because several files are in transit at one time. See additional note for Multi-Client.

  4. Download large groups of smaller files - Large groups of small files will download more slowly than individual files, but multi-client compensates for this just like on local file systems because several files are in transit at one time. See additional note for Multi-Client

  5. Multi-client transfers - When executing multi-client transfers, the FileCatalyst client may try to initiate many requests in quick succession. If the External File System is unable to process the rapid influx of requests, this may result in some requests being rejected or the UI becoming unresponsive.

  6. Renaming files - Renaming a file is slower the larger the file is. There is no concept of renaming a file in Azure Blob. FileCatalyst must copy the file to the new name, then delete original. It is handled transparently, but the bigger the file, the longer it takes to copy, and hence rename. This means uploading with temporary file names for example will have delays when being renamed at the end of the transfer. The rename is still fairly fast for smaller files but can take several seconds for multi-GB files; in extreme cases the client might also experience a timeout despite the rename being successful. It is recommended that the "Extended Read Timeout" value of the client be increased to accommodate if this issue arises.

  7. Renaming folders - Renaming folders is not possible in FileCatalyst as they are virtual, and would require every file under them to be renamed. As per the previous limitation (renaming files), this would be extremely slow. For this reason renaming folders is currently not implemented.

  8. MD5 checksums - Calculating MD5 checksums requires all bytes of the file to be read by the FileCatalyst Server. Since the files are on Azure Blob, and presumably the server is in Azure cloud as well, the entire file is transferred from Blob storage to you server to calculate the checksum. This means MD5 calculations will take longer than they would on a local file system.

  9. HTTP Uploads - This feature uploads a file in chunks, appending to the file each time. Because of these appends, HTTP uploads may be slower against an Azure Blob EFS than against a local file system.

  10. Upload with Incremental / Deltas - It is highly recommended that this feature be avoided unless bandwidth usage is of higher concern than overall transfer time, as it may be more efficient to re-transfer the entire file than to perform the deltas process. The deltas process performs many small read operations against Azure Blob. These small read operations have much lower performance than larger read operations such as downloading an entire file. Subsequently, the deltas process will be considerably slower against an Azure Blob EFS than against a local file system. Furthermore, if this feature must be used, it is strongly recommended that the user adds a 10+ second delay to progressive transfers or disable progressive transfers entirely.

  11. File size considerations - the maximum size of a single file in Azure Blob is 4.75TB. This is due to a limit in both maximum block size (100MB) and in maximum number of parts per blob (50000).

  12. TCP downloads are slow - FileCatalyst uses HTTP GET with a "range" header in the background when it performs read operations on objects. Because the TCP transfer buffers are small (only 16KB) these read operations are not very efficient, and hence downloads are slow. This will be addressed in a future release.

  13. LINK feature will not work if user is on the root path - FileCatalyst LINK requires the ability to create a directory FC_OUTBOX to store files uploaded to the server. If the user home directory is stored in the root path of the Microsoft Azure account (as opposed to inside a defined container), the server will not be able to create the required folder to temporarily hold files. It is recommended that the user home directory be defined inside an existing container.