For example, type log or .log to exclude *.log files. Enables you to choose whether to receive Cloud Sync notifications in Cloud Managers Notification Center. For example, heres a recommendation for an Azure NetApp Files relationship: The deployment of the Google Cloud data broker has been simplified. You can now update the Box credentials for an existing sync relationship. View the list of supported sync relationships.
You must now select an NFS version or SMB version when setting up a new sync relationship for an NFS or SMB server. A new Compare by setting is now available for each sync relationship. When syncing data, Cloud Sync copies ACLs between NFS servers that use NFS versions 4.0, 4.1, or 4.2. When a Google Cloud Storage bucket is the target in a sync relationship, you can now choose the storage class that you want to use. Two new sync relationships are supported: IBM Cloud Object Storage to IBM Cloud Object Storage. When creating a sync relationship between an Azure Blob container and an NFS or SMB server, you must now provide the storage account connection string. StorageGRID Webscale is now supported as the source in a sync relationship. Cloud Sync now provides recommendations when it identifies ways for you to accelerate or optimize a sync relationship. View the supported storage classes. Choose whether Cloud Sync should compare certain attributes when determining whether a file or directory has changed and should be synced again. Cloud Sync now provides a Google login form and a quick deployment wizard. A new reporting feature provides information that you can use with the help of NetApp personnel to tune a data brokers configuration and improve performance. On the Bucket Setup page, set up the bucket: Choose whether to enable S3 bucket encryption and then select an AWS KMS key, enter the ARN of a KMS key, or select AES-256 encryption. Learn more about this setting. It must be a regular connection string, not a shared access signature (SAS). Learn how to deploy a data broker in Google Cloud. The standalone Cloud Sync service has been retired. Cloud Sync now supports sync relationships from SFTP to S3 by using the API. When setting up a relationship, you need to provide a project ID, client email, and private key for a Cloud Storage service account. You can add the name when you create the relationship and any time after. Each report provides in-depth details about a path in a sync relationship: how many directories and files there are, the distribution of file size, how deep and wide the directories are, and more. Cloud Sync then shows you the right commands to use. We made several enhancements to data broker groups: You can now move a data broker to a new or existing group. If you extended a Cloud Sync license that you purchased from NetApp, you can add the license again to refresh the expiration date. Documentation for each Cloud Sync API is available from https://api.cloudsync.netapp.com/docs. Go to the Cloud Sync section in the Cloud Manager documentation. Cloud Sync now supports copying ACLs from object storage to SMB shares. You can now choose whether to copy access control lists (ACLs) between a source NFS server and a target NFS server when using NFS version 4 or later. These reports are now supported with object storage. These stats enable you to see more details about the most recent data sync. The on-prem data broker is now supported with additional Linux distributions: SUSE Linux Enterprise Server 12.4 SP5 and 15 SP1. You will need to register your support serial number to activate support before you can contact NetApp technical support. The following steps provide an example of how to create a sync relationship from an NFS server to an S3 bucket. Learn more about using a HashiCorp Vault with a data broker. For example, you can filter on sync relationships that have a failed status. When an Amazon S3 bucket is the target in a sync relationship, you can now enable S3 bucket encryption by selecting an AWS KMS key, entering the ARN of a KMS key, or by selecting AES-256 encryption. The following image shows the Schedule setting for a new sync relationship: While this image shows the Schedule setting for an existing sync relationship: Additional statistics are available for each sync relationship on the Sync Relationships page. Copy ACLs from an SMB server by enabling a setting when you create a relationship or after you create a relationship. We enhanced how Cloud Sync copies access control lists (ACLs) between SMB servers. You can now update Cloud Sync licenses that you extended. Two new settings are available for sync relationships: When you enable this setting, Cloud Sync deletes files from the source location after it copies the files to the target location. To get the result of the operation, you need to use the following API: The result is an array with one object that includes an ID field. Open the file and update it as follows: Choose to delete files from the target location, if they were deleted from the source. Cloud Sync syncs data at the next scheduled time. This may take a few minutes. When an SMB server is the source, this setting enables you to sync files that were created after a specific date, before a specific date, or between a specific time range. On the Define Sync Relationship page, choose a source and target. You now choose whether the data broker should have AWS access, GCP access, or both, and the type of proxy configuration that you have. You can now sync data from an NFS server or SMB server to Azure Data Lake Storage Gen2. Choose to delete files from the source location after Cloud Sync copies the files to the target location. This message can help to ensure that youre getting the latest features and functionalities. Define the number of times that Cloud Sync should retry to sync a file before skipping it. When you call the list-s3-buckets API, a result is not immediately returned: The result is HTTP status code 202, which means the message was accepted, but was not processed yet. This setting enables you to define whether Cloud Sync should cancel a data sync if the sync hasnt completed in the specified number of hours or days. If there are any changes, then it syncs those files and directories. You can schedule a relationship to sync data as often as every 1 minute. Learn more about data-in-flight encryption. You no longer need to use a CloudFormation template to install the data broker in AWS. You can now view and select objects from S3 buckets that belong to other AWS accounts if they are shared with your account. The following video provides a short demo: Choose to sync all files regardless of their size or just files that are in a specific size range. View the list of supported sync relationships. This advanced setting enables you to choose whether Cloud Sync should compare certain attributes when determining whether a file or directory has changed and should be synced again. Copying metadata is supported with "cloud-to-cloud" relationships between any of the following endpoints: Choose to exclude files that were recently modified prior to the scheduled sync. Define the file types to include in each sync: files, directories, and symbolic links. Google Cloud Documentation: Creating and Managing Service Account Keys. When setting up a sync relationship, you can choose from Google Cloud buckets in different projects, if you provide the required permissions to the data brokers service account.
On the Settings page, define how source files and folders are synced and maintained in the target location: Choose a recurring schedule for future syncs or turn off the sync schedule. A new Sync Timeout setting is now available for sync relationships. Click View details about sync failures (as shown in the image above) and youll see a page similar to the following: Click Download Data Broker Logs to download logs that can help with troubleshooting.
If you want to create sync relationships for any of the following, then you first need to create or discover the working environment: Create or discover the working environment. Obtain a user token from NetApp Cloud Central. This sync relationship was previously supported with the API only. For example, when you create a new sync relationship, you select the data broker group to use with the relationship, rather than a specific data broker.
When you create a sync relationship to or from Google Cloud Storage, Cloud Sync no longer prompts you to provide a project ID, client email, and private key for a Cloud Storage service account. Cloud Sync now copies metadata between Google Cloud Storage and S3 providers (AWS S3, StorageGRID, and IBM Cloud Object Storage). The ID field represents the last message that the server sent. For more details, refer to the following pages: Installing the data broker on a Linux host. Cloud Sync is integrated with Cloud Data Sense. View the list of supported sync relationships. After you initiate a data sync from Cloud Data Sense, all of the source information is contained in a single step and only requires you to enter a few key details.
For example, the following image shows that 684 directories were scanned and no files were marked for copy or for deletion: Cloud Sync now supports deploying a data broker in Google Cloud Platform. Note that the IAM role associated with your data broker will need the following permission to use this feature: This permission is automatically added to any new data brokers that you create. In the Manage Data Brokers tab, we also show the number of sync relationships that a data broker group is managing. You cant abort syncs, modify settings, or view reports. A new Notifications setting is now available for sync relationships. Cloud Sync supports the following storage classes: You can now view the errors found in reports and you can delete the last report or all reports. For example: You would now make the following API call using the ID that you just received: The result is an array of messages. When AWS S3 is the target in a sync relationship, Cloud Sync tags S3 objects with metadata thats relevant to the sync operation. You can now use Cloud Sync with S3 buckets that are protected with AWS KMS encryption. The Cloud Sync capabilities that are available through the web UI are also available through the RESTful API. Learn how to update credentials. The credentials that you provide while using the Cloud Sync service are stored directly on the data broker machine. A data broker that uses a proxy server can now be part of a sync relationship that includes an Azure Blob container. You can now sync data between two Azure Blob containers. You can now sync data between NFS servers using data-in-flight encryption. Sync relationship settings were moved to a new page when setting up a relationship and when editing a relationship. The sync status for an in-progress sync now shows an estimated time remaining that is more precise. Cloud Sync now supports additional sync relationships for Box when using the Cloud Sync API: Learn how to set up a sync relationship using the API. We changed how you interact with data brokers by emphasizing data broker groups. This does not break the sync relationship. Two new regions are now supported in AWS: Middle East (Bahrain) and Asia Pacific (Hong Kong). Review the details of the sync relationship and then click Create Relationship. When you create a sync relationship, Cloud Sync now displays all volumes on a source Cloud Volumes ONTAP system, on-premises ONTAP cluster, or FSx for ONTAP file system. Learn more about these reports. Theres no need to rescan the source at scheduled intervals. In this update, we enhanced the integration by making it easier to create the sync relationship. For NFS, you can select version 3, 4.1, or 4.2. Cloud Sync now provides a Microsoft Azure login form and a quick deployment wizard. The service account associated with your data broker needs the following permissions to use this setting: Learn more about the Continuous Sync setting. For SMB, you can select version 1.0, 2.0, 2.1, or 3.0. Creating a sync relationship from SFTP to Amazon S3 is now supported in the user interface. NFS to NFS sync relationships are now supported. You can enable notifications for successful data syncs, failed data syncs, and canceled data syncs. When setting up a sync relationship to an Amazon S3 bucket, the Sync Relationship wizard now enables you to define the tags or metadata that you want to save on the objects in the target S3 bucket. Learn how to accelerate sync performance. Select a working environment that matches any of the types listed above. Learn more about installing the data broker in Google Cloud. Prior to this update, the only data broker supported with this feature was the On-Prem Data Broker option with a CentOS 7.0 host.
Two new sync relationships are supported: You can now provide a meaningful name to each of your sync relationships to more easily identify the purpose of each relationship. Previously, Cloud Sync would only display the volumes that matched the selected protocol. Heres an example: Grouping data brokers together can help improve the performance of sync relationships. The most typical issues are related to connectivity or permissions. Learn more about creating and viewing reports to tune your configuration. The only response from the server is HTTP status 202. Specify file extensions to exclude from the sync by typing the file extension and pressing Enter. A few bugs were fixed. Learn how to deploy the data broker in GCP. The deployment of the Azure data broker has been simplified. This may take a few minutes. Cloud Sync now supports sync relationships between ONTAP S3 Storage and a Google Cloud Storage bucket from the user interface. When creating a new sync relationship, you can choose an S3 bucket that is not associated with your AWS account. Credentials are now locally encrypted on the data broker machine. The API response includes an "expires_in" field that states when the token expires. You can access the option when selecting a directory: The following options are available when filtering source objects: Incremental updates from an S3 bucket to an NFS or SMB server are no longer event-driventhey are based on a sync schedule. We improved the sync performance of folders that contain greater than 10,000 files. After the initial copy, the service syncs any changed data every 24 hours. You can now restart a data broker from Cloud Sync. You still have the option to use a CloudFormation template, if youd rather not provide an access key. Choose all files regardless of their last modified date, files modified after a specific date, before a specific date, or between a time range. The following example shows an API call to create a data broker in Microsoft Azure. When the data broker copies data from the source to the target, it goes through the private link. Starting today, this feature works with any type of data broker: the AWS, Azure, Google Cloud Platform, or on-prem data broker. This setting is available only when creating a sync relationship and when you sync from an S3 bucket or Google Cloud Storage to S3, Google Cloud Storage, Azure Blob storage, StorageGRID, or IBM Storage. Youll need to add the token and account ID to the Authorization header when making API calls. Learn how to install the data broker on an existing Linux host. To get started with the Cloud Sync API, you need to obtain a user token and your Cloud Central account ID. You can now set up sync relationships to sync data between: An AWS S3 bucket and an Azure Blob container, An AWS S3 bucket and a Google Cloud Storage bucket. Use these steps to sync data to or from a supported storage type other than Amazon FSx for ONTAP, Azure NetApp Files, Cloud Volumes ONTAP, or on-prem ONTAP clusters. File metadata is now copied between sync relationships that include an S3 bucket and StorageGRID. NetApp periodically updates Cloud Sync to bring you new features, enhancements, and bug fixes. When creating a sync relationship that includes Azure Data Lake, you need to provide Cloud Sync with the storage account connection string. Select Sync data from this location or Sync data to this location and follow the prompts to set up the sync relationship. Learn more about changing the settings for a sync relationship. For Azure and Google Cloud Storage, only the metadata option is available. When creating a sync relationship, you can now add tags to the object storage target in a sync relationship. Cloud Sync is now integrated with NetApp Cloud Central, which enables centralized user authentication. For example, you can exclude files modified 30 seconds before the scheduled sync. You can now download data broker logs at any time. Cloud Sync now supports Box as the source in a sync relationship to Amazon S3, StorageGRID, and IBM Cloud Object Storage when using the Cloud Sync API. This same feature is available when syncing data to StorageGRID and IBM Cloud Object Storage. Cloud Sync now enables you to update the data broker with the latest credentials of the source or target in an existing sync relationship. Thanks for your patience. When a sync is in progress, Cloud Sync now displays the remaining estimated time to finish syncing the data. This setting is available when syncing from a Google Cloud Storage bucket to S3, Google Cloud Storage, Azure Blob storage, StorageGRID, or IBM Storage. You can now change the sync schedule for a relationship to as frequently as 5 minutes. The default is 24 hours. You can disable tagging of S3 objects, if its not desired in your environment. Click Add to the list to select an S3 bucket that is not associated with your AWS account. Previously, we only supported copying ACLs from an SMB share to object storage. Before you can create some types of sync relationships, youll first need to create a working environment in Cloud Manager. This setting is available only when syncing from an S3 bucket to S3, Google Cloud Storage, Azure Blob storage, StorageGRID, or IBM Storage. EFS to S3 and S3 to EFS sync relationships are now supported. If this happens, you can now click Add Share Manually and enter the name of an SMB share. These credentials are now encrypted using HashiCorp Vault. You cant edit this option after you create the relationship. Cloud Sync now displays the number of failed transfers. Cloud Sync now supports deploying the data broker in the AWS Asia Pacific (Jakarta) region. Review details about using your own IAM role. Filtering source objects enables you to define how source files and folders are synced and maintained in the target location. You should consult with NetApp to understand when to change the configuration and how to change it. We redesigned the table view on the Dashboard for ease of use. When setting up the sync relationship, you simply need to enter the Azure storage account name and the access key for the storage account. After the initial data sync, Cloud Sync listens for changes on the source S3 bucket and continuously syncs any changes to the target as they occur. The on-prem data broker can run any supported operating system. This update includes performance improvements and usability enhancements to the data broker creation wizard. When you sync data to or from Amazon S3, you can choose whether to use an S3 private link. Thanks for your patience. SMB to SMB sync relationships are now supported. You can now sync NFS data from Azure NetApp Files to Azure NetApp Files using data-in-flight encryption. Cloud Sync now gives you the option to enter an AWS access key and then fill out a quick deployment wizard. Then you can select the Blob container. Specific permissions must be applied to the S3 bucket. After the initial data sync, Cloud Sync listens for changes on the source Google Cloud Storage bucket and continuously syncs any changes to the target as they occur. After logging in to Cloud Manager, you can switch to the Sync tab at the top and view your relationships, just like before. When you deploy a data broker in Google Cloud, you can now choose whether to enable or disable a public IP address for the virtual machine instance. You can now set up the data broker to access credentials from an external HashiCorp Vault by authenticating with a Google Cloud service account. Learn more about ONTAP support for Amazon S3. EFS to NFS and NFS to EFS sync relationships are now supported. Enable this option to copy object storage metadata and tags. If a user changes the metadata on the source, Cloud Sync copies this object in the next sync, but if a user changes the tags on the source (and not the data itself), Cloud Sync doesnt copy the object in the next sync. Azure NetApp Files (SMB) is now supported as the source or target in a sync relationship. If Cloud Sync is unable to retrieve the exports, click Add Export Manually and enter the name of an NFS export. If you enable this setting, it affects other features as follows: The following settings are reverted to their default values: Sync Timeout, Recently Modified Files, and Date Modified. A new setting enables you to continuously sync changes from a source S3 bucket to a target. Note that the IAM role associated with your data broker will need the following permissions to use this setting: These permissions are automatically added to any new data brokers that you create. Cloud Sync now provides additional stats about the last data sync for each sync relationship: How many directories and files were scanned, How many directories and files were marked for copy and marked for deletion. Learn more about installing the data broker in AWS. You can choose to enable or disable Cloud Sync from comparing the following attributes: mtime: The last modified time for a file. On the Tags/Metadata page, choose whether to save a key-value pair as a tag on all files transferred to the S3 bucket or to assign a metadata key-value pair on all files. This is in addition to copying metadata. If you click More info, Cloud Sync filters the dashboard to show you more information about that specific relationship. Choose all files regardless of their last modified date, files modified after a specific date, before a specific date, or between a time range.