Storage Mover Not Listing All AWS S3 Buckets During Job Creation

Thank you for the detailed guidance. I have verified all the points you mentioned.

However, after waiting 6 hours (instead of 1 hour), the inventory shows 94 out of 120 buckets. The scan is bringing in buckets from other regions that we don't need, while failing to retrieve the specific buckets we actually want to migrate.

I suspect the issue might be related to the object count in the buckets. The S3 buckets that are currently not being listed contain over 700 million objects each.

Could the high object count be causing the inventory scan to fail or timeout for these specific buckets?

Some additional context:

• The missing buckets are in the eu-west-1 region as expected
• IAM permissions include s3:ListBucket, s3:GetBucketLocation, s3:GetObject (note: s3:ListAllMyBuckets permission is not included, but other buckets are being retrieved successfully without this permission)
• No VPC endpoint or IP restrictions in the bucket policies
• All buckets are within the same AWS account
• The buckets contain standard S3 objects (no Access Points, Outposts, or Glacier)

The main concern is whether Azure Storage Mover has limitations when processing buckets with extremely large object counts (700M+ objects), and if there are any workarounds or configuration adjustments to handle such large-scale buckets.

Could you please advise on:

1. Any known limitations for high object count buckets?
2. Recommended approaches for inventorying buckets with 700M+ objects?
3. Whether we should consider splitting the inventory process or using alternative methods?

Thank you for your assistance.Thank you for the detailed guidance. I have verified all the points you mentioned.

However, after waiting 6 hours (instead of 1 hour), the inventory shows 94 out of 120 buckets. The scan is bringing in buckets from other regions that we don't need, while failing to retrieve the specific buckets we actually want to migrate.

I suspect the issue might be related to the object count in the buckets. The S3 buckets that are currently not being listed contain over 700 million objects each.

Could the high object count be causing the inventory scan to fail or timeout for these specific buckets?

Some additional context:

• The missing buckets are in the eu-west-1 region as expected
• IAM permissions include s3:ListBucket, s3:GetBucketLocation, s3:GetObject (note: s3:ListAllMyBuckets permission is not included, but other buckets are being retrieved successfully without this permission)
• No VPC endpoint or IP restrictions in the bucket policies
• All buckets are within the same AWS account
• The buckets contain standard S3 objects (no Access Points, Outposts, or Glacier)

The main concern is whether Azure Storage Mover has limitations when processing buckets with extremely large object counts (700M+ objects), and if there are any workarounds or configuration adjustments to handle such large-scale buckets.

Could you please advise on:

1. Any known limitations for high object count buckets?
2. Recommended approaches for inventorying buckets with 700M+ objects?
3. Whether we should consider splitting the inventory process or using alternative methods?

Thank you for your assistance.

it isn’t a region cap, your missing buckets are almost certainly a combo of inventory scope + IAM browse permissions + preview scale limits. Fix it like this: (1) On the multicloud connector, edit Inventory and uncheck “all regions,” include only eu-west-1, and let a rescan finish (it can take a while). (2) Give the connector role explicit S3 browse rights: add s3:ListAllMyBuckets, s3:GetBucketLocation, and s3:ListBucket (these are commonly required for UIs to list buckets), then rescan.  (3) Be aware of preview limits: cloud-to-cloud jobs currently support 10M objects per job (and 10 concurrent jobs per subscription). With buckets holding 700M+ objects, even when they appear you’ll need to split migration into many jobs by prefix (e.g., key ranges/folders) or use an alternate mover.  (4) For truly huge namespaces, consider tools built for very large object counts (parallel AzCopy S3→Blob, or Data Factory with prefix filters) and run in batches; Storage Mover’s general scale guidance tops out at ~500M items tested, so your 700M buckets are beyond that comfort zone.  (5) If after (1)–(2) the buckets still don’t show, re-deploy the connector (CloudFormation) and verify the Inventory solution is enabled before Storage – Data Management; bucket discovery can take up to ~an hour after first connect.

Thank you for the detailed guidance. However, I should clarify that I have not yet applied the suggestions you mentioned, and the region is already set to eu-west-1 only.

This morning, the scan successfully detected buckets with over 700 million objects, and I was able to complete migration of partial data from those buckets using Storage Mover. However, I'm now experiencing a different issue: the bucket list in the Connector's resources has been reset and is showing only a few buckets again, whereas previously it displayed the full list including the large buckets.

Could you help me understand what might be causing this bucket list reset behavior? The successful scan and migration this morning confirms that the connector was working properly at that time. However, the inventory seems to be reverting to a limited bucket view without any configuration changes on my part.

Specific observations:

• Morning scan: Successfully listed 700M+ object buckets and completed partial migration
• Current state: Bucket list has reset to showing only a subset of previously visible buckets
• No configuration changes were made between successful scan and current limited view
• Region setting: Already configured to eu-west-1 only (not "all regions")
• IAM permissions and bucket policies: No changes made since successful morning scan

Is this related to inventory caching, scan timeouts, or some form of automatic inventory refresh cycle that might be filtering out large buckets? I'd appreciate guidance on maintaining consistent bucket visibility in the Storage Mover inventory.

Thank you for your assistance.

Hello @윤호성,

Hope you're doing well!

Thank you for sharing these additional details.

From your description, the fact that previously visible S3 buckets (including the large ones with 700M+ objects) showed up during the morning scan but later reset to only a smaller subset doesn’t appear to be a region-related issue. Instead, it’s more likely related to how the Storage Mover Inventory solution refreshes and manages visibility.

1)The Inventory solution can be set up with a Periodic sync schedule, which means the connector will automatically re-scan your AWS account at regular intervals. As a result, the list of buckets may change over time depending on the filters applied and the progress of the current scan. If a scan of a large bucket doesn’t finish before the next refresh starts, that bucket may temporarily disappear from the visible list until the scan completes successfully.

Could you please check whether Periodic sync is enabled and confirm the interval that’s configured? You might want to try increasing the interval or turning it off temporarily. Also, re-saving the region filter (set to eu-west-1 only) can help ensure it’s being applied consistently.

2)The connector’s internal cache can reset when the connector is redeployed or when the Inventory solution is reconfigured, disabled, or re-enabled. Similarly, restarting or resetting the agent can trigger a refresh. In these situations, some buckets that were previously visible may temporarily disappear until the next full scan completes.

Take a look at the connector’s deployment or change history to see if there was a reset or redeployment. If you find one, make sure the Inventory solution has been turned back on and that the correct filters are applied.

3)Buckets with very large object counts (such as your 700M+ ones) can take a long time to scan. If the refresh cycle or a timeout happens before the scan finishes, those buckets might temporarily disappear from the visible list until the next scan completes.

If logging is enabled, review the connector logs to see whether large bucket scans are being interrupted. For better performance, you might want to split the migration by prefix or folder so that each job handles a smaller set of objects at a time.

You can refer to the public document below, which provides step-by-step guidance on setting up multi-cloud connectors, configuring endpoints, and securely migrating data from AWS S3 to Azure Blob Storage.

Supporting document:

https://learn.microsoft.com/en-us/azure/storage-mover/cloud-to-cloud-migration