Sharding a Cross-Cloud Bucket Migration, and Why AWS DataSync Wasn't Fit for Purpose

Introduction

AWS DataSync is fine for simple, homogeneous transfers and for anything staying inside AWS. It is not fit for purpose for cross-cloud transfers where the source provider uses a different checksum algorithm.

Migrating Google Cloud Storage (GCS) to Amazon S3, AWS DataSync silently wrote every file in the large buckets as 0 bytes, because it could not reconcile GCS's CRC32c checksums with what S3 expects. The fix was to shard the job by bucket and match each shard to a tool that could actually handle it: DataSync for the trivial buckets, rclone's native GCS backend for everything else. The most dangerous part was not the outright failure. It was the 0-byte files that looked like success.

The job

Migrate a set of GCS buckets to S3 for a platform moving off Google Cloud. The buckets varied enormously in scale: some held only a few thousand objects, while the largest held several million. That spread in size and complexity is what made sharding the natural unit of work, because each bucket could be evaluated, and tooled, on its own.

First approach: AWS DataSync, sharded per bucket

DataSync was the obvious first choice. It is managed, it advertises GCS-to-S3 transfers, and it parallelises for you. I set it up in Terraform (an IAM role for the transfer, per-bucket location definitions) and kicked off one task per bucket, planning an initial full load followed by scheduled hourly syncs.

The results split cleanly by complexity:

• The small, simple buckets succeeded.
• The larger buckets failed, but not with a clean error. The files arrived in S3 as 0 bytes.

Root cause: CRC32c versus what S3 will accept

GCS checksums objects with CRC32c. S3 does not use CRC32c as its default integrity mechanism. When DataSync runs in its agent mode, the AWS SDK receives the CRC32c hash from GCS and cannot produce a valid S3 object from it. The transfer "completes", but each object is written as 0 bytes.

This is a checksum-algorithm mismatch at the storage-protocol layer, not a permissions or config error, which is why it is so easy to misdiagnose. Direct access to the source buckets with the GCS tooling worked fine, which pointed at DataSync's write path rather than at source connectivity.

I then tried rclone with its S3-compatible GCS backend (treating GCS as an S3 endpoint). Same failure, one layer down: that backend forwards the CRC32c hash in the PutObject call, and S3 rejects it outright:

BadDigest: The CRC32 you specified did not match the calculated checksum

So the S3-compatible shortcut has the identical defect. The lesson generalises: any tool that treats GCS as if it were S3 inherits GCS's checksum semantics and breaks at the S3 write.

The fix: rclone's native GCS backend

rclone's native google cloud storage backend understands CRC32c correctly. It does not try to pass a GCS checksum through to S3. It reads the object, writes it to S3, and lets each side use its own integrity mechanism. This was the only approach that transferred the complex buckets intact.

The config was two named remotes, one native GCS, one S3:

[gcs]
type = google cloud storage
service_account_file = /root/.config/rclone/gcs-key.json

[s3]
type = s3
provider = AWS
env_auth = true
region = us-east-2

The subtle bit: type = google cloud storage, not type = s3. That one line is the difference between a working transfer and a wall of BadDigest errors.

The sharding strategy

Shard by bucket, run shards in parallel, size the parallelism per shard.

I reused the DataSync agent EC2 that Terraform had already provisioned (an m5.2xlarge in a private subnet, reachable over SSM with no SSH or bastion, an IAM role for S3 write). The DataSync Terraform did not go to waste even though the tool did: it defined the bucket locations and stood up the compute that rclone then ran on.

Each bucket got its own rclone copy in a separate screen window so they ran concurrently and survived disconnects:

rclone copy gcs:<source-bucket> s3:<target-bucket> \
  --transfers 32 --checkers 32 \
  --size-only \
  --progress --stats 30s \
  --s3-upload-concurrency 4

• --transfers 32 --checkers 32: 32 parallel file transfers and 32 parallel existence/size checks. This is the intra-shard parallelism that makes a multi-million-file bucket finish.
• --s3-upload-concurrency 4: multipart concurrency per large object.
• --size-only: see the next section. This one is not optional here.

The large buckets ran in their own long-lived windows; the small ones ran sequentially in a spare window. Verification was rclone size on both sides, comparing total object count and total bytes.

The barrier that nearly shipped corrupt: 0-byte files that looked done

This is the war story worth leading the article with.

DataSync's failed runs did not leave the target buckets empty. They left them full of 0-byte objects stamped with the source files' original modification times. So when rclone ran afterwards with its default comparison (size and modtime), it saw a target object whose modtime matched the source, concluded the file was already transferred, and skipped it, leaving the 0-byte version in place. The migration reported success. The data was not there.

Two things saved it:

1. --size-only, which tells rclone to compare on size alone and ignore modtime. That forces it to notice the 0-byte object differs from the real file and re-copy it. On a clean target you would not need this. On a target polluted by a previous tool's failed run, it is mandatory.
2. An explicit audit that counts 0-byte objects directly rather than trusting a "files transferred" number:

rclone ls s3:<bucket> | awk '$1 == 0' | wc -l

Generalised lesson: idempotency is only as good as the equality check underneath it. "Skip if it already exists" is a lie if "exists" is defined by metadata that a previous broken run set to look correct. Verify on content or size, and count what you actually have rather than trusting the tool's success summary.

Other barriers worth a line each

• Secret handling. The GCS service-account key lived in AWS Secrets Manager, was pulled to disk on the agent only for the duration of a run, and was deleted straight afterwards. Never leave a cross-cloud credential sitting on the box.
• Running as the wrong user. Over SSM you land as ssm-user, but the rclone config lived under /root. Without sudo -s first, rclone reported didn't find section in config file ("gcs"), which reads like a config error and is actually a whose-home-directory error.
• Naming mismatch across clouds. GCS bucket names used underscores; the S3 targets used hyphens. Cross-cloud migrations rarely get to keep identical names, and a silent typo here sends data into the wrong or a non-existent bucket. Map names explicitly rather than assuming they carry across.
• Verification doubles as repair. Because rclone copy only transfers what is missing or different, the verify step is also the fix: if counts do not match, re-run the same command and it fills the gaps. That only holds once --size-only is in place, per the 0-byte trap above.

The real thesis: tier your tools, do not force one

The mistake is assuming a managed, "any source to any target" service is a universal solvent. AWS DataSync makes assumptions about checksums and metadata that hold within AWS and for simple homogeneous transfers, and break precisely at the cross-cloud edge where the source provider does integrity differently. GCS uses CRC32c; S3 does not treat it as authoritative; DataSync cannot bridge that in agent mode.

The right shape was tiered and sharded:

• Keep the managed tool for the shards where it works (small, simple buckets). No need to hand-roll those.
• Drop to a purpose-built tool with a correct native backend (rclone's GCS backend) for the shards where the managed tool fails.
• Reuse the failed tool's infrastructure (the agent EC2, the location and IAM definitions) rather than tearing it down. The compute and access were fine; it was the transfer engine that was wrong.

Sharding by bucket is what made this possible. It let me evaluate each shard's result independently, isolate the failures to the complex buckets, and swap the engine for just those without redoing the whole job.

Takeaways

• AWS DataSync is an in-AWS and simple-transfer tool. Cross-cloud with a different checksum algorithm on the source is its blind spot.
• GCS CRC32c does not survive a naive write to S3. Use a tool with a native GCS backend, not one that treats GCS as S3-compatible.
• A silent failure that leaves 0-byte files is worse than a loud one. Verify on content or size, never on metadata a broken run may have set.
• Idempotent "skip if exists" is only safe if "exists" means "correct", not "present". --size-only over the default modtime comparison was the difference.
• Shard the job so you can match each shard to the tool that fits it, and so a failure is contained to one shard instead of the whole migration.
• Reuse infrastructure even when you abandon the tool it was built for.