One Bucket, Two Terraform Owners - the Last apply Wins

It started as an ordinary cleanup problem. Users upload media files (recordings and images) through presigned URLs. The server issues an upload URL, the client uploads straight to S3, then calls a commit API to say "register this key as a real asset." The problem is what happens when someone gets a presign but never commits. The app crashes, the network drops, the user leaves the screen, and the bucket is left with an object that isn't registered anywhere.

I wanted a lifecycle rule to clean these up, but there was no way to write one. Committed and uncommitted objects were mixed under the same prefix, so any rule that says "delete old things" would delete real assets too. A daily upload quota kept the pile from growing fast, but the fact remained: there was no path to reclaim the space.

The design: what isn't committed lives in tmp

The backbone of the fix is key namespace separation.

1. presign issues a temporary key under the tmp/ prefix.
2. When commit passes validation (existence check via HEAD, Content-Type, size limit), it promotes the object to its final key with CopyObject and deletes the tmp original.
3. Objects whose commit never arrives stay in tmp/, and a lifecycle rule expires them after 7 days.

Now the lifecycle rule only has to look at tmp/. Real assets are outside its blast radius from the start. The clients didn't need to change. I read all three upload flows to confirm this: every one of them uses the key returned in the commit response for its follow-up calls, so the server can change the key shape without them noticing. Commits for old-format keys already in flight at deploy time still go through the existing path.

One trap here. This bucket has versioning enabled. On a versioned bucket, expiration doesn't delete an object. It only adds a delete marker, and the original bytes stay behind as a noncurrent version. Without a paired noncurrent_version_expiration (1 day), the cleanup runs and not a single byte is reclaimed.

The design review paid for itself too. Putting a review on the promotion logic surfaced three confirmed defects. A hole where an admin-only custom key path could commit a tmp/ key as the final key, letting the lifecycle rule delete a live asset (blocked tmp final keys at the shared commit boundary). The promotion's sourceKey acting as an arbitrary copy primitive (enforced the equation between tmp and final keys). And an idempotency problem where two concurrent commits of the same tmp key make one side's copy die with a 404. All three were fixed in code before moving on.

And then apply failed

I added the lifecycle rule in Terraform and merged. The staging apply failed.

The error was a timeout writing the lifecycle configuration, but digging in, the timeout was a symptom. The structure was the problem. Two Terraform resources each owned this bucket's lifecycle. One lived inside the image resize module (the module attached its own rule for its own purposes). The other lived in the per-environment bucket config file (where I had just added the tmp rule). Same physical bucket, two owners.

S3's lifecycle API has no "add one rule" operation. PutBucketLifecycleConfiguration replaces the entire document. Terraform's aws_s3_bucket_lifecycle_configuration sits on top of that, so the resource overwrites the bucket's whole lifecycle document with just the rules it knows about. With two owners, each keeps pushing a document containing only its own rules. If they meet inside one apply, the concurrent writes time out. If they meet separately, it's worse: the apply succeeds without any error, and the last winner erases the other side's rules.

That's the coldest part of this incident. The timeout was actually the lucky outcome, because it was a loud failure. If this structure had stayed alive, the next apply that touched the image resize module for any reason would have silently deleted the tmp expiration rule I had just added. The plan would have shown nothing but a one-line in-place update on that bucket resource, and uncommitted objects would have started piling up forever again. No monitoring anywhere would have noticed that a cleanup rule had vanished.

In fact, this exact spot had been flagged once during code review, as a suspicion that "these two resources seem to touch the same bucket." At the time it was filed as a potential risk. It took only minutes after the merge for potential to become real, when the apply failed.

The fix: one owner

The direction was clear. Each bucket must have exactly one lifecycle owner. I removed the lifecycle resource from the module and made the per-environment config file the single owner, moving the module's rule over as well.

The tool for this was Terraform's removed block. If you just delete the resource declaration from the module, Terraform tries to destroy the real lifecycle configuration. That opens a window where the rules currently live in production get deleted and then recreated. Give the removed block destroy = false and Terraform forgets the resource in state only. The real thing stays untouched while it disappears from the ledger, so ownership transfers to the environment config with zero downtime.

After the fix, the staging apply passed, and after rolling out to prod I checked the actual bucket state directly through the AWS API. Both buckets had the tmp expiration rule alive, and all three pre-existing rules intact. The reason I didn't stop at terraform plan is exactly what this incident taught me: a plan is just each owner's ledger, and in a document-overwrite structure, two ledgers know nothing about each other.

What it leaves behind

A document-level resource must have exactly one owner. Not just S3 lifecycle. Bucket policies, CORS configuration, notification configuration - every Terraform resource built on a "replace the whole thing" API has the same property. If declarations pointing at the same target exist once in a module and once in the environment config, you don't get a compile error. You get a silent overwrite at runtime. Before adding a new rule, first go find out whether that document already has an owner.

One more. Deleting a resource declaration and deleting the real thing are different acts, and Terraform expresses that distinction with the removed block. When the goal is an ownership transfer, the moment you go through destroy, zero downtime is gone.

Finally, a reclamation path starts with key design. When data with different lifespans - committed and uncommitted - lives in the same namespace, no cleanup rule can be written safely. If the lifespans differ, separate where they live first. For the record, this approach does not retroactively reclaim the uncommitted objects that already piled up. Those objects can't be distinguished from real assets (that was the starting point of this whole problem), so I deliberately chose to apply the scheme to new uploads only.