Re-point the staging files bucket

The staging Mongo database is a restore of a production dump, so every document in adb-files.files carries fileLocation.bucket = prod-files-life-connect-fr-5824. The download path signs its S3 GET against the bucket stored on the document, not against the Worker's configured bucket, so the staging credentials hit the production bucket, S3 answers 403, and adb-files maps that to 502: every GET /files/:id fails on staging. tools/scripts/files-bucket-migrate.mjs copies each object into the staging bucket and then re-points fileLocation.bucket. It is dry-run by default, resumable, idempotent, and never deletes anything.

At a glance

Why the bucket is read from the document

This is faithful behaviour, not a port bug — do not "fix" it by forcing the configured bucket:

• JVM oracle — adb-files/.../service/impl/FileServiceImpl.java builds the read as GetObjectRequest.builder().bucket(location.getBucket()) (both the /files/{id} download and the PDF-concat read).
• Port — services/files/src/download.mjs calls getObjectResponse({ ...cfg, bucket: loc.bucket }, key) and maps a non-404 S3 failure to 502.

The object key is the JVM's too: an explicit fileLocation.key wins, otherwise organizations/<organisationId>/files/<fileId> (AwsS3FileLocation.FILE_KEY_PATTERN).

Why the copy carries the object's content-type

The persisted document has no contentType: FileMetaDataEntity (the @Document(collection = "files")) has no such field — contentType lives only on the transport model FileMetaData, where it signs the upload and is then dropped. download.mjs therefore resolves the response type as doc.contentType ?? <the header S3 returns> ?? "application/octet-stream". A copy that forgot the source header would leave every migrated blob as application/octet-stream on staging while production serves application/pdf / image/jpeg — so the migration reads the object with a GET it keeps the headers of, and replays the type on the staging PUT.

S3 user metadata (x-amz-meta-lc_target_id, stamped by the browser POST-policy upload) is not copied: nothing in the JVM or the port ever reads it back. If that ever changes, the copy needs to forward x-amz-meta-* too.

Prerequisites

• pnpm install at the repo root (the script needs the mongodb driver and aws4fetch).
• A read-only IAM credential on the production bucket (s3:GetObject on prod-files-life-connect-fr-5824/*, nothing else). Do not reuse a read-write production key.
• A credential allowed to s3:GetObject and s3:PutObject on the staging bucket, plus s3:ListBucket on the bucket itself. Without ListBucket, S3 answers a GET on a missing key with 403 instead of 404; the script refuses to guess and reports every document as failed (staging S3 GET … responded 403). That is deliberate — treating a denied probe as "absent" would re-point documents at objects nobody verified.
• A staging Atlas connection string with write access to adb-files.
• Enough local bandwidth for the blobs — every object missing from staging is streamed through the workstation (download from production, upload to staging).

Environment

Credentials and URIs are read only from the environment; nothing sensitive is passed on the command line. Export them in a shell that is not logged:

export PROD_S3_ACCESS_KEY_ID=…          # READ-ONLY production credential
export PROD_S3_SECRET_ACCESS_KEY=…
export PROD_S3_REGION=eu-west-3
export PROD_S3_BUCKET=prod-files-life-connect-fr-5824

export STAGE_S3_ACCESS_KEY_ID=…
export STAGE_S3_SECRET_ACCESS_KEY=…
export STAGE_S3_REGION=eu-west-3
export STAGE_S3_BUCKET=stage-files-life-connect-fr-2729

export STAGE_MONGO_URI='mongodb+srv://…'  # staging Atlas
# export STAGE_MONGO_DB=adb-files          # optional, this is the default

The script refuses to start if PROD_S3_BUCKET and STAGE_S3_BUCKET are the same, and lists every missing variable at once. --check does not need the production credential at all.

Every run opens with a header naming the cluster it will act on, credentials stripped:

mode: dry-run
mongo: mongodb+srv://stage-cluster.abcd.mongodb.net -> adb-files.files
buckets: prod-files-life-connect-fr-5824 -> stage-files-life-connect-fr-2729

Read that mongo: line before typing --apply. The script writes to whatever STAGE_MONGO_URI points at; a production connection string pasted there would be re-pointed at a bucket it cannot read. The dry-run prints the same header, so the check costs nothing.

Steps

1. Census (read-only)

node tools/scripts/files-bucket-migrate.mjs --check

Prints the document count per fileLocation.bucket and, over a bounded sample (--sample, default 20), how many of those objects are already in the staging bucket. Nothing is copied or written.

2. Dry run (the default)

node tools/scripts/files-bucket-migrate.mjs
node tools/scripts/files-bucket-migrate.mjs --limit 20   # a smaller first look

Reports what would be copied and re-pointed. It probes the staging bucket but never reads production, never writes, and never touches the manifest. --limit takes a positive integer — --limit 0 is refused rather than read as "no limit"; omitting the flag is the only way to ask for every document.

3. Apply

Start small, then run the rest:

node tools/scripts/files-bucket-migrate.mjs --apply --limit 20
node tools/scripts/files-bucket-migrate.mjs --apply

Per document: copy the object into staging when it is missing → verify it is readable there → only then $set fileLocation.bucket. Each completed document is appended to the manifest (.files-bucket-migrate.manifest.jsonl by default, --manifest <path> to relocate it). Interrupt it with Ctrl-C and re-run the same command: manifest entries are skipped, already-copied objects are not re-uploaded, and already-re-pointed documents are left alone.

A document that fails (unreadable blob, denied read) is logged, recorded in the manifest as failed, and the walk continues; the process exits 1 if anything failed. Re-running retries exactly those.

4. Verify

node tools/scripts/files-bucket-migrate.mjs --check

Every document should now be counted under the staging bucket. Then, in the staging SPA, open a document (a rent receipt or an agency logo) and confirm GET /files/:id returns 200 instead of 502.

What it deliberately does not do

• No deletes. Production objects stay, staging objects are never removed, no document is dropped. Cleaning up an over-copied staging bucket is a separate, manual decision.
• No writes to production. Production is only ever read, with the read-only credential.
• No whole-fileLocation rewrite. Only the bucket field is $set, so _class, an explicit key, and anything else the dump carries survive. The JVM's MetaDataUpdateRepositoryImpl.patch replaces the whole sub-document because it writes a freshly built AwsS3FileLocation; a migration must not drop fields, so it changes the one field that moves.
• No Worker-side override. Nothing in services/files changes; the parity with the JVM read path is preserved.

Open questions

• The production env block of services/files/wrangler.jsonc still names S3_BUCKET_FILES: files.life-connect.fr in eu-west-1, which matches neither the audited bucket (prod-files-life-connect-fr-5824) nor its region (eu-west-3). That is a separate cutover decision, flagged in the file itself — this runbook only concerns staging.
• Objects whose fileLocation has neither a key nor organisationId/fileId are skipped and reported. If --check shows a non-zero count there, those documents need an individual look.
• The copy is sequential and streams through the workstation. If the full run is too slow for the volume, an S3 server-side copy (CopyObject between the two buckets, same region) would be the next step — it needs a credential trusted by both buckets, which is why it is not the default.