How do I automatically add object retention to new files in my bucket?

[Nyxton]

Hello

For a backup project I maintain a bucket with object retention enabled. I need new files which enter the bucket to automatically be retained until a specified time. I currently use a simple script which iterates over all the objects and locks it using gcloud cli, but this isn't production worthy.

It is weird to me how you can't select a directory and just retain all the objects in it as they say folders can't be retained, which is true I suppose, but it could just retain all objects in the folder instead of saying it can't do this.

Is there a way to automate this process, or is there no other way than using a script or doing it manually?

Thank you in advance

[robertcarlos]

Hi @Nyxton,

Welcome to Google Cloud Community!

I would suggest using Cloud Storage Object Lifecycle as this contains the option to retain objects that became noncurrent when a live object has been modified or deleted, as long as Cloud Storage object versioning is enabled.

Please refer to the screenshot below:

You can access this option by clicking on the bucket that you want to edit, then click on LIFECYCLE, finally ADD A RULE.

You can refer to this documentation on managing object lifecycles that you can follow or refer to this configuration examples for Object Lifecycle Management that you can check for your reference.

Hope this helps.

[Nyxton]

Hi @robertcarlos 

I have looked at the possibility of using both Object versioning and retention, but this does not solve the problem at hand for me. This does not ensure immutability of data. If there would be an intrusion and a malicious user got to the bucket, he would be able to disable object versioning and delete previous generations of files, unless I am completely overseeing something.

Object retention does solve this problem, as when an object its retention is set to locked, even an intruder wouldn't be able to delete it. It is just that automating that procedure is slow when doing it manually, or not production worthy when doing it with a script over CLI.

With kind regards

[Nyxton]

an update to this: I decided to ditch the idea of automating object retention, as Google Cloud doesn't really support this. Instead I now use  object versioning with a soft delete policy, which ensures data recovery in the case of a breach. The soft delete policy still is kind of buggy, and the docs point to a 404, but for now it suffices.

[Markus0815]

Hi@Nyxton 
Thanks for your question, I'm looking for exactly the same solution as I want to use an S3 bucket as a WORM backup target. My thought is also to use a script which iterates over all the objects and locks it.
So I'm interested to know why you don't use this solution anymore.
Best regards

[Nyxton]

Hi Marcus

while definitely possible, I don't think a script to be good practice to solve this issue. There are important bits of data in the bucket that I use, so I don't trust a user written script to do the job.

On a sidenote: I eventually choose to protect the data another way and leave object / bucket retention altogether. I now use a soft delete policy so that I can always restore the data if something were to happen to it. 

[alexmoore]

Hiya,

I'm curious to understand why the lifecycle policy wasn't what you were after, what was it your script was doing that was different?

Thanks

Alex