Gsutil cp from storage bucket to compute instance using Identity-Aware Proxy (IAP)

[jomram]

If I create a compute engine instance with no external IP assigned. I can successfully login to the node using gcloud compute ssh "vm_name" --tunnel-through-iap 

I would now like to transfer files to the vm from a storage bucket using gsutil, for example (after connecting to vm with ssh) :

gsutil cp gs://bucket_name/file_name outfile_name

I receive, after a while, this message:

INFO 1106 09:11:03.773168 retry_util.py] Retrying request, attempt #1...

The transfer is never successful. I think it cannot connect.

It seems that IAP works over port 22 at least, but does not work with gsutil. I realise the instance has no internet connection, as it has no external IP assigned, but I am trying to use gsutil to copy a file from a Google Cloud bucket in the same region, so anticipated this would work. 

How can I get gsutil to work in this case? (or otherwise how can I transfer a file from a storage bucket to an instance without an external IP - I could not find it documented how to do this)

Thanks

[christianpaula · Accepted solution]

Hi @jomram,

Welcome to Google Cloud Community!

A few approaches to achieve your goal of transferring files from a storage bucket to an instance without an external IP:

1. Use a temporary external IP:

• Assign a temporary external IP address to your instance.
• Connect to the instance using SSH with the temporary IP address and tunnel through IAP.
• Use gsutil to copy the files from the bucket to the instance.
• Remove the temporary external IP address after the transfer is complete.

This approach is straightforward, but it requires managing the external IP address lifecycle.

2. Use Cloud Storage Transfer Service:

• Configure Cloud Storage Transfer Service to transfer files from your bucket to the instance.
• You can schedule the transfer or trigger it manually.
• Cloud Storage Transfer Service can access Cloud Storage buckets without requiring an external IP on the instance.

This approach is automated and doesn't require manual IP management, but it might involve additional configuration depending on your specific needs.

3. Use a jump server:

• Create a jump server in your VPC with an external IP address and IAP enabled.
• Connect to the jump server using SSH with tunnel through IAP.
• Use gsutil on the jump server to copy the files from the bucket to the instance using its internal IP address.

This approach requires managing another server, but it allows you to keep the target instance without an external IP for security reasons.

4. Use a Cloud Storage FUSE client:

• Install a Cloud Storage FUSE client on the instance.
• Mount the bucket as a local filesystem using the FUSE client.
• You can then access the files in the bucket directly as if they were on the local filesystem.

This approach provides seamless access to the bucket, but it requires installing additional software on the instance and might not be suitable for all workloads.

Choose the approach that best suits your needs and security requirements. Remember to remove the temporary external IP address if you use that option and ensure proper IAM permissions are configured for each method.

[jomram]

Thank you, that's really helpful, for the 1st option (which seems to be the most straightforward for my purposes/maintaining my current code) can an ephemeral external IP be assigned (and removed again) after instance creation?

 Or is this only possible with a static IP? (I'm just concerned if there are any additional costs associated with a static IP that I do not require).

Also, for FUSE, can you mount a storage bucket multiple times, i.e. the same bucket mounted on multiple instances? If so, this might be a good option. 

Thanks again for your help

[jomram]

I'm not sure about the FUSE question, but to remove the external IP:

gcloud compute instances delete-access-config [VM_NAME] --access-config-name="external-nat"

to re-add an ephemeral external IP:

gcloud compute instances add-access-config [VM_NAME] --access-config-name="external-nat"