Secure Signed URLs on Google Cloud Storage

Ad-hoc Secure Data Upload to Cloud Storage

Thanks to Paul Williams and Thinh Ha for their ideas and advice on Go

Overview

A key problem for sharing data is to ingest securely to cloud storage systems with enterprise standard controls. Whilst there are many methods for automated ingestion, user-driven ad-hoc upload of data is common practice but with a less well-established pattern, especially where the users sit outside the security boundaries of the cloud platform, such as outside a VPCSC boundary.

One solution for adhoc uploads is to generate a signed url - a i.e. URL which contains authentication information in the string and can be used to access an object on Cloud Storage. A signed URL can be used for both uploads and downloads of objects. The signed-url authenticates using a service account with permissions restricted to the relevant cloud storage bucket.

Generation of the signed url can be done several ways, either via gsutil or through a client library. Here, the process is to authenticate as the users’ GCP identity, then impersonate the service account before generating the particular URL using the service account’s authentication.

Most of the GCP client (Java, Python, NodeJS) libraries are implemented such that authentication for the service account comes via a key. The use of service account keys is often blocked at the organisational level and is generally bad practice. However, the Go client library allows for service account impersonation without a private key, meaning that signed urls can be generated securely.

I will demonstrate how to setup a service account, configure a go script with the appropriate and generate a signed URL for use in upload.

Run-through

You will need a GCP organisation (myorg) with service account keys disabled and a GCP project within it (myproject). Your GCP identity will need the following permissions

• Service Account Admin (roles/iam.serviceAccountAdmin) - to create the service account,
• Service Account User (roles/iam.serviceAccountUser) - to use the service account
• Service Account Token Creator (roles/iam.serviceAccountTokenCreator)- to generate a token for the service account. Both your account and the service account will need permission to access resources within the VPC-SC perimeter.

Create a bucket for upload

Choose a bucket name and use the gsutil application:

gsutil mb gs://BUCKET_NAME

Configure the service account

Create the service account (SA_NAME)

gcloud iam service-accounts create SA_NAME \
    --description="DESCRIPTION" \
    --display-name="DISPLAY_NAME"

And allow the service account to access Google Cloud Storage:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="serviceAccount:SA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
    --role="ROLE_NAME"

Where ROLE_NAME = “roles/storage.objectAdmin” (give more fine-grained permissions for your project).

Configure the binary

Download the go script to generate the signed URL

$ git clone git@github.com:alexshires/google-cloud-demos.git

And navigate to the code

$ cd gen-signed-url

The key parts are setting up the impersonation at:

ts, err := impersonate.CredentialsTokenSource(ctx, impersonate.CredentialsConfig{
    	TargetPrincipal: "signurl-service-account@ashires-joonix-test-1.iam.gserviceaccount.com",
    	Scopes:          []string{"https://www.googleapis.com/auth/cloud-platform"},
    	// Optionally supply delegates.
    	// Delegates: []string{"bar@project-id.iam.gserviceaccount.com"},
    })

and then generating the url with the impersonated service account

opts := &storage.SignedURLOptions{
        Scheme: storage.SigningSchemeV4,
        Method: "PUT",
        Headers: []string{
            "Content-Type:application/octet-stream",
        },
        Expires: time.Now().Add(12 * time.Hour),
        GoogleAccessID: "signurl-service-account@ashires-joonix-test-1.iam.gserviceaccount.com",
    }

In a suitable terminal, edit line 25 and 56 of the main.go file for your service account’s full identity: (SA_NAME@PROJECT_ID.iam.gserviceaccount.com).

Other configuration for the program is on line 55, for the duration of validity for this URL and line 53 for the content type of object expected to be uploaded.

To run, compile the go program:

$ go mod init example.com/ashires/gensignedurl
$ go build example.com/ashires/gensignedurl

Execute the go program as follows, with the bucket name from above and the object name

$ ./gensignedurl BUCKET_NAME OBJECT_NAME

This will return a URL of the form, which you can share for parties interested in uploading to the specific bucket.

Future changes

There are a few ways this can be extended / wrapped up for a user-friendly enterprise experience:

• Generate the URL through a cloud function
• Generate the URL based on the object name through a form
• Generate the URL and upload in a client-side script for automatic upload for a large set of objects from a third party.