Uploading files to to Google Cloud Storage using `gcloud` Gem

This post goes through how to set up your Rails application to upload files to Google Cloud Storage using Google's gCloud gem and fog gem. <!–more–>

Unless you really need to use the gCloud gem, I highly recommend you follow this tutorial instead. It walks you through how to achieve uploads to Google Cloud Storage using the Carrierwave gem which, in my opinion, is a much better solution than gCloud.

The ability to upload PDF files was a major requirement of the Drumiverse. Storing files in the Rails /public directory does not work if you are deploying to Heroku. The reason for this is because Heroku recreates your app from scratch every time your dyno is restarted after some period of inactivity.

The solution to this is to store persistent data on an external storage provider like Google Cloud Storage. They charge a few cents per GB storage, which is very reasonable. Unless you are storing large files that take up hundreds of gigabytes, or terabytes, then your monthly bill will be negligible.

Setting up a Project and Storage Bucket

1. On your dashboard click on "Create a Cloud Storage Bucket". Enter a name for your project and select a location.
2. Go to your new Project and navigate to the storage section.
3. Create a new Bucket by entering a unique name. The cheapest storage class is Durable Reduced Availability (DRA), which sounds much worse than it is. From my understanding, it takes a few more microseconds before files are served following a file request from your rails app. That is perfectly ok. You can select the other storage class which costs only a few fractions of a cent more.

4. Once your bucket is created click on the three vertical dots to configure your bucket. Click on "Edit object default permissions".

5. Add a new item to the table. Select User from the dropdown.

6. enter "allUsers" in the text field

7. select "Reader" from the second drop down

8. Click Save

9. Do the same under "Edit Bucket Permissions"

10. Next, click on the "Settings" on the left sidebar, and then click on the "Interoperability" tab

11. Enable interoperability and create a new key. Keep this web page open because we need these keys in a few minutes.

Setting up your Rails application

Add some Gems

Add the following gems and run bundle install

#Google Cloud Storage Integration
gem 'gcloud'
gem 'fog' # they recommend using fog-google, I had issues doing that.

Create a Scaffold

If you haven't already created a scaffold, do so now. Your scaffold must include the attribute file:string which we will use later to store the public URL of the uploaded file.

To add this to an existing model, create a migration.

rails g migration AddFileToMODELNAME file:string

Replace MODELNAME with the name of your existing model. Don't forget to rake db:migrate before moving on to the next step.

Add Configuration Lines to Application.rb

Add the following lines to your config/application.rb file. Replace BUCKETNAME with the name of the bucket you created earlier. If your rails server fails to start then it is probably because you skipped this step.

config.x.settings = Rails.application.config_for :settings
config.x.fog_dir = 'BUCKETNAME'

Adding an Initializer

We have to create an initializer for the fog gem. In config/initializers create a new file called fog_cloud_storage.rb and add the following code to it. (Just copy and paste, you don't have to customize anything here)

if Rails.env.test?

    Fog.mock!
    FogStorage = Fog::Storage.new(
        provider: "Google",
        google_storage_access_key_id: "mock",
        google_storage_secret_access_key: "mock"
    )
    FogStorage.directories.create key: "testbucket", acl: "public-read"
    StorageBucket = FogStorage.directories.get "testbucket"

else
    config = Rails.application.config.x.settings["cloud_storage"]

    FogStorage = Fog::Storage.new(
    provider: "Google",
    google_storage_access_key_id: config["access_key_id"],
    google_storage_secret_access_key: config["secret_access_key"]
)

StorageBucket = FogStorage.directories.new key: config["bucket"]

end

Settings File

Next, we have to tell our application the interoperability keys we generated before. Create the following file:

default: &default
cloud_storage:
  bucket: BUCKET_NAME
  access_key_id: YOUR_ACCESS_KEY
  secret_access_key: SECRET_ACCESS_KEY

development:
<<: *default

    production:
    cloud_storage:
    bucket: BUCKET_NAME
    access_key_id: YOUR_ACCESS_KEY
    secret_access_key: SECRET_ACCESS_KEY

Make sure you use different keys for development and production and that you do not commit these keys into your repository, but instead inject them as environment variables. {: .notice–danger}

Add uploading logic to the model

Open your model file that contains the file:string attribute you added before. Add the following lines. Do not refactor anything until you know uploading works. (I wasted a lot of time refactoring and breaking things along the way. This code is harder than it looks.) This code was taken from Google's example project demonstrating how to upload to Google Cloud Storage. (I refactored it a bit.)

# used to temporarily store the file url from the form in the model
attr_accessor :file_url

private

after_create :upload_file, if: :file_url

def upload_file
    if not file_url.blank?
        pdf_file = StorageBucket.files.new(
                key: "uploads/#{id}/#{file_url.original_filename.gsub(/[^0-9A-z.\-]/, '_')}",
                body: file_url.read,
                public: true
        )

        pdf_file.save

        update_columns file: pdf_file.public_url
    end
end


before_destroy :delete_file, if: :file #checks whether model has a file column set

def delete_file
    bucket_name = StorageBucket.key
    file_uri = URI.parse file #must be file, the database column

    if file_uri.host == "#{bucket_name}.storage.googleapis.com"
        # Remove leading forward slash from image path
        # The result will be the image key, eg. "uploads/:id/:filename"
        file_key = file_uri.path.sub("/", "")
        file = StorageBucket.files.new key: file_key
        file.destroy
    end
end


before_update :update_file, if: :file_url #must be :file, checks whether there is a file to delete.

def update_file
    delete_file if file?
    upload_file
end

Prepare your View

If you used the scaffold generator to create your view, the you can easily add a file upload button to the form partial. (I am using simple_form. If you are using form_for, just google how to add a file field to a form.)

<%= f.file_field :file_url %>

Note that we modified the type of input as well as the attribute that the input value is stored in.

Allowing :file_url in params

The attribute :file_url is a temporary variable declared in our model. We use this to store the :file_url from the form. The actual file we want to store in the database is assigned automatically in the upload_file method in our model. In your controller, scroll down to the bottom and change the :file attribute to :file_url Note that the line of code should not contain the :file attribute. Like I said, the model will assign that after a file is uploaded.

Conclusion

That should be all you have to do to upload files to Google Cloud Storage. You can display the uploaded images in your view using an image tag. In app/views/MODELNAME/show.html.erb, add this line:

<%= image_tag @post.file %>

Depending on the name of your model, you might have to change the @post to @modelname.

This code uploads the selected file to Google Cloud Storage, saving it in a subdirectory denoted by the record's ID number. Deleting or updating a record in rails will delete and update the file on your storage server as well.

If there are any mistakes or inconsistencies in this post, please point them out in the comments.