Serving raster tiles from Google Cloud Storage to CartoDB

With Google Maps Engine (GME) closing down next year, GME users are looking for alternatives. CartoDB offers a great solution for mapping vector data, but some users also have raster imagery they use in maps. For those users, here’s a reliable and simple way to keep hosting imagery in the Google cloud after GME shuts down.

In brief, the steps are:

• download the source imagery from Google (or pull out the original imagery from the desk drawer);
• chop the raw imagery into web map tiles;
• upload the tiles to Google cloud storage; and finally,
• configure your CartoDB visualizations to use your tiles in the Google cloud.

Get Raster Data

From Google Maps Engine

If you have existing raster data already loaded in Google Maps Engine (GME), you can download the original files and reprocess them using our technique. Log into your GME account, and navigate to your raster data source. There is a “Download original files” link in the details page of each raster data source in GME.

Once you have downloaded the original data, you are ready to start creating a tile cache.

Sample Data

To just follow along with this tutorial, download a sample image file, from an open data site. The City of Kamloops has an accessible collection, for example.

Generate a Tile Cache

To generate a tile cache, we will use the gdal2tiles.py program, which comes with the GDAL image processing library. You can get gdal2tiles.py for your platform by:

• Windows: Installing GDAL using OSGeo4W.
• Max OSX: Installing the GDAL OSX framework build or using a package manager (MacPorts or HomeBrew) to install GDAL.
• Linux: Installing GDAL using your package manager (this is quite variable depending on distribution), or building from source.

Once you have the program installed, generating a tile cache is a one-line command:

gdal2tiles.py <your_input_file> <your_tiles_directory>

So for example:

gdal2tiles.py 5255C.tif kamloops

Faster, Flexible Tiles

The gdal2tiles.py program is pretty good, but it has a couple shortcomings for bulk tile creation:

• it runs “single-threaded” so it cannot take advantage of multi-core CPUs; and,
• it outputs PNG tiles only, which results in very poor image compression for aerial/satellite imagery.

There is an enhanced version, gdal2tilesp.py that fixes both limitations. Once you have installed GDAL as described above, you should be able to just download gdal2tilesp.py and copy it next to the original version to access the new functionality.

Running gdal2tilesp.py on 4 CPUs and outputting JPEG tiles (which makes sense for imagery) looks like this:

gdal2tilesp.py --format=JPEG --processes=4 5255C.tif kamloops

The parallel process is faster than the single process, and more importantly the JPEG storage is much smaller than PNG storage for imagery.

Upload the Tile Cache

The first time you upload to Google Cloud Storage, you will need to do a bit of setup:

1. Create a new Google Cloud Platform “project”.
2. Install the Google Cloud SDK, which has tools for uploading data to Google Cloud Storage.

3. Authorize Google Cloud SDK to access your Google Account, by running the following command:

    gcloud auth login --project YOUR_PROJECT
   

Now you are ready to upload the tile cache.

1. Create a new bucket in Google Cloud Storage to hold your tile cache. Note that when you upload data to your bucket, you will incur some costs. You can estimate the charges using the calculator provided by Google. For a small cache (1-2 Gb) the charges are only pennies a month.

2. Navigate to your tile cache directory and run the upload command, which will recursively copy all the tiles up into the cloud. Note that you will need both your project name and bucket name to run the command.

    gsutil -m \
      -h 'Cache-Control:public,max-age:31536000,x-goog-project-id:YOUR_PROJECT' \
      cp -a public-read -R ./YOUR_TILES_DIR gs://YOUR_BUCKET/*
   

Use the Tiles as a CartoDB Base Map

Now that the tiles are stored in the cloud, you can easily add them to our CartoDB maps as a new basemap.

1. Open up a visualization and click the “Change basemap” option at the lower left.
2. Select the “Yours” option to add a custom basemap.

3. Use the XYZ tab and enter in the URL for your tile cache. If you’ve followed the instructions above, it should be formatted like this:

    http://storage.googleapis.com/YOUR_BUCKET/YOUR_TILES_DIR/{z}/{x}/{y}.jpg
   

Here’s an example of the Kamloops aerial basemap with some zoning data overlaid on top. Crisp local imagery and great registration to the overlaid data too!

Conclusion

Any raster input file that GDAL can read (and that is most of them) can be converted into a tiled basemap using this simple technique.

The same technique can also be used along with GDAL Virtual Raster Table driver and the gdalbuildvrt tool to convert large collections of multiple input source rasters into tile caches ready for upload into the Google Cloud. More on that in a future post.