With the news that Google is shuttering its Google Maps Engine product (GME) in less than a year, many new clients have come to CartoDB in search of a “new home” for their geospatial needs. For organizations still looking at migrating from GME to CartoDB, our partners over at CloudSherpas have covered in detail both “why” CartoDB is a great choice and the technical “how” of the transition (hint: it’s easy!). But any large organizational software migration can be challenging to wrap one’s head around, which is why we wanted to go a step further and compare, 1-on-1, some common workflows and uses of GME (right from the GME documentation) and how one would accomplish those same tasks within CartoDB. Read on to discover just how easy it can be to move to CartoDB!
(from GME’s “Upload and style vector data for viewing on mapsengine.google.com”)
A vector data file defines geographic features: points, lines and polygons. See the topic on supported formats in the CartoDB Documentation.
Once your data is imported, you can use CartoCSS to define visual aspects for your points, lines, and polygons, such as point marker styles (fill color, opacity, width, etc); line styles (dashing, width, color, etc); polygon styles (fill color, opacity, border width, etc); and label styles (text face, size, positioning, etc).
Workflow in brief:
Upload the vector data file (along with the associated sidecar files in a zip file if necessary, for example a zipped shapefile) by using the CartoDB Editor or making a request to the Import API directly (example). (Supported formats.)
Go to your CartoDB Dashboard (YOUR_USERNAME.cartodb.com), go to “Your Datasets”, and open the new Dataset that was created after your file was imported.
Style the data by configuring a wizard or by manually editing the CartoCSS.
Set the Dataset’s privacy settings to determine who can see the data. Consider that you can still create a “public map” out of private data, or a private map out of public data. See “What are private maps?” for more detail.
Create a new Map by clicking the “Visualize” button in the top right corner of the CartoDB Editor and enter a name.
Share your new Map by clicking the “Share” button in the top right corner of the CartoDB Editor and use one of the given options (Get the link, embed it, or use CartoDB.js).
(from GME’s “Insert vector data incrementally”)
Instead of uploading all the features at once via a file upload, as described in the above scenario, you can also insert new data over time as needed.
This workflow is useful when your application needs to collect data or offers a user interface where people can add features, for example.
Workflow in brief:
Create a new empty Dataset in the CartoDB Editor.
Navigate to your new Dataset.
Using the “Add column” button on the grey right sidebar and the column headers, create any new columns that you need and set their data types.
If it’s a private Dataset, you’ll need to copy your API key. (Find your API key).
Add features via the CartoDB SQL API.
You can interact with the CartoDB SQL API directly with REST calls, or use one of the official client libraries. Here’s an example of inserting a point into a table with the python client library:
fromcartodbimportCartoDBAPIKey,CartoDBExceptionAPI_KEY='YOUR_CARTODB_API_KEY'cartodb_domain='YOUR_CARTODB_DOMAIN'cl=CartoDBAPIKey(API_KEY,cartodb_domain)try:printcl.sql("insert into mytable (the_geom,foo,bar) values (CDB_LatLng(40.7127,-74.0059),'testing',123)")exceptCartoDBExceptionase:print("some error ocurred",e)
(from GME’s “Upload imagery for viewing on mapsengine.google.com”)
A raster image is a geo-referenced image. CartoDB supports GeoTIFF images. See the topic on supported formats in the CartoDB Documentation.
Workflow in brief:
Create a new empty table in the CartoDB Editor.
Upload the GeoTIFF raster data file.
Go to the CartoDB Editor (YOUR_USERNAME.cartodb.com).
You will see a new Raster Table item in your table list. Currently, you can only interact with images in CartoDB over the SQL API, i.e., you won’t be able view your GeoTIFF in the CartoDB Editor.
For the import, follow the example in the Import API documentation. Here’s an example of creating a map with a CartoDB Raster Table and CartoDB.js.
For smaller images or small sets of images, it’s often far easier to use either the Google Maps API to create a ground overlay(s) with the image(s), or CartoDB.JS (which follows Leaflet.js functionality) to create an image overlay(s) with the image(s).
(from GME’s “Upload and process large volumes of imagery”)
If you are interested in using large volumes of imagery (usually from aerial photographs), read this full guide on “Serving raster tiles from Google Cloud Storage to CartoDB”.
(from GME’s “Integrate Google Maps Engine data into a custom map”)
There are a number of options for integrating and visualizing CartoDB Datasets in your own application.
(from GME’s “Update your data on Google Maps Engine”)
The CartoDB SQL API provides full SQL access (and thus, CRUD operations - create, read, update, delete) to public and private (with an API key) CartoDB Datasets. Each Dataset is a table in CartoDB’s PostgreSQL database. For optimization tips on updating, inserting, and “upserting” data through the SQL API, check out this blog post.
You can also use the CartoDB Editor - either the SQL query pane or clicking and typing in table cells in the “Data View” to update records in a Dataset.
(from GME’s “Synchronize your data store with Google Maps Engine”)
This workflow is useful if you want to use CartoDB as a geospatial analysis and/or publishing platform for analyzing and/or exposing data to customers or the world, and to treat another platform or database as the canonical data store. In this way you can take advantage of the PostGIS geospatial analysis and mapping features, sharing capabilities, and scalability offered by CartoDB.
In such a case, you will need to keep the Google Maps Engine data in sync with your canonical data store. There are a few ways to do this, each with their own advantages and complexity:
“Pull” data into CartoDB on a schedule with Sync Tables:
Most CartoDB plans, and all Enterprise plans, offer “sync tables”, which can pull data from a remote file and refresh their contents on a periodic schedule specified by the user. The remote files that sync tables pull from may come from services like Google Drive or Dropbox, or a public URL accessible to CartoDB via HTTP or HTTPS.
Sync tables can be created easily, either with the CartoDB Editor (no programming required) or programmatically via the Import API.
Sync tables require your data to be accessible to CartoDB - public, essentially.
Sync tables are “destructive”; each time the sync happens, CartoDB deletes its old version of the table and recreates a new table from the remote source. Another way to think of this is that CartoDB’s version of the data gets overwritten each sync, not“appended to.”
“Push” data into CartoDB with the SQL or Import APIs, or OGR2OGR:
If your canonical data store is not public; or you don’t want CartoDB to overwrite its version on each sync and want to instead “append” new data; and your data store or middleware has the ability to run scripts and/or call APIs, these “push” methods are for you. You can issue SQL queries using the SQL API, including UPDATE and INSERT, to “push” updated data to CartoDB. You can also push data into new tables with the Import API. Finally, you can use the ogr2ogr tool in the open-source GDAL library to append data into existing CartoDB tables as discussed in this blog post.
You can issue fine-grained SQL queries to CartoDB to update only a few properties of a particular record, or insert (“append”) only a few records, rather than having to sync the whole dataset/table. The rest of the data in the CartoDB table will be unaffected.
This method may also be useful for uploading “snapshots” of your data to CartoDB. Your canonical data store can “dump” or otherwise export a snapshot of data on a schedule, and then push the snapshot to CartoDB with any of the SQL API, Import API, or ogr2ogr.
Direct Connections to External Databases:
CartoDB is built on PostgreSQL, and thus supports PostgreSQL’s “Foreign Data Wrappers” (FDW) technology for connecting directly to external databases. There are two ways to use FDWs with CartoDB, and the choice depends on the volume of data you are syncing and whether your database needs to process the data before sending it to CartoDB. Both methods require low latency, direct connections to your database. If you are interested in these FDW approaches, please contact us for more details so we can help find the best fit for your needs.
FDW with Materialized Views and refresh: This method works if the data in your canonical data store that you are synchronizing with CartoDB is relatively small (< 100MB), and CartoDB can do any querying or processing necessary. A “materialized view” uses a query to fetch its data (like a view), and also stores this data (like a table). Materialized views combined with an FDW allow you to issue a traditional SQL query to CartoDB, which then translates and submits that query to your canonical data store in a format your canonical data store accepts. When CartoDB receives the queried data, it translates the data back into a tabular format which can be mapped and analyzed from then on within CartoDB. Each time a query or “refresh” command is issued, CartoDB can update the contents of its materialized view from your canonical data store.
FDW with Live Queries: This method is best suited for situations where your data is complex or high volume, and your canonical data store must execute computationally complex queries or processing to prepare data to be mapped in CartoDB. Using a more complex FDW, CartoDB can issue queries to your canonical data store, where your canonical data store translates the query, processes the data, and sends the processed data back to CartoDB in a SQL-compatible tabular format, ready for mapping.
Manual export/upload:
If for some reason you cannot use scripts or APIs with your canonical data store, or CartoDB cannot access data stored there via an HTTP/S URL, and FDWs are also not an option, as a last resort you can export data files and upload them to CartoDB manually.
Export the data as a file from the canonical data store.
Upload the data file using the CartoDB Editor, which will create a Dataset in CartoDB.
Create a new Map with the Dataset as a layer.
Analyze and style using SQL and CartoCSS. Set the Map’s privacy and share it with a link, embed code, or CartoDB.JS.
When the time comes to update the data, you can optionally delete the outdated dataset in CartoDB, re-export a file from your canonical data store, and upload it to CartoDB. Keep in mind that once you delete a Dataset in CartoDB, any Maps that used that Dataset will also be deleted. If you create a new Dataset, your Maps will still work, but you will have to edit them to point to the new Dataset.
(from GME’s “Upload data to Google Maps Engine and query it SQL-style”)
This workflow is useful if you want to use CartoDB as a canonical data store or as an exploratory data analysis environment. It’s not essential to create a map with your data in CartoDB if you just want to query and analyze it. Unlike Google Maps Engine, CartoDB supports the complete set of Structured Query Language (SQL) as the PostgreSQL database, in addition to the geospatial-specific SQL functions of PostGIS.
Workflow in brief:
Upload data to a CartoDB table. The data does not necessarily even need to be geospatial in nature if you are not planning to create a map with it.
Use the Editor’s SQL pane, or the SQL API to issue dynamic queries. The SQL API can return the queried data in a variety of formats, including GeoJSON.
Unlike Google Maps Engine, each table in your CartoDB database is accessed via the same SQL API endpoint, just as if you were querying from the “psql” interactive terminal. For details on SQL commands, see the PostgreSQL documentation and the PostGIS documentation.
