Last week’s Partner Summit was a smash hit! With representatives from more than 50 partner companies, ranging from small geo-consulting firms to large systems integrators to other SaaS businesses, this two-day event had all the right voices. Here’s a quick recap and some thoughts on the future.

On Thursday, our day of technical workshops saw new and experienced CartoDB developers building a consumer insights app that filters AirBnB housing by things like proximity to transit. Meanwhile, a packed series of business development talks walked through CartoDB user personas and how each can leverage the stack, tools for giving a fantastic CartoDB demo, and the many resources available to our partners to help them best serve their clients.

But with this event, we also wanted to ask a bigger question: where is the location intelligence industry headed?

We were excited to have a fantastic roster of industry leaders respond to this question: Chris Holmes (Planet Labs), Jonathan Lowe (BCG), Herwig Baumberger (Wabion), Bobby Shackleton (Bloomberg LP), and Alberto Olivas (Mexico City). The answers all pointed to a smart future involving fresh, ubiquitous data: flocks of tiny satellites, self-driving cars, network-analysis for business managers, and government IT where location is at the center of the technology stack.

On Friday, we asked Paul Ramsey to weigh in, he in turn asked us to “relax your mind a bit and think: what would this trend look like at basically zero cost, and ten times more powerful?” We’ll leave you with his take:

CartoDB15 - Trends In Business and Technology, The Future and All That from CartoDB on Vimeo.

Happy data mapping!

Proportional symbol maps are used to represent point data that are attached to a specific geographic location (like a city) or data aggregated to a point from an area (like a state). The area of each symbol on the map (usually a circle) is scaled according to its value at a given geographic location using either absolute scaling or range-grading. The result is a map where larger symbols indicate higher values and smaller ones, lower values. This type of map is flexible in that you can represent raw data values (total population) or data that are normalized (percentage of population).

In this blog, I’ll walk through the basic principles of proportional symbols and some best practices when designing them. Then, I’ll cover the steps to make a map of 2015 urban population by country using the absolute scaling method.

Proportional Symbol Maps

Absolute Scaling vs. Range-Grading

There are two methods used to create proportional symbols: absolute scaling and range-grading. With absolute scaling, the area of each symbol on the map is scaled proportionately to its value in the data. With range-grading, values are broken into ranges (typically 3-5) using a classification method where symbols are sized based on the range they fall into (this is similar to the choropleth technique used for polygon data).

Use absolute scaling when you want your map reader to estimate relative magnitudes between individual values. Use range-grading to show magnitudes using classes that represent a range of values.

You will hear proportional symbol maps referred to in different ways. In the CartoDB Editor, we call them Bubble maps. Others refer to absolute scaled symbols as proportional symbol and range-graded as graduated symbols. Another way they are referred to is classed (range-graded) and unclassed (absolute scaling).

Best Practices

There are some important design considerations when making proportional symbol maps:

• Avoid a homogeneous looking map. It is important that there is enough variation between symbol sizes representing the range of values. With absolute scaling if there isn’t a lot of variation between the values in your data, there will be little variation in symbol size. With range-grading, make sure there is enough differentiation in symbol size from class to class so they are easily distinguishable from one another

• Use the right method for your data. Research shows that map readers have a hard time estimating the area of a symbol on a map and even more difficulty between symbols. Make sure your data are appropriate for absolutely scaled symbols. If not, use range-grading to limit the number of symbols that need to be interpreted

• Think about your map reader. Cartographic research also shows that map readers interpret the areas of circles and squares best on proportional symbol maps. Definitely experiment with other symbol types (even text!) but when in doubt, default to these

• Legends are important. For absolute scaling, your legend should provide symbols that are sized close to the largest and smallest values in the data. And of course with range-grading, each symbol should be sized as it is on the map with its associated values listed

• Symbol legibility. Make sure to order your data so symbols representing smaller values don’t get lost underneath symbols with larger values. In areas where there is a concentration of symbols, the key is to help each one stand out against the other. This can be done using a couple of design tricks like symbol outlines

For more information on proportional symbols, definitely check out Cartography: Thematic Map Design by Borden Dent where a lot of this information comes from!

Let’s Get Mapping!

Before we dive into the data preparation and map making process, let’s take a look at some details about the final map:

• It was designed to be viewed from zooms 2-5
• Symbols are sized using absolute scaling
• Only countries with urban populations of greater than 1 million are included
• The color of each country point relates to its continent
• It uses a simple basemap to promote visual hierarchy as discussed in this blog
• The final map projection is World Robinson (for more on projections in CartoDB see this blog)
• Zoom in on the map below to see the multi-scale legend, that uses custom data and CartoCSS for its design (this will be covered in another blog post!)

As mentioned earlier, with absolute scaling, the area of each symbol is proportionate to its value in the data. In future versions of CartoDB.js and the CartoDB Editor, you will be able to do this on the fly. Until then, this blog outlines the process to make a proportional symbol map using absolute scaling.

The topics covered are:

• Data Prep
• Formula overview for absolute scaling
• Calculating symbol sizes in CartoDB
• Applying symbology based on values
• Final touches

Data

The data used for this map come from the World Bank Data Bank.

To get the data ready for the map:

• I downloaded total urban population projections by country for 2015 in .csv format
• Next, I uploaded the .csv to CartoDB where it was georeferenced using country name
• I converted the resulting polygons to points using ST_Centroid
• Finally, I assigned a continent to each point in the data using a method similar to the one outlined here

I won’t outline each step in detail, but if you would like to follow along, you can download the prepared dataset here.

The attributes in the dataset are:

country_name: the name of the country
country_code: the three letter country code
continent_name: the continent the country belongs to
pop_2015: the total count for 2015 urban population

Note: the SQL and CartoCSS in this blog assume that the name of your table is world_population_2015

Formula Overview

To determine the proper symbol size for each country, we’ll do the following:

1. Pick a maximum symbol size that will be used for the largest value in the data
2. Calculate the square root for each value in the data
3. Use those two pieces of information to calculate the symbol size for each country

To simplify this to an equation:

symbol size = (maximum symbol size) * (square root / max square root)

Where:

maximum symbol size = the size of the largest symbol on our map
square root = the square root of each value in the data
max square root = the square root of the largest value in the data

Note: For a more detailed discussion on this calculation, see Chapter 9 of Borden Dent’s book!

Symbol Size Calculation in CartoDB

1. Choose a Maximum Symbol Size

The maximum symbol size is assigned to the largest value in the data and is the starting point from which all other symbols will be scaled. Choose a maximum symbol size that won’t cause too much clutter or overpower the map.

For the final map (after some trial and error), I chose a maximum symbol size of 75px based on the values in the data, the opening scale, and extent.

Note: You should adjust this number depending on your map’s extent, scale, and the range of values. Using the method outlined below, you can easily adjust this number and recalculate symbol sizes.

2. Add Fields

We need to add two numeric fields to our table: square_root where we’ll calculate the square root for each pop_2015 value, and symbol_size where we’ll calculate the symbol size for each country.

• Go to DATA VIEW
• In the bottom right hand corner, click the button to Add Column

• Name the column square_root and hit enter
• Change the column type to number by selecting it from the drop-down
• Follow the same steps to add another column named symbol_size

3. Calculate Square Root

• From DATA VIEW open the SQL tray
• Using the SQL query below, we’ll update our square_root field with the square root of each value in the pop_2015 field:

UPDATEworld_population_2015SETsquare_root=SQRT(pop_2015)

• Click Apply Query

4. Calculate Symbol Size

Now that we have decided on our maximum symbol size (75) and have the square_root field populated, we can use that information to calculate the symbol_size for all other values in the data.

Let’s take a look at our equation again:

symbol size = (maximum symbol size) * (square root / max square root)

Where:
symbol size = our symbol_size field
maximum symbol size = 75
square root = our square_root field
max square root = 27620.2642999664 (the square_root value for China)

With all of the information that we need, we can construct the equation using SQL to update the symbol_size field:

• From Data View open your SQL tray
• Copy/paste the SQL below to calculate a symbol size for each value in the data

UPDATEworld_population_2015SETsymbol_size=75.0*square_root/27620.2642999664

• Click Apply Query

If we look at the resulting sizes, we can see that the country with the largest population (China) has a symbol size of 75 and all other values are scaled accordingly:

Symbol Design

Now that we have the additional attributes in the data, we’ll walk through the steps to size, color, and define the drawing order of the symbols in MAP VIEW.

• From DATA VIEW click the option to VISUALIZE to create a map from our data table

1. Symbolize by Continent

On the final map, each symbol is colored according to its continent using the continent_name field.

• Open the styling Wizard
• Select the Category option and choose continent_name for Column

2. Size Symbols

Next, we’ll size each point symbol using the values from our [symbol_size] field.

• Open the CartoCSS tray for the layer
• Replace the value for marker-width: from 10 to [symbol_size] so the first block of global styling looks like:

#world_population_2015{marker-fill-opacity:0.9;marker-line-color:#FFF;marker-line-width:1;marker-line-opacity:0.8;marker-placement:point;marker-type:ellipse;marker-width:[symbol_size];marker-allow-overlap:true;}...

• Click Apply Style

If you aren’t satisfied with the symbol sizes after seeing them on the map, you can change the maximum symbol size value and rerun the equation to calculate new values for the symbol_size field.

3. Color Symbols

Similar to the drought monitor map, I chose six different colors for each continent, assigned them as variables, and then replaced the default category colors. I also made a slight adjustment to the default marker-line-opacity from 1 to 0.8.

• In the CartoCSS tray, copy and paste the six color variables (or choose your own!) and paste them above the first block of CartoCSS

@africa:#75445C;@asia:#AF6458;@europe:#D5A75B;@northam:#736F4C;@oceania:#5b788e;@southam:#4C4E8F;

• Next, assign the color variables to each continent name
• For example, the CartoCSS for Africa would look like this:

#world_population_2015[continent_name="Africa"]{marker-fill:@africa;}

• Click Apply Style to see the changes

4. Drawing Order

As outlined in the best practices section, we need to make sure that smaller symbols are being drawn on top of larger symbols so important information does not get covered up. Looking around the map, we can see that there are cases where this is happening.

To fix this, we will change the default drawing order (from ascending) to descending using a field (symbol_size) in our data along with the SQL operator ORDER BY. This will force larger symbol_size values to draw first and smaller values to draw on top.

• Open the SQL tray
• Copy/paste the following

SELECT*FROMworld_population_2015ORDERBYsymbol_sizeDESC

• Click Apply Query to see the changes

Final Touches

1. Symbol Size Through Zoom

Since the map will be viewed from zooms 2-5, we need to size the symbols appropriately based on viewing scale.

We’ll do this with zoom dependent styling that increases the symbol_size by a factor of 2 at each zoom level for the marker-width property.

• Open the CartoCSS tray for the layer
• We’ll add the additional marker-width styling to the global properties of the layer

#world_population_2015{marker-fill-opacity:0.9;marker-line-color:#FFF;marker-line-width:1;marker-line-opacity:0.8;marker-placement:point;marker-type:ellipse;marker-width:[symbol_size];marker-allow-overlap:true;//scale symbols based on zoom level
[zoom=3]{marker-width:[symbol_size]*2;}[zoom=4]{marker-width:[symbol_size]*4;}[zoom=5]{marker-width:[symbol_size]*8;}}

• Click Apply Style

At this point, your map should look something like this where symbols are colored by continent, sized by the value they represent in the data, smaller symbols are drawing on top of larger ones, and a symbol’s size scales between zooms 2-5:

2. Increase Buffer Size

If you zoom into China, you’ll notice that at zoom 4, the symbol cuts off along a tile boundary.

We can fix this by increasing the map’s buffer-size, which adds additional pixels around each tile.

• Copy the following

Map{buffer-size:256;}

• Paste it above the color variables for continent colors
• Click Apply Style

And the result:

3. Basemap

For the final map, I designed a simple basemap in World Robinson. For tips on designing a basemap for thematic overlays in a variety of projections, see this post and this post.

Alternatively, you can use one of the default CartoDB basemaps like Dark Matter (the lite version):

4. Other Map Elements

As discussed earlier, legends are important for proportional symbol maps. For the final map, I made a legend with three data points, assigned them large, medium, and low values and then scaled them using the same formula that was used for all other symbols. I then positioned the circles and text using CartoCSS through zoom level.

I followed the same method for the map title, description, and data source text.

Experiment!

Proportional symbol maps are some of my favorite! There are a lot of interesting stories that can be told using this method in a variety of creative ways.

For example, the final map from the blog can be taken further to include the World Bank’s population projections for multiple time periods:

or to visualize 50+ years of cholera outbreaks around the world:

or for tiger counts in India:

… and many, many more …

Happy proportional symbol mapping!

Have you ever had trouble making your map look legible when it has a large numer of overlapping elements? Have you ever wanted to control subtle effects when one geometry overlays multiple geometries of different colors? You can solve all of these problems and more by using the composite operations baked into the CartoCSS map design language!

In our new Map Academy lesson How to Use Composite Operations in CartoCSS, we will go over many of the common composite operations and when and how to use them. The subtle effects produce a big win for cartographic design!

This lesson is the latest in our Intermediate Design course. Comp-ops give you power over how much the elements of your map stand out compared to others by letting you fine-tune how color and transparency blend between your map layers. The lesson discusses the purpose of the composite operations available through the wizard menus, and then show how to go beyond them with your custom CartoCSS!

We have much more coming on the subject of design possibilities within CartoDB. If you want more, be sure to check out the other lesson in our Intermediate Design course, and the newly added CartoCSS documentation!

Besides the new content, check out our new Map Academy site design! Now it’s easier for you to choose a lesson according to level, navigate via the sidebar, and keep track of your course progress. This design change is just the start of a lot more to come to Map Academy.

We hope you find the new site design and new lesson useful. Share your comp-op maps with us in the lesson comments! If you have feedback on this lesson or want to suggest another, please let us know at support@cartodb.com.

Happy map making!

Join CartoDB scientists Andy Eschbacher and Stuart Lynn, fellow geospatial data wranglers, and web cartographers at our Intermediate SQL workshop on Tuesday, January 12th!

We strongly recommend taking our first two courses on SQL and PostGIS in our
Map Academy to prepare because this course will not be introducing basic levels of SQL.

In this workshop you’ll learn to:

• Use aggregate functions and GROUP BY
• Subqueries
• Super fun spatial queries
• Ultra cool tricks

Where can you find us? That’s easy. This workshop will be at our Brooklyn office in Bushwick (201 Moore Street, Brooklyn, NY) on Tuesday, January 12 at
6:30 PM.

Can’t make this session or simply can’t get enough of CartoDB? Join us for our twice-monthly Friday open office hours at our spacious Brooklyn office. Bring that project you need a little extra help on, come learn that webmapping skill you’ve always wanted to pick up, or just come to have a coffee!

Happy data mapping!

Many universities around the world use CartoDB, for research and in the classroom, to create interactive tools for educators and departments. Types of visualizations vary from historical occurrences, natural events, predictions, economic trends, population performance, and disease evolution.

View how test scores are visualized per school district in polygons to show elementary school student performance in New York City.

Or visualize how far birds migrate in a single night thanks to our torque feature:

Discover how to use CartoDB as a powerful education and research tool in our webinar next Thursday January 21:

Register

Andy Eschbacher, our Map Scientist at CartoDB, is hosting this webinar. Andy has been fascinated by maps since he was a boy and now he enjoys researching and improving our tool to let you create amazing data-driven maps!

Happy data mapping!

The NYPD recently made waves with an open data release, sharing fairly detailed crime data about the “seven major felonies” for the first 3 quarters of 2015. The dataset has a row for every incident, including location (as lat/lon), date/time, etc. However, as with many crime datasets, the point locations have been adjusted to mid-block or nearest intersection so that you can’t determine exactly where the reported crime occurred. As you might expect, there are many locations in NYC where multiple crimes occurred on the same block/intersection, so we end up with many rows that have the exact same point coordinates. This is a mapper’s conundrum, as point markers will simply overlap each other, only allowing you to see the one point when there are many beneath it.

Some Workarounds to the ‘Overlapping Points Problem’

We’ll use a simple dummy dataset that has 10 points in one location, and 5 points in another location. When rendered by CartoDB, we see 2 points when there are actually 15:

Heatmaps

A heatmap overlay “heats up” an area on the map based on the number of points, or based on some aggregation of a numeric attribute. This obscures the point markers themselves, but at least the data are all visually accounted for.

Clustering

Clustering replaces many markers with a single marker, either with a label or other visual representation of the represented points. When you zoom in closer, the algorithm determines whether there is still overlap. If there is none, it breaks up the cluster. Of course, if the points have the exact same coordinates, the cluster will never split up. Go ahead zoom all the way in… the clusters will remain:

Spidering

Spidering, or spiderifying is a visualization technique where overlapping points can be offset around their mapped location, and visually linked to it with a line drawn on the map. This is sort of an “exploded view”, when there are multiple markers in the same area, allowing the user to interact with each one, but indicating that the marker has been shifted away from the location it represents.

This functionality is built into the leaflet.MarkerCluster plugin, which makes pretty animated clusters. Here’s an example using the same dummy data (click each cluster to see the spiderify animations):

Crime Wafers, Mentos, Chip Stacks - A SQL Hack in CartoDB

While tinkering with the NYC Felonies data shortly after its release, I was struggling with the overlapping points problem, and tried to implement a CartoDB-based approach to spidering based on this example by our very own Andrew Hill.

Because of the density of the felonies data, the results of trying this technique were a huge mess. Even after reigning in the spirals, the clusters were still too close together and the map was unusable.

I eventually figured out a way to modify the example so that instead of spreading out the points in a spiral pattern, I could simply offset each point along the y axis. The result was a felony map that shows “stacks” of markers at each location, allowing both visual styling and interactivity with every point in the dataset. Take a look:

So what’s the secret sauce? The example below shows the same dummy data from before rendered using this method. I’ll walk through through each step of the complex SQL query, explaining what each part does.

Step 1:

WITHmAS(SELECTarray_agg(cartodb_id)id_list,the_geom_webmercator,ST_Y(the_geom_webmercator)yFROMchriswhong.stack_dummyGROUPBYthe_geom_webmercatorORDERBYyDESC),

This step groups the points by their geometry, so points with identical locations will be grouped. It stashes the cartodb_ids for each group into an array using array_agg(). The groups are then sorted by their y coordinate (ST_Y is used to generate a new column with the y value), so that groups that are further south appear first. This will ensure that the more southern “stacks” are rendered in the foreground.

Now we’ve got one row per location, each with an array of unique ids from the source data associated with that location.

Step 2:

fAS(SELECTgenerate_series(1,array_length(id_list,1))p,unnest(id_list)cartodb_id,the_geom_webmercatorFROMm),

With this step, we’re going to un-group the data. generate_series() assigns a number for each row within its group, which we’ll use later to offset the geometries. The cartodb_ids are pulled out of the array we stashed them in, so now we’ve got a row for each felony in the original dataset, with it’s original geometry, its cartodb_id, and its number within its previous location group.

Step 3:

SELECTST_Translate(f.the_geom_webmercator,0,f.p*12)the_geom_webmercator,f.cartodb_id,q.categoryFROMf,chriswhong.stack_dummyqWHEREf.cartodb_id=q.cartodb_id

The last step generates the result set that is actually used in the map. ST_Translate() is used to create new geometries for each point based on the “location group” numbering we added in step 3. The x value of the new geometry remains the same as the original, but the y value is calculated as ST_Y(f.the_geom_webmercator) + f.p*12. (12 is the y-offset in web mercator units, you can modify this to tighen or spread out the points) The first point in the location group will be 12 web mercator units north of the original point, the second one will be 24, and so on.

A little CartoCSS makes the markers into ovals instead of circles for a 3D-ish look of stacked chips.

marker-type:ellipse;marker-width:10;marker-height:8;

This approach with a hard-coded offset will only work at a single zoom level. If you zoom in on the map above, you’ll see the points appear to spread out. This is due to the fact that 12 web mercator units takes up double the distance on your screen each time you zoom in. To overcome this in the felony map, we must dynamically define the offset based on the zoom level.

Here’s a little javascript function that that returns a y-offset for a given zoom level.

functiongetYOffset(zoom){varyOffset=655360;//offset in webmercator units at zoom level 0for(vari=0;i<zoom;i++){yOffset=yOffset/2;}returnyOffset;}

For each zoom level above zero, the y-offset is cut in half. The offset is then used with a SQL template in cartodb.js to re-render the map whenever the user zooms, and the “stacks” appear the same no matter what zoom level you’re on Check out the full code for the felonies map here.

Next time you find yourself facing the overlapping points problem, give this method a try and stack your chips!

Happy Stacking!

Organization owners are now able to manage users in their organization with our new Enterprise User Management API!

The Enterprise User Management API enables organization administrators to:

• Create new users in the organization
• Delete existing users from the organization
• Update users’ quota

Here’s how it works…

Instead of having to login to the Editor to manage users, Admins (organization owners) will now be able to manage them programmatically through this API. Similar to the Editor’s functionality, Admins will be able to add/remove users, update their email accounts and passwords, as well as assign a specific amount of the data quota to them.

This new API is perfect for doing integrations, specifically with a client with an atypical authentication system, or on-premises users who would like to create applications on top of CartoDB and need to add/remove users dynamically.

The API is available in SAAS and in on-premises installations. We have advanced auth integrations like, LDAP, for on-premises installations and custom auth integrations on request. Please don’t hesitate to contact our sales team to know how to integrate your company directory with CartoDB.

Is the Enterprise Management API right for you?

Imagine your organization has a CSV file with info (username, email, password, etc.) of various employees and those employees would like to have CartoDB accounts. Now, a programmer can write script that grabs that CSV file information and sends it to the API - in no time at all!

For more information on this organization ONLY functionality you can read here.

A warm welcome to Javier Goizueta, who joined CartoDB to take our data services to the next level.

He’s got a lot of experience in GIS-related software development, working at various consultancy firms. After getting too tired of C++ he’s been having fun with Ruby lately, but seems to have a soft spot for Julia, Elixir, or CoffeeScript too.

He’s fond of finding opportunities to apply his love for math & physics in his work, although he sometimes may get too meta…

We’ve heard rumors about him and stringed instruments and other weird hobbies but we can’t confirm anything yet.

“Find me the nearest…” is a common way to start a geographical query, and we’ve talked about how to write such queries for single points, but it can be tricky to carry out in bulk.

CartoDB supports high performance “nearest neighbor” queries, but getting them to run in bulk on a whole table can be tricky, so let’s look at the SQL we need to make it happen.

This post will use data from the City of VictoriaOpen Data Catalogue if you want to follow along:

• fire hydrant points; and
• parcel polygons.

The goal is to create a map of parcels that colors each parcel by how far away it is from the nearest fire hydrant – the kind of map that might be useful for fire department planners.

Nearest To One Location

The underlying functionality to get the nearest neighbor is a PostGIS ordering operation (the “<->” symbol below) that sorts results of a query in distance order relative to a point.

SELECTcartodb_id,fulladdrFROMparcelsshpORDERBYthe_geom<->CDB_LatLng(48.43,-123.35)LIMIT1;

By limiting our result to just the first entry in the sorted list, we naturally get the nearest parcel to our query point.

The trouble is, this trick only works one record at a time. If we want to map the distance each parcel is from its nearest fire hydrant, we need some way of running this query 30,288 times (that’s how many parcels there are in total).

Nearest To Many Locations

To create a query where every resulting row is tied to a “nearest neighbor” we have to iterate a PostGIS ordering operating once for each input row. Fortunately, we can do this using a “lateral join”.

SELECTparcels.*,hydrants.cartodb_idashydrant_cartodb_id,ST_Distance(geography(hydrants.the_geom),geography(parcels.the_geom))ASdistanceFROMparcelsshpASparcelsCROSSJOINLATERAL(SELECTcartodb_id,the_geomFROMhydrantsshpORDERBYparcels.the_geom_webmercator<->the_geom_webmercatorLIMIT1)AShydrants

You can see our original nearest neighbor query embedded after the “lateral” keyword: for each parcel, we’re finding the nearest hydrant.

Since there is always at least one nearest hydrant, we can use a “cross join” that takes every combination of records (just one, in our case) on each side of
the join.

Once we have the nearest hydrant, we just calculate a little extra information for the parcel/hydrant pair: how far apart are they? To get an accurate answer, we do the calculation using the “geography” type, as the distance between two geographies will be calculated exactly on the spheroid.

ST_Distance(geography(hydrants.the_geom),geography(parcels.the_geom))

To get a neat visualization, use the “chloropleth” wizard to build a basic 5-category style, then go into the CSS code and tidy up the breaks to nice even numbers (every 30 meters, in this case).

#parcelsshp{
  polygon-fill: #eee;
  polygon-opacity: 0.8;
  line-color: #ddd;
  line-width: 0.5;
  line-opacity: 0.8;
}
#parcelsshp [ d <= 150] {
   polygon-fill: #BD0026;
}
#parcelsshp [ d <= 120] {
   polygon-fill: #F03B20;
}
#parcelsshp [ d <= 90] {
   polygon-fill: #FD8D3C;
}
#parcelsshp [ d <= 60] {
   polygon-fill: #FECC5C;
}
#parcelsshp [ d <= 30] {
   polygon-fill: #FFFFb2;
}

Tidying Up

The map we get from the lateral join is almost perfect. We can color parcels by distance, and get an attractive rendering.

However, there are a few areas where the coloring is slightly off, because the City of Victoria parcel data includes multiple polygons for parcels that are strata titles (condominiums).

To remove the duplicates, we add a “distinct on” clause that strips out duplicate geometries. And to remove unattributed parcels we also filter out all parcels with a null value in the parcel number (“pid”).

SELECTparcels.*,hydrants.cartodb_idashydrant_cartodb_id,ST_Distance(geography(hydrants.the_geom),geography(parcels.the_geom))asdistanceFROM(SELECTDISTINCTON(the_geom)*FROMparcelsshpWHEREpidISNOTNULL)ASparcelsCROSSJOINLATERAL(SELECTcartodb_id,the_geomFROMhydrantsshpORDERBYparcels.the_geom_webmercator<->the_geom_webmercatorLIMIT1)AShydrants

And now we have the completed map, calculating the nearest hydrant to all parcels in real time.

Calculating nearest neighbors is fast, but not instantaneous. For this example, the nearest hydrant for each parcel calcuation takes about 3 seconds for 30,000 parcels. So don’t try this on more than a few hundred thousand records, or your query will probably time out before your map has a chance to draw.

Happy data mapping!

GKudos, a CartoDB partner, is a GIS visualizations firm that specializes in the management, analysis, and publication of geo-referenced information. One of their goals is to help customers have increased productivity and competitiveness through the use of information technology and telecommunications.

In their quest to create software tools that are personalized according to the expectation and needs of each client. GKudos has made CartoDB available to their clients and mappers through a series of tools and data visualizations!

Through GKudos technology you can upload File Geodatabase data layers to CartoDB using QGIS. Now something that was once only available to ESRI customers is available to all, through CartoDB.

Their main objective is to allow a better process for decision-making and a more productive work environment through a unified view of organizational information using location intelligence.

Now you can give some love to your data using ArcGIS for desktop and CartoDB. With CartoDB Toolbox you can upload data from ArcMap or ArcCatalog to CartoDB. You can use the import tool like any other toolbox, you just need your user name, api key, and feature classes.

The following tutorial will show you how to create a simple geoprocessing model to convert raster datasets to beautiful vector web maps using ArcGIS Desktop, CartoDB Toolbox and the CartoDB Editor. Also, last week GKudos launched a new article about the automatization process with Python and Arcpy.

GKudos offers users the custom application development using modern technologies that facilitate processing, visualization, and analysis of information through the web. You can view their data-driven visualizations here.

Happy Data Mapping!

Real estate is a great sector to leverage deep insights using location intelligence. Properati, a property search portal with a strong presence in Latin America, aims to improve the experience of purchase, sale and rental of real estate. Currently the site operates in Argentina, Brazil, Chile, Mexico, and Colombia with over one million properties listed.

Using CartoDB, Properati developed an analysis of property values in San Pablo, Brazil. This visualization indicates home and commercial real estate values measured against other properties in the same neighborhood.

As cities move to urbanize and meet the demands of growing populations biking has become a major movement, as it becomes increasingly important for people to bike commute to and from work and home.

Here are two data visualizations that depict all the properties for sale or rent that are close to the bike paths in Curitiba, Brazil and whether the streets are close to the bikeways.

Ultimately, Properati’s goal is to assist buyers and enable them to make rational decisions based on market data and sellers so they can better know their customers and interpret their property searches. When looking for a property using Properati and the CartoDB platform you can see list prices for each property, the price per square meter per neighborhood, the type of property in the area, whether the neighborhood is commercial or residential, proximity to green spaces, and public transport. You can view all this information and more, block by city block in Sao Paulo, Brazil.

We love to see great examples of location intelligence and big data applied to all sectors! Take a look at Properati’s open data portal and CartoDB’s gallery
for inspiration.

Can’t get enough of location intelligence and real estate? Not to worry, we’re hosting a real estate webinar on March 17. Discover insights on present and future real estate.

Happy data mapping!

The news heard by millions of straphangers around New York City last week was Metropolitan Transit Authority’s (MTA) proposed repair to the blighted Canarsie Tunnel, the vital connection used by the L train to connect Manhattan and Brooklyn under the East River. Due to extensive damage caused by flooding from 2012’s Hurricane Sandy, the restoration will have large consequences for L train riders in the near future. How massive and where those consequences can show up, can be understood better using a bit of location intelligence.

As the MTA weighs its options — which apparently include a full closing of the L train during repairs, or a longer period with weekend-only closures — CartoDB started digging into open data to shed some light on how disruptive the L train closure would be to the people living in Brooklyn.

As a side note, if you are publishing any stories about the L train, feel free to use any of the maps below (or above), no need to even ask. You can find instructions for embedding each over here.

Overview of the L Train

The L train is a major east-west subway artery of New York City, connecting Manhattan with many parts of Brooklyn. If you know people who live in Williamsburg and commute to Manhattan in the morning, you’ve probably heard stories of morning commutes where the only place to put your arms is on the heads of other people, packed like sardines all around you. In recent years, it has been one of the fastest growing subway lines in terms of ridership, with the Bedford Avenue stop in Williamsburg seeing27,224 average weekday customers back in 2014. Besides the crowds, the L is a pretty convenient train for New York because of its early adoption of advanced signaling which allows for a more efficient volume of trains to travel on its tracks.

What data is available to understand more about the L?

To start answering this question, we turned to data from the American Community Survey (ACS), which describes the demographics of people, and LODES, which describes the dynamics between home and work. Both are products of the United States Census.

To look at the relationship of these datasets to the L train, we used location data on L entrances. Gathering all of this data, we are able to assemble information about Brooklyn inhabitants who work in Manhattan and who are likely to use the L based on proximity. We calculate proximity to the L entryways from every block in Manhattan using walking directions calculated by Mapzen’s Valhalla.

Using a series of common table joins and geospatial joins on a census block’s closeness to an L entrance, we created a summary table of census block groups that are:

1. Closer to L entrances than any other subway line
2. Have counts of the number of workers who live in Brooklyn but work in Manhattan (via LODES)
3. Demographic information (ACS)
4. Routing information from the center of a Census Block to the nearest L entrance which gave us an estimate of walking distance and walking time

Let’s take a look!

Why is the L so important?

Some things you might notice right away is the span of coverage in the northwest and southeast of Brooklyn. In both cases, the L train serves many city blocks that aren’t well serviced by other trains.

First, the further east people are located on the L, the more viable alternatives they have to reach Manhattan in the same or similar travel time by changing to another line before the tunnel (e.g., the A at Broadway Junction). Second, we limited our blocks to those that were within a 30 minute walk or less of the subway and inside of Brooklyn. The US Census only collects one mode of transportation to work, so while many people probably take a bus to the subway, the data doesn’t capture those cases.

While you can see Manhattan-bound subway riders along the entire length of the L, we were curious what relationships there were between travel-time and proportion of people who work in Manhattan. What we found was that three different breakdowns of travel time all showed a negative correlation (p<2e-04) with travel time. The three breakdowns are the amount of time it takes to walk to the subway station, the amount of time riding the subway to Manhattan, and the total commute time.

If you are interested in the raw numbers, he are some important ones:

• A recent estimate by the MTA said that there were 200,000 daily riders using the L to cross between Manhattan and Brooklyn.
• According to the US Census and our estimates based on proximity, 114,067 people use the L to commute to work in Manhattan.
• In all of Brooklyn, 541,803 commute to Manhattan for work using any subway line as their primary method.

These people can’t just move to new homes if the L is out of service for an extended period of time. So let’s look a bit more at who some of the people are and then what a scenario of shuttle buses would look like.

Affected populations

There are many demographic variables that we can explore, but here we will narrow our focus to poverty status as defined by the ACS to look at the proximity of affected populations to the L.

Of our estimated population of L riders, we see approximately 18,889 Brooklyn households in poverty where the nearest subway stop is the L. We can visualize the distribution of data using a choropleth as follows:

We further analyzed the data with CartoDB to identify areas in Brooklyn where significant groupings of like blockgroups (measured by poverty rates) appear. This is valuable for planning outreach and potentially targeted alternatives if the L were to close for a significant amount of time. We are looking for areas with high or low rates of poverty status as larger clusters, with the goal of identifying broader areas that would not have the financial resources and job flexibility to overcome a major transit disruption.

As you can see in the above map, smaller clusters in Bushwick and larger ones in Brownsville and East New York have significant areas of low poverty rates and are in very close proximity to the L. These clusters are likely not served entirely by the L train, as they are also situated close to M, J, and A stations. Depending on capacity of those alternate options, these might be places to explore additional bus services to Manhattan.

Greenpoint, while situated very close to the L, has low rates of poverty as compared to neighboring census block groups. It is also on the G, so has transfers to Manhattan-bound trains to the north and south, as well as close access to the Williamsburg and Queensboro Bridges and Queens Midtown Tunnel if the MTA were to start shuttle service. While these people could be served by busses shuttling them to Manhattan, one could also guess that a good number of this affuent population would be targets of commercial modes of transportation during a disruption of L service. Keep your eye out for new Uber and Lyft advertisements along those dense walking corridors.

Is there a realistic alternative?

When Gothamist covered the potential shutdown, they called it a disruption of epic proportions. We were curious just how much of a disruption a full closure of the L train would cause. The MTA suggested that one of the fastest ways to fix the L would be to close the Canarsie Tunnel completely, potentially replaced by shuttle bus services. Would that be realistic?

To answer that question, we used CartoDB to analyze the best option for riders given different bus coverage. Based on the location of the Williamsburg Bridge, we made the assumption that buses would originate at or close to the Lorimer stop in Williamsburg and end in the Lower East Side or Union Square in Manhattan. Using the Valhalla router again, we found that the fastest a person could get to Manhattan via this new bus line would be around 10 minutes without traffic. To get to any location on the L line in Manhattan would take closer to 20 minutes by bus.

If the shuttle bus takes 20 minutes, we are able to calculate the best choices for riders along the L needing to get to Manhattan. Should they go all the way to Lorimer and take the shuttle bus or should they transfer earlier at Myrtle-Wyckoff to the M train or at Broadway Junction to the A? The answer is shown in the map below.

We found that for a 20 minute bus shuttle service, all but 5 stops on the L, riders would be better off transferring to another existing line. The challenge that we realized was that the shuttle is the best option for almost 75,000 riders per day. Given large buses with 65 passengers this would still take 1154 bus trips per day, or a bus every minute and a half or so. That’s pretending there are only completely full buses or rush hours. To handle this capacity they’d have to build a new bridge.

Yet still at the various speeds of shuttle bus connections, how many subway passengers would still be better off taking the bus than back tracking to the M? What we found was that the numbers slowly decline up to 28 minutes. If the shuttle buses took 28 minutes they would still be the better option for 53,000 passengers over back tracking to the M. That is still 816 buses per day. If the shuttle buses took 29 minutes, everyone along the L is better off just taking the L to the M or the A transfers to get to Manhattan.

Interested in more?

We’ll be watching this topic closely. If it gets interesting, we’ll post some follow-up and deeper looks at what it would mean to close the L.

If you want to learn more about the methods we used above or about how to do advanced analyses of data with CartoDB, get in touch!

If you ended up on this blog post because you are just interested in tunnel repairs but are now disappointed, here are some links for you:

• If you are curious what it’s like down there MTA Video Release: Canarsie Tube Signal and Track Recovery Operations
• What does it cost to build a tunnel? Not much in 1918

CartoDB is very excited about the upcoming FOSS4G-NA conference!

We’ve given sessions at the conference for the last 2 years and hope 2016 won’t be an exception. Held in Raleigh, North Carolina from May 2-5, this is the premiere conference for free, open source software designed for geospatial in North America. Thousands of participants and hundreds of companies converge to share, demo, and talk about the things we love most.

In order to see us there we need your help!

FOSS4G NA has opened up community voting to allow participants to have a say in what sessions they would like to see at this year’s conference. You can vote for as many panels as you’d like! Voting remains open until February 8, 2016, 11:59 EST.

Here are CartoDB’s proposed sessions:

• Wrapping up Python into a Cloud-based PostgreSQL by Andy Eschbacher

• Building Urban Scratchoff - An Experimental Technique Using Canvas Tiles by Chris Whong

• Big Metadata: Mapping the Web’s Data Resources by John Krauss

• The Information Commons: Ecosystems, Public Discourse, and Open Source GIS by Santiago Giraldo

• Decision Making Data: The Open Source Future of Government Technology by Sanitago Giraldo

• Workshop: Building Dynamic Maps with CartoDB by Santiago Giraldo

• Machine Learning on Geospatial Datasets for Segmentation, Prediction and Modeling by Stuart Lynn.

Cast your votes for our 5 star panels!

We can’t wait to see you at FOSS4G-NA 2016!

After COP21, it is expected that more and more cities will work towards implementing a solution to climate changes that have adverse affects on infrastructure and populations.

A CartoDB partner, Vizonomy, created a risk platform analysis dashboard, using open government data - for this specific reason. The dashboard allows communities and businesses to see infrastructure and populations vulnerable to a rise in sea level and flooding.

Now governments and cities can visualize exposure to threats such as flooding, fire, and other natural disasters to determine high-risk areas. Vizonomy helps create city plans and programs that are resilient and responsive for a more sustainable future.

Discover how Vizonomy used CartoDB to create a dashboard that assess climate risk and improves municipal response times:

Learn more!

Local resiliency officers or urban planners can run customized analysis using federal standards at the building, neighborhood, or city level. The CartoDB-based platform uses the most up-to-date datasets available from the government: FEMA floodplains and National Oceanic and Atmospheric Administration data along with more than 70 other data layers representing buildings, schools, populations and more—through an automatic sync of updated information in the cloud.

What a great way to usher in change for 2016!

Happy data mapping!

With less than a month away before the first U.S. presidential primaries, all eyes are on the headlines for the latest jabs by Trump, Hillary, and the gang. Just how do journalists get the spin on the topics and trends this election season?

This year Google sponsored a state-of-the-art spin room for the sixth Republican Party debate. Many questions posed to the candidates and seasoned politicians come from the live streaming of google searches, tweets and other social media feeds. That’s where the citizens discussion happens on real time. So having that piece of information in a concise way, provides journalist with great context for their live reports. Check out an example map being used in action on a very large screen.

“It’s great to be able to display this amazing map data with such clarity. Reporters filmed in front of it all day and it added to the story,” said Simon Rogers, Data Journalist and Editor at Google.

The CartoDB visualization that made it into the spin room displayed the ‘Top Searched Republican Candidate by County.’ The interactive map featured the most searched republican candidate on Google by county since November, in addition to infographics with municipality details and a scroll functionality with most searched candidate by county and battleground states.

Political representative maps are focused on getting insights from data, allowing journalist to ask the hard hitting questions. CartoDB is focused on providing representation and analysis that allowed them to follow in real time the pulse of the debate from the citizen side in real time.

Will be great to hear from you other interesting maps and Location Intelligence performed with CartoDB in relation with democrazy.

Journalist seeking to recreate the magic of the spin room can find resources and other analysis tools at CartoDB.

Happy data mapping!

Mobile World Congress 2016 is now just a month away! We are happy to participate again in the world’s largest conference for the mobile industry, organized by GSMA. This global event in the mobile tech industry will take place in Barcelona at Fira Gran Via from February 22-25.

CartoDB’s Javier, Sergio, Miguel, Jaime and the rest of the team will be at booth 8.0I13 in Hall 8 with an announcement on February 23rd that will surely set the world of LI spinning!

MWC 2015 was a record breaking year with just under 100,000 attendees, nearly 2,200 exhibitors, thousands of C-level leaders, more than 140 inspiring sessions, and a cadre of press and industry analysts.

We’ll also be hosting an exclusive event alongside our friends at Mapzen at C3Bar on Tuesday, February 23rd from 7:30 pm. We ask that you RSVP so you don’t miss it.

Everyone who is part of the industry is in Barcelona for MWC. If you’re in mobile, or an industry that supports mobile, or are looking to make contacts in the mobile industry MWC 2016 is the hottest place to be!

“Mobile powers our lives. Mobile is connectivity. Mobile is identity. Mobile is commerce. Mobile is inclusive. There is no clearer way to say it. Everything is mobile, but more importantly: Mobile is Everything.” - MWC 2016.

If you’d like to schedule a one-on-one meeting with us while we’re there, please contact us at: sales@cartodb.com.

PS: You can also find us at the 4YFN, so look out for us there too!

Happy data mapping!

For two days, CartoDB will be at the Big Data Innovation - Las Vegas Summit! From January 28-29, we will participate in world-class presentations, panels, discussions, and networking.

The summit’s agenda will explore topics and themes on how to better embrace data. Topics include:

• Data Analytics
• Data Science
• Emerging Algorithms
• Machine Learning
• Big Data Technologies
• Data Security
• Data in the Cloud

Join us at the Four Seasons Las Vegas where we’ll be demonstrating the power of location intelligence for big data. CartoDB’s Santiago Giraldo Anduaga, is the official chairperson and will have a ten minute talk, introducing CartoDB and demoing the CartoDB platform. Sebastien Burgess and Santiago will also be attending one-on-one meetings with top companies in the data analytics industry.

If you’d like to schedule a meeting with us, please reach out to sales@cartodb.com.

Viva data mapping in Las Vegas!

Wait… My call’s dropping… I think my phone is breaking up. Darn it! I’m not getting any wifi signals so all my iMessages are turning into green SMS texts instead of the placating blue. Many telecommunications agencies work relentlessly to provide solutions to many of these and other customer related problems.

Crowdx, a telecommunications analytics firm, partnered with CartoDB to help remedy these cellular woes. Aggregating large quantities of data was difficult before with many companies not having the bandwidth to properly measure customer bases or the most up-to-date tools to perform a comprehensive analysis.

Discover how Crowdx used CartoDB to create a dashboard that analyzes millions of data points to provide better solutions for telecommunications agencies:

Learn more!

CartoDB loves great examples of applied data analysis and location intelligence. To find other examples of actionable insights and big data, visit our
resource center.

Happy data mapping!

What’s the busiest airport in the United States? That simple question led Geographica on a path to insights and analyses of airports and flights all around the globe. Geographica specializes in software solutions to sort, analyze, and visualize data. Using CartoDB, they developed a data-driven visualization to derive insights on thousands of data points.

Flight routes isn’t the only data displayed. This interactive map allows you to view the 50 busiest airports in the world, and the number of passengers.

We sat down with Geographica to find out more about this amazing project and why CartoDB worked for them.

Why did you embark on this mission?

After a casual discussion about which airport was the busiest, our curiosity piqued. After some research, it was determined that Atlanta was the busiest. We were amazed but even more curious. We started asking even more questions like: Why Atlanta? How are other countries similar? How many flight routes are there from Asia to Europe?

Why did you choose CartoDB?

It really came down to three reasons:

• The versatility that CartoDB offers through its ‘out of the box’ features
  
• The platform is incredibly easy to use and can handle all of the datasets that were uploaded. Using postGIS made the analysis very powerful, as well.
  
• The APIs allow us to create application on top of CartoDB that were
  fully flexible.

One of our favorite things about the Geographica’s flight route tracker is the level of analysis it incorporates. This project allows viewers to consider modern issues that might affect traditional flight patterns.

To learn more about great projects done by Geographica and other CartoDB partners, visit our resource center.

Happy data mapping!

Did this past weekend’s East Coast blizzard leave you with the feeling of wishing you could pinpoint where you built that wicked igloo during Snowstorm Jonas?

Not to worry, every igloo deserves a place on the map now - with or without specific geo-coordinates to set it apart. CartoDB’s data services team has created an updated and even better Geocoder API, taking a major leap towards the democratization of location based services.

Now CartoDB users will have the possibility of turning string data columns into numerical coordinates. Just as we offer free basemaps, the Geocoder API is part of the package. The functionality was previously only available through the Editor and now users can access it from the Platform.

The Geocoder API will let developers, partners, and techies georeference their data on their 3rd party applications using a set of functions on the SQL API - this is the main differentiator from the already existing geocoding functionality at CartoDB.

The CartoDB Geocoder API provides different levels of granularity and lets you convert plain text to countries, administrative boundaries, named places, and zip code geometries for free – programatically. It also allows for street level geocoding at a set price per 1000 rows (credits will go against your quota). For Google users, it provides integration with the Google Geocoder as well.

Happy data mapping!