Rent and real estate conditions are fascinating subjects for maps. While they may be the maps that haunt us when in moving-in/out mode, they are likewise the ones that help us mostly clearly comprehend the human, economic, and social landscape of our surroundings.

Read on to continue in our series on Map of the Week posts related to “making rent”, this installment—Montréal.

Historically, the affect of human migration patterns has dramatically changed the topography of our world, and this helps explain some of our obsession with the patterns of human mobility in metropolitan areas. Presently, the obsession continues, with more factors complicating how we deal with dual changes in pricing, stability, and our concept of community. Certain months of the year define the cultural patterns of human movement in most cities: the beginning of September marks a return to school, and likewise some rental flux in New York, the end of August marks an exodus from San Francisco for those planning a pilgrimage to Burning Man.

This Map of the Week chronicles a map project built to track the July migrations in Montréal; for this we welcome our Ambassador Florent Daudens, journalist at Radio Canada, to discuss this project and the peculiar conditionals of its genesis (with translations en anglais, and en françaiségalement).

Migration Data in Montréal

Montreal has a curious tradition: many people move on July 1. Which causes an intense hunting for flats in the previous months.

Two journalists from Radio-Canada, Pasquale Julien Harrison-Julien (@pasqualehj) and (myself) Florent Daudens (@fdaudens), wanted to know how much the average rent is in the city, and the suburbs.

We created two maps. The first one displays the average rent prices in 35 areas of the city and its suburbs, by number of rooms, according to their sample. The second one compares these data with the ones of the Canada Mortgage and Housing Corporation (CMHC), the government agency that oversees the market. It shows neighbourhoods where new rents are higher than the average, suggesting that this is where gentrification occurs.

We first searched the CMHC data. The catch: this data establishes an overall picture, both for tenants who retain the same lease for 20 years, as those who wish to rent today. They do not tell us how much the rent is for those seeking an apartment today.

To arrive at the final result, we had to follow several steps :

• Scrapping classified ads on Kijiji for two weeks
• Geocoding and cleaning in OpenRefine
• Summarizing data in spreadsheets
• Distributing the points in polygons by zones and creation of the SHP in QGIS

In the end, we mapped 10,000 ads in 35 zones.

Adventures in Multilingual Mobile Map Projects

With these data, we could begin our interviews. Comparing and validating data, getting different perspectives and going beyond only a visualization to a data journalism that provides context and helps to decrypt the situation. Where is
my flat?

Then we turned to CartoDB because we had several requirements:

• Visualize our database with maps in English and French (Oh yeeah, Canada is a bilingual country)
• Display an overview of all areas, but also allow each user to zoom in on his neighbourhood
• Filter by number of rooms
• Provide a stable mobile view

Polygons were included in our SHP files and CartoDB’s import wizard recognized them easily.

So we started to style our two maps in French in CartoDB’s GUI to generate chloropleth maps. Then, we adjusted the color slices to produces uniforms brackets for all types of apartments and thus keep the same color scale between all number of rooms.

In addition, to zoom in on a specific area, we wanted to allow the user to filter by number of rooms, so that s/he can fully compare with his/her own situation. Therefore, we used cartodb.js with createLayer method, to be able to filter
with SQL.

To set the style of each filter, we copied the CSS generated by the CartoDB wizard and simply changed the ID of the column for each apartment type.

For example:

#database_name[1bedroom<=800]{polygon-fill:#fee0d2;}#database_name[1bedroom<=600]{polygon-fill:#fff5f0;}#database_name[2bedrooms<=800]{polygon-fill:#fee0d2;}#database_name[2bedrooms<=600]{polygon-fill:#fff5f0;}

As for the tooltip on each area, we set it up with the CartoDB’s GUI after several unsuccessful attempts in the code editor. This seems simpler with createVis than with createLayer.

Once our code was ready for the first map, we just had to make a few modifications for the second one which displays the gap between our data and those from the CHMC. With our code finally structured, it was easy to create the English version. We only had to translate the text in the tooltips.

ENGLISH

FRENCH

Reflections on a Rental Realities

The final results were published in our article on CBC.ca accompanied by a french version of the same article.

ENGLISH

FRENCH

Looking back, we see some opportunities for improvement. We would have liked to style the map background, and also to limit the levels of minimum and maximum zoom, as well as the perimeter of the map to guide the user.

That said, we were able to configure our maps to be closest as possible to the reality of our readers. Tens of thousands of them have felt challenged by the subject and read the article. Moreover, we also showed our maps on our TV channel, and thus make a complex subject accessible thanks to its visual dimension. We even managed to speak about it on our radio channel!

Multi-media maps at their finest, we hope you enjoyed! You can reach out to Florent via Twitter to learn more about his maps, or check out his public profile on CartoDB.

Meantime, happy data mapping!

Fi-Ware is a public-private partnership launched by the European Union in 2014 to promote the development of smart cities apps based on open standards and open source code. By providing tools and components in areas such as data management, Internet of Things interoperability, and information security, the initiative seeks to provide simple, standards-based interconnectivity between apps across any vertical.

The Fi-Ware Development team at Telefónica has recently built a connector for CartoDB. The python script parses data from a Fi-Ware “Context Broker” (a service which allows for easy publishing of/subscribing to frequently updated data). By leveraging CartoDB’s SQL API, the connector updates rows in a CartoDB table with the latest data from the context broker, keeping the map’s underlying data as fresh as possible.

The new connector can be seen in action powering this CartoDB map of Bus Locations in the city of Santander, Spain. You can see near real-time vehicle locations as markers, or view a torque animation of location data collected over the previous 24 hours.

The Fi-Ware to CartoDB connector is now listed on CartoDB’s booming external libraries list. Take a look to see how other platforms and companies are leveraging the power of CartoDB’s APIs to bring their data to life as maps.

Happy Mapping!

Here at CartoDB we are always looking for new ways to enable our educational communities. From our commitment to Education and Research in the Classroom to our popular Educator’s Nights and workshops at universities, we’re working hard to spark new ideas in geospatial visualization and analysis across all professional fields.

Wether you’re new to CartoDB or a seasoned senior, all students can now link their Google Apps-based university email accounts directly into CartoDB! In a few clicks this union lets students skip the sign up process and get right to mapping while also receiving a pretty sizable bump up in data storage space and student resources. Talk about elevating education!

Getting started is easy. Simply follow this Student sign up link, click the ‘Login With Google’ button, and CartoDB takes care of the rest.

Building on the powerful resources we work hard creating for our educational communities, we have made it even easier for students to get started with CartoDB. For professors from across the world, you now use less class time getting students up and going!

Together, we can create a culture of learning!

Happy Mapping!

As a child, nothing was more exciting to me than a chance to ride on an airplane. And after enjoying playing with the seatbelt buckle and feeling the crazy push of take-off acceleration, I would usually settle in and page to the back of the in-flight magazine where the airline route maps were: where were we going today, and where could we go tomorrow?

Destination Map

We can build route maps for any city in the world using airport and route data from OpenFlights.org. Start by uploading the airports.csv and routes.csv files
into CartoDB.

We can see every destination available starting from Vancouver, Canada (airport code “YVR”) by making some custom SQL to join the airports table to the routes table and restricting to just the “YVR” routes:

SELECTa2.cartodb_id,a2.the_geom_webmercator,a2.city,r.airlineFROMairportsa1JOINroutesrONr.airport_st=a1.code_iataJOINairportsa2ONr.airport_end=a2.code_iataWHEREa1.code_iata='YVR'ANDr.codeshareISNULL

That’s the data we want, but without the flight lines it lacks a sense of movement.

Simple Route Map

Our query is already joining the airports twice: once for the origin and once for the destination airport, so we can turn the end points into a line very easily using the ST_MakeLine() function:

SELECTa2.cartodb_id,a2.city,r.airline,ST_Makeline(a2.the_geom_webmercator,a1.the_geom_webmercator)asthe_geom_webmercatorFROMairportsa1JOINroutesrONr.airport_st=a1.code_iataJOINairportsa2ONr.airport_end=a2.code_iataWHEREa1.code_iata='YVR'ANDr.codeshareISNULL

That looks much better! But there’s something wrong about this map – actually two things wrong.

First, the routes are all straight lines, and they should be great circle routes, that’s how the airplanes fly!

Second, some of the routes go the wrong way around the world: no airline would fly from Vancouver to Sydney via Africa!

Great Circle Route Map

If we convert our lines into great circle routes, we can maybe kill both of these birds with one stone, since the great circle routes will go the right direction.

SELECTa2.cartodb_id,a2.cityAScity,r.airline,ST_Transform(ST_Segmentize(ST_Makeline(a2.the_geom,a1.the_geom)::geography,100000)::geometry,3857)asthe_geom_webmercatorFROMairportsa1JOINroutesrONr.airport_st=a1.code_iataJOINairportsa2ONr.airport_end=a2.code_iataWHEREa1.code_iata='YVR'ANDr.codeshareISNULL

This is a bit complex, but reading the nested functions outwards starting from the ST_MakeLine(), we:

• Cast the point-to-point line to the “geography” type, which understands edges (also known as connections between nodes) as great circles; then
• Use the ST_Segmentize(geography) function to add points to the line along the great circle (so when we put it back on the flat map, it’ll appear curved); then
• Cast the line back into the “geometry” type we use for flat mapping; and finally
• Transform the coordinates of the line using ST_Transform(geometry, srid) to the web mercator projection we use on our flat maps.

The end result is really, really close!

But what is going on with those horizontal lines?

Great Circle Route Map with Dateline Fix

There’s a gap, right where the horizontal line appears.

Everything is fine until an edge on the great circle route tries to cross the dateline. Then it zings around the world in order to hook up to the next edge. Fundamentally our map still does not understand the circularity of the world, even though the edges we built do understand it. We have to work around the limitations of the flat map, by chopping our data at the dateline to avoid having edges that cross it.

-- First build our lines just as before, this can be any raw data you
-- need to feed into a dateline chopping process
WITHlinesAS(SELECTa2.cartodb_id,a2.city,r.airline,ST_Segmentize(ST_Makeline(a2.the_geom,a1.the_geom)::geography,100000)::geometryasthe_geomFROMairportsa1JOINroutesrONr.airport_st=a1.code_iataJOINairportsa2ONr.airport_end=a2.code_iataWHEREa1.code_iata='YVR'ANDr.codeshareISNULL),-- Now break the input data into two sets, one to split and one to leave 
-- unprocessed. Objects that cross the dateline will appear to be very wide 
-- (as they zing across the world) so we'll only chop features that are very 
-- wide. This is just for efficiency.
tosplitAS(SELECT*FROMlinesWHEREST_XMax(the_geom)-ST_XMin(the_geom)>180),-- Narrow objects we'll leave un-chopped.
nosplitAS(SELECT*FROMlinesWHEREST_XMax(the_geom)-ST_XMin(the_geom)<=180),-- In order to chop the objects we need to get them into a space where they make
-- "sense" so we shift any vertex that is < 0 to the right by 180 units, 
-- effectively building a map ranging from 0 to 360 units centered around the 
-- dateline, instead of the usual map from -180 to 180 centered at Greenwich.
-- Then if we split at 180, we get a nice cut, just where we want it.
-- We split by removing a very narrow gap from the objects, centered at 180, wide
-- enough so that we don't get any wee rounding errors at the dateline.
splitAS(SELECTcartodb_id,city,airline,ST_Difference(ST_Shift_Longitude(the_geom),ST_Buffer(ST_GeomFromText('LINESTRING(180 90, 180 -90)',4326),0.00001))ASthe_geomFROMtosplit),-- Merge the split features back with the unprocessed features.
finalAS(SELECT*FROMsplitUNIONALLSELECT*FROMnosplit)-- Transform them all into web mercator for mapping. 
-- The web mercator map projection handles the longitudes > 180 as if they were 
-- negative longitudes, so there is no need to convert them back.
SELECTcartodb_id,city,airline,ST_Transform(the_geom,3857)ASthe_geom_webmercatorFROMfinal

And it works!

Of course, this is a route map of all flights leaving Vancouver (YVR), so it’s not exactly the kind of map you’d find in a in-flight magazine. However, it’s easy to build such a map, just by changing set of input airports we use to build
the routes.

Where the existing query says:

WHEREa1.code_iata='YVR'ANDr.codeshareISNULL

Replace the airport filter with an airline filter of “AC” to get an Air Canada route map:

WHEREr.airline='AC'ANDr.codeshareISNULL

Or try “UA” for a United map, or “DL” for a Delta map.

Happy flying!

Spain is the number one tourism country in the world. With sights like the Museo del Prado and Royal Palace of Madrid, it is hard not to see how such spectacular beauty could ever be overlooked.

To assist in visualizing the majesty of this European gem, Vizzuality and CartoDB are proud to announce our latest release: an interactive tool to analyze tourist spending in Spain during the summer of 2014. Using BBVA Data and Analytics, you can see how tourists of different nationalities spent their time in Spain. Take a look at this UN-BBVA-LIEVABLE visualization and welcome guest blogger, Jamie Gibson from Vizzuality!

With anonymized data, on 5.4 million credit card transactions, we worked out how to visualize tourist spending effectively while optimizing the speed and performance of the application. Equipped with our unique blend of pioneering design principles and innovative coding, we delivered “a piece of artwork” - almost fit for the Prado!

Vizzuality and CartoDB are committed to making data easy to look at and understand. The designers at Vizzuality paid close attention to crafting an interface that lends itself to clear presentation and powerful interrogation of the visualization. They also structured the visualization to break down big data into small digestible chunks through the four tabs, providing multiple angles into the data, and a host of opportunities to gain a deep understanding of all the facets of this dataset.

Over five million records is a rather large amount of data, especially when you add a geospatial dimension. Instead of simplifying or splitting up the transactions to make them easy to process - we wanted the visualisation to be as accurate as possible - the whole dataset was loaded into CartoDB and queried when needed using CartoDB’s SQL API. The CartoDB platform is great for handling large amounts of data and the high user load we expected.

To produce a cohesive visualization made up of multiple maps and timelines that run in unison, Vizzuality modified and extended CartoDB’s Torque.js library, allowing the user to quickly jump back and forth in time and drill down to the transactions they want to see. To avoid having to worry about scaling and handling large volumes of users, we chose to build an entirely static, single-page application, allowing us to focus on building a great experience, instead of building infrastructure.

So what did people find out? El Pais, the biggest newspaper in Spain, pointed out that US and British tourists spend more on eating and drinking than anyone else, Barcelona gets the largest amount of tourist spending, and Madrid is the main base for Mexican and Venezuelan tourists. Meanwhile, the variation in tourist spending in different places was picked up by El Diario Vasco and discussed by members of Meneame.

It’s been great to see so many people enjoying the application and we’re really proud of what we’ve achieved together with our friends at BBVA’s Centre for Innovation and Data and Analytics team. On to the next visualization!

With the recent discovery of water on Mars, what better time than now to think of fresh new ways of space exploration and data visualization, as well as how we view the third rock from the sun?

On October 12 to 14, join our CEO, Javier de la Torre at ESRIN in Frascati, Italy to explore the new challenges and opportunities for Earth Observation (EO) research created by the rapid advances in Information and Communications Technologies (ICT). This includes open tools and software, data-intensive science, virtual research environment, citizen science and crowdsourcing, advanced visualization, e-learning and education of the new generation of
data scientists.

Javier will deliver a keynote at the innovative Earth Observation Scientific Exploitation Programme: Earth Observation Open Science 2.0. His address titled, Making the Most of Earth Observation through Visualization is at session C1A during the Scientific and Visualization portion of the program.

The program’s objective is to maximize the scientific benefits of EO data by capitalizing on the digital revolution and is organized by The European Space Agency (ESA), an intergovernmental organization dedicated to the exploration
of space.

Participation to the conference is free and will be followed by a Hackathon on October 15 and 16.

Happy data mapping and see you in Frascati!

If you’ve ever seen an animated map on the web, or read our technical blogs for the past year, you might be familiar with Torque, CartoDB’s library for animating time-series data on maps. You can already build a range of Torque maps with point data in our Editor GUI: plotting dynamic events, social reactions, traffic patterns, categorical comparisons, and hotspots of activity.

This week, we went further, open sourcing Torque.js, and building documentation for developers to get started customizing Torque code for their projects and contributing to the library at large.

TECHNIQUE IN TORQUE

Torque’s approach to rendering spatiotemporal data relies on methods for spatial aggregation, economic encoding, and tilecube rendering. There are some great tutorials to illustrate how you can invoke Torque in the CartoDB platform.

This week, we’re launching a persistant reference page to showcase all of the resources available for working with Torque.js.

READ THE DOCS

Integrating Torque.js in your code is easy, you can learn more about it here in our docs and on our landing page.

We’ve also updated our CartoCSS docs to include properties specific to
Torque maps.

LEARN FROM SAMPLES

Our standard Torque demo illustrates the functionality in a friendly and filterable sandbox, but we also have an index of example code snippets that show the breadth of options when designing maps with Torque.js, from voronoi hacks, to bubble maps and beyond!

BUILD THE FUTURE OF TORQUE

In the future, we’ll be including animation capability for not just points, but also lines and polygons! We’re updating our docs daily, and answering issue questions where possible. This is a collaborative effort and we totally want our community involved. Creative folks in our community have built Torque projects to plot time-series data globally, pairing Torque with other services, and partnering it with other mapping libraries. We’re excited to see what’s next!

BROWSE THE RESOURCES

Looking for more info, or something specific not linked above? Check out the following links:

LIBRARY:

• Torque.js: https://github.com/CartoDB/torque
• Torque Tiles: https://github.com/CartoDB/torque/blob/master/dist/torque.js
• Non-Torque Tile Generation: https://github.com/CartoDB/torque_cube

DOCS:

• Landing Page: https://cartodb.com/torque
• Torque README: https://github.com/CartoDB/torque#torque
• Torque Spec: https://github.com/CartoDB/tilecubes/blob/master/2.0/spec.md
• CartoCSS Torque Methods: http://docs.cartodb.com/cartodb-editor.html#cartocss-properties-for-torque-style-maps

TUTORIALS:

• Intro to Torque: http://docs.cartodb.com/tutorials/introduction_torque.html
• Torque with Heat: http://docs.cartodb.com/tutorials/heatmap.html

EXAMPLES:

• Torque.js Examples: https://github.com/CartoDB/torque/tree/master/examples
• Torque Demo: http://cartodb.github.io/torque/

It’s been featured in tutorials for journalists, in partnership with other mapping libraries, in Twitter data maps; we’re so proud to make it more open to the world of cartographers and developers.

Check it out, set it up, and happy mapping !

To an outside observer, serving a population of 150,000 map creators and millions of map viewers might seem like a simple matter, but to us it’s a big complex deal, and we’re always looking for ways to make the CartoDB
platform faster!

Because we serve so many users, and thus so many apps, finding even small efficiencies can generate big savings on the amount of infrastructure we need to deploy, which is good for the environment and good for the business.

We work with spatial data to draw maps, and one thing that disguishes spatial data from regular data is size– it’s really, really large. Just one country polygon can take up as much storage as several pages of text.

If we can reduce spatial data in size, we win twice. A big polygon takes up a lot of memory on a map rendering server, and a lot of network bandwidth as it is moved from the database server to the rendering server. And if we do the job right, any changes in the maps will be visually undetectable.

The input data for this map were 24kb:

The input data for this map were 554kb:

See the difference? Me neither.

There are two ways to make spatial objects smaller:

• Reduce the number of vertices used in their representation, and
• Reduce the amount of space used by the representation itself.

To improve CartoDB efficiency this summer, we worked on both approaches, changing the SQL Mapnik generated during rendering requests to lower the size of the generated spatial data.

Reducing vertex count

As a loose rule, since the map output will be displayed on a screen with some kind of resolution, any vertex density that exceeds the output resolution is wasted information. (That isn’t quite true, since most modern map renderers use “sub-pixel” shading to convey higher density information, but past a certain point, like 1/10 of a pixel, extra vertices aren’t helping much).

We started with a simple approach, just using the ST_Simplify() function to reduce the number of vertices in our geometry. This worked great, creating compact geometries that still captured the intent of the original, but it had two problems:

• It was a bit too slow for large objects.
• It was too hard on small geometries – not only did it simplify them, but if they fell below the simplification tolerance, it dropped them altogether. This could result in surprising gaps in maps that were zoomed out.

Pre-filtering to make things faster

Our initial filtering used the long-standing ST_SnapToGrid() function, since snapping coordinates to a tolerance and removing duplicates is a very
fast operation.

Unfortunately, snapping at high tolerances can lead to an output that doesn’t follow the original line particularly closely, and can contain some odd effects.

For PostGIS 2.2, ST_RemoveRepeatedPoints() includes an optional parameter, to consider any point within a tolerance to be “repeated”. The cost is only the distance calculation, so it’s no more expensive than snapping, and the results are much closer to the original line.

Retaining small geometries

We had to enhance the ST_Simplify() function to optionally retain geometries that were simplified out of existence. For example, look at this map of building footprints (in red), simplified to a 30 meter tolerance (in green).

Many of the buildings are smaller than 30 meters, and simply disappear from the green map. To create an attractive rendering, we want to keep the tiny buildings around, in their simplest form, so they fill out the picture. For PostGIS 2.2, we added a second parameter to ST_Simplify() to “preserve collapsed” geometries rather than throwing them out.

Reducing representation size

If you don’t work with strongly-typed programming languages you probably don’t put a lot of thought into how much space a representation takes, but different representations can take drastically different amounts of size to store the
same value.

The number “5” can be stored as a:

• a 1-byte character,
• or a 2-byte short integer,
• or a 4-byte integer,
• or an 8-byte double,
• or an 8-byte long integer.

So, for this example, there’s an eight-times storage size win available, just for choosing a good representation.

The default implementation of the PostGIS plug-in for Mapnik uses the OGC “well-known binary” (WKB) format to represent geometries when requesting them from PostGIS. The WKB format in turn specifies 8-byte doubles for representing vertices. That can take a lot of space, space that maybe we don’t need to use.

Accurately storing the distance from the origin to a vertex takes pretty big numbers, sometimes in the millions or 10s of millions. That might legitimately take 3 or 4 bytes, or more if you need sub-meter precision.

But storing the distance between one vertex and the next vertex in a shape usually only takes 1 or 2 bytes, because the points aren’t very far apart.

So, there is a big space savings available in using “delta” coordinates rather than “absolute” coordinates.

Also, the image resolution we are going to be rendering to implies an upper limit to how much precision we will need to represent in each delta. Anything more precise than 1/10 of a pixel or so is wasted energy.

To make all this a reality, we made use of the work Niklas Aven has been doing on “Tiny well-known binary”, a representation that retains the logical structure of WKB (simple feature geometry types) but packs them into the minimum possible size using delta encoding and variable length integers.

All this calculating of deltas and minimum sized integers takes a fair amount of CPU time, but in our application the trade-off is worth it. Our databases very happily serve thousands of users without putting pressure on the CPUs – all the pressure is on the network and renderer memory.

We have made sure that tiny well-known binary is ready for release with PostGIS 2.2 as well.

Putting it all together

When we put it all together and tested it out, we were very happy, particularly for rendering large complex features.

This set of Amtrak routes, drawn in purple, is rendered without filtering, and using WKB as the representation format. So the image is as “correct” as possible. Because the routes have some small back-and-forth patterns in them below the pixel scale you can see the renderer has furred the lines in places.

In terms of size, the original data are:

• 147,000 total vertices, and
• 2.3Mb when encoded in WKB.

That’s a lot of data to throw over the wire, just to draw some purple lines.

Here’s the same data, rendered after being filtered (using repeated point removal), simplified (but not dropping features below tolerance) and encoded in TWKB. Our size is now:

• 14,000 total vertices, and
• 29Kb when encoded in TWKB.

That’s right, we are now only transporting 1% of the data size, to create basically the same image. In some ways this image is even nicer, lacking the furry parts created by the overly-precise WKB data.

Final answer

And when we rolled it out to the real world, did it perform as we hoped? Yes, it did indeed, visibly reducing our network overhead and memory usage when it rolled out: the red graph is network usage, and you can see when the deployment took place.

We’re running a patched version of PostGIS 2.1 right now, updated with all the changes we require. These changes are all part of the PostGIS 2.2 release, which will come out very soon. We hope others will find these updates as useful as
we have.

Yet, there are still many things to optimize, would you like to help? Join us!

Happy data mapping!

Howdy Mappers! CartoDB is happy to be a platinum sponsor of the Texas GIS Forum, the state’s premier conference for the geospatial professional community.

The Texas GIS Forum is hosted from October 26th to the 29th at the Commons Learning Center at the J.J. Pickle Research Campus in Austin, TX.

Join CartoDB in Austin, where geography and technology intersect, to hear about the latest advancements in the private and public sector and take the opportunity to touch base with long-time colleagues — as well as make new connections. The Texas GIS Forum is the can’t miss event for the statewide GIS community!

To learn more about GIS and CartoDB checkout these awesome events!

• Wednesday, October 28th Andy Eschbacher is giving a half-hour talk on
  GIS Lessons: Learning to Query and Code with CartoDB Academy from
  2:30 p.m. to 3 p.m.
  
• Wednesday, October 28th CartoDB is sponsoring a Forum Social at Punch Bowl. You can find drinks, bowling, darts, and ping pong. Join CartoDB at 6:00 p.m. Bring your badge to get two drinks tickets and use reserved bowling, darts, and ping pong! Classic arcade games are also available. Learn more about this GIS forum here.
  
• Thursday, October 29th Santiago Giraldo will give a half-hour talk on
  The Information Commons: Cloud-based GIS from 11:00 a.m. to 11:30 a.m.

For more information and the full agenda please visit Texas GIS Forum!

Happy data mapping!

From real estate and government organizations, to the financial sector—location intelligence is used by many industries to uncover a myriad of insights in big data. Media and Creative agencies can also harness the power of insight and uncover many secrets through a great data-driven visualization for increased brand engagement and international exposure.

We would like to spotlight one of CartoDB’s currently most-viewed visualizations, Tecnilógica’s Ashley Madison infidelity map, Malfideleco.

Having the trendiest map at the moment has been especially revealing for Tecnilógica, a creative digital agency that specializes in website design
and development.

To create a map showing infidelity on a global level, Tecnilógica began analyzing leaked Ashley Madison data. “As CartoDB partners, we are always thinking ‘Can we make a map with this?’ When we discovered the geographical data in the AshMad database, there was no other option than to say, ‘Let’s make a map,’” said Juan Alonso, CTO of Tecnilógica.

To do this, the Tecnilógica team spent around six hours cleaning up the dataset and filtering sensitive information - in order to protect the account holder’s privacy. Then they made a visualization that presented a global view of the cities with the greatest number of accounts. This process only took them two hours. How easy!

With more than 5.5 million map views in less than 20 days, this data-driven map has produced a number of reactions, both Spanish and international. Tecnilógica has received requests for more information from countries all around the world.

To learn more about how CartoDB helped Tecnilógica visualize this trending media topic, download our case study and start creating engaging visualizations!

Learn more!

At CartoDB we really hope to see more successful user experiences like this one!

Happy data mapping!

At CartoDB we listen to what our users need and want. We are constantly brainstorming and coming up with new ideas, but it is always the very best in user experience that we hope to deliver. This month we’ve launched two brand new sections on our website: Explore and /me!

Visit Explore to see the top trendy maps of the day. The new explore section allows users to browse the most popular public CartoDB data-driven visualizations. Now you can see the top 10 maps from all CartoDB users based on three criteria: likes, views, and date. You can also filter your list by “maps”, “datasets”, or “maps and datasets”! Additionally, you can order visualizations by most ‘likes’ and views.

The “map of the moment” will be displayed as the top page banner. And of course to see even more data-driven visualizations select ‘view more’.

Start creating data-driven visualizations and maybe yours will be a
“map of the moment”!

But before you start taking your visualizations to the top of the data mapping charts, customize and track your profile activity.

Take a look at CartoDB’s co-founder Sergio Álvarez’s profile. He’s included information like: twitter handle, email address, and how long he’s been a CartoDB member. Pretty nifty, right?

To customize your profile and make it even cooler than Sergio’s, you only need to click on your avatar, go through your account settings, and change your profile information. Update your:

• Avatar
• Name
• Website
• Location
• Description
• Twitter Username
• Disqus Shortname
• Hire Availability

…to make CartoDB truly yours.

Follow these steps and start getting even more out of CartoDB. And of course, continue enjoying what you love most about CartoDB. To learn about these and all the other developments and features at CartoDB sign up for one of our webinars!

Happy data mapping!

Businesses and real estate agencies can save time and money by adding spatial analytics to their intelligence systems. Location analysis can provide insights that support and improve decision making from marketing to supply chain logistics and operations. Rilos, an international business-to-business company, fused business intelligence and geographic analysis to discover powerful new insights. Now that’s location intelligence.

Using CartoDB’s location and data analysis driven platform, Rilos created Geomatrix, an online customer-centric solution, dedicated to quickly producing user-friendly analytical reports directly for employees and clients, bypassing the need for GIS and other specialists.

The Geomatrix app aggregates big data, technologies, and algorithms in a single online platform to generate site studies. CartoDB’s Torque feature creates visualizations using chronological time-stamped data. This dynamic capability provided Rilos and their clients a new look at data analytics over specific time periods. With Geomatrix delivering analysis directly to clients, Rilos was able to focus on the creation of new custom content and functional modules. Geomatrix subscribers can audit the commercial environment of their existing point of sales portfolio, compare this data with operational results, and process a correlation analysis to determine which external parameters are impacting point of
sales turnover.

Geomatrix proves that CartoDB's location intelligence leads to business insights
and gives clients a competitive advantage.

Learn more!

Rilos selected CartoDB to help develop Geomatrix after several months of testing many different platforms. By choosing CartoDB, Rilos was able to get Geomatrix up and running quickly, receive custom technical support and regular additions of new functions and modules.

CartoDB’s location intelligence enriches business intelligence with deep insights from geographical and customer data. To learn more about how CartoDB helped Rilos harness the power of location intelligence, download our case study and see how you can start putting business intelligence to work for you.

We really hope to see more successful user experiences like this one!

Happy data mapping!

The National Oceanic and Atmospheric Administration (NOAA) has just partnered with Amazon Web Services to make a huge amount of historic and current radar data publicly available. In this tutorial we will show you how to use CartoDB to map this data to explore weather!

Weather effects us all in many different ways, be it knowing if its hot enough to go to the beach or getting advanced warning of a destructive storm approaching your city. One of the major world-wide organizations tasked with monitoring our weather is NOAA who among other things operate a radar network that covers all of the US.

We will walk you through how to get the data off Amazon S3 and into CartoDB to make both high-resolution polygons and torque-based animations using the NEXRad data.

Accessing the data.

The data lives on Amazons S3 services which is a great service for storing large amounts of data. To grab the data out of there we will need to know the structure of the data. The data lives in the “noaa- nexrad-level2” bucket and is split in to files and stored in a directory structure that follows a format of:

/Year/Month/Day/NEXRAD Station/filename

This allows us to grab just the data we want from the dataset. This can be done either via amazons command line utilities.

To list files available for a given year month day and station we can use
the command:

aws s3 ls --recursive s3://noaa-nexrad-level2/2010/01/01/`

and then download one of those files using the command:

  aws s3 cp {path} ./

You can also access the files via the AWS API kits for your favorite language or libraries like boto for python which we will show an example of in the
dynamics maps.

Static maps.

Once we have downloaded the data, we need to convert it to a format that CartoDB can load. Lets grab the data for Hurricane Arthur which hit off the coast of the North Carolina. Using the map above we see that the MHX station is close to the storm site so we can see the files available using this command:

aws s3 ls --recursive s3://noaa-nexrad-level2/2014/07/03/KMHX/

Let’s grab the file at 2014/07/03/KMHX/KMHX20140703_182118_V06.gz

aws s3 cp s3://noaa-nexrad-level2/2014/07/03/KMHX/KMHX20140703_182118_V06.gz ./

Now let’s open this in the climate toolkit. The first thing we want to do is select the elevation at which we’ll slice the data. Let’s take it as 0.54. This should look like this:

Frame the view using the zoom and move tools, and then we are ready to export the data to a shapefile. Click the data tab on the side then click on export. Enter in a location to save the data, select the elevation we had before, 0.54. When asked what the spatial extent is, just click next, it will grab this info from the view window. Similarly hit next on the following screen then start export. This will produce the shapefile for you. It will take a while.

It should have produced 4 files. One with the extension .dbf, .prj, .shp and .shx, select all of them and create a zip file from them. Next simply upload this
to CartoDB.

Once it’s loaded, head over to map view and select the positron dark basemap and then select the choropleth wizard. Set the Polygon Stroke to 0 and the column
to value.

You should now have an awesome map of our Hurricane!

Dynamic maps.

To make the map animate, we are going to have to use Torque which unfortunately doesn’t work just yet with polygons. Instead we are going to have to convert the data to a CSV and upload it to CartoDB.

We have created a python script to do just this which you can find here. You will need to install a few packages which we recomend you do with Anaconda.

conda install boto
conda install -c https://conda.anaconda.org/jjhelmus pyart
pip install pygeo

Once you have done this we can select the data for your dynamic map by changing this line:

keys = grab_list_of_files("2014", "07", "04", station="KMHX")

We have set it up here to grab all the data for Hurricane Arthur on the 4th of July 2014. Running the script will take a little while but will eventually produce a csv file called results.csv. Upload this to CartoDB and select Torque Cat as your map-type in the Wizard.

Select the CartoCSS panel and paste in the following CartoCSS.

Map {
-torque-frame-count:30;
-torque-animation-duration:5;
-torque-time-attribute:"date";
-torque-aggregation-function:"avg(value)";
-torque-resolution:1;
-torque-data-aggregation:linear;
}

#result2{
  comp-op: source-over;
  marker-fill-opacity: 1.0;
  marker-line-color: #FFF;
  marker-line-width: 0;
  marker-line-opacity: 1;
  marker-type: ellipse;
  marker-width: 1.2;
  marker-fill: #0F3B82;
}
#result2 [ value <= 47.0025244471123] {
   marker-fill: #B10026;
}
#result2 [ value <= 31.6189224706048] {
   marker-fill: #E31A1C;
}
#result2 [ value <= 28.5837251630989] {
   marker-fill: #FC4E2A;
}
#result2 [ value <= 26.3768800405451] {
   marker-fill: #FD8D3C;
}
#result2 [ value <= 24.6271245698714] {
   marker-fill: #FEB24C;
}
#result2 [ value <= 23.1575184005245] {
   marker-fill: #FED976;
}
#result2 [ value <= 21.8729825923365] {
   marker-fill: #FFFFB2;
}

You should now have an awesome animated version of the Hurricane Arthur map like this one:

We have only scratched the surface of what is possible with the NEXRad data in CartoDB. Have fun playing with this data which is being updated all the time.

Happy weather mapping!

Looking for a digital service that offers an understanding of data complexity that leads to smart decisions, that make people’s lives easier? Looking for a conference that can offer all these things too?

Aurelia Moser, one of CartoDB’s Data Scientist, will be at Extract San Francisco, a full day conference and workshop dedicated to telling data stories that will entertain, educate, and inspire. Discover everything you wanted to know about data, from people who know it best. Aurelia’s workshop Making Dynamic Maps with CartoDB will be at 10:20 on Saturday, October 30. She will also be hosting a booth during the entire event. Stop by and learn about how easy it is to gain insight from location intelligence!

Last year Extract San Francisco sold out in advance. This year organizers imprort.io moved the conference to Dogpatch Studios in San Francisco and rented all three floors to accomodate the crowd of data lovers.

Speakers come from across the data spectrum and from some of the most successful and innovative companies in the business. Join Aurelia, and many other data scientist and professionals, to extract insights from big data and learn how to start using CartoDB!

Extract is your chance to learn from the biggest game changers in data! If you haven’t purchased your ticket yet, ACT NOW!

To prepare for Extract, take advantage of CartoDB Resources to guide you to better data-driven results and solutions.

Happy data mapping!

Amidst the growing number of civically engaged public events, federally led hackathons, and public development projects, the question has arisen, where can all these projects and information sources be consolidated?

Commons Lab Inventory, is a project with the mission of tackling global affairs and informing actionable ideas with an emphasis on issues related to science and technology. Their main goal is to support a growing community of federal agencies engaging the public through crowdsourcing, citizen science, and open source innovation.

For the first time researchers, government officials, and citizens interested in developing solutions through data can filter by project topic, agency sponsor or partner, geographic scope, participant age, or intended outcome.

“Our partnership with CartoDB began after we already had a prototype. CartoDB offered a complete re-design of the interface (as well as implementing important functionality) that was much more consistent with our brand,” said Anne Bowser, co-director of Commons Lab.

The main push behind this project is to get U.S. government agencies to identify the synergies between them and publicize and mobilize public participation and innovation in science, technology, and policy, through research development. Right now, there are 102 projects that belong to 19 agencies. Bowser has identified this as an opportunity, explaining, “We also see the inventory as a research tool, valuable for providing a ‘snapshot’ of what federally funded citizen science looks like in 2015. Here, the current database could be particularly valuable if it was expanded to an international scope — or, if federal projects were compared to projects in other databases.”

The Commons Lab database illustrates what is possible by highlighting existing projects, and shows what is successful by illustrating different aspects of project design. The database can show areas where where certain gaps may lie.

The platform is a web application based on HTML5 and Javascript technologies, using CartoDB as data and visualization backend. A CartoDB partner, iCarto, has worked jointly with the Commons Lab team in designing a new interaction scheme, developing an user interface more consistent with its brand and creating a data model to support its process and work methodology.

The Commons Lab project has been selected for inclusion by the White House in its Citizen Science Toolkit. “The inventory is currently featured on the landing page of the Federal Crowdsourcing and Citizen Science Toolkit, and we couldn’t be happier,” Bowser says.

What’s next for the project? Sharing the open source code, so that others may build on CartoDB’s work directly. Bowser would like the Commons Lab to assist other agencies in their efforts to use existing code to create a database but are just as excited about a “project database that could be linked to actual datasets that the public could access and use.”

Happy data mapping!

Most maps that we see on the web use the Web Mercator projection. Web Mercator gained its popularity because it provides an efficient way for a two-dimensional flat map to be chopped up into seamless 256x256 pixel map tiles that load quickly into the rectangular shape of your browser.

If you asked a cartographer which map projection you should choose for your map, most of the time the answer would not be Web Mercator. What projection you choose depends on your map’s extent, the type of data you are mapping, and as with all maps, the story you want to tell.

Well, get excited because with a few lines of SQL in CartoDB, you can free your maps from Web Mercator!

This blog covers how you can go from the standard Web Mercator:

To Albers Equal Area Conic (a popular choice for thematic maps of North America):

Projections in CartoDB

Every CartoDB account comes with a set of default projections. Even if the projection you are looking for isn’t in the default set, no problem! In a few steps, you can start using nearly any projection you want for your web maps.

Even better, this is all done on the fly and you can project the same dataset in multiple ways for different maps.

For a more detailed discussion on projections, see this tutorial.

Getting Started

Let’s start by accessing the list of default projections available in CartoDB:

• From your account open any SQL tray, and type in

SELECT*FROMspatial_ref_sys

• Next, click Apply Query

What you will see is a table load with all of the projections that you can use for your maps. Take some time to sort through the table to see what is available. As mentioned before, even if you don’t see the projection you are looking for, its ok!

Adding a Projection

Our final map is in Albers Equal Area Conic centered on North America. I know this projection isn’t in the default list, so let’s add it.

To add a projection, we need to insert its definition into the spatial_ref_sys table. There are a couple of great websites out there where you can copy and paste the definition that you need. Two of the ones that I’ve found most useful are spatialreference.org and epsg.io.

• In a web browser go to epsg.io
• In the search bar, type Albers Equal Area

• Scroll to the bottom of the first page and click on the link for North America Albers Equal Area Conic
• Under the Export list on the left hand side of the page, click on PostGIS and copy the projection definition text

• Back in your CartoDB dashboard, paste the definition text into a SQL tray and click Apply Query

• Your projections table is now updated with North America Albers Equal Area Conic (SRID 102008)

Let’s Make a Map!

Now that we have added the projection definition to CartoDB, we can use its SRID to project any data layers on the fly. In this example, we’ll use two datasets from Natural Earth (land and ocean) that are available in CartoDB’s Data Library.

• From your Maps dashboard, click the option NEW MAP
• In the Add Dataset window, we’ll search for available Natural Earth datasets by typing ne_50m into the data search bar
• From the available list, highlight Land (ne_50m_land) and Ocean (ne_50m_ocean) and click the option to CREATE MAP

• Next, we’ll project each layer using ST_Transform and the projection’s SRID

• Copy/paste or type in the following query into each layer’s SQL tray and click Apply Query

ne_50m_land

SELECTST_Transform(the_geom,102008)ASthe_geom_webmercatorFROMne_50m_land

ne_50m_ocean

SELECTST_Transform(the_geom,102008)ASthe_geom_webmercatorFROMne_50m_ocean

The land and ocean datasets should now be projected and your map probably looks something like this:

Ok! Let’s add some final touches to the map.

• Since we are using the Albers projection centered on North America, let’s zoom our map to focus on that part of the world
• Next, we’ll remove Positron as our basemap and instead use a white background. To do this click on the option to Change Basemap and choose white (#FFFFFF) for your Custom color

• As a final design touch, change the color of the land and ocean. If you would like to use similar colors to the final map, here is the CartoCSS:

ne_50m_land

#ne_50m_land{polygon-fill:#98B087;}

ne_50m_ocean

#ne_50m_ocean{polygon-fill:#B8D0CB;}

And here is our final map! (If you would like to add graticule lines to your map you can download them from Natural Earth, and add the ST_Transform SQL from above.)

Coming Soon

In the coming weeks, look for more detailed blog posts going over some advanced cartographic effects on a variety of maps… most of which are NOT Web Mercator!

Other Projections and Additional Resources

And for fun, here are some other projections that you might like to use in your maps. This CartoDBlock has a more detailed overview with links to the projection text and SQL examples.

North Pole Azimuthal Equidistant

World Bonne

Lambert Conformal Conic centered on Asia

Winkel Tripel

We’ll be hosting a CartoCamp here at the CartoDB offices in NYC on Tuesday, November 10th. Join us to learn more about advanced mapping techniques (including projections) with CartoDB! You can sign up here.

Happy Mapping!

In the ever­ changing and rapidly growing ecosystems that define cities, the CartoDB platform has become imperative in enabling citizens, city agencies, and private industries to implement easily adaptable, real ­time enabled, and analytically robust data­ driven solutions.

Join CartoDB at Smart City Expo World Congress in Barcelona, November 17-19, as we continue to investigate the most innovative developments transforming our cities. The event is the best meeting point for companies, public administrations, entrepreneurs, and research centers to show, learn, share, network, and gather inspiration to support the development of cities of the future.

Organized by Fira Barcelona and hosted by Ajuntament de Barcelona, SCEWC aims to define what smart cities are, what their challenges include and examine which solutions and responses are most relevant.

Participants will have the opportunity to share knowledge, innovative projects, and strategies with experts and leaders from the smart city community to tackle the challenges faced by modern urban developments.

Applications of CartoDB technology have revolutionized the smart cities paradigm and become an essential tool for civic insight by providing unparalleled ease of integration, data processing capacity, and analytical speeds with cutting ­edge design.

With over 125 global cities and more than 200 companies and institutions confirmed in attendance, come share visions and solutions towards becoming a more livable and sustainable society. Let’s build the future of cities together.

By the year 2050, an additional 2.8 billion people will be living in urban areas compared to nowadays. What’s more, at our current rate of emissions, by 2032 we will have exhausted our carbon budget and pushed global temperatures above pre-industrial levels. What kind of cities will we have created by then?

We hope to see you at Gran Via Exhibition Centre, Avinguda de Joan Carles I, 64, L’Hospitalet de Llobregat (Barcelona). We’ll also be hosting a new session of our famous CartoDBeers, an informal gathering to talk data, visualizations, and tech…stay tuned for more info!

To learn more about the innovative ways CartoDB is shaping the geospatial community and the data-driven world visit our resource center today.

Happy data mapping!

This past Tuesday marked the 111th anniversary of the New York City Subway, an iconic transport system which to-date has the most active stations of any underground system in the world. Subways operate in over 55 countries globally, throughout a growing collection of 150-some cities; and the art of mapping subterranean transit continues to fascinate information designers and commuters alike.

This past week, Cooper Union hosted a sold-out event called “The Subway Map: The Last 50 Years, The Next 50 Years” for a packed crowd of New Yorkers, and in honor of this, we’ve assembled a quick collection of some of the most interesting subway maps in CartoDB, along with a little tutorial on using subway data in your own maps.

Getting Transit Data

The New York City Subway lines are available in CartoDB’s Common Data Library, available to all users. You can also check out Steven Romalewski’s datasets of MTA data, and otherwise check out OKFN’s Transit data dashboard, for a collection of all available transit data by city. You can also help “groom” the datasets by submitting changes for versioning or recommendations for review.

You can also fork these public datasets in various formats on CartoDB:

• 2015 NYC subway entrances
• 2015 NYC subway routes

A Tour of Subway Maps

Many members of our community have made pretty clever maps with ridership data.

SUBWAY FOUNDATIONS

Many users have built functional maps of NYC subway design. Like this one featuring subway entrances and lines in NYC.

Or this one featuring the real-time repair status of MTA lines.

SUBWAY DANGER ZONES Sarah Ryley’s map of subway danger zones was also a Map of the Week and otherwise well-featured in the NY Daily News.

SUBWAY DESERTS Sometimes the absence of transport is information nearly as critical as its presence. Chris Whong mapped a lack of access using buffers around subways to showcase transportation “deserts” in his map of New York City.

SUBWAY ECONOMY

Sometimes the micro-economies that subway access can highlight become topics for some pretty awesome maps, like this one overlaying average income on a distance map to the NYC subways.

Visualizing Subway Ridership

Beyond mapping the lines and stops, you might also be interested in the human impact of these transit systems, on integrating ridership data into your study of subways. Below is an example of a Torque map that shows ridership entry data for rail stations in Chicago. The oscillations in station-point size reflect the number of people entering a station on any particular day, with a noticeable dip on the weekends. The data took some munging, but you can find subway ridership data in many city open data portals.

Thanks for reading, and happy subway mapping!

CartoDB has developed a new connector to better integrate your business data – now upload and connect files to Box. Just plug any Box file into CartoDB by selecting the source on the connect dataset window.

Login into your CartoDB dashboard -> create a new map -> connect dataset

Box is a certified under the Federal Risk and Authorization Management Program, a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. You don’t have to worry about the security of your data with the ease of this integration.

Also, enjoy real-time syncing with the files you have in Box. Enjoy secure, effortless data visualizations across your business.

With CartoDB you’ll never need to download big company files from your computer. Once your dataset is in Box you can import it. Access synced data anywhere you’re connected. Make better decisions for your business based on real-time updates.

Start leveraging location intelligence for your business today!

To learn more about the power of location intelligence visit our resource center.

Happy data mapping!

CartoDB’s strength is its agility working alongside other excellent tools in a mapping workflow. QGIS (Quantum GIS) is one of those tools, and its position as a no-cost open-source GIS software leader makes it an accessible, and affordable choice in exploring data. Now, the two can work together much
more seamlessly.

The CartoDB QGIS plugin makes possible viewing, creating, editing or deleting data from a CartoDB account in QGIS. It’s so simple that once you set it up it feels like an integrated part of the QGIS interface.

One-button upload and download, extends your data exploration to any analysis workflow you might normally do in QGIS, and imports the new polygon, point and line data you create into your CartoDB account. The plugin also gives a SQL filtering import, making it possible to get only a subset of your data, at will.

What this means is that no matter what type of CartoDB account you are using, you can access your stored datasets and add them as layers to your QGIS projects for further processing.

Once imported, you can edit them using the QGIS built-in tools and robust marketplace of plugins, blending it with existing data layers to build complex projects in a CartoDB/QGIS workflow. Some potential uses:

• Deeper print capabilities for CartoDB: We have a great Static API to use in print work, but sometimes a job will need higher-resolution print capabilities. The CartoDB plugin enables this to happen in the QGIS Print Composer, a great way to get GIS into print production.

• Rapid attribute editing: Changing large amounts of data and fine-tune attributes all at once in the QGIS interface.

• Geo-processing Tools: Use QGIS analysis tools – voronoi diagrams, nearest neighbor and more – on your CartoDB data, then re-upload it with new vector data. Combine the analysis power of CartoDB and QGIS in a workflow.

Here’s and example of a Voronoi diagram done on New York City’s Citibike bike-sharing stations, using the QGIS CartoDB plugin:

There’s a lot more you can do with the CartoDB/QGIS workflow. The plugin can be downloaded from directly within the QGIS plugins menu, and installed in seconds. Try it out for yourself today.

Big thanks to Kudos for their help making this possible!

Happy data mapping!