Geo Code

The following post was written by our newest team member @deland, and it is something between an introduction to the Hyperpublic API and a documentation of a first experience with it. The actual use case described below is just a toy application - but it provided an opportunity for some cool visualizations. We hope you enjoy it and come up with some cool uses of your own!

If you haven't already, the first thing you're going to need to do to get started with the API and follow along (or experiment on your own) is to register for a Hyperpublic API key. It's free! It's fun! There are API wrappers for many programming languages so you can choose your favorite and get started right away. I'm going to be describing my method using Python, but the syntax in all the languages is pretty straightforward.

We're going to focus on pizzerias in New York City because everyone loves pizza (and because we had to choose something). The Hyperpublic API is a window into the world indexed by (exact) location. We can query the API by sending it addresses, zip codes, phone numbers, or latitudes and longitudes as well as categorical information (more on this in a moment). The query we send the API is then resolved to a specific latitude and longitude, and nearby places satisfying the query are returned. This means if we send the API a generic location like "New York, NY", a specific latitude and longitude is set, and then the search is executed. What this really means is you may not get what you wanted if you use such a generic location! The parameters available to query the API are described here, as is the structure of the documents that will be returned to you.

The categorical information in included in the query quite easily. We ask for places with a specific category (restaurant, office, etc) or we can ask for the database to be searched for other text (via the parameter 'q', for query). All locations with a place-name or tag matching your text will be returned. So we will query the API using the location parameter and also q = 'pizza'. For reasons that will become clear soon, I took the result of the queries and stored them in a local Mongo database . To be able to get all pizzarias in new york city, I looped over all the zipcodes in new york. The API only returns 50 locations at a time, but we can ask for more. Here is the relevant code:

from hyperpublic import *
...
for zip in zipcodes:
   inserted = 0
   notinserted = 0
   for p in range(1,rounds + 1):
      try:
         items = hp.places.find(location=str(zip),q="pizza", page = p, page_size = 50)
      except:
         break
      for item in items:
         try:
            dbh.nypizza.insert(item, safe = True)
            inserted += 1;
         except DuplicateKeyError:
            notinserted += 1
    print "%d items inserted and %d duplicates found for zipcode %d" % (inserted, notinserted, zip)


Here, my local mongodb connection is called dbh and the collection is called nypizza. Because some zip codes don't cover that much area - many of these queries will return the same pizzerias as other queries. This may not be a problem depending on your application, but you also might want to be aware of it (I just prevented mongo from duplicating records by inserting a key on the id field). Now we have the location of all (I get 2439) pizza places in the city - how easy was that?!

Probably we want to visualize all the locations at once. There are many options for doing this - and many existing python libraries to make our lives easier. For visualization only, my favorite tool (so far) was created by Seth Golub. It creates a heatmap based on density of points, and can overlay the 'heat' on top of maps coming from OpenStreetMap . This image is available under CC-BY-SA.

Depending on your needs, you can also overlay this information onto Google Earth maps or Gmaps. The nicest tool I found for this is hosted here. The interface is incredibly easy. Here is the output (it spits out kml data which google earth can read, and can also be accessed by google maps):

You might start wondering, "if I wanted to go on a 'pizza-slice' tour of new york city, and I wanted the longest sequence of pizzerias separated by .1 miles without visiting any place twice - what would I do?". Coincidentally, I wondered the exact same thing, and wrote a short function to search the graph for the longest such path. (For those who care, finding the longest path in a graph with cycles is known to be NP-hard. So don't try this on your big graphs.) Here is the result, you can visit 32 pizzerias (labeled, in order,it A-Z, then 0-5). Best of luck.

On Thursday, November 3rd, Hyperpublic's head of engineering, Doug Petkanics, will be teaching a Skillshare class called "APIs For Beginners." The class will start with the basics, and will cover the following:

The last few months have been busy at Hyperpublic and we’d love to tell you about two great local data APIs: Places+ and Geo Deals and Events.

Places+ returns nearby businesses and other points of interest (POI). Our places data set has freshest and most accurate information from the web. You can query the API from many popular languages and use it in your web and mobile applications.

Geo Deals and Events is a collection of real time local offers including daily deals, meet ups, exhibitions and other types of events that can be integrated into any app or web service. When your users interact with these offers, you get paid through our local offers monetization plan.

For this post, we'll focus on calling the API through simple URLs that you can try in your browser. We’ll walk through what the requests and results look like and how they're constructed.

The Places+ API returns points of interest around a specific location. Here are places around Hyperpublic HQ:

Click and you'll get a block of raw JSON. 

That's an array of JSON objects: each one represents a place in the Places+ database. Each place includes useful information like its name, lists of tags and properties, and its locations with street address and latitude/longitude.

In most cases, API calls are simply GET requests to (carefully constructed) URLs. Let’s review each piece of the example above:

• https://api.hyperpublic.com/api/v1 — This is the base URL for Hyperpublic API requests. Hyperpublic requires SSL for all requests.
• /places — Include this path after the base URL to access the Places+ endpoint. Other endpoints use different paths.
• ?...&...&... — These are query parameters. Following the ? is an &-separated list of key=value pairs. You use these to define your search and pass your authorization credentials:
  • address=416%20W%2013th%20St%2C%20NY%2C%20NY%2C%2010014 — The address parameter takes a URL-encoded address. For this example, we URL encoded "416 W 13th St., New York, NY, 10014".
  • client_id=..., client_secret=... — These are the Client ID and Client Secret values you're given when you register an application. They must be included in every API call. (The values used in this post are just for documentation and examples. Don't use them in your own applications!)

By default, all location queries return at most 10 results inside a 2 km radius. You can override the defaults by passing the relevant query parameters. To get more results, include limit=50, or to search within a 0.5 km radius, include radius=0.5. How about the closest Japanese restaurant to Madison Square Garden? Combine category=japanese with the lat, lon, and limit parameters, and you're there:

You can get a specific place by making requests to /places/ID, where ID is the value of the id field in that place’s JSON representation. Here's how to get that same sushi restaurant without searching:

Calling the Geo Deals and Events endpoint is as easy as substituting /offers for /places in the request URL. The basic search and show functionalities are the same as with Places+, but some of the query parameters are different. Let's find offers near Prospect Park, Brooklyn, that cost less than $50:

And we can request a specific offer the same way we requested a specific place:

This is a basic introduction to the Hyperpublic API; check out the API documentation for a full reference. The documentation pages will be updated with new features as we roll them out, so check back often! Our Geo Code blog will have more posts to show how to use language-specific libraries to easily integrate Hyperpublic into your applications.

Got questions? Post to our API Developers Google Group and we'll be glad to help you out. Happy hacking!

The engineering team at Hyperpublic has been hard at work over the last few weeks re-architecting our platform in order to migrate from a relational database (PostgreSQL) to a NoSQL datastore (MongoDB). We completed the migration about two weeks ago, and nothing major has broken down so far, so it's about time to do a recap so that the community can benefit from what we learned along the way. This post starts off anecdotal and moves to technical, and hopefully after reading it you'll have a good background on the reasons why you may want/need to migrate and how to go about performing the migration.