HowTo Geolocate NST API Reference

From MediaWiki
Jump to navigationJump to search

NST Geolocation API Overview

While the NST system provides many different and specialized geolocation renderings, it also provides a generic set of APIs to allow you to plot IPv4 addresses on a map. In general, this is accomplished as follows:

  • You set up your NST system such that is enabled to geolocate IPv4 addresses.
  • You produce a file (or stream) containing a list of IPv4 addresses.
  • You feed your list of IPv4 addresses into one of the NST rendering engines.
  • You get a rendering of your geolocated IPv4 addresses.

NST Geolocation APIs

ASCII Output API (nstipgeolocate)

The nstipgeolocate command can take a list of IPv4 addresses as input and produce a ASCII output file containing geolocated results in various formats.

The following shows the help output when nstipgeolocate is invoked with the --help option: 

[root@dhcp143 ~]# nstipgeolocate --help
Usage: nstipgeolocate [options]

This program reads a list of IPv4 addresses from stdin or the command line and
attempts to look up the associated latitude and longitude coordinate values to
geolocate the IPv4 addresses passed.  The results are then echoed to standard
out in the format specified on the command line indicating the locations (if
possible) for each IPv4 address processed.  An associated resource file
(Custom: "$HOME/.etc/nst/nstipgeolocate.py" or Global:
"/etc/nst/nstipgeolocate.py") can be configured to provide latitude and
longitude coordinate adjustments, private IPv4 Address/Network coordinate
locations and the selection of the Geolocation database source. Alternatively,
on can set the python environment variable: 'PYTHONPATH' to a directory
containing a "nstipgeolocate.py" resource file.  This program exits with the
following return codes (useful if calling from a script): 0 - No issues
encountered. 1 - A unknown issue, typically a Python error in the
configuration file. 2 - The GeoIP implementation method was configured, but
the GeoIP database did not exist. To test, check the exit status of:
(nstipgeolocate </dev/null &>/dev/null).

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -i 'IP0[ IP1[...]]', --ip-list='IP0[ IP1[...]]'
                        Specify a list of IPv4 space separated addresses on
                        command line instead of reading stdin
  -f plain|csv|xml|xmlext|wui|json, --format=plain|csv|xml|xmlext|wui|json
                        The output format to generate: plain, csv (Comma
                        Separated Values), xml (short XML format), xmlext
                        (extended XML format), json (JavaScript Object
                        Notation), or wui (for NST WUI page) [default: plain]
  -n, --disallow-host-names
                        Only IPv4 addresses are processed, ignore host names
                        or Fully Qualified Domain Names (FQDN)
  -r, --resolve-host-names
                        Try to resolve Fully Qualified Domain Names (FQDN) for
                        each IPv4 address
[root@dhcp143 ~]#

While there are more than three different output formats available, we will demonstrate the csv, xmlext and json outputs as they are well suited for further processing by your own custom code.

Example IP File

The following IPv4 address will be saved to the file /tmp/ip-list.txt and used as the input to the example invocations of the nstipgeolocate command. 

108.107.15.248
118.90.104.224
12.12.28.31

Comma Separated Values (CSV) Output

The following demonstrates how to produce a ASCII table of geolocated information for the IPv4 addresses in our file. The ASCII output will be in a comma separated value (CSV) format suitable for importing into a spreadsheet or processing with your own custom code: 

[pkb@rice ~]$ nstipgeolocate -r -f csv < /tmp/ip-list.txt 
HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone
108-107-15-248.pools.spcsdns.net,108.107.15.248,33.9393997192,-84.2077026367,Norcross,United States,US,USA,Georgia,GA,Norcross,,404,524,America/New_York
ip-118-90-104-224.xdsl.xnet.co.nz,118.90.104.224,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland
12.12.28.31,12.12.28.31,61.1692008972,-149.844299316,Anchorage,United States,US,USA,Alaska,AK,Anchorage,,907,743,America/Anchorage
[pkb@rice ~]$

XML Output

The following demonstrates how to produce a ASCII XML file of geolocated information for a set of IPv4 addresses. This XML file is suitable for processing in a variety of methods and is what we use when building KML and KMZ output for many of our custom Google Earth renderings. 

[pkb@rice ~]$ nstipgeolocate -r -f xmlext < /tmp/ip-list.txt 
<?xml version="1.0" encoding="UTF-8"?>
<!-- NST Tools Generated: Wed May 04 07:14:30 EDT 2011
'nstipgeolocate' Host Geolocate Document -->
<geoip>
  <properties>
    <comment>nstipgeolocate Host Geolocate Document</comment>
    <entry key="document_source">NST tools generated: nstipgeolocate</entry>
    <entry key="document_version">2130.000</entry>
    <entry key="create_time_formatted">2011-05-04 07:14:30.447865000-0400</entry>
    <entry key="create_time_secs">1304507670.447865</entry>
  </properties>
  <location>
    <hostname>108-107-15-248.pools.spcsdns.net</hostname>
    <addr>108.107.15.248</addr>
    <lat>33.939400</lat>
    <lon>-84.207703</lon>
    <properties>
      <entry key="city">Norcross</entry>
      <entry key="region_name">Georgia</entry>
      <entry key="alias">Norcross</entry>
      <entry key="region">GA</entry>
      <entry key="area_code">404</entry>
      <entry key="time_zone">America/New_York</entry>
      <entry key="metro_code">524</entry>
      <entry key="country_code3">USA</entry>
      <entry key="dma_code">524</entry>
      <entry key="country_code">US</entry>
      <entry key="country_name">United States</entry>
      <entry key="quality">6</entry>
    </properties>
  </location>
  <location>
    <hostname>ip-118-90-104-224.xdsl.xnet.co.nz</hostname>
    <addr>118.90.104.224</addr>
    <lat>-36.866699</lat>
    <lon>174.766693</lon>
    <properties>
      <entry key="city">Auckland</entry>
      <entry key="region_name">Auckland</entry>
      <entry key="alias">Auckland</entry>
      <entry key="region">E7</entry>
      <entry key="time_zone">Pacific/Auckland</entry>
      <entry key="country_code3">NZL</entry>
      <entry key="country_code">NZ</entry>
      <entry key="country_name">New Zealand</entry>
      <entry key="quality">6</entry>
    </properties>
  </location>
  <location>
    <hostname>12.12.28.31</hostname>
    <addr>12.12.28.31</addr>
    <lat>61.169201</lat>
    <lon>-149.844299</lon>
    <properties>
      <entry key="city">Anchorage</entry>
      <entry key="region_name">Alaska</entry>
      <entry key="alias">Anchorage</entry>
      <entry key="region">AK</entry>
      <entry key="area_code">907</entry>
      <entry key="time_zone">America/Anchorage</entry>
      <entry key="metro_code">743</entry>
      <entry key="country_code3">USA</entry>
      <entry key="dma_code">743</entry>
      <entry key="country_code">US</entry>
      <entry key="country_name">United States</entry>
      <entry key="quality">6</entry>
    </properties>
  </location>
</geoip>
[pkb@rice ~]$

JavaScript Object Notation Output

The following demonstrates how to produce a ASCII file in JavaScript Object Notation (JSON) format for a set of IPv4 addresses. This JSON file is suitable for processing in a variety of scripting languages or can be used directly by JavaScript code running within a web browser. 

[pkb@rice ~]$ nstipgeolocate -r -f json < /tmp/ip-list.txt 
[
  ["108.107.15.248", "108-107-15-248.pools.spcsdns.net", [33.939399719238281, -84.20770263671875, {
    "city": "Norcross", "region_name": "Georgia", "alias": "Norcross", "region": "GA", "area_code": 404,
    "time_zone": "America/New_York", "metro_code": 524, "country_code3": "USA", "postal_code": null,
    "dma_code": 524, "country_code": "US", "country_name": "United States", "quality": "6"
  }]],
  ["118.90.104.224", "ip-118-90-104-224.xdsl.xnet.co.nz", [-36.86669921875, 174.76669311523438, {
    "city": "Auckland", "region_name": "Auckland", "alias": "Auckland", "region": "E7", "area_code": 0,
    "time_zone": "Pacific/Auckland", "metro_code": 0, "country_code3": "NZL", "postal_code": null,
     "dma_code": 0, "country_code": "NZ", "country_name": "New Zealand", "quality": "6"
  }]], 
  ["12.12.28.31", "12.12.28.31", [61.169200897216797, -149.84429931640625, {
    "city": "Anchorage", "region_name": "Alaska", "alias": "Anchorage", "region": "AK", "area_code": 907,
    "time_zone": "America/Anchorage", "metro_code": 743, "country_code3": "USA", "postal_code": null,
    "dma_code": 743, "country_code": "US", "country_name": "United States", "quality": "6"
  }]]
]
[pkb@rice ~]$

NOTE: New lines and white space characters were manually added to the above output to make it slightly easier to read.

Unfortunately, this format is not human readable (especially if you are not familiar with JSON). However, if you are familiar with JSON files, you will find it quite useful. We use this output format frequently via AJAX calls from our NST WUI when presenting information on a Google Map rendering within a web browser.

Name Resolution Warning

In the above examples, we included the -r option which instructs the nstipgeolocate to try and resolve host names. Since there were only 3 IPv4 addresses to process this was not a big deal. However, if you have a lot of IPv4 addresses to process you may want to omit this option as it adds a significant amount of time to perform the lookups.

The following shows that the time to process 564 IPv4 processes jumped from 1.073 seconds to 92.536 seconds once we enabled the resolve host names option: 

[pkb@rice ~]$ wc /tmp/ip-list-full.txt 
 564  564 7923 /tmp/ip-list-full.txt
[pkb@rice ~]$ time nstipgeolocate -f csv < /tmp/ip-list-full.txt >/dev/null

real	0m1.073s
user	0m0.230s
sys	0m0.086s
[pkb@rice ~]$ time nstipgeolocate -r -f csv < /tmp/ip-list-full.txt >/dev/null

real	1m32.536s
user	0m0.456s
sys	0m0.153s
[pkb@rice ~]$


ASCII Output As A Service

While the nstipgeolocate command is quite useful if you are on the NST system itself, it can't be run if you are on a different system. You can however, use the NST WUI web service page to run the equivalent nstipgeolocate commands remotely. The following outlines how to do this.

The Base URL

The following URL is used to request that a set of IPv4 address be geolocated (change 192.168.1.143 to the address of your NST system): 

https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php

In order to use the above URL, you must provide a set of parameters. These parameters can either be appended directly on the end of the URL (a GET request), or submitted separately via a POST request. Examples of both methods will be described later. The GET approach is simpler, but the POST method is typically better suited when you need to geolocate a lot of IPv4 addresses.

Here are the parameters which can be included:

op=nstipgeolocate
This parameter is required and tells the service that you want to geolocate a set of IPv4 addresses.
iplist=IP0[+IP1[+IP2...]]
This parameter is required and allows you to tell the service what IPv4 address you would like to have geolocated. Each IPv4 address in the list is separated by a + symbol (which corresponds to the escaped space character).
format=csv|xmlext|json|...
This parameter is required and allows you to tell the service what type of output you would like back. If omitted, it defaults to xmlext (extended XML output). The common output formats are csv, xmlext, and json. For a full list of available output formats, look at the help output of the nstipgeolocate command.
resolve=true|false
This parameter can be used to turn on (or off) name resolution. Name resolution tends to be very slow when processing large sets of data. For example, it may only take a couple of seconds to process a thousand IPv4 addresses without name resolution and several minutes if name resolution is enabled. It is recommended you only set this option to true when processing a small set of IPv4 addresses.

A wget Example (using GET)

The following demonstrates using the NST web service to geolocate two IPv4 addresses using a simple wget invocation: 

[pkb@rice ~]$ URL='https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php?op=nstipgeolocate&resolve=false&format=csv&iplist=108.102.15.248+118.90.104.224';
[pkb@rice ~]$ wget --quiet --no-check-certificate -O /tmp/ip-list.csv --user root --ask-password "${URL}"
Password for user “root”: 
[pkb@rice ~]$ wc /tmp/ip-list.csv  3   5 399 /tmp/ip-list.csv
[pkb@rice ~]$ cat /tmp/ip-list.csv
HostName,IpAddress,Latitude,Longitude,Alias,CountryName,CountryCode,CountryCode3,RegionName,Region,City,PostalCode,AreaCode,DmaCode,TimeZone
108.102.15.248,108.102.15.248,32.8097991943,-96.7993011475,Dallas,United States,US,USA,Texas,TX,Dallas,,214,623,America/Chicago
118.90.104.224,118.90.104.224,-36.8666992188,174.766693115,Auckland,New Zealand,NZ,NZL,Auckland,E7,Auckland,,0,0,Pacific/Auckland
[pkb@rice ~]$

A wget Example (using POST)

The following takes a list of 564 IPv4 addresses in the file /tmp/ip-list.txt found above and builds a files containing the appropriate parmaters to use the POST method. The wget command is then used to POST this information to the web service and store the results as the XML file /tmp/ip-list.xml. 

[pkb@rice ~]$ wc /tmp/ip-list.txt 
 564  564 7923 /tmp/ip-list.txt
[pkb@rice ~]$ head -3 /tmp/ip-list.txt 
108.107.15.248
109.154.109.210
109.230.220.96
[pkb@rice ~]$ rm -f /tmp/ip-list.post
[pkb@rice ~]$ SEP="op=nstipgeolocate&format=xmlext&iplist=";
[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do echo -n "${SEP}${ip}" >> /tmp/ip-list.post; SEP="+"; done
[pkb@rice ~]$ URL='https://192.168.1.143/nstwui/apps/geo/ipgeolocate-ajax.php';
[pkb@rice ~]$ wget --no-check-certificate -O /tmp/ip-list.xml --user root --ask-password --post-file /tmp/ip-list.post --quiet "${URL}" 
Password for user “root”: 
[pkb@rice ~]$ wc /tmp/ip-list.xml
 10748  17690 377318 /tmp/ip-list.xml
[pkb@rice ~]$ tail -21 /tmp/ip-list.xml 
  <location>
    <hostname>99.59.87.159</hostname>
    <addr>99.59.87.159</addr>
    <lat>41.867500</lat>
    <lon>-87.674400</lon>
    <properties>
      <entry key="city">Chicago</entry>
      <entry key="region_name">Illinois</entry>
      <entry key="alias">Chicago</entry>
      <entry key="region">IL</entry>
      <entry key="area_code">312</entry>
      <entry key="time_zone">America/Chicago</entry>
      <entry key="metro_code">602</entry>
      <entry key="country_code3">USA</entry>
      <entry key="dma_code">602</entry>
      <entry key="country_code">US</entry>
      <entry key="country_name">United States</entry>
      <entry key="quality">6</entry>
    </properties>
  </location>
</geoip>
[pkb@rice ~]$

URL (Google Maps)

Strengths/Weaknesses

  • Extremely simple to implement.
  • Highly interactive.
  • Requires web browser and access to a NST system configured for geolocation.
  • Does not scale well. Both the length of a URL string and the number of locations you can plot on Google Maps can be a limiting factor (not a good solution if you have thousands of IPv4 addresses to plot).
  • Optional host name resolution is fairly quick.

URL API Reference

This is a extremely simple API, you start with a base URL of (change the 192.168.1.143 to the IP address of your NST system): 

https://192.168.1.143/nstwui/apps/geo/map.html

The above URL will simply pull up a empty Google Map. However, the web page which renders the map also accepts the following parameters (all are optional):

NstMapIpList=IP0[+IP1[+IP2...]]
This parameter allows you to tell the page what IPv4 address you would like to have geolocated. Each IPv4 address in the list is separated by a + symbol (which corresponds to the escaped space character).
NstMapResolve=true|false
This parameter can be used to turn on (or off) name resolution. If name resolution is enabled, the names will be shown when you click on the markers on the map and pull up the information balloon.
NstMapFullScreen=true|false
If this parameter is set to true, then the map will be opened in Full Screen mode (you will only see the map in the browser window - there will be no scroll bar).
NstMapFitScreen=true|false
If this parameter set to true, then the map will be opened in Fit Screen mode (you will the map and the control area in the browser window - there will be a scroll bar).
NstMapMarkerDefault=NUMBER
This parameter controls which marker symbol is used for the IPv4 addresses that are plotted on the map. It is essentially a index (starting from 0) corresponding to the available markers shown under the Options panel.
NstMapMarkerMoved=NUMBER
This parameter controls which marker symbol is used for the IPv4 addresses that have been moved (dragged by the user) the map. It is essentially a index (starting from 0) corresponding to the available markers shown under the Options panel.
NstMapShowShadows=true|false
This controls whether or not marker shadows are shown.
NstMapAutoCenter=true|false
This parameter controls whether or not the map should try to center itself on one of the markers. This is typically only useful when you know you are plotting exactly one marker.
NstMapTypeId=hybrid|roadmap|terrain|satellite|sparse
This parameter can be used to force the initial type of map shown to the end user.
NstMapZoom=NUMBER
This parameter can be used to force the initial zoom level in the range of 0 to 15 where 0 is zoomed all the way out and 15 is zoomed all the way in.
NstMapCenterLat=LATITUDE
This parameter can be used to force the initial latitude (in the range of -90.0 to 90.0) for the location that the map will center on.
NstMapCenterLon=LONGITUDE
This parameter can be used to force the initial longitude (in the range of -180.0 to 180.0) for the location that the map will center on.

NOTE: You will typically omit all but the NstMapIpList parameter and let everything else default to the user's preferences.

Example URL

The following builds a URL string which uses the NST system 192.168.1.143 to render three IPv4 addresses (108.107.15.248, 109.154.109.210, and 98.251.209.196) on a web page (we also included the NstMapResolve parameter to force name resolution lookup for the IPv4 addresses passed): 

https://192.168.1.143/nstwui/apps/geo/map.html?NstMapResolve=true&NstMapIpList=108.107.15.248+109.154.109.210+98.251.209.196

If your NST system is set up for geolocation, you should be able to:

  • Copy/paste the above URL into your browser.
  • Change the 192.168.1.143 to the IP address of your NST system.
  • Press the enter key.
  • See the three IPv4 addresses plotted on a Google Map.

You should be able change the three IPv4 addresses assigned to the NstMapIpList parameter in the URL shown above to your own custom set of IPv4 addresses.

Example Commands

Typing in IPv4 addresses by hand gets old quick. So let's take a look at a real world example where we:

  • Extract a list of IPv4 addresses from a log file.
  • Use a few commands to build our URL.
  • Open the URL in the Google Chrome browser.

For this example, we will extract some IPv4 addresses from a Apache access file. Here is a snippet of the log file we will be extracting the IPv4 addresses from: 

66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nst/images/nstwiki_110x32.gif HTTP/1.1" 200 2443 "http://www.networksecuritytool...
66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nst/images/nstpro_110x32.gif HTTP/1.1" 200 2375 "http://www.networksecuritytool...
66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nstpro/images/npc_conv_ge_thumb.png HTTP/1.1" 200 58331 "http://www.networksecuritytool...

This is a easy log file to extract the IPv4 addresses from. We'll simply combine the use of the awk command (to print the first word of each line) with the use of the sort command to filter out duplicates: 

[pkb@rice ~]$ gzip -dc < /tmp/access_log.gz | awk -- '{ print $1; }' | sort -u >| /tmp/ip-list.txt
[pkb@rice ~]$ wc /tmp/ip-list.txt
 564  564 7923 /tmp/ip-list.txt
[pkb@rice ~]$

The following takes a list of 564 IPv4 addresses found above and builds a very long URL. It then opens the URL using Google Chrome: 

[pkb@rice ~]$ MAP_URL="https://192.168.1.143/nstwui/apps/geo/map.html?NstMapResolve=true&NstMapIpList=";
[pkb@rice ~]$ SEP="${MAP_URL}"; URL="";
[pkb@rice ~]$ for ip in $(cat /tmp/ip-list.txt); do URL="${URL}${SEP}${ip}"; SEP="+"; done
[pkb@rice ~]$ google-chrome "${URL}"
Created new window in existing browser session.
[pkb@rice ~]$

Here is what shows up in the browser from running the commands shown above:

Geolocating IPv4 Using A URL

KML Output (Google Earth)

Producing KML output involves the following steps:

  • Get a list of IPv4 addresses you would like to geolocate.
  • Use the nstipgeolocate command or ipgeolocate-ajax.php web service to convert to XML (the xmlext format as detailed in the "ASCII Output API (nstipgeolocate)" and "ASCII Output As A Service" sections above).
  • Apply a XSL style sheet to convert the XML file into a KML file.
  • Open the KML file in Google Earth.

Creating a XSL style sheet to convert the XML file produced by nstgeolocate to KML is a non-trivial task. However, a working template is available at: http://nst.svn.sourceforge.net/viewvc/nst/trunk/include/xsl/geoip2kml.xsl.

Getting a list of IPv4 Addresses

How you do this is up to you. There are many locations where you can get IPv4 addresses. For this example, we'll grab them from a Apache access_log file (where Apache puts the IPv4 address as the first word of each log entry): 

[pkb@rice ~]$ gzip -dc /tmp/access_log.gz | head -1
66.92.170.40 - - [03/May/2011:09:16:53 -0600] "GET /nst/images/sf-icon2.gif HTTP/1.1" 200 2934 "http://www.networksecuritytool...
[pkb@rice ~]$ gzip -dc /tmp/access_log.gz | awk -- '{ print $1; }' | sort -u >| /tmp/ip-list.txt
[pkb@rice ~]$ head -3 /tmp/ip-list.txt 
108.107.15.248
109.154.109.210
109.230.220.96
[pkb@rice ~]$ wc /tmp/ip-list.txt 
 564  564 7923 /tmp/ip-list.txt
[pkb@rice ~]$

We now have a file with 564 unique IPv4 addresses in it.

Convert To XML

We'll now use the nstipgeolocate command to convert the IPv4 address list into a XML file with the geolocated values associated with each IPv4 address: 

[pkb@rice ~]$ nstipgeolocate --format xmlext < /tmp/ip-list.txt >| /tmp/ip-list.xml
[pkb@rice ~]$

Convert To KML

We'll now use the xsltproc and the template XSL file from the NST Subversion repository to convert the XML file into a KML file for Google Earth: 

[pkb@rice ~]$ wget -O /tmp/geoip2kml.xsl 'http://nst.svn.sourceforge.net/viewvc/nst/trunk/include/xsl/geoip2kml.xsl'
--2011-05-04 17:27:34--  http://nst.svn.sourceforge.net/viewvc/nst/trunk/include/xsl/geoip2kml.xsl
Connecting to 192.168.1.2:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/xml]
Saving to: “/tmp/geoip2kml.xsl”

    [ <=>                                         ] 2,868       --.-K/s   in 0.002s  

2011-05-04 17:27:34 (1.57 MB/s) - “/tmp/geoip2kml.xsl” saved [2868]

[pkb@rice ~]$ xsltproc /tmp/geoip2kml.xsl /tmp/ip-list.xml >| /tmp/ip-list.kml
[pkb@rice ~]$

NOTE: You will likely want to customize the /tmp/geoip2kml.xsl file. It's just meant to serve as a starting point.

Open In Google Earth

You should be able to open the KML file produced using a tool like Google Earth: 

[pkb@rice ~]$ google-earth /tmp/ip-list.kml
[pkb@rice ~]$

Here is what the KML rendering resembles:

Geolocating IPv4 Using KML API

Mercator (Bitmap Image)

***Note: Section Under Construction***
Retrieved from "https://wiki.networksecuritytoolkit.org/nstwiki/index.php?title=HowTo_Geolocate_NST_API_Reference&oldid=4078"