WHEREHOO
SOFTWARE

please contact the author for source code access

Server
Current Java source

Sample
Clients
(Java)
For either the query or insert client, download the shared classes too

Shared classes

Query

Insert

(PHP)
Query

Importer
(Java)
Command line tool to add records to Wherehoo, including binary data

Importer info and source code

 

WHEREHOO TECHNICAL OVERVIEW

Author: Jim Youll
Last update: 22 December 2001, 12:20 EST
This version of the documentation corresponds with version 2.12 of the Wherehoo server.

Want to write code to talk to the Wherehoo server? Skip down to the section labeled "WHEREHOO TCP INTERFACE"

STUFF AND BOTHER

What is in a Wherehoo record?

A Wherehoo record contains some digital data or a pointer to data, plus the coordinates (Latitude,Longitude,Height) of the location on Earth of the thing represented by the data, and a time window during which the data was at that place. For example, a Wherehoo record bound to the location of JP Licks ice cream shop on Newbury St. in Boston, MA may contain the URL of a review of the shop, or a JPEG image of the cafe. The coordinates are the latitude/longitude/height of the cafe as read from a GPS receiver. The time window of a record containing a review of the shop might run from the date the review was written to "infinity" (assuming the shop is still open) . The time window for a photograph of people taken at the cafe might just be one second long - the photograph is placed at the exact location and moment that it was made...


EXAMPLE RECORD:

AREA ((42.348025,288.912257))
BEGIN-TIME 1999-02-11 00:00:00-05
END-TIME 10000-01-01 00:00:00-05
DATA http://boston.citysearch.com/E/V/BOSMA/0005/52/12/

Small binary data blocks can be stored in the server so that simple devices do not have to make arrangements with a second server via a second protocol to host information about themselves. For example, a video return drop box may want to report that it contains tapes. Using Wherehoo, the box can maintain its own location and status within Wherehoo, rather than using additional protocols and servers to store state data.

By default, Wherehoo examines the "present" at the moment of the query rather than looking into the future or past. In a default search, only records that are active at the moment of the query are returned. Querys may include being/end time data, to tell the server what time window should be included in the search. All time data are sent to Wherehoo as relative offsets from "now," rather than absolute dates/times. Wherehoo is thus resilient with regard to differing time zones and unsynchronized clocks. Each device storing data in Wherehoo (and each Wherehoo server) need only be true to itself when dealing with begin and end times.

QUERY PARAMETERS AND PROCESS

Wherehoo draws a square around "location", with the centers of all edges at distance "search radius" from "location." This results in the inclusion of approximately 27% more area in the search (the area of a square of width=2R minus the area of a circle having diameter=2R). It is likely that a sufficiently-populated Wherehoo database will return some records bound to locations with a straight-line distance beyond the specified search radius. Given the intended use of the search results ("on the street") and the fact that straight-line distances to even relatively nearby locations do not precisely describe the distance one must travel to reach that place on roads and sidewalks, this should not pose a problem for agents that anticipate (or don't care about) this possible outcome. Presently all searches are conducted without regard to the height of an object. This behavior may change in future versions of the server.

RESPONSE TO QUERIES

Wherehoo answers queries by sending a series of record headers, one per line, describing data located in the search area. The records are not sorted. Each record is presented to the client via a header that describes the record, including: size of data field, whether or not there is metadata, the data's MIME type, protocol needed to retrieve the data, and the range, bearing and heading from the current location to the location represented by the record.

Once a client has the record header, it may "SKIP" the record, ask for the textual metadata describing the record (if there is any), ask for the DATA field contents, or say "BYE" to terminate the connection.

TCP INTERFACE

The Wherehoo server talks on port 5859 via TCP

wherehoo.org and wherehoo.media.mit.edu are Wherehoo servers in the world. There are others. In the future there may be more, but clients will only need to know about one Wherehoo server to find all the others.

The communications protocol is reminiscent of SMTP. Each command is issued on a single line, command first, parameters following, then a carriage return. A command is separated from its parameters by one or more blank spaces. Case does not matter.
If a command is sent more than once, the most recent values are used.

Sample command: LLH 42 -071 0

Unlike SMTP, there is no required sequence for commands to the server. All parameters sent by the client are analyzed together upon receipt of the client's EOD (EOD is signaled by a '.' on a line by itself). However all data needed by the server must be in place before the client sends its EOD.

Insert operations require prior authorization. Clients must IDT themselves if they are doing INSERTs. An authentication code is required. At present these are issued by the Wherehoo administrator. In the future they will be generated and issued automatically.

Queries do not require authentication. However, a client's IDT yields an IDT from the server, containing server operating parameters including timeouts and size limits for data fields that are useful to querying agents. All agents may benefit from knowing these limits before trying to interact with the server so programmers are encouraged to IDT themselves to the Wherehoo server, and to use the returned values in their code.

Not all commands are required for all operations. Many of the commands (particularly when querying) are optional, and default to useful values if not explicitly set by the client. When a command is sent, however, all its parameters are mandatory.

 

MESSAGES FROM THE CLIENT TO THE SERVER
Identification and authentication
IDT identity Identity - identifies client. Must match an IDT on file, for INSERT actions, or the insert will fail.
SHA signature SHA-1 message digest used as a signature over data
Query and record insertion
ACT QUERY Action: Run a query and return the matching records
ACT COUNT Action: Run a query and return the number of matching records
ACT INSERT Action: Insert a record into the database and return the uniqueid of the record.
ACT DELETE Action: Delete a record from the database and return either "ACK" for success or "NAK" for failure. Records may only be deleted if the IDT of the creator and the UID of the record are presented and match the values that were stored in the record when it was created.
LLH Lat Lon Ht [Lat Lon Ht...]

Location in Lat/Lon/Height (type double)
LLH values must be in the range -360.0 <= (L1,L2) <= 360.0

To describe a polygon, provide the Latitude and Longitude of the polygon's points. For example,suppose we have three point polygon ((lat1,lon1),(lat2,lon2),(lat3,lon3)), which is located at the height of h.

To enter it into wherehoo, the command is:
LLH lat1 lon1 h lat2 lon2 h lat3 lon3 h

The current system supports 2-dimensional spaces only, so the height provided with each point must be the same.

 

XYZ L1 L2 L3 Current location in X/Y/Z coordinates (type double) XYZ coordinates are no longer supported
Specifically for queries
BEG yy mo dd hh mi ss

Return records that were alive on or after this offset from NOW (NOW is calculated at the moment the query is run). If specifying BEG in a query, always use END as well or results may be confusing. Offsets may be positive or negative.

If BEG and END are not specified (they should be used only when really necessary) the records returned are those that are alive at the moment the query is run, that is, "now"

END yy mo dd hh mi ss

Return records that were alive on or before this offset from NOW (NOW is calculated at the moment the query is run). If specifying END in a query, always use BEG as well. Offsets may be positive or negative.

If BEG and END are not specified (they should be used only when really necessary) the records returned are those that are alive at the moment the query is run, that is, "now"

HDG heading

Heading for directionally-focused searches (0.0 <= heading <= 360.0, type double). Default is 0°=due North.

As of version 0.70, LIM is not compatible with a directional search. Do not use LIM and HDG together.

LEN length If not using RAD, search length in meters (0° and 180° at current heading, type integer)
LIM limit

Limit maximum number of records returned (type integer). limit must be >= 0.
If limit = 0 (the default) there is no limit and all records are found and returned.
This command is provided so that clients can help the server do less work, by limiting the number of records retrieved i the background SQL query.

As of version 0.70, LIM is not compatible with a directional search. Do not use LIM and HDG together.

MET metadata Return only records with the substring metadata anywhere in their META fields.
Matches are case-insensitive.
Use sparingly. Records with no metadata, and those with similar but non-exact matches, will not be returned when this parameter is included.
MET is quite useful when looking for a set of records tagged with similar metadata (e.g. you know something was inserted with specific metadata, and only want those records)
MIM mimetype

Return only records whose header indicates they are exactly this MIME type (full-string match is performed)
If the thing pointed to by this record is another Wherehoo server, the MIME type will be 'application/x-whserver'
If this parameter is sent, only exact matches to mimetype will be returned.
If this parameter is not sent with the query, records with all mime types are returned.

PJT heading range "Project" the user's location from the stated Latitude and Longitude (LLH command) to a new place at a position that is "range" distance from the current Lat,Lon at the given "heading". This is useful for things that need to pretend they are at a place "away from" the current location. The internal calculation is as precise as (and derived from) the other distance and heading calculations built into the server. (-360.0 < heading < 360.0)
PRO protocol

Return only records whose protocol is exactly protocol
If the bits in this record's DATA field are the thing pointed to (DATA is not the URI of some location, but IS the data for that place - a binary block, some words, some music, some code, whatever), then the protocol is 'WHEREHOO'

If this parameter is not sent with the query, records with all protocols are returned

The table of protocols is maintained manually.
Please bring missing or new protocols to the attention of Wherehoo project leads.
Permitted protocols are listed at http://wherehoo.media.mit.edu/about/supported-protocols.php

RAD radius Search radius, in meters (0°, 90°, 180° and 270° around current location at current heading, type integer). Sets WID = LEN = radius. Alternatively, use WID and LEN to specify a non-square search area.
SHP shapecode Shape of directional search {rect_ctr | rect_fwd}
WID width If not using RAD, search width in meters (90° and 270° at current heading, type integer)
META After receiving descriptive data about a Whereoo record, client wants the record's metadata
DATA After receiving descriptive data about a Whereoo record, client wants the actual data block
SKIP After receiving descriptive data about a Whereoo record, client does not want the data block. Skip to next record
Specifically for record insertion
BEG yy mo dd hh mi ss Offset from NOW for the beginning date/time of this record, defaults to 00/00/0000 if not specified
"NOW" is calculated at the moment the record is inserted or updated (when server receives client's EOD marker)
DAT data_size

Here comes the binary data.

The client should send:
DAT data_size
to tell the server the data's coming
, then a CRLF (like all other commands), then...
data_size bytes of data
followed immediately by
the 20-byte SHA-1 signature, created using the data plus the client's secret (arranged in advance)

Here is a class that holds the data block and provides method SHAhash(mysecret) to return the SHA-1 signature:

import java.security.*;
public class dataBlock {
public byte[] data;

public byte[] SHAhash(String mysecret) {
byte[] mdfinal;
try {
MessageDigest md = MessageDigest.getInstance("SHA-1");
md.update(data); // the data and...
md.update(mysecret.getBytes()); // my secret
mdfinal = md.digest(); // yield the secure hash
} catch (NoSuchAlgorithmException nsae) { return null; }
return mdfinal;
}
}

The total bytes your client sends must be exactly data_size + 20
1 <= Data_size <= max_data (max_data is stated by the server in response to the client's IDT)
If data_size is greater than max_data, then only the first max_data + 20 bytes will be read and the client will be disconnected as the server tries to interpret the additional data as commands.
So, don't do that.

Immediately after sending the 'DAT data_size' line, send your bytes of data plus the 20-byte signature, all as one contiguous binary block. Then go back to line-reading and watch for "ACK". The server will answer with an ACK after it's received your data_size bytes plus the 20-byte signature.
If there's something wrong with your data stream, or if data_size is < 1, the server will return a NAK and break its connection.

WHERE DO I GET AN IDENTITY AND SECRET SO I CAN ADD RECORDS?
For now, we give them to testers after manually adding the IDT and signature pair to the server's user database. Please write to request one. Eventually, they'll be generated and exchanged with clients automatically.

END yy mo dd hh mi ss Offset from NOW for the ending date/time of this record, defaults to 12/31/9999 if not specified
"NOW" is calculated at the moment the record is inserted or updated (when server receives client's EOD marker)
MET metadata

Text metadata describing the thing pointed to by this record.
The length of the metadata should be <= the max_meta value provided by the server in response to the client's IDT.
If the metadata block is longer than max_meta, it will be truncated to the first max_meta characters.

MIM mimetype

MIME type of the data pointed to by (or contained in) this record.
For maximum happiness, use well-known MIME types.
If the thing pointed to by this record is another Wherehoo server, the MIME type should be 'application/x-whserver'

NOTE: Beginning with version 0.630, 11-1-00, MIME types are no longer validated against a list. We still recommend using well-known MIME types (else nobody will be able to use your records)!

PRO protocol

The protocol an agent will use to retrieve the thing pointed to by this record.
For maximum happiness, use standard protocols. These are checked against a table built from ftp://ftp.iana.org/in-notes/iana/assignments/service-names. If the protocol is not found, Wherehoo will not insert the record.
The additional protocol 'WHEREHOO' is also accepted.
If the bits in this record's DATA field are the thing pointed to (DATA is not the URI of some location, but IS the data for that place - a binary block, some words, some music, some code, whatever), then the protocol is '
WHEREHOO'
The table of protocols is maintained manually. Please bring missing or new protocols to the attention of Wherehoo project leads.
Permitted protocols are listed at http://wherehoo.media.mit.edu/about/supported-protocols.php

Specifically for record deletion
UID uniqueid The 32-character uniqueid of the record that is to be deleted. This information was reported to the creating client at the time the record was created and cannot be retrieved otherwise. Clients that want to delete or update their records will need to store and manage the records' UIDs.
Connection control
DBG debug_mode Mode of debugging {on | off }. Default is OFF. When ON, state info is sent back toward you.
Don't use unless debugging the server or your communications stream will be full of unwanted goo
NOP

No-op to forestall rx timeout by server if your client is running slowly or to see if server is still listening
Server answers with ACK if it's still listening

BYE Close our connection now
. EOD. Client is finished sending data to server. Server should run the query now.

 

wherehoo_server version timeout min_expiration max_meta max_data

Server's response to an IDT from the client. All fields are variable length, space-delimited
example: wherehoo_server 0.480 30 20 1024 65535

wherehoo_server is a constant text string
version is the version number
timeout is the data timeout in seconds. Silence from client for this duration drops connection
min_expiration is the minimum number of seconds "later than the beginning time" that a record may end... This is to avoid thrashing the database with too-short-lived records
max_meta is the maximum length of the text metadata describing the contents of a Wherehoo record's data field

max_data is the maximum number of bytes of binary data that may be stored inside a Wherehoo record's data field

OK Search parameters accepted. Server is running query or insert operation.
NAK command_list These commands had invalid or out of bounds parameters. Query not running.
ACK Response to a NOP. Server is alive and listening.
. EOD (End of data). There are no more records to send you.
BYE Server is closing the connection
 

 

DATA FROM THE SERVER TO THE CLIENT

In response to a query from the client, the server sends back individual records.

1. A line of text containing descriptive data about the record's content is sent to the client
2. The client should check the descriptive data for compatibility ("can I actually handle this data type? protocol? data size?")
3. The client
issues one of these instructions on a line by itself:

DATA if it wants the data (binary; byte count was stated in bytes field of record header - see below)
META if it wants the text metadata describing the contents of DATA (one line, size limit of max_meta)
SKIP to move to the next record
BYE to close the connection.

4. If the client sent "DATA" or "META" the requested stuff is sent. If the client asks for META and there isn't any (should have read the record header!) an empty line will be sent.
5. If client sent META, the server loops to #3, waiting for another command regarding this record (because the client might want DATA next).
... otherwise the server returns to #1 and sends descriptive data for the next record, or EOD (".") if there are no more records.

Record descriptive data appears on one line, space-delimited, in this format:
bearing compassdirection distance ttl bytes protocol mimetype meta

bearing Bearing from your location to this object, in degrees, at your current HEADING
compassdirection Compass direction from your location to this object {N | NE | E | SE | S | SW | W | NW }
distance Straight-line distance from your location to this object, in meters
ttl Time-to-live of this record in the database, in seconds
TTL longer than 99999999 seconds, just over 3 years, come back as 99999999.
bytes Number of bytes of data in the object's data block (to be sent if client replies DATA)
protocol If object's data block is a URI, the protocol to be used for fetching the data
If object's data block is stored in the Wherehoo server, the protocol is "WHEREHOO"
mimetype The MIME data type of the data represented by the object's data block, whether served directly from WHEREHOO or some other server (protocol tells you where it's coming from)
meta "META" if textual metadata exists for this record, "NONE" if no metadata exists
Client replies META to tell server to send this text.

 

SAMPLE CLIENT-SERVER INTERACTION - A Query
From the client to the server From the server to the client
IDT jim  
  wherehoo_server 2.1 30 20 1024 65535
ACT QUERY  
LLH 72.019 288.908 0  
RAD 250  
SHP rect_fwd  
HDG 0  
.  
  OK
  96 N 1443 99999999 33212 WHEREHOO IMAGE/JPG META
DATA  
  (33212 bytes of data)
NEXT  
  138 NW 230 99999999 74 HTTP TEXT/HTML NONE
DATA  
  (74 bytes of data)
  .
  BYE

 

SAMPLE CLIENT-SERVER INTERACTION - An insert
From the client to the server From the server to the client
IDT jim  
  wherehoo_server 0.56 30 20 1024 65535
ACT INSERT  
LLH 72.019 288.908 0  
MIM TEXT/PLAIN  
PRO WHEREHOO  
DAT 12  
(12 bytes of data)  
  ACK
MET This is some metadata describing the content of the DAT field  
.  
  OK
  c08b9ac6d59dba9be0106096626809b2e4098f0a
  .
  BYE

 

SAMPLE CLIENT-SERVER INTERACTION - A delete
From the client to the server From the server to the client
IDT jim  
  wherehoo_server 0.56 30 20 1024 65535
ACT DELETE  
UID c08b9ac6d59dba9be0106096626809b2e4098f0a  
.  
  OK
  ACK
  .
  BYE

 

Hard limits
Maximum RAD, WID, or LEN MAXWIDLEN = 999999
Maximum length of identity for IDT MAXIDT = 10
Unique id string returned after a successful INSERT action 40 hex characters(0..F) Fixed length
   

 

Things to think about
Self-determination of coverage area How does a network of Wherehoo servers self-configure?
Query routing and server acquisition of geodata  

 

Revision History
   
2.02 12/12/01 More forgiving - receipt of invalid commands no longer triggers "BYE" and disconnect from server. Invalid commands are silently ignored.
Also, bug in RAD command corrected - RAD was not setting the LEN and WID parameters as it should have.
2.00 12/10/01 Databases transitioned to Postgresql 7.1, transition from (X,Y) single-point data to polygonal areas (using the geometric search abilities of Postgresql). Significant for client code: Server no longer transparently sends the next record after a DATA block has been sent. Client must request the next record explicitly. Officially, the command "NEXT<cr>" should be sent. However, at present, any <cr>-terminated message from the client that does not match an existing Wherehoo command will cause the next record to be sent. This will be changed in the future, so use NEXT<cr> to get the next record!
0.630 11/1/00

Removed checking of MIME type presented with the MIM command
If MIM or PRO are set (or both) when performing a query, the server will return only thos records that exactly match the provided MIME type and/or protocol.
Example:

ACT QUERY
LLH 42.361 -071.088 0
RAD 10000
MIM text/html
PRO http
.

This query only returns records that point to web pages accessible via the HTTP protocol

0.623 Catch datalen > MAXDATA and return DAT error
0.621 10/29/00 Revised diagnostic byte printouts, solving the 1429-byte-problem
revisions to DAT read section, problem using .read(,,) switched to .ready() plus .read() with a wait loop when anticipating more data
0.610 10/8/00 Added some diagnostics. nothing major.
0.602 10/6/00 Add PJT command to project self to some other place (for periscope, other lookahead apps)
0.590 10/1/00 Allow BEG, END offsets to expand search window beyond records that are alive "now"
0.580 9/3/00 Allow META for queries, match meta to any substring in the meta field
0.570 9/17/00 Add SHA1 signature as 20 bytes appeneded to DAT bytes. Checks signature by generating SHA1 sig against asserted IDT's secret