|
WHEREHOO
SOFTWARE
please
contact the author for source code access
Server
Current Java source
Sample
Clients
(Java)
For either the query or insert client, download the
shared classes too
Shared
classes
Query
Insert
(PHP)
Query
Importer
(Java)
Command
line tool to add records to Wherehoo, including binary data
Importer
info and source code
|
WHEREHOO
TECHNICAL OVERVIEW
Author: Jim Youll
Last update: 22 December 2001, 12:20 EST
This version of the
documentation corresponds with version 2.12
of the Wherehoo server.
Want to write code to talk
to the Wherehoo server? Skip down to the section labeled "WHEREHOO
TCP INTERFACE"
STUFF
AND BOTHER
What
is in a Wherehoo record?
A Wherehoo record
contains some digital data or a pointer to data, plus the coordinates
(Latitude,Longitude,Height) of the location on Earth of the thing represented
by the data, and a time window during which the data was at that place.
For example, a Wherehoo record bound to the location of JP Licks ice
cream shop on Newbury St. in Boston, MA may contain the URL of a review
of the shop, or a JPEG image of the cafe. The coordinates are the latitude/longitude/height
of the cafe as read from a GPS receiver. The time window of a record
containing a review of the shop might run from the date the review was
written to "infinity" (assuming the shop is still open) .
The time window for a photograph of people taken at the cafe might just
be one second long - the photograph is placed at the exact location
and moment that it was made...
EXAMPLE RECORD:
Small binary
data blocks can be stored in the server so that simple devices do not
have to make arrangements with a second server via a second protocol
to host information about themselves. For example, a video return drop
box may want to report that it contains tapes. Using Wherehoo, the box
can maintain its own location and status within Wherehoo, rather than
using additional protocols and servers to store state data.
By default,
Wherehoo examines the "present" at the moment of the query
rather than looking into the future or past. In a default search, only
records that are active at the moment of the query are returned. Querys
may include being/end time data, to tell the server what time window
should be included in the search. All time data are sent to Wherehoo
as relative offsets from "now," rather than absolute dates/times.
Wherehoo is thus resilient with regard to differing time zones and unsynchronized
clocks. Each device storing data in Wherehoo (and each Wherehoo server)
need only be true to itself when dealing with begin and end times.
QUERY
PARAMETERS AND PROCESS
Wherehoo
draws a square around "location", with the centers of all
edges at distance "search radius" from "location."
This results in the inclusion of approximately 27% more area in the
search (the area of a square of width=2R minus the area of a circle
having diameter=2R). It is likely that a sufficiently-populated Wherehoo
database will return some records bound to locations with a straight-line
distance beyond the specified search radius. Given the intended use
of the search results ("on the street") and the fact that
straight-line distances to even relatively nearby locations do not precisely
describe the distance one must travel to reach that place on roads and
sidewalks, this should not pose a problem for agents that anticipate
(or don't care about) this possible outcome. Presently
all searches are conducted without regard to the height of an object.
This behavior may change in future versions of the server.
RESPONSE
TO QUERIES
Wherehoo answers
queries by sending a series of record headers, one per line, describing
data located in the search area. The records are not sorted. Each record
is presented to the client via a header that describes the record, including:
size of data field, whether or not there is metadata, the data's MIME
type, protocol needed to retrieve the data, and the range, bearing and
heading from the current location to the location represented by the
record.
Once a client
has the record header, it may "SKIP" the record, ask for the
textual metadata describing the record (if there is any), ask for the
DATA field contents, or say "BYE" to terminate the connection.
TCP
INTERFACE
The Wherehoo
server talks on port 5859 via TCP
wherehoo.org
and wherehoo.media.mit.edu are Wherehoo servers in the world.
There are others. In the
future there may be more, but clients will only need to know about one
Wherehoo server to find all the others.
The communications
protocol is reminiscent of SMTP. Each command is issued on a single
line, command first, parameters following, then a carriage return. A
command is separated from its parameters by one or more blank spaces.
Case does not matter.
If a command is sent more than once, the most recent values are used.
Sample command:
LLH 42 -071 0
Unlike SMTP,
there is no required sequence for commands to the server. All parameters
sent by the client are analyzed together upon receipt of the client's
EOD (EOD is signaled by a '.' on a line by itself). However all data
needed by the server must be in place before the client sends its EOD.
Insert operations
require prior authorization. Clients must IDT themselves if they are
doing INSERTs. An authentication code is required. At present these
are issued by the Wherehoo administrator. In the future they will be
generated and issued automatically.
Queries do not
require authentication. However, a client's IDT yields an IDT from the
server, containing server operating parameters including timeouts and
size limits for data fields that are useful to querying agents. All
agents may benefit from knowing these limits before trying to interact
with the server so programmers are encouraged to IDT themselves to the
Wherehoo server, and to use the returned values in their code.
Not all commands
are required for all operations. Many of the commands (particularly
when querying) are optional, and default to useful values if not explicitly
set by the client. When a command is sent, however, all its parameters
are mandatory.
|
MESSAGES
FROM THE CLIENT TO THE SERVER
|
| Identification
and authentication |
| IDT
identity |
Identity
- identifies client. Must match an IDT on file, for INSERT actions,
or the insert will fail. |
| SHA
signature |
SHA-1
message digest used as a signature over data |
| Query
and record insertion |
| ACT
QUERY |
Action:
Run a query and return the matching records |
| ACT
COUNT |
Action:
Run a query and return the number of matching records |
| ACT
INSERT |
Action:
Insert a record into the database and return the uniqueid
of the record. |
| ACT
DELETE |
Action:
Delete a record from the database and return either "ACK"
for success or "NAK" for failure. Records may only be
deleted if the IDT of the creator and the UID of the record are
presented and match the values that were stored in the record when
it was created. |
| LLH
Lat Lon Ht [Lat Lon Ht...] |
Location
in Lat/Lon/Height (type double)
LLH values must be in the range -360.0 <= (L1,L2) <= 360.0
To describe a polygon,
provide the Latitude and Longitude of the polygon's points. For
example,suppose we have three point polygon ((lat1,lon1),(lat2,lon2),(lat3,lon3)),
which is located at the height of h.
To enter it into wherehoo,
the command is:
LLH lat1 lon1 h lat2 lon2 h lat3 lon3 h
The current system
supports 2-dimensional spaces only, so the height provided with
each point must be the same.
|
XYZ
L1 L2 L3 |
Current
location in X/Y/Z coordinates (type double) XYZ coordinates
are no longer supported |
| Specifically
for queries |
BEG
yy mo dd hh mi ss
|
Return records that
were alive on or after this offset from NOW (NOW is calculated
at the moment the query is run). If specifying BEG in a query,
always use END as well or results may be confusing. Offsets may
be positive or negative.
If BEG and END are
not specified (they should be used only when really necessary)
the records returned are those that are alive at the moment the
query is run, that is, "now"
|
| END
yy mo dd hh mi ss |
Return records that
were alive on or before this offset from NOW (NOW is calculated
at the moment the query is run). If specifying END in a query,
always use BEG as well. Offsets may be positive or negative.
If BEG and END are
not specified (they should be used only when really necessary)
the records returned are those that are alive at the moment the
query is run, that is, "now"
|
| HDG
heading |
Heading for directionally-focused
searches (0.0 <= heading <= 360.0, type double). Default
is 0°=due North.
As of version 0.70,
LIM is not compatible with a directional search. Do not
use LIM and HDG together.
|
| LEN
length |
If
not using RAD, search length in meters (0° and 180° at current
heading, type integer) |
| LIM
limit |
Limit maximum number
of records returned (type integer). limit must be >=
0.
If limit = 0 (the default) there is no limit and all records
are found and returned.
This command is provided so that clients can help the server do
less work, by limiting the number of records retrieved i the background
SQL query.
As of version 0.70,
LIM is not compatible with a directional search. Do not
use LIM and HDG together.
|
| MET
metadata |
Return
only records with the substring metadata anywhere in their
META fields.
Matches are case-insensitive.
Use sparingly. Records with no metadata, and those with similar
but non-exact matches, will not be returned when this parameter
is included.
MET is quite useful when looking for a set of records tagged with
similar metadata (e.g. you know something was inserted with specific
metadata, and only want those records) |
| MIM
mimetype |
Return only records
whose header indicates they are exactly this MIME type (full-string
match is performed)
If the thing pointed to by this record is another Wherehoo server,
the MIME type will be 'application/x-whserver'
If this parameter is sent, only exact matches to mimetype
will be returned.
If this parameter is not sent with the query, records with all
mime types are returned.
|
| PJT
heading range |
"Project"
the user's location from the stated Latitude and Longitude (LLH
command) to a new place at a position that is "range"
distance from the current Lat,Lon at the given "heading".
This is useful for things that need to pretend they are at a place
"away from" the current location. The internal calculation
is as precise as (and derived from) the other distance and heading
calculations built into the server. (-360.0
< heading < 360.0) |
| PRO
protocol |
Return only records
whose protocol is exactly protocol
If the bits in
this record's DATA field are the thing pointed to (DATA
is not the URI of some location, but IS the data for that place
- a binary block, some words, some music, some code, whatever),
then the protocol is 'WHEREHOO'
If this parameter is
not sent with the query, records with all protocols are returned
The table of protocols
is maintained manually.
Please bring missing or new protocols to the attention of Wherehoo
project leads.
Permitted protocols are listed at http://wherehoo.media.mit.edu/about/supported-protocols.php
|
| RAD
radius |
Search
radius, in meters (0°, 90°, 180° and 270° around
current location at current heading, type integer). Sets WID = LEN
= radius. Alternatively, use WID and LEN to specify a non-square
search area. |
| SHP
shapecode |
Shape
of directional search {rect_ctr | rect_fwd} |
| WID
width |
If
not using RAD, search width in meters (90° and 270° at current
heading, type integer) |
| META |
After
receiving descriptive data about a Whereoo record, client wants
the record's metadata |
| DATA |
After
receiving descriptive data about a Whereoo record, client wants
the actual data block |
| SKIP |
After
receiving descriptive data about a Whereoo record, client does not
want the data block. Skip to next record |
| Specifically
for record insertion |
| BEG
yy mo dd hh mi ss |
Offset
from NOW for the beginning date/time of this record, defaults to
00/00/0000 if not specified
"NOW" is calculated at the moment the record is inserted
or updated (when server receives client's EOD marker) |
| DAT
data_size |
Here comes the binary
data.
The client should send:
DAT data_size
to tell the server the data's coming,
then a CRLF (like all other commands), then...
data_size
bytes of data
followed immediately by
the 20-byte SHA-1 signature, created using the data plus
the client's secret (arranged in advance)
Here is a class that
holds the data block and provides method SHAhash(mysecret) to
return the SHA-1 signature:
import java.security.*;
public class dataBlock {
public byte[] data;
public byte[] SHAhash(String mysecret) {
byte[] mdfinal;
try {
MessageDigest md = MessageDigest.getInstance("SHA-1");
md.update(data); // the data and...
md.update(mysecret.getBytes()); // my secret
mdfinal = md.digest(); // yield the secure hash
} catch (NoSuchAlgorithmException nsae) { return null; }
return mdfinal;
}
}
The total bytes your
client sends must be exactly data_size + 20
1 <= Data_size <= max_data
(max_data is stated by the server in response to the client's
IDT)
If data_size is greater than max_data, then only the first max_data
+ 20 bytes will be read and the client will be disconnected as
the server tries to interpret the additional data as commands.
So, don't do that.
Immediately after sending
the 'DAT data_size' line, send your bytes of data plus the 20-byte
signature, all as one contiguous binary block. Then go back to
line-reading and watch for "ACK". The server will answer
with an ACK after it's received your data_size bytes plus the
20-byte signature.
If there's something wrong with your data stream, or if data_size
is < 1, the server will return a NAK and break its connection.
WHERE DO I GET AN
IDENTITY AND SECRET SO I CAN ADD RECORDS?
For now, we give them
to testers after manually adding the IDT and signature pair to
the server's user database. Please write to request one. Eventually,
they'll be generated and exchanged with clients automatically.
|
| END
yy mo dd hh mi ss |
Offset
from NOW for the ending date/time of this record, defaults to 12/31/9999
if not specified
"NOW" is calculated at the moment the record is inserted
or updated (when server receives client's EOD marker) |
| MET
metadata |
Text metadata describing
the thing pointed to by this record.
The length of the metadata should be <= the
max_meta value provided by the server in response to the client's
IDT.
If the metadata block is longer than max_meta, it will be truncated
to the first max_meta characters.
|
| MIM
mimetype |
MIME type of the data
pointed to by (or contained in) this record.
For maximum happiness, use well-known MIME types.
If the thing pointed to by this record is another Wherehoo server,
the MIME type should be 'application/x-whserver'
NOTE:
Beginning with version 0.630, 11-1-00, MIME types are no
longer validated against a list. We still recommend using well-known
MIME types (else nobody will be able to use your records)!
|
| PRO
protocol |
The protocol an agent
will use to retrieve the thing pointed to by this record.
For maximum happiness, use standard protocols. These are checked
against a table built from ftp://ftp.iana.org/in-notes/iana/assignments/service-names.
If the protocol is not found, Wherehoo will not insert the record.
The additional
protocol 'WHEREHOO' is also accepted.
If the bits in this record's DATA field are the thing pointed
to (DATA is not the URI of some location, but IS the data for
that place - a binary block, some words, some music, some code,
whatever), then the protocol is 'WHEREHOO'
The table of protocols is maintained manually. Please bring missing
or new protocols to the attention of Wherehoo
project leads.
Permitted protocols are listed at http://wherehoo.media.mit.edu/about/supported-protocols.php
|
| Specifically
for record deletion |
| UID
uniqueid |
The
32-character uniqueid of the record that is to be deleted. This
information was reported to the creating client at the time the
record was created and cannot be retrieved otherwise. Clients that
want to delete or update their records will need to store and manage
the records' UIDs. |
| Connection
control |
| DBG
debug_mode |
Mode
of debugging {on | off }. Default is OFF. When ON, state info is
sent back toward you.
Don't use unless debugging the server or your communications stream
will be full of unwanted goo |
| NOP |
No-op to forestall
rx timeout by server if your client is running slowly or to see
if server is still listening
Server answers with ACK if it's still listening
|
| BYE |
Close
our connection now |
| . |
EOD.
Client is finished sending data to server. Server should run the
query now. |
|
|
| wherehoo_server
version timeout min_expiration max_meta max_data |
Server's response
to an IDT from the client. All fields are variable length,
space-delimited
example: wherehoo_server 0.480 30 20 1024 65535
wherehoo_server
is a constant text string
version is the version number
timeout is the data timeout in seconds. Silence from client
for this duration drops connection
min_expiration is the minimum number of seconds
"later than the beginning time" that a record may end...
This is to avoid thrashing the database with too-short-lived records
max_meta is the maximum length of the text metadata describing
the contents of a Wherehoo record's data field
max_data is the
maximum number of bytes of binary data that may be stored inside
a Wherehoo record's data field
|
| OK |
Search parameters
accepted. Server is running query or insert operation. |
| NAK
command_list |
These commands
had invalid or out of bounds parameters. Query not running. |
| ACK |
Response
to a NOP. Server is alive and listening. |
| . |
EOD (End
of data). There are no more records to send you. |
| BYE |
Server is
closing the connection |
| |
|
DATA
FROM THE SERVER TO THE CLIENT
|
|
In response to a query
from the client, the server sends back individual records.
1. A line of text
containing descriptive data about the record's content is sent
to the client
2. The client should check the descriptive data for compatibility
("can I actually handle this data type? protocol? data
size?")
3. The client
issues one of these instructions on a line by itself:
DATA if
it wants the data (binary; byte count was stated in bytes
field of record header - see below)
META if it wants the text metadata describing the contents
of DATA (one line, size limit of max_meta)
SKIP
to move to the next record
BYE to close the connection.
4. If the client
sent "DATA" or "META" the requested stuff
is sent. If the client asks for META and there isn't any (should
have read the record header!) an empty line will be sent.
5. If client sent META, the server loops to #3, waiting for
another command regarding this record (because the client might
want DATA next).
... otherwise the server returns to #1 and sends descriptive
data for the next record, or EOD (".") if there are
no more records.
Record descriptive
data appears on one line, space-delimited, in this format:
bearing compassdirection
distance ttl bytes protocol mimetype meta
|
| bearing |
Bearing from your location
to this object, in degrees, at your current HEADING |
| compassdirection |
Compass direction from
your location to this object {N | NE | E | SE | S | SW | W | NW
} |
| distance |
Straight-line distance
from your location to this object, in meters |
| ttl |
Time-to-live of this
record in the database, in seconds
TTL longer than 99999999 seconds, just over 3 years, come back as
99999999. |
| bytes |
Number of bytes of data
in the object's data block (to be sent if client replies DATA) |
| protocol |
If object's data block
is a URI, the protocol to be used for fetching the data
If object's data block is stored in the Wherehoo server, the protocol
is "WHEREHOO" |
| mimetype |
The MIME data type of
the data represented by the object's data block, whether served
directly from WHEREHOO or some other server (protocol tells you
where it's coming from) |
| meta |
"META" if
textual metadata exists for this record, "NONE" if no
metadata exists
Client replies META to tell server to send this text. |
|
SAMPLE
CLIENT-SERVER INTERACTION - A Query
|
| From
the client to the server |
From
the server to the client |
| IDT jim |
|
| |
wherehoo_server
2.1 30 20 1024 65535 |
| ACT QUERY |
|
| LLH 72.019
288.908 0 |
|
| RAD 250 |
|
| SHP rect_fwd |
|
| HDG 0 |
|
| . |
|
| |
OK |
| |
96 N 1443
99999999 33212 WHEREHOO IMAGE/JPG META |
| DATA |
|
| |
(33212 bytes
of data) |
| NEXT |
|
| |
138 NW 230
99999999 74 HTTP TEXT/HTML NONE |
| DATA |
|
| |
(74 bytes
of data) |
| |
. |
| |
BYE |
|
SAMPLE
CLIENT-SERVER INTERACTION - An insert
|
| From
the client to the server |
From
the server to the client |
| IDT jim |
|
| |
wherehoo_server
0.56 30 20 1024 65535 |
| ACT INSERT |
|
| LLH 72.019
288.908 0 |
|
| MIM TEXT/PLAIN |
|
| PRO WHEREHOO |
|
| DAT 12 |
|
| (12 bytes
of data) |
|
| |
ACK |
| MET This
is some metadata describing the content of the DAT field |
|
| . |
|
| |
OK |
| |
c08b9ac6d59dba9be0106096626809b2e4098f0a |
| |
. |
| |
BYE |
|
SAMPLE
CLIENT-SERVER INTERACTION - A delete
|
| From
the client to the server |
From
the server to the client |
| IDT jim |
|
| |
wherehoo_server
0.56 30 20 1024 65535 |
| ACT DELETE |
|
| UID c08b9ac6d59dba9be0106096626809b2e4098f0a |
|
| . |
|
| |
OK |
| |
ACK |
| |
. |
| |
BYE |
|
Hard
limits
|
| Maximum RAD, WID, or
LEN |
MAXWIDLEN = 999999 |
| Maximum length of identity
for IDT |
MAXIDT = 10 |
| Unique id string returned
after a successful INSERT action |
40 hex characters(0..F)
Fixed length |
| |
|
| Things
to think about |
| Self-determination of coverage area |
How does a network of Wherehoo servers self-configure? |
| Query routing and server acquisition of geodata |
|
| Revision
History |
| |
|
| 2.02
12/12/01 |
More forgiving
- receipt of invalid commands no longer triggers "BYE"
and disconnect from server. Invalid commands are silently ignored.
Also, bug in RAD command corrected - RAD was not setting the LEN
and WID parameters as it should have. |
| 2.00
12/10/01 |
Databases
transitioned to Postgresql 7.1, transition from (X,Y) single-point
data to polygonal areas (using the geometric search abilities of
Postgresql). Significant for client code: Server no longer transparently
sends the next record after a DATA block has been sent. Client must
request the next record explicitly. Officially, the command "NEXT<cr>"
should be sent. However, at present, any <cr>-terminated message
from the client that does not match an existing Wherehoo command
will cause the next record to be sent. This will be changed in the
future, so use NEXT<cr> to get the next record! |
| 0.630
11/1/00 |
Removed checking of
MIME type presented with the MIM command
If MIM or PRO are set (or both) when performing a query, the server
will return only thos records that exactly match the provided
MIME type and/or protocol.
Example:
ACT QUERY
LLH 42.361 -071.088 0
RAD 10000
MIM text/html
PRO http
.
This query only
returns records that point to web pages accessible via the HTTP
protocol
|
| 0.623 |
Catch datalen
> MAXDATA and return DAT error |
| 0.621
10/29/00 |
Revised
diagnostic byte printouts, solving the 1429-byte-problem
revisions to DAT read section, problem using .read(,,) switched
to .ready() plus .read() with a wait loop when anticipating more
data |
| 0.610
10/8/00 |
Added some
diagnostics. nothing major. |
| 0.602
10/6/00 |
Add PJT
command to project self to some other place (for periscope, other
lookahead apps) |
| 0.590
10/1/00 |
Allow BEG,
END offsets to expand search window beyond records that are alive
"now" |
| 0.580
9/3/00 |
Allow META
for queries, match meta to any substring in the meta field |
| 0.570
9/17/00 |
Add SHA1
signature as 20 bytes appeneded to DAT bytes. Checks signature by
generating SHA1 sig against asserted IDT's secret |
|