Gathering Search Result from AltaVista

Submitting the Query
Two main classes are responsible for submitting the Query (request) and retrieving the Search Results (response). The SearchResultsCollector and the SearchEngineFacade. The collector runs in a separate Thread and is responsible for passing the query request to a dynamically dispatched SearchEngine. Running the Collector as a separate thread enables us to spawn multiple collectors and run them concurrently. Each collector, then, has a specific Search Engine tied to it.

Design Pattern
In designing this architecture, we implemented the Strategy Design Pattern. This pattern provides a model solution for implementing multiple interchangeable algorithms. In our implementation, we created an abstract search engine class. This abstract class can support multiple specific concrete classes. Each concrete class represents a single unique Search Engine.

Discovery
Java provides some nice features for managing HTTP connections. Specifically, we utilized the URL and HttpURLConnection classes to connect to the desired Search Engine. For additional information see Java Tutorial (Networking).

Parsing Search Results
The response from the Search Engine takes the form of an HTML Page. In this form, the information is useless. The parsers' job is to find a way to identify and extract the desired information, from the HTML Page, and store it in a structure that the VRML understands.

There are various methods/tools for parsing HTML pages. The JavaCC HTML Parser builds a formatted tree, based on a specified grammar. The Metamata Parser is another such parser utility. While we could've incorporated these or other parser utilities, we chose to write a simplified AltaVista specific Search Engine parser.
Note: The AltaVista Search Engine separates Search Results with Table Tags.

Communicating with the ASP
The AdditionalResultsFacade is a SearchEngineFacade that's intended for internal use. It has the responsibility of obtaining serialized QueryResults from the ASP and converting them into deserialized QueryResult instances, that can be used by the VRML Page.

HTTP Headers
Because we have a consumer/producer problem, such that the Collector is producing results and the Additional Results is consuming them, we need to maintain a concept of the remote producer's state. This state is passed to the AdditionalResultsFacade class in the form of an HTTP header. Again, we found that Java provided us with HttpURLCollection methods to interrogate HTTP Header Keys and to obtain HTTP Header values.

Sample Code
  if theConnection.getHeaderFieldKey(i).equals("collector-state"))
     m_RemoteCollectorState = Integer.parseInt(theConnection.getHeaderField(i));

Maintaining State
One benefit of ASP is that it can maintain Session State. However, in order to maintain state, the ASP requires a token from the client. In our case, the client is the AdditionalResultsFacade. Again, we found that Java provided us with methods that allow us to set the HTTP Cookie.

Sample Code
  theConnection = (HttpURLConnection) theURL.openConnection();
  theConnection.setRequestProperty("Cookie", m_SessionID);
  if ( (theConnection.getResponseCode() == -1) || (theConnection.getResponseCode() !=
                                                        HttpURLConnection.HTTP_OK) )
        throw new IOException("Bad response from URLconnect");