Search Appliance SBE

 

Thunderstone Search Appliance Manual

Submission Format

 

Data is submitted to the Search Appliance with an HTTP POST request sent to a similar URL as the admin interface (e.g. http://.../dowalk), but with /recvdata.xml appended. E.g.:

http://www.example.com/texis/dowalk/recvdata.xml

The following POST variables must be set in the request. Be sure to URL-encode the values:

Specifying all fields manually

Below is an example data document where all fields are specified. Be sure to HTML-encode values.

 

<?xml version="1.0" encoding="UTF-8"?>
<ThunderstoneReplication
      xmlns:dt="urn:schemas-microsoft-com:datatypes"
>
  <Item>
    <Type>I</Type>
    <Size>150369</Size>
    <Visited>2005-10-25 15:25:18</Visited>
    <Dlsecs>0</Dlsecs>
    <Depth>0</Depth>
    <Url>http://www.example.com/dir/page.html</Url>
    <Title>Sprocket Specifications</Title>
    <Body>...</Body>
    <Keywords>sprockets, gears, hubs</Keywords>
    <Description>Sprocket details</Description>
    <Meta></Meta>
    <Category>Mechanical</Category>
    <Modified>2005-10-25 11:21:07</Modified>
    <NextCheck>2005-10-25 16:25:18</NextCheck>
    <Views>0</Views>
    <Clicks>0</Clicks>
    <CTR>0.000000</CTR>
    <Pop>0</Pop>
    <MimeType>text/html</MimeType>
    <Charset>UTF-8</Charset>
    <Refs dt:dt="bin.base64">...</Refs>
    <Errors dt:dt="bin.base64">...</Errors>
    <RawData dt:dt="bin.base64"></RawData>
  </Item>
</ThunderstoneReplication>

Any element whose text data might not be XML-safe (e.g. binary chars in the <Body>) should be base64-encoded, and the attribute dt:dt="bin.base64" set in the tag. E.g. the <Refs> and <Errors> elements' text data are always base64-encoded. Note that the XML namespace prefix dt should also then be set to urn:schemas-microsoft-com:datatypes in the root <ThunderstoneReplication> element.

The elements are:

Uploading a binary file

If you have a binary file, such as a PDF or an Office document, you can send it with the dataload API and let the Search Appliance extract the text from it.

 

<?xml version="1.0" encoding="UTF-8"?>
<ThunderstoneReplication
      xmlns:dt="urn:schemas-microsoft-com:datatypes"
>
    <Item>
        <Type>I</Type>
        <Url>http://www.example.com/dataload.pdf</Url>
        <RawData dt:dt="bin.base64">0M8R4KGxGu....</RawData>
    </Item>
</ThunderstoneReplication>

The elements are:

Combining the two: binary files with custom fields

It is possible to specify both a <RawData> document, and fields such as <Title>, <Description>, etc. The binary document will be processed, and any other fields provided will override the values that came from the document.

This can be useful in situations where you have a Content Management System (CMS) that contains metadata about a document that doesn't actually occur anywhere in the document. You can do a custom dataload that pushes in the document, and the custom Title/Description/etc.

Additional Fields

Each profile-specific Additional Field is optionally sent in a single element named after the field, with the XML namespace prefix u. The value of the field is the content of the XML element. Note that the u XML namespace prefix should be declared in the root <ThunderstoneReplication> node, as shown earlier.

For example, an Integer field Quantity and a Text field State may be given as:

<u:Quantity>57</u:Quantity>
<u:State>NY</u:State>

Other Details

The optional <Refs> element lists the links (references) from the given document, for parent-child linking. Its text value is a base64-encoded XML document with the following format when decoded:

<results xmlns:dt="urn:schemas-microsoft-com:datatypes">
  <result>
    <Url>http://www.example.com/dir/page.html</Url>
    <Ref>http://www.example.com/dir/otherpage.html/</Ref>
  </result>
  ...
</results>

Each <Url> should be the same as the <Url> in the above <Item> block. The <Ref> is a single link from the page. Only one <Ref> may be listed per <result>; additional links should be sent with additional <result> elements.

The optional <Errors> element contains any errors to be logged for the document. Note that this may be empty or not present if no errors are to be logged. Its text value is a base64-encoded XML document with the following format when decoded:

<results xmlns:dt="urn:schemas-microsoft-com:datatypes">
  <result>
    <Url>http://www.example.com/dir/page.html</Url>
    <Reason>Document not found: 404 (Not Found)</Reason>
  </result>
  ...
</results>

As with the <Refs> element, the <Url> must correspond with the original <Item> <Url>, and multiple errors must be listed in separate <result> elements.


Copyright © Thunderstone Software     Last updated: Mar 19 2020