gate

Class DocumentFormat

java.lang.Object

gate.util.AbstractFeatureBearer

gate.creole.AbstractResource

gate.creole.AbstractLanguageResource

gate.DocumentFormat

All Implemented Interfaces:

LanguageResource, Resource, FeatureBearer, NameBearer, Serializable

Direct Known Subclasses:

TextualDocumentFormat, TikaFormat

public abstract class DocumentFormat
extends AbstractLanguageResource

The format of Documents. Subclasses of DocumentFormat know about particular MIME types and how to unpack the information in any markup or formatting they contain into GATE annotations. Each MIME type has its own subclass of DocumentFormat, e.g. XmlDocumentFormat, RtfDocumentFormat, MpegDocumentFormat. These classes register themselves with a static index residing here when they are constructed. Static getDocumentFormat methods can then be used to get the appropriate format class for a particular document.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,String>	element2StringMap This map is used inside uppackMarkup() method...
protected static Map<String,MimeType>	magic2mimeTypeMap Map of Set of magic numbers to MimeType.
protected Map<String,String>	markupElementsMap Map of markup elements to annotation types.
protected static Map<String,DocumentFormat>	mimeString2ClassHandlerMap Map of MimeTypeString to ClassHandler class.
protected static Map<String,MimeType>	mimeString2mimeTypeMap Map of MimeType to DocumentFormat Class.
protected static Map<String,MimeType>	suffixes2mimeTypeMap Map of Set of file suffixes to MimeType.

Fields inherited from class gate.creole.AbstractLanguageResource

dataStore, lrPersistentId

Fields inherited from class gate.creole.AbstractResource

name

Constructor Summary

Constructors

Constructor and Description
DocumentFormat() Default construction

Method Summary

Modifier and Type	Method and Description
void	addStatusListener(StatusListener l)
protected static boolean	areEqual(MimeType aMimeType, MimeType anotherMimeType) Tests if two MimeType objects are equal.
protected static MimeType	decideBetweenThreeMimeTypes(MimeType aMimeTypeFromWebServer, MimeType aMimeTypeFromFileSuffix, MimeType aMimeTypeFromMagicNumbers) This method decides what mimeType is in majority
protected static MimeType	decideBetweenTwoMimeTypes(MimeType aMimeType, MimeType anotherMimeType) Decide between two mimeTypes.
protected void	fireStatusChanged(String e)
static DocumentFormat	getDocumentFormat(Document aGateDocument, MimeType mimeType) Find a DocumentFormat implementation that deals with a particular MIME type, given that type.
static DocumentFormat	getDocumentFormat(Document aGateDocument, String fileSuffix) Find a DocumentFormat implementation that deals with a particular MIME type, given the file suffix (e.g.
static DocumentFormat	getDocumentFormat(Document aGateDocument, URL url) Find a DocumentFormat implementation that deals with a particular MIME type, given the URL of the Document.
static DocumentFormat	getDocumentFormat(MimeType mimeType) Find the DocumentFormat implementation that deals with the given MIME type.
Map<String,String>	getElement2StringMap() Get the element 2 string map
FeatureMap	getFeatures() Get the feature set
Map<String,String>	getMarkupElementsMap() Get the markup elements map
MimeType	getMimeType() Gets the mime Type
static MimeType	getMimeTypeForString(String typeString) Utility method to get a MimeType given the type string.
Boolean	getShouldCollectRepositioning()
static Set<String>	getSupportedFileSuffixes() Utility method to get the set of all file suffixes that are registered with this class.
static Set<String>	getSupportedMimeTypes()
protected static MimeType	guessTypeUsingMagicNumbers(InputStream aInputStream, String anEncoding) This method tries to guess the mime Type using some magic numbers.
void	removeStatusListener(StatusListener l)
protected static MimeType	runMagicNumbers(Reader aReader) Performs magic over Gate Document
void	setElement2StringMap(Map<String,String> anElement2StringMap) Set the element 2 string map
void	setFeatures(FeatureMap features) Set the features map
void	setMarkupElementsMap(Map<String,String> markupElementsMap) Set the markup elements map
void	setMimeType(MimeType aMimeType) Set the mime type
void	setShouldCollectRepositioning(Boolean b)
Boolean	supportsRepositioning() If the document format could collect repositioning information during the unpack phase this method will return true.
abstract void	unpackMarkup(Document doc) Unpack the markup in the document.
abstract void	unpackMarkup(Document doc, RepositioningInfo repInfo, RepositioningInfo ampCodingInfo)
void	unpackMarkup(Document doc, String originalContentFeatureType) Unpack the markup in the document.
static boolean	willReadFromUrl(String mimeTypeStr, URL docUrl) Utility function to determine if reading from the URL will be done by a registered DocumentFormat.

Methods inherited from class gate.creole.AbstractLanguageResource

cleanup, getDataStore, getLRPersistenceId, getParent, isModified, setDataStore, setLRPersistenceId, setParent, sync

Methods inherited from class gate.creole.AbstractResource

checkParameterValues, flushBeanInfoCache, forgetBeanInfo, getBeanInfo, getInitParameterValues, getInitParameterValues, getName, getParameterValue, getParameterValue, getParameterValues, init, removeResourceListeners, setName, setParameterValue, setParameterValue, setParameterValues, setParameterValues, setResourceListeners, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface gate.Resource

getParameterValue, init, setParameterValue, setParameterValues

Methods inherited from interface gate.util.NameBearer

getName, setName

Field Detail

mimeString2ClassHandlerMap

protected static final Map<String,DocumentFormat> mimeString2ClassHandlerMap

Map of MimeTypeString to ClassHandler class. This is used to find the language resource that deals with the specific Document format

mimeString2mimeTypeMap

protected static final Map<String,MimeType> mimeString2mimeTypeMap

Map of MimeType to DocumentFormat Class. This is used to find the DocumentFormat subclass that deals with a particular MIME type.

suffixes2mimeTypeMap

protected static final Map<String,MimeType> suffixes2mimeTypeMap

Map of Set of file suffixes to MimeType. This is used to figure out what MIME type a document is from its file name.

magic2mimeTypeMap

protected static final Map<String,MimeType> magic2mimeTypeMap

Map of Set of magic numbers to MimeType. This is used to guess the MIME type of a document, when we don't have any other clues.

markupElementsMap

protected Map<String,String> markupElementsMap

Map of markup elements to annotation types. If it is null, the unpackMarkup() method will convert all markup, using the element names for annotation types. If it is non-null, only those elements specified here will be converted.

element2StringMap

protected Map<String,String> element2StringMap

This map is used inside uppackMarkup() method... When an element from the map is encounted, The corresponding string element is added to the document content

Constructor Detail

DocumentFormat

public DocumentFormat()

Default construction

Method Detail

supportsRepositioning

public Boolean supportsRepositioning()

If the document format could collect repositioning information during the unpack phase this method will return true.
You should override this method in the child class of the defined document format if it could collect the repositioning information.

setShouldCollectRepositioning

public void setShouldCollectRepositioning(Boolean b)

getShouldCollectRepositioning

public Boolean getShouldCollectRepositioning()

unpackMarkup

public abstract void unpackMarkup(Document doc)
                           throws DocumentFormatException

Unpack the markup in the document. This converts markup from the native format (e.g. XML, RTF) into annotations in GATE format. Uses the markupElementsMap to determine which elements to convert, and what annotation type names to use.

Throws:

DocumentFormatException

unpackMarkup

public abstract void unpackMarkup(Document doc,
                                  RepositioningInfo repInfo,
                                  RepositioningInfo ampCodingInfo)
                           throws DocumentFormatException

Throws:

DocumentFormatException

unpackMarkup

public void unpackMarkup(Document doc,
                         String originalContentFeatureType)
                  throws DocumentFormatException

Unpack the markup in the document. This method calls unpackMarkup on the GATE document, but after it saves its content as a feature attached to the document. This method is useful if one wants to save the content of the document being unpacked. After the markups have been unpacked, the content of the document will be replaced with a new one containing the text between markups.

Parameters:

doc - the document that will be unpacked

originalContentFeatureType - the name of the feature that will hold the document's content.

Throws:

DocumentFormatException

getSupportedMimeTypes

public static Set<String> getSupportedMimeTypes()

decideBetweenThreeMimeTypes

protected static MimeType decideBetweenThreeMimeTypes(MimeType aMimeTypeFromWebServer,
                                                      MimeType aMimeTypeFromFileSuffix,
                                                      MimeType aMimeTypeFromMagicNumbers)

This method decides what mimeType is in majority

Parameters:

aMimeTypeFromWebServer - a MimeType

aMimeTypeFromFileSuffix - a MimeType

aMimeTypeFromMagicNumbers - a MimeType

Returns:

the MimeType which occurs most. If all are null, then returns null

decideBetweenTwoMimeTypes

protected static MimeType decideBetweenTwoMimeTypes(MimeType aMimeType,
                                                    MimeType anotherMimeType)

Decide between two mimeTypes. The decistion is made on "Priority" parameter set into decideBetweenThreeMimeTypes method. If both mimeTypes doesn't have "Priority" paramether set, it will return one on them.

Parameters:

aMimeType - a MimeType object with "Prority" parameter set

anotherMimeType - a MimeType object with "Prority" parameter set

Returns:

One of the two mime types.

areEqual

protected static boolean areEqual(MimeType aMimeType,
                                  MimeType anotherMimeType)

Tests if two MimeType objects are equal.

Returns:

true only if boths MimeType objects are different than null and their Types and Subtypes are equals. The method is case sensitive.

guessTypeUsingMagicNumbers

protected static MimeType guessTypeUsingMagicNumbers(InputStream aInputStream,
                                                     String anEncoding)

This method tries to guess the mime Type using some magic numbers.

Parameters:

aInputStream - a InputStream which has to be transformed into a InputStreamReader

anEncoding - the encoding. If is null or unknown then a InputStreamReader with default encodings will be created.

Returns:

the mime type associated with magic numbers

runMagicNumbers

protected static MimeType runMagicNumbers(Reader aReader)

Performs magic over Gate Document

getDocumentFormat

public static DocumentFormat getDocumentFormat(Document aGateDocument,
                                               MimeType mimeType)

Find a DocumentFormat implementation that deals with a particular MIME type, given that type.

Parameters:

aGateDocument - this document will receive as a feature the associated Mime Type. The name of the feature is MimeType and its value is in the format type/subtype

mimeType - the mime type that is given as input

getDocumentFormat

public static DocumentFormat getDocumentFormat(Document aGateDocument,
                                               String fileSuffix)

Find a DocumentFormat implementation that deals with a particular MIME type, given the file suffix (e.g. ".txt") that the document came from.

Parameters:

aGateDocument - this document will receive as a feature the associated Mime Type. The name of the feature is MimeType and its value is in the format type/subtype

fileSuffix - the file suffix that is given as input

getDocumentFormat

public static DocumentFormat getDocumentFormat(MimeType mimeType)

Find the DocumentFormat implementation that deals with the given MIME type.

Parameters:

mimeType - the MIME type you want the DocumentFormat for

Returns:

the DocumentFormat associated with the MIME type or null if the MIME type does not have a registered DocumentFormat

getDocumentFormat

public static DocumentFormat getDocumentFormat(Document aGateDocument,
                                               URL url)

Find a DocumentFormat implementation that deals with a particular MIME type, given the URL of the Document. If it is an HTTP URL, we can ask the web server. If it has a recognised file extension, we can use that. Otherwise we need to use a map of magic numbers to MIME types to guess the type, and then look up the format using the type.

Parameters:

aGateDocument - this document will receive as a feature the associated Mime Type. The name of the feature is MimeType and its value is in the format type/subtype

url - the URL that is given as input

getFeatures

public FeatureMap getFeatures()

Get the feature set

Specified by:

getFeatures in interface FeatureBearer

Overrides:

getFeatures in class AbstractFeatureBearer

getMarkupElementsMap

public Map<String,String> getMarkupElementsMap()

Get the markup elements map

getElement2StringMap

public Map<String,String> getElement2StringMap()

Get the element 2 string map

setMarkupElementsMap

public void setMarkupElementsMap(Map<String,String> markupElementsMap)

Set the markup elements map

setElement2StringMap

public void setElement2StringMap(Map<String,String> anElement2StringMap)

Set the element 2 string map

setFeatures

public void setFeatures(FeatureMap features)

Set the features map

Specified by:

setFeatures in interface FeatureBearer

Overrides:

setFeatures in class AbstractFeatureBearer

setMimeType

public void setMimeType(MimeType aMimeType)

Set the mime type

getMimeType

public MimeType getMimeType()

Gets the mime Type

getMimeTypeForString

public static MimeType getMimeTypeForString(String typeString)

Utility method to get a MimeType given the type string.

getSupportedFileSuffixes

public static Set<String> getSupportedFileSuffixes()

Utility method to get the set of all file suffixes that are registered with this class.

willReadFromUrl

public static boolean willReadFromUrl(String mimeTypeStr,
                                      URL docUrl)

Utility function to determine if reading from the URL will be done by a registered DocumentFormat. This tries to find out if there is a registered DocumentFormat for the mime type string or document URL. If yes, it is checked if that document format implements DirectLoadingDocumentFormat. If yes, returns true, otherwise returns false. If both the mime type and docUrl are null or empty, the default behavior of false is returned.

Parameters:

mimeTypeStr - the mime type string parameter for the Document

docUrl - the sourceUrl parameter for the Document

Returns:

true if the URL will be read directly by a document format.

removeStatusListener

public void removeStatusListener(StatusListener l)

addStatusListener

public void addStatusListener(StatusListener l)

fireStatusChanged

protected void fireStatusChanged(String e)