Introduction

This document describes the setup and the solution used for building the FileNet Extract Connector.The connector aims only the Content Engine (CE) component of FileNet. The Process Engine (PE) component is outside the scope of this connector. The FileNet Extract Connector is capable of creating a generic data model to accommodate the storage of FileNet data into a MongoDB instance, recursively extracting all the objects from a FileNet hierarchical structure, storing associated object metadata, version extraction, export of permissions and export of member information. The transformation of the extracted records to allow the loading into a target system is outside the scope of this document.

Prerequisites

In order to successfully run the FileNet connector scripts, the following requirements must be met:

  • The scripts code must be opened in Xill IDE 3.1
  • A MongoDB instance must be running locally on the standard port (127.0.0.1:27017)
  • Access to a FileNet user environment (FileNet server) is required
  • Access to the FileNet Content Engine (CE) Java API is required; the Java API must be configured to run through the WSI (Web Services Interoperability) protocol, by using the following WSI url: http://<server-name>:9080/wsi/FNCEWS40MTOM
  • The FileNet proxy has to be accessible and running. The proxy can be accessed through the following url:http://<server-name>[:port]/filenetproxy/v1/
  • The authentication to the Content Engine requires: a username, a password and a stanza. The stanza is a JAAS (Java Authentication and Authorization Service) configuration that contains a list of login modules. Since the authentication is done over the CEWS (Content Engine Web Services) transport, FileNetP8WSI needs to be used as stanza.
  • Administrator privileges for the FileNet CE are required in order to have access to the user account information and object permissions

Before running any of the connector's extract robots, the access credentials to the FileNet CE, the proxy details and the object store must be specified in the "xill.properties" file.


Unified Data Model Setup

Generally the Unified Data Model (UDM) is built based on the dataset that needs to be migrated. For FileNet, however, some configuration may be required to align the Content Engine object types with the content types used in UDM. Since different UDM content types contain not only standard metadata, but also their own custom metadata, a way should be found to create content types with custom metadata in FileNet as well. This functionality can be configured in FileNet by using the class concept. The class in FileNet is a specification for a set of objects that have the same properties. The Content Engine has some pre-defined classes: "Document", "CustomObject", "Folder", but one can also create custom subclasses. These subclasses will then inherit the properties of the parent class, but will also have their own set of custom properties. To give an example, a "Letter" with custom properties such as: originator, recipient, subject of a message can be created in FileNet as a subclass "Letter" of the "Document" class. The "Letter" subclass will inherit the "Document" properties, such as: title, but unlike the rest of the "Documents", it will also have its own properties: originator, recipient, subject of message and others. Such configurations can be done in the Administrative Console for Content Engine (http://<server-name>:9080/acce).

Custom mapping of content types can be added to the connector when export of particular content types is required. Below there is an example of a mapping for the content type "Letter". In the mapping, the metadata of the content type "Letter" (subclass of the "Document") from FileNet is being used to populate the metadata of the content type "Letter" in UDM.

function mapLetter(document, data) {
    document += {
        "originator" : {
            "name" : data.properties.OriginatorName.value,
            "address" : data.properties.OriginatorAddress.value
        },
        "recipient" : {
            "name" : data.properties.RecipientName.value,
            "address" : data.properties.RecipientAddress.value
        },
        "message" : {
            "date" : Date.parse(data.properties.MessageDate.value, "yyyy-MM-dd'T'HH:mm'Z'"),
            "subject" : data.properties.MessageSubject.value
        }
    };
    return document;
}
Most of the decorators created for the FileNet Extract Connector are standard decorators. In addition to the standard decorators, a separate decorator is created for FileNet-specific objects.
"filenet" : {
	"id" : {
		"type"    : "STRING",
                "required": true
	},
        "path" : {
                "type"    : "STRING",
                "required": false
	}
}
The rest of the custom decorators are dataset-specifc and they differ per migration.

Solution

The FileNet Connector has been implemented using the Content Engine Java API. First, the REST API has been considered for building the connector, but there is no REST API provided for the Content Engine. Since the Content Engine Java API offers complete API functionality, it is easy to use and provides good supportability, it was chosen over the Content Engine Web Service (Disadvantages of using CEWS) for implementing the FileNet connector. The API has been configured to communicate with the CE server through the Web Services Interoperability (WSI) protocol.

This document does not go into details in what regards the Content Engine building design and the FileNet Java API. Other articles exist that cover these aspects. For documentation on the Content Engine Architecture, please refer to the following link:http://www.developer.com/java/other/article.php/3793406/Connecting-to-IBM-FileNet-40-Outside-of-a-Java-Application-Server.htm For documentation on the FileNet Java API and the FileNet concepts in general, please refer to the following link:http://www.ibm.com/support/knowledgecenter/en/SSNW2F_5.2.1/kc_gen/ce_dev_ce_toc-gen1.html However aspects such as: why was the FileNet Proxy built, how it works, how is a FileNet connection established and what features the FileNet export incorporates,are described further in the document.

FileNet Proxy

At the beginning the connector was built by directly employing XILL routines written in Java, based on the CE Java API. That approach failed since the CE resides within a Java Enterprise Edition Application Server and the Java API requires a LDAP context when connecting to the CE, which cannot be provided with XILL IDE. Therefore a FileNet Proxy-middleware has been developed to allow communication with the CE through the FileNet Java API. The proxy has been developed using the RESTful Web Services with Spring. There are several ways to start the proxy. The current solution is running the proxy application packaged as a JAR file from XILL IDE. When the proxy runs, one can use the FileNet Extract Connector to extract data from CE and can also get interactive documentation with the swagger-enabled API (http://localhost[:port]/swagger-ui.html). The swagger UI can be used to get the complete list of all the RESTful URIs employed when building the connector.

FileNet Connection

The connection with the FileNet as well as the rest of the API calls to the FileNet are done through the FileNet proxy. In order to establish a connection, the following FileNet connection information is needed:
  1. FileNet Credentials (username and password)
  2. Stanza (FileNetP8WSI should be used as stanza as explained in the Prerequisites section )
  3. WSI URI (see Prerequisites section)
  4. Proxy URI (see Prerequisites section)

Below there is a code snippet that shows the REST call used in the connector for FileNet authentication.

function connect(server, proxy){
    var body = {
      "urlString"   : server.apiUrl,
      "userName"    : server.user,
      "password"    : server.password,
      "stanza"      : server.stanza
    };
    
    var result = post(server, proxy.proxyUrl :: "core/session/open", body);
    return result.body.localSessionId;
}

FileNet Export

Once a connection with FileNet is established, the export connector starts downloading content from the FileNet CE by means of a standard scraper. The FileNet content metadata is also being extracted and it is being stored in UDM. A separation of the content types in UDM is supported, with the mention that content type mappings have to be added to the connector separately since they usually differ per project. Furthermore, the FileNet export connector allows version export, fine-grained extraction of permission information and a basic extraction of user and group metadata. All the above-mentioned features are implemented with REST calls with RESTfull URIs generated by the FileNet proxy. An example of a post request to download a FileNet document binary is shown below:
function downloadBinary(server, proxy, documentId, path, objectStoreId, localSessionId){
    var body = {
        "xill:objectStoreId": objectStoreId,
        "localSessionId"    : localSessionId,
        "documentId"        : documentId,
        "path"              : path
    };
    var result = post(server, proxy.proxyUrl :: "command/getBinary", body);
    return result.body;
}

Closing remarks

The FileNet Export connector offers a good head-start for migration projects from FileNet to other ECM systems. The built-in functionality covers to a large extent the expected features of a FileNet content extraction. Adding custom content type mappings should suffice for realizing migrations involving FileNet exports.