Introduction

This document provides an overview of a migration process from an arbitrary source system to the OpenText Content Server. Apart from that, it describes the setup and the solution used for building the OpenText Import Connector. The Import Connector is accountable for importing documents to the OpenText Content Server together with metadata, versions, permissions and member information.

Market demands

The OpenText import has been created to satisfy a set of market demands. One demand is the need for process automation. Migrations that imply manual operations are error-prone, difficult to control and usually require more time to perform. On the other hand, an automated migration process is much easier to monitor, is less error-prone and thus, more accurate. Another demand is scalability. It should be possible to easily extend the migration setup with additional software/hardware to improve performance. Traceability is another market demand. It confers an overview of how the migration evolves: how many documents are staging, how many documents are migrated, how many documents failed to be loaded, etc. Usually, it is preferred to employ migration software that is self-contained, without making use of any additional modules. The communication with the OpenText Content Server has to be realized only over approved API's. Furthermore, it should be possible to easily add migration-specific features with minimum scripting. Handling/logging exception is also listed among the market demands.

OpenTex Migration process

The OpenText Import Connector as well as the OpenText Export Connector are not meant to be used as independent applications. They are usually part of a bigger process, known as a migration process. As such, an import to OpenText does not consist of just the OT Import Connector. It usually implies a series of steps: an extract from a source system, a mapping of the data to the UDM, another extraction from the UDM, a mapping of the data from UDM to OpenText and finally an import to the OTCS. The role of the OpenText Connector in this process is that of a communication interface between the Xillio Core and the OpenText Content Server. The code for the OpenText Connector is fixed and independent of the source system or the migration project.

In any migration, it is customary to have a standard data representation that is common for all the systems. This is included in the UDM Standard. The UDM Standard is built with decorators from the list of standard decorators. The list can be found in Standard Decorators. For an import to OTCS, the UDM requires also of a custom part. The UDM Custom consists of at least several OpenText-specific decorators: categories, acl, owner, ownerGroup, openText, etc. Then, depending on the Source System, it may also contain a set of other custom decorators.

The communication with the UDM usually requires a mapping. The extraction from the Source System provides data in a format specific for the Source System. To store data to UDM, a mapping is required that would transform the data into the correct UDM representation. Also, when importing to OTCS, data is being extracted from UDM, transformed and provided in the appropriate format for the import. These operations are flexible and can be extended with custom functionality depending on the source/target system.

Furthermore, as part of the migration process, extra information can be provided in addition to the information extracted from the Source System. For example, when the Source System is a FileShare, the extraction does not include any information regarding permissions, categories or versions. Such information can be provided separately, for example, in an Excel sheet. The additional information has to be mapped to the UDM as well.

Per project, there might be variations in requirements. These requirements can be modeled as business rules and introduced in the migration process either before loading to UDM or immediately before importing to OpenText. If keeping the business rules changes is required for future reference, then the former procedure is recommended.

Prerequisites

In order to successfully run the OpenText migration scripts, the following requirements must be met:

  • The scripts code must be opened in Xill IDE 3.0
  • A MongoDB instance must be running locally on the standard port (127.0.0.1:27017)
  • Access to an OTCS (OpenText Content Server) user environment is required
  • Access to the Content Server API is required; the following REST API urls can be used in Windows: http://{servername}/OTCS/cs.exe/api/v1, http://{servername}/OTCS/llisapi.dll/api/v1/
  • Administrator privileges for the OTCS is required in order to have access to the user account information and object permissions
  • CWS (Content Web Services) has to be setup and and CWS Client Proxies must be created 

Before running any of the import robots, the access credentials to the OpenText Content Server must be set in the"settings.xill" file:

"OTServer": {
            "apiUrl" : "http://{server}/cs.exe/api/v1/",
            "credentials":{
                "username" : "username",
                "password" : "password"
            }
       }

The import starts by recursively filling in one or more OpenText root folders with data from one or more folder locations. Therefore, the entry point(s) for the importer must be specified beforehand in the "settings.xill" file.

"targetSystems" : {
            "OpenText": {
                "name": "name",
                "roots" : {
                    "1111111": [
                            "D:\\Projects\\FileShare"
                    ]
                }
            }
        }

To be able to use the CWS services for permission import, valid user credentials have to be provided in the settings.xill" file.

 "cws" : {
            "cwsUrl" : "http://{server}/cws/",
            "credentials":{
                "username" : "username",
                "password" : "password"
            }
        }

Unified Data Model Setup

The required setup for using UDM depends on the migration project, but it will always follow at least the basic UDM flow: extract data from the Source System, mapping (template) data to UDM content types, load data to UDM, extract data from UDM, mapping (template) data to convert from UDM to target content types and load data to OpenText. As mentioned in the Migration Process section, it is always possible to enrich the UDM with additional metadata, should these metadata be provided.

The content types created in UDM reflect the functionality of both the content types from the source systems as well as the content types from the target systems. Since in a migration to OpenText, any system can be considered a source system and the creation of the content types in UDM is dependent on both the source system and the target system, it is not possible to give in this document valid content types, independent of the source system.

Still, to exemplify how it works, let's consider a migration from FileShare to OpenText. Both the Document content type from OpenText and the File content type from FileShare can be modeled as a Document content type in UDM. The Folder content type from OpenText and the Folder content type from FileShare can be modeled as a Folder content type in UDM.

The Folder content type contains a set of standard decorators that comprises all the FileShare data, but also some custom decorators to model OpenText-specific features. For example:

var openText = {
     "id" : {
          "type" : "NUMBER",
          "required" : false
     },

     "type" : {
          "type" : "STRING",
          "required" : false
     },

     "typeName" : {
         "type" : "STRING",
         "required" : false
     },

     "description" : {
         "type" : "STRING",
         "required" : false
     }
};

var acl = {
    "aclPermissions" : {
        "type" : "LIST",
        "required" : false
    }
};

var owner = {
    "id" : {
        "type" : "STRING",
        "required": false
    },
    "type" : {
        "type" : "STRING",
        "required": false
    },
};

The Document content type contains the custom decorators specified before, but also a categories decorator:

var categories = {
    "categoryList": {
        "type" : "LIST",
         "required" : false
    }
};

Solution

From the available OpenText API's, we have considered the following ones for our solution:

  • OpenText Object Importer
  • XML Import  
  • REST API
  • Content Web Services API 
  • Screen scraping

Since the OpenText Object Importer is a separate OpenText module, we did not use it for import. The XML Import has also been excluded because it is not applicable for parallel import of documents with a hierarchical structure. The screen scraping has not yet been used, but it is something we might consider when it comes to functionality that cannot be imported with any other available OpenText API's.

The solution we have implemented uses the REST API for the most of the import functionality. However, the REST API is not complete enough. To compensate, we use the CWS, that allows the import of permissions to OTCS.

After setting up the Unified Data Model, the following solution steps are followed:

  • Extraction of the content types from the UDM
  • Mapping of UDM content types to the OpenText content types  
  • Load of the mapped content types to the OpenText Content Server 

Each step is implemented in a separate robot. The reason for that is twofold. First, it makes the implementation more readable and easy to follow. Secondly, it is a rather convenient splitting for the multi-threading algorithm that we have implemented. The extraction of the content types from UDM provides the starting point for the import. Once extracted, the UDM content types are mapped to the OpenText content types. This is doen in parallel. The import of the content types to OTCS is also done in parallel. Except for folders and binaries, it is also possible to import versions, permissions, users, groups and categories.

In what regards performance improvement, there are several steps we have considered. We have started with a single server to run XILL and a single OpenText Content Server. At the beginning, only one request at a time was sent from XILL to OpenText. The first performance improvement was to send multiple requests to OTCS in parallel. Since OTCS needs time to process each request, when sending multiple requests in parallel, the OTCS may not be able to process all the requests quick enough. The solution was to add a new content server to mirror the first one and perform import to both servers at the same time. Also, multiple XILL instances can be added to ensure that XILL sends enough requests to the content server(s).

Closing remarks

The import connector described in this document provides an easily configurable, fast and powerful solution for importing content to OTCS (OpenText Content Server) environments. However, it contains (on purpose) restricted functionality when it comes to processing information contained within non-standard OpenText objects, such as Agendas, Channels, Discussions, Task Lists  etc. The processing of these objects has to be tailored accordingly with the different source systems.