Configuration Files - Payload Definition

03 December 2018

1. Payload Definition

A Payload definition defines what data to be extracted from the ENOVIA/3DExperience database and how this data is structured and how it can be validated. The extracted data is called "payload" and used when transferring data to another system.

The payload definition itself is an XML file containing the rules for this. This configuration file is stored inside a directory called "payload" below the "cfg" directory.

1.1. Payload / Fractions

The generated payload consist of at least one fraction (or section) where each is generated by different configuration settings.

You may for example have one fraction whose data is generated by applying a table (defining the select statements) on a set of business object and/or connections, another fraction generated by executing some MQL statement and a third (or more) fraction generated by using custom Java code.

Below is a image illustrating this.

Overview

Fraction 1 in the image above uses a data-set to define what objects to be included in the fraction. The table defines what metadata to be extracted from ENOVIA/3DExperience.

The result in the final Payload is a section wrapped inside the "Documents" element.

There exists many possibilities to impact the generated XML. You may for example provide additional so called "XML Specification" into the Payload definition that changes the raw XML extract from ENOVIA/3DExperience.

Finally, you may transform the generated payload for example by using a configured stylesheet (XSLT template).

1.2. Configuration Details

The root element of the payload definition is "Payload".

The allowed attributes on the <Payload> element are shown in the table below.

Attribute Required Default Description

omitRootElement

No

False

Can be used to omit generating the root element. Note that this is only possible if the Payload definition contains one fraction. Otherwise, the generated XML payload data would be invalid.

rootElement

No

payload

Can be used to configure the root element name

contentType

No

text/xml

Can be used to change the output content type

encoding

No

According to configured encoding, typically utf-8

The available child elements are listed in the table below.

Element Description

<JavaContent>

Defines a Java class that produces some content. The class must implement the interface com.technia.tif.enovia.payload.PayloadFraction.

The name of the class is provided within the attribute className or in the body of this element (between start/end of element).

<TableContent>

Defines a dataset together with a table to define what meta-data from the business objects and/or connections to be part of the payload.

The data set is provided within the child element called <DataSet> and the table is provided within the child element called <Table>.

See Table Content for more information.

<MQLContent>

Can be used to execute arbitrary MQL code. The code to be executed is defined between the start/end tag. By default, TIF assumes that the output from your MQL command is XML data. If that is not the case, you can use some additional configuration options.

<MQLContent
    xmlOutput="false"
    splitBy="\n" <!-- split on cr -->
    outerElement="name_of_outer_element"
    itemElement="name_of_element_holding_each_value">...</MQLContent>

<InputContent>

Can be used to include input payload that is provided to the job.

See this chapter for more information.

<Template>

Used for allowing a template to produce some content as a part of the payload.

<Payload>
    <Template>tvc:template/Test.ftl</Template>
</Payload>

<XMLSpec>

Defines the default XML specification (rules) how the extracted data should be formatted.

Note that each <TableContent> fraction may define its own rules.

See chapter below

<Transformer>

Defines how to transform the generated payload data.

You can declare multiple transformer elements if needed. Each will be ran in a pipeline with the output from the previous transformer as input to the current.

See <<Transformer,chapter> below for details

<Validator>

Defines an XML schema file to be used for validation of the final payload data.

The value of the validator element is a reference to a file of type “xsd”.

See chapter below for details.

1.3. Conditionally Include/Exclude Fraction

The payload fractions JavaContent, TableContent, MQLContent and Template, all supports the attributes if, unless and filter.

The filter attribute may be used to point our a Java class of type com.technia.tif.enovia.payload.InclusionEvaluator that will be used to decide if the fraction should be part of the payload or not.

Specifying the filter attribute causes the if/unless attributes to have no effect.

The if and/or unless attribute can be used to specify a parameter, or a comma separate list of parameters that either should be present (if) or not-present (unless) in order for the fraction to be included in the final payload.

1.4. Table Content

A table driven payload fraction uses a table (or system table) together with a data-set.

Element Description Example Required

<Table>

Points out the table (XML defined, or System table) that defines the columns (with selectables) that defines what meta data to be extracted from ENOVIA/3DExperience.

You can point out a table in different ways, see below for some different examples.

<Table ref="tvc:table/NameOfTable.xml" />
<Table>tvc:table:namespace/NameOfTable.xml</Table>
<Table>Name of System Table</Table>

No, but recommended.

If not defined a default table is used that will include all basic information, all attributes and all file information from the objects.

This might result in bad performance or too much data being extracted.

<DataSet>

Points out the data-set containing the definition how to find/query/filter out objects being part of this payload fraction.

 
<DataSet ref="tvc:dataset/NameOfDataSet.xml" />
<DataSet>tvc:dataset:namespace/NameOfDataSet.xml</DataSet>
<DataSet>NameOfDataSet.xml</DataSet>

Note that the first and last example are equivalent.

The dataset also supports the attribute cacheResult, which may be used to reuse the output from the data-set if you use the same data-set across multiple fractions.

<DataSet cacheResult="true" ref="tvc:dataset/NameOfDataSet.xml" />

The default value for cacheResult is FALSE. If the value is false, the result will be removed from the cache. Example: The same data set is declared in 3 different fractions. You only set the cacheResult to TRUE on the first one. This will cause the data set to be evaluated on the first and on the third fraction.

No, but recommended.

If not defined, the objects, which this payload fraction is operating upon, will use the business object(s) as given as input to the payload generator.

<XMLSpec>

Defines XML specification (rules) how the extracted data should be formatted.

See this page

No. If not defined, the default XML format is used.

<XMLSerializer>

May be used to point out a custom XML serializer. The class you point out must implement the interface com.technia.tif.enovia. payload.tvc.xml.XMLSerializer

 
<XMLSerializer className="com.acme.integrations.xml.MyXMLSerializer"/>

or

<XMLSerializer>com.acme.integrations.xml.MyXMLSerializer</XMLSerializer>

No. If not defined, the default XML serializer is used.

<Transformer>

May be used to point out for example an XSLT stylesheet used for transformation of the generated XML.

You can apply multiple transformer elements.

 
<Transformer>MyStylesheet.xsl</Transformer>

or

<Transformer xslt="MyStylesheet.xsl" />

No.

<Template>

May be used to point out a template (Freemarker) to be used for generating the data.

See the chapter Template below

 
<Template>MyTemplate.ftl</Template>

or

<Template>tvc:template:namespace/MyTemplate.ftl</Template>

No.

The following table lists additional attributes that may be used on the <TableContent> element:

Attribute Description Example Required

outerElement

May be used to define the element that will group the table-content fraction.

 
<TableContent outerElement="Documents">
    <DataSet>...</DataSet>
    <Table>...</Table>
    <Template>...</Template>
    <Transformer>MyTransformer.xsl</Transformer>
</TableContent>

No

paginationSize

Defines a pagination size that results in that data is retreived in chunks from the database. May be useful for larger datasets in order to reduce footprint, however, the performance will be slightly degraded since more DB calls will be made.

Default is -1, e.g. disabled

 
<TableContent paginationSize="100">
    ....
</TableContent>

No

sortingDisabled

May be used to disable the default sorting according to the table column definitions. May be used to increase performance for larger data-sets.

Default is false, e.g sorting is enabled according to Table Column definition(s)

 
<TableContent sortingDisabled="true">
    ....
</TableContent>

No

1.5. Input Content

Input content fraction includes input payload that is provided to the job. Typically this fraction can be applied when the job is initiated via <NewJob> event handler by other job that provides some input.

The following table lists additional sub elements that may be used on the <InputContent> element:

Element Description Example Required

<Transformer>

May be used to point out a transformer, such as an XSLT stylesheet, that can be used for transformation of the input.

Note that you can use multiple transformers if needed.

 
<Transformer xslt="MyStylesheet.xsl"/>

or

<Transformer type="json" />

No

1.6. Template

The traditional way of producing data in a payload is to generate the XML with or without using a so called <XMLSpec> to configure the generated XML.

In addition, one usually need to implement a XSLT stylesheet to transform the XML data into the final format.

An alternative approach is to use a template for this.

TIF bundles the Apache Freemarker library, which is a template engine that is capable of generating output based upon data in an object model or model.

The model is data that is passed in to the template, e.g. the extracted data for a <TableContent> fraction and other parameters passed into the TIF job, which the payload is generated for.

Example configuration of a Payload using a single Table Content fraction with a template:

<Payload>
    <TableContent>
        <Table>tvc:table/Test.xml</Table> (1)
        <Template>tvc:template/Test.ftl</Template> (2)
        <Transformer>tvc:xslt/Test.xsl</Transformer> (3)
    </TableContent>
</Payload>
1 Specify what table to be used
2 Specifies the template to be used
3 OPTIONAL: If your template produces XML data, you can still use a transformer to convert the data into another format.

An example Freemarker template is shown below:

<#ftl output_format="XML">
<objects>
    <#list data.rows as row>
        <row>
            <#list row.cells as cell>
                <cell name="${cell.column.name}">
                    <#list cell.values as cellValue>
                        <value>${cellValue.value}</value>
                    </#list>
                </cell>
            </#list>
        </row>
    </#list>
</objects>

For complete details regarding template authoring, please visit http://freemarker.org/docs/ref.html

The data model passed into the template engine are:

paginated

A boolean indicating if the template will be called multiple times due to pagination

pageCount

The number of pages

currentPage

The current page

paramMap

The parameters passed to the Job. The data-type for this object is Map<String, String[]>

rpe

ENOVIA/3DExperience RPE parameters from when the Job was initiated. The data type for this object is Map<String, String>

data

The extracted data. Contains the following methods

The elements <XMLSpec> and <XMLSerializer> has no meaning when a template is being defined.

1.7. MQLContent

The MQL content fraction may be used to include output from a MQL command into the generated XML. Your MQL command may for example execute some other program declared in the database, or use some built-in MQL commands for queries against the ENOVIA/3DExperience database.

The <MQLContent> element is defined as illustrated below:

<MQLContent
    xmlOutput="true | false"
    splitBy="\n"
    outerElement="name_of_outer_element"
    itemElement="name_of_element_wrapping_each_item_splitted_on">

    put the MQL code here
</MQLContent>

The attributes available on the <MQLContent> element are described in the table below:

Attribute Default Description

xmlOutput

true

Defines if the MQL command returns XML formatted code.

splitBy

N/A

If the output should be splitted upon some character, you may specify that here.

You may use "\n", "\r" and "\t" to specify new-line, carriage-return or tab.

outerElement

Defines the element, which the output from this fraction should be wrapped inside.

itemElement

item

If you use the splitBy feature, each item in the split-array will be encapsulated with the specified element.

txType

Defines the type of transaction used when executing the MQL command.

Possible values are:

  • inactive

  • update

  • read

executeAsSuperUser

false

If true, the MQL command will be executed with admin privileges (super user).

1.8. JavaContent

Using a custom Java class for extraction of a payload fraction might be useful in some cases.

In the payload configuration, you just declare it like this:

<Payload>
    <JavaContent>com.acme.integrations.payload.ECODetails</JavaContent>
</Payload>

or

<Payload>
    <JavaContent className="com.acme.integrations.payload.ECODetails"/>
</Payload>

And the class you point out should look like this:

package com.acme.integrations.payload;

import com.technia.tif.enovia.payload.PayloadFraction;
import com.technia.tif.enovia.payload.PayloadFractionContext;
import com.technia.tif.core.ExtractionException;

public class ECODetails implements PayladFraction {
    public void createData(PayloadFractionContext ctx) throws ExtractionException {
        Writer w = ctx.getWriter();
        // ... implement logic here ...
    }
}

You only need to implement one method called "createData".

1.9. Transformer

The transformer element defines how to transform the data.

You can combine different type of transformations by providing multiple <Transformer> elements.

On the <Transformer> element you can use the attribute "xslt" to point out an XSLT stylesheet that you use to transform the source XML data with.

TIF includes Saxon-HE 9.7 that "provides a basic XSLT 2.0 processor as defined in section 21 of the XSLT 2.0 Recommendation".

Example:

<Payload>
    ...
    <Transformer xslt="tvc:xslt:namespace/MyStylesheet.xslt"/>
</Payload>

For backward compatibility we also support specifying the XSLT stylesheet as body text of the Transformer element.

<Payload>
    <Transformer>tvc:xslt:namespace/MyStylesheet.xslt</Transformer>
</Payload>

The stylesheets should be put inside a directory called "xslt" below the "cfg" directory.

You may omit the "tvc:xslt" prefix. For example just specifying xslt="MyStylesheet.xslt" implies referring to "tvc:xslt/MyStylesheet.xslt".

Information passed as Parameters or RPEs to the job can be used during the transformation. The prefix for parameter is job.parameter and job.rpe for RPEs.

Example where the transactionId is sent as a parameter to the job and written to the XML:

<xsl:stylesheet ...>
	<xsl:param name="job.param.transactionId" />
	<xsl:template name="Part">
		<part>
			<transactionId>
				<xsl:value-of select="$job.param.transactionId" />
			</transactionId>
		</part>
		...
	</xsl:template>
</xsl:stylesheet>

1.9.1. JSON Transformer

XML payload can be transformed to JSON by specifying attribute type with value json.

An example:

<Payload>
    ...
    <Transformer type="json">
        <!-- Optional properties -->
        <Property ... />
    </Tranformer>

</Payload>

The transformer is based on XSLTJSON XSLT stylesheet version 1.0.93. Refer the documentation for supported properties that can be used to control the stylesheet.

1.9.2. Custom Transformer

You may also implement a custom Java class that transforms the payload. The class is defined with attribute className. Optionally, one or more properties can be supplied with <Property> child element.

An example configuration:

<Payload>
    ...
    <Transformer className="com.acme.MyCustomTransformer">
        <Property name="First Property" value="Value 1" />
        <Property name="Second Property" value="Value 2" />
    </Transformer>

</Payload>

The class must implement the interface com.technia.tif.core.transform.Transformer.

The definition is as follows:

package com.technia.tif.core.transform;

public interface Transformer {

    void transform(TransformContext ctx) throws TransformerException;

}

Example: Excel Transformer

See the source code: POIExcelTransformer.java.

The class transforms flat, unformatted XML payload to an Excel XLSX format.

Example payload configuration for executing the transformer class:

<Payload>
	<TableContent>
        <!-- NOTE: Table columns must have <Label> configured -->
        <Table>tvc:table/MyTable.xml</Table>
	</TableContent>
    <Transformer className="com.acme.tif.transform.POIExcelTransformer" />
</Payload>
The example is based on Apache POI version 3.17.

1.10. Validator

After the data has been extracted and optionally transformed by your transformer (XSLT stylesheet) you may want to validate the XML data against a schema in order to ensure the validity of the payload.

You may specify a Validator within the payload configuration as shown below:

<Payload>
    ...
    <Validator schema="tvc:xsd/MySchema.xsd"/>

    OR

    <Validator>tvc:xsd/MySchema.xsd</Validator>
</Payload>

The XML Schema files should be put inside a directory called "xsd" below the "cfg" directory.

You may omit the "tvc:xsd" prefix. For example just specifying schema="MySchema.xsd" refers to "tvc:xsd/MySchema.xsd".

1.10.1. Custom Validator

You may also implement a custom Java class that validates the payload. The class is defined with attribute className. Optionally, one or more properties can be supplied with <Property> child element.

An example configuration:

<Payload>
    ...
    <Validator className="com.acme.MyCustomValidator">
        <Property name="First Property" value="Value 1" />
        <Property name="Second Property" value="Value 2" />
    </Validator>

</Payload>

The class must implement the interface com.technia.tif.core.validation.Validator.

The definitions are as follows:

package com.technia.tif.core.xml;

import com.technia.tif.core.ValidationException;

public interface Validator {

    void validate(ValidationContext ctx) throws ValidationException;
}

Example: JSON Validator

See the source code: JustifyJSONValidator.java.

The class validates JSON payload against a JSON schema file that is defined in property "schema".

Example payload configuration for executing the validator class:

<Payload>
	<TableContent>
		...
	</TableContent>
    <!-- Transforms XML payload to JSON -->
    <Transformer type="json" />
    <Validator className="com.acme.tif.validation.JustifyJSONValidator">
        <!-- Point out JSON schema file -->
        <Property name="schema" value="tvc:json/MySchema.json"/>
    </Validator>
</Payload>
The example is based on Justify JSON validator version 0.10.0.

1.11. Content Type and Encoding

It is possible to explicitly configure content type and encoding of payload configuration. This might be useful for example when using a custom Java class for extraction of a payload fraction.

The following table lists optional attributes that may be used on the <Payload> element:

Attribute Description Example

contentType

May be used to define the content type of the payload.

 
<Payload contentType="application/json">
    ...
</Payload>

encoding

May be used to define the encoding of the payload.

 
<Payload encoding="UTF-8">
    ...
</Payload>

When applying XSLT transformer using <Transformer> element, content type and encoding can be overridden in XSLT with <xsl:output> element. See the following table:

Attribute Description Example

media-type

May be used to define the content type of the transformed payload.

 
<xsl:stylesheet>
    <xsl:output media-type="application/json" />
    ...
</xsl:stylesheet>

encoding

May be used to define the encoding of the transformed payload.

 
<xsl:stylesheet>
    <xsl:output encoding="UTF-8" />
    ...
</xsl:stylesheet>