Configuration Files - Dataset Definition

1. Dataset Definition

The dataset feature allows an administrator to define complex data retrievals within an XML configuration file.

The result from a dataset can either be a structure consisting of objects and connections or a flat list of objects (each row in a flat list can also be associated with a connection).

A data set is defined in XML and is stored as a resource below the ${TIF_ROOT}/modules/enovia/cfg folder within your TIF installation.

The type of this resource is called dataset. Below is a table showing some examples of the paths and their corresponding resource name to a particular data set.

File System Path	Resource Name
${TIF_ROOT}/modules/enovia/cfg/dataset/MyDataSet.xml	tvc:dataset/MyDataSet.xml
${TIF_ROOT}/modules/enovia/cfg/tvx/enc/dataset/MyDataSet.xml	tvc:dataset:tvx:enc/MyDataSet.xml
${TIF_ROOT}/modules/enovia/cfg/dataset/test/MyDataSet.xml	tvc:dataset/test/MyDataSet.xml
${TIF_ROOT}/modules/enovia/cfg/tvx/enc/dataset/test/MyDataSet.xml	tvc:dataset:tvx:enc/test/MyDataSet.xml

File System Path

Resource Name

${TIF_ROOT}/modules/enovia/cfg/dataset/MyDataSet.xml

tvc:dataset/MyDataSet.xml

${TIF_ROOT}/modules/enovia/cfg/tvx/enc/dataset/MyDataSet.xml

tvc:dataset:tvx:enc/MyDataSet.xml

${TIF_ROOT}/modules/enovia/cfg/dataset/test/MyDataSet.xml

tvc:dataset/test/MyDataSet.xml

${TIF_ROOT}/modules/enovia/cfg/tvx/enc/dataset/test/MyDataSet.xml

tvc:dataset:tvx:enc/test/MyDataSet.xml

2. XML Format

The root element of the XML definition for a dataset is <DataSet>.

Below the root element, you will declare a so called loader.

There exist a number of different loaders, some of them require input, which either comes from the caller such as object-ids provided in request parameters or from a child loader, while some loaders doesn’t require input.

The table below describes the built-in loaders together with a short description and reference to the chapter where they are described in more detail.

Loader XML Element Description Requires Input

Loader	XML Element	Description	Requires Input
Expand	<Expand>	Performs an expansion. The way the expansion is done, is configured through child elements.	Yes
Inquiry	<Inquiry>	Executes an inquiry. Either the inquiry is defined inline in the XML, or you reference an existing inquiry.	No
Query	<Query>	Performs a temp query. The query parameters are configured as child elements.	No
Java Program Object	<JPO>	Executes a Java Program Object.	No
Select	<Select>	Performs a select to retrieve the desired objects.	Yes
Flatten	<Flatten>	Converts a structure into a flat list. E.g. removes all hierarchy information	Yes
End Items	<EndItems>	Removes all nodes except the leafs from a structure.	Yes
Remove Duplicate Objects	<RemoveDuplicates>	Ensures that each object only appears once.	Yes
Filter	<Filter>	Performs a filtering of the rows based upon an expression.	Yes
Java	<Java>	Defines a loader implemented in Java	No
Source	<Source>	Explicitly obtains the objects provided by the client; typically the values either from the `emxTableRowId` parameter or the `objectId` parameter.	No
PersonObject	<PersonObject>	Gives the object-id of the person object associated with the current user.	No
CompanyObject	<CompanyObject>	Gives the object-id of the company object associated with the current user.	No
RemoveRoots	<RemoveRoots>	Removes the root nodes from a structure.	Yes
DataSetRef	<DataSetRef>	Includes the content from another data-set	No
Structure	<Structure>	Converts a flat list of objects into a structure, where the root nodes of the structure originate from the rows in the flat list.	Yes
Union	<Union>	Combines the result from several loaders into one flat list OR one structure.	Yes
Intersection	<Intersection>	Takes the id’s that are present in all child loaders.	Yes
Complement	<Complement>	Loads data from one or more child loaders, and exclude data by defining other child loaders that defines what to exclude.	Yes
DoPrivileged	<DoProvileged>	Runs the child loaders with system privileges	Yes
LatestInState	<LatestInState>	Finds the latest revision of the object in a particular state.	Yes

Expand

Performs an expansion. The way the expansion is done, is configured through child elements.

Yes

Inquiry

Executes an inquiry. Either the inquiry is defined inline in the XML, or you reference an existing inquiry.

Query

<Query>

Performs a temp query. The query parameters are configured as child elements.

Java Program Object

<JPO>

Executes a Java Program Object.

Select

Performs a select to retrieve the desired objects.

Yes

Flatten

Converts a structure into a flat list. E.g. removes all hierarchy information

Yes

End Items

Removes all nodes except the leafs from a structure.

Yes

Remove Duplicate Objects

Ensures that each object only appears once.

Yes

Filter

Performs a filtering of the rows based upon an expression.

Yes

Java

<Java>

Defines a loader implemented in Java

Source

Explicitly obtains the objects provided by the client; typically the values either from the emxTableRowId parameter or the objectId parameter.

PersonObject

Gives the object-id of the person object associated with the current user.

CompanyObject

Gives the object-id of the company object associated with the current user.

RemoveRoots

Removes the root nodes from a structure.

Yes

DataSetRef

Includes the content from another data-set

Structure

Converts a flat list of objects into a structure, where the root nodes of the structure originate from the rows in the flat list.

Yes

Union

<Union>

Combines the result from several loaders into one flat list OR one structure.

Yes

Intersection

Takes the id’s that are present in all child loaders.

Yes

Complement

Loads data from one or more child loaders, and exclude data by defining other child loaders that defines what to exclude.

Yes

DoPrivileged

Runs the child loaders with system privileges

Yes

LatestInState

Finds the latest revision of the object in a particular state.

Yes

3. Nesting of Loaders

A loader might require input, while some doesn’t. The loaders that requires input can either use input from a loader defined as a child element OR it can use the input as provided by the client (typically the id of the object, which the integration job operates upon).

Below is an example how to nest the loaders:

<DataSet>
    <RemoveDuplicates>
        <Filter usesBusinessObject="true">
            <Expression>attribute[Unit of Measure] == kg</Expression>
            <EndItems>
                <Expand>
                    <Inquiry>
                        <Name>tvc:inquiry/MyInquiry.xml</Name>
                    </Inquiry>
                </Expand>
            </EndItems>
        </Filter>
    </RemoveDuplicates>
</DataSet>

Note that this example is not complete; some configuration details has been left out for readability purpose.

What this example actually does is:

Use an inquiry to load some objects
The objects loaded from the inquiry is expanded according to the Expand definition
All rows except the end items within the structure is removed
A filter is applied to the objects, which will remove all objects that doesn’t fulfil the filter-expression
Finally, all objects appearing more than once are removed.

This illustrates how to accomplish a quite complex data-retrieval just by using a combination of different loaders. If you would have implemented this code as for example a JPO, you would typically need to implement +100 rows of Java code.

3.1. Expansion

A complete definition how to expand a structure is shown below. Note that some elements are optional.

<DataSet>
    <Expand>
        <From>true</From>
        <To>false</To>
        <Depth>1</Depth>
        <TypePattern>
            <Type>type_Part</Type>
            <Type>...</Type>
        </TypePattern>
        <RelationshipPattern>
            <Relationship>relationship_SparePart</Relationship>
            <Relationship>...</Relationship>
        </RelationshipPattern>
        <ObjectWhere>...expression...</ObjectWhere>
        <RelWhere>...expression...</RelWhere>
        <LoadRootInquiry>
            name of inquiry that provides the root objects
        </LoadRootInquiry>
    </Expand>
</DataSet>

Setting the depth to 0, you will expand as deep as possible. This can only be done in one direction.

Note that the LoadRootInquiry element can be used to specify an inquiry that loads the root nodes. This would be the same as doing:

<Expand>
    <Inquiry>
        <Name>name of inquiry</Name>
    </Inquiry>
</Expand>

The difference is that the first approach will execute slightly more efficient.

3.2. Inquiry

Executing an inquiry can be done in two different ways. Either you will reference an existing inquiry by its name, or you can define the inquiry inline. Below are two examples illustrating both ways.

<Inquiry>
    <Name>name of inquiry to run</Name>
</Inquiry>

Alternatively, you can also say:

<Inquiry name="name of inquiry"/>

Or by defining the inquiry inline:

<Inquiry>
    <Code>temp query bus Part * * limit 100</Code>
    <Pattern>*|*|*|${OID}</Pattern>
    <Format>${OID}</Format>
</Inquiry>

3.3. Query

A complete definition how to perform a query is shown below. Note that some elements are optional.

<Query>
    <ExpandType>true</ExpandType> <!-- whether or not to find sub-types -->
    <FindLimit>0</FindLimit> <!-- the find limit -->
    <TypePattern>
        <Type>type_Part</Type>
        <Type>type_AnotherType</Type>
    </TypePattern>
    <NamePattern>
        <Name>A*</Name>
        <Name>B*</Name>
    </NamePattern>
    <RevisionPattern>
         <Revision>A</Revision>
         <Revision>B</Revision>
         <Revision>C</Revision>
    </RevisionPattern>
    <OwnerPattern>
        <Owner>A user</Owner>
    </OwnerPattern>
    <VaultPattern>
        <Vault>A vault</Vault>
        <Vault>Another vault</Vault>
    </VaultPattern>
    <Where>$&lt;attribute[attribute_Weight]&lt; &gt; 100</Where>
</Query>

3.4. JPO

The JPO loader can be used to invoke a JPO. The example below illustrates how to configure such loader.

<JPO>
    <Name>MyJPO</Name>
    <Method>myMethod</Method>
</JPO>

Alternatively, you can also say:

<JPO name="MyJPO" method="myMethod"/>

The JPO method must return a MapList where each Map in the list contains the id and optionally the id[connection] key.

3.5. Select

The select loader selects id’s based upon the input to the select loader. Below is an example how to do so:

<Select>
    <Statement>from[Documents].to.revisions.id</Statement>
    <Statement>from[Specs].to.revisions.id</Statement>
</Select>

By default, the select loader assumes that you select on business objects, and the select returns other business object id’s. If you select on relationships, or your select clause returns connection id’s, see below for details.

In this case, the select is done upon the implicit input objects as provided by the client. You can use a child-loader also; in that case, the select is based upon the objects provided by that loader.

it is possible to have several statements. Each statement must of course return object-ids.

If you have select statements that operates on connection id’s; then you can specify so as shown below:

<Select selectFromConnections="true">
    <Statement>to.id</Statement>
</Select>

If your select statement would return connection id’s instead of object id’s, then you can specify so as shown below:

<Select selectReturnsConnections="true">
    <Statement>to.id</Statement>
</Select>

If you only have one select statement, this can be written as:

<Select statement="from[Documents].to.id"/>

3.6. Flatten

The flatten loader doesn’t have any configuration options. The only requirement is that the input to this loader is a structure; if not, nothing will be done by this loader. Example:

<Flatten>
    <Expand>
        ...
    </Expand>
</Flatten>

3.7. End Items / Leafs

The end items loader doesn’t have any configuration options. The only requirement is that the input to this loader is a structure; if not, nothing will be done by this loader.

Example:

<EndItems>
    <Expand>
        ...
    </Expand>
</EndItems>

3.8. Remove Duplicates

The remove duplicates loader requires input, either a structure or a flat list. In case of a structure, the removal of duplicates is only made on the root-nodes in the structure.

Example:

<RemoveDuplicates>
    <EndItems>
        <Expand>
            ...
        </Expand>
    </EndItems>
</RemoveDuplicates>

If you need to retain occurrences of the same object if the connection id differs, specify so using the attribute "ignoreConnectId" as shown below:

<RemoveDuplicates ignoreConnectId="false">
    <EndItems>
        <Expand>
            ...
        </Expand>
    </EndItems>
</RemoveDuplicates>

3.9. Filter

The filter loader can be used to filter out objects based upon an expression. This loader requires input. The example below expands a structure, makes it flat, then applies the filter and removes all the rows not matching this filter.

<Filter>
    <Expression>$&lt;attribute[attribute_Classification]&gt; == ABC</Expression>
    <Flatten>
        <Expand>
            ...
        </Expand>
    </Flatten>
</Filter>

Alternatively, you can also say:

<Filter expression="$&lt;attribute[attribute_ABC]&gt; == 'DEF'"/>

Note the following:

The expression is evaluated using the ENOVIA select statement evaluate[${EXPRESSION}]. All rows returning TRUE will be accepted, others will be filtered out.
If you need to combine criteria using AND/OR operators; you might in some cases need to use the AND or OR key words instead of using the && and || operators. This is due to a bug/behaviour of ENOVIA.

For example the expression shown below:
```
<Expression>attribute[Attribute 1] == Value1 || attribute[Attribute 2] == Value2</Expression>
```
Might need to be rewritten as shown below:
```
<Expression>attribute[Attribute 1] == Value1 OR attribute[Attribute 2] == Value2</Expression>
```
E.g. the operators "||" and "&&" could be changed to "OR" and "AND".

To verify your expressions, you can from MQL do:
```
<MQL> pri bus T N R select evaluate[attribute[Attribute 1] == Value1];
```
If your expression should be evaluated against the connections instead of over the business objects, which is the default, then you can say so as shown below:
```
<Filter usesBusinessObject="false">
```

3.10. Java

To use a custom loader within a data-set that is implemented in Java, this can be done through the example below:

<Java className="com.acme.MyDataLoader"/>

The class specified must implement one of the interfaces below:

com.technia.tvc.core.db.dataset.loader.DataLoader
com.technia.tvc.core.db.dataset.loader.DataLoaderWithInput

If your custom Java class supports load by a child loader, you need to implement the DataLoaderWithInput interface.

3.11. Source

The source objects as provided by the client through for example request parameters will be used by the data-loaders that require input, but doesn’t have any input configured in the data-set XML file through any child loaders, implicitly.

To explicitly define that you will use the source objects, then you can define this in the Data set XML file like shown below:

<Filter expression="...">
    <Source/>
</Filter>

3.12. Person Object

The id of the person object associated with the current user can in some cases be useful to start from when for example expanding a structure. Below is an example how to use it:

<EndItems>
    <Expand>
        ...
        <PersonObject/>
    </Expand>
</EndItems>

This example will expand the person object and finally filter out all rows in the structure except for the "end items".

3.13. Company Object

The id of the company object associated with the current user can in some cases be useful to start from when for example expanding a structure. Below is an example how to use it:

<EndItems>
    <Expand>
        ...
        <CompanyObject/>
    </Expand>
</EndItems>

This example will expand the company object and finally filter out all rows in the structure except for the "end items"

3.14. Remote Roots

In some cases when you have expanded a structure, you only want to remove the root nodes. This can be accomplished as shown below:

<RemoveRoots>
    <Expand>
        ...
    </Expand>
</RemoveRoots>

The <RemoveRoots> element doesn’t have any configuration options.

3.15. Dataset Reference

A dataset might use the outcome from another dataset. This can be accomplished as shown below:

<DataSetRef name="tvc:dataset/MyDataSet.xml">

3.16. Structure

In conjunction with the Flatten loader, which converts a structure into a flat list, you can use the Structure loader to convert a flat list into a structure.

Each row in the flat list used as input to this loader will be set as root node in the returned structure. Note that if the input to this loader is a structure, no change is made.

You can configure this loader with some additional attributes:

ignoreRelIds

A Boolean specifying if to exclude any relationship information as provided by the child loader.

Default is false.

Example:

<Structure ignoreRelIds="false">
    <Query>
        ...
    </Query>
</Structure>

3.17. Union

To combine the result from several loaders, you can use the "Union" loader. This can be accomplished as shown below:

<DataSet>
    <Union>
        <DataSetRef name="tvc:dataset/Example.xml"/>
        <Query>
            <TypePattern>
                <Type>type_SomeType</Type>
            </TypePattern>
            <FindLimit>10</FindLimit>
            <Where>current == Obsolete</Where>
        </Query>
        <Query>
            <TypePattern>
                <Type>type_AnotherType</Type>
            </TypePattern>
            <FindLimit>10</FindLimit>
            <Where>current == Create</Where>
        </Query>
        <Query>
            <TypePattern>
                <Type>type_ThirdType</Type>
            </TypePattern>
            <Where>current == Release</Where>
            <FindLimit>10</FindLimit>
        </Query>
    </Union>
</DataSet>

In this example, we combine the result from three queries and the result from a referenced data-set.

If any of the loaders contains structural data, the outcome of this loader becomes a structure.

To force flat list, you can use the attribute "forceFlat" on the <Union> element and set the value to true. E.g.

<DataSet>
    <Union forceFlat="true">

3.18. Intersection

This loader will return the ID’s that are part of all child loaders. Example:

<DataSet>
    <Intersection>
        <DataSetRef name="tvc:dataset/First.xml"/>
        <DataSetRef name="tvc:dataset/Second.xml"/>
    </Intersection>
</DataSet>

The result from the example below will contain the objects that are in both the results from the specified data-sets.

By default, this loader uses the object-id’s. However, you can configure it to use the relationship id’s instead. This is accomplished as shown below.

<DataSet>
    <Intersection useRelationship="true">

3.19. Complement

To exclude data from a loader, you can use the Complement-loader. It allows you to define nested loaders, which defines the maximum objects/connections to be included. From this maximum set, you can exclude objects/connections.

See the example below which queries the system for all Person objects, but excludes the one representing the current user.

<Complement>
    <Include>
        <Query>
            <TypePattern>
                <Type>type_Person</Type>
            </TypePattern>
        </Query>
    </Include>
    <Exclude>
        <PersonObject/>
    </Exclude>
</Complement>

Both the Include and Exclude elements accepts multiple loaders.

3.20. DoPrivileged

By default, any data loader within a data set is run with the same context (credentials) as the user who is logged in. In some cases, you might want to run a query as a super user in order to by-pass any access checks that are expensive to evaluate.

To do so, you can accomplish this as shown below:

<DataSet>
    <DoPrivileged>
        <Query>
            ...
        </Query>
    </DoPrivileged>
</DataSet>

3.21. Latest In State

This loader is used to get only the latest revision of the object which is in the specific state. For example this can be used to only get the latest "Released" revision of a part. Example of this scenario.

<DataSet>
    <LatestInState>
        <State>Released</State>
        <Query>
            <TypePattern>
                <Type>Part</Type>
            </TypePattern>
            ...
        </Query>
    </LatestInState>
</DataSet>