TIF ENOVIA/3DExperience Connector - Transfer Data
An integration job that will extract data from the ENOVIA/3DEXPERIENCE database and send it to some destination is called "Transfer Data Job".
To define this kind of job, you need to create a job definition/configuration that defines first how to extract the data (e.g. the payload) but also define the destinations, which the data is transferred to.
Transfer data jobs support job events for handling errors etc. Read more in the "Job Event" chapter.
Before digging into the details of the configuration format for such a job, we will start by showing an example of how such a configuration can look alike.
<Job>
<TransferData>
<Payload>tvc:payload/PartReleased.xml</Payload>
<Destinations>
<SysOut/>
<File id="fileDest1"/>
<Http id="httpDest1"/>
<RabbitMQ id="rabbitMQ1">
<RetryAttempts>10</RetryAttempts>
<RetryDelay>5000,10000,20000,30000,60000</RetryDelay>
<Header name="type" value="${job.rpe.TYPE}" type="string"/>
<HeaderProvider>name of class implementing com.technia.tif.enovia.job.destination.HeaderProvider</HeaderProvider>
</RabbitMQ>
</Destinations>
</TransferData>
<Events>
<Error>
<SendMail>
<TO>...</TO>
<CC>...</CC>
<Subject>...</Subject>
<Message>
message...
${STACK_TRACE}
</Message>
</SendMail>
</Error>
</Events>
</Job>
The root tag in the definition is always <Job>
. The first child element of the Job element defines the kind of job, in this case we will use the <TransferData>
element to define the integration job.
Below the TransferData element, the following child elements are available.
<Payload>
-
Defines the name of the payload definition that defines how the payload is being created.
<Destinations>
-
Defines the destinations, to which the payload will be transferred.
<TransactionType>
-
Defines the ENOVIA/3DEXPERIENCE transaction type to be used during the job.
<WithContext>
-
Optional configuration for specifying what user to perform operation as
Each of these elements are defined in more detail in the sub-pages.
Payload
The Payload element points out the configuration that defines how to generate the payload (e.g. the data) from ENOVIA/3DEXPERIENCE.
The value of the Payload element is either the XML configuration of type "payload" OR a Java class. See below for the two alternatives to define this.
Referencing a Payload Definition
Refer to payload configuration that contains the rules how to generate the payload.
<Job>
<TransferData>
<Payload>tvc:payload:namespace/ReleasedPart.xml</Payload>
</TransferData>
</Job>
The referenced Payload configuration is the file cfg/namespace/payload/ReleasedPart.xml
.
See page Payload Definition for details on how to define the payload definition.
Referencing a Custom Payload Generator
Refer to payload generator implemented in Java.
<Job>
<TransferData>
<Payload>java:com.acme.integration.MyPayloadGenerator</Payload>(1)
</TransferData>
</Job>
1 | Note that we use the "java:" prefix. The class must implement the interface com.technia.tif.enovia.payload.Payload |
Destinations
The Destinations element within the <TransferData>
parent element contains the target destination (or destinations),
which the generated Payload will be transferred to.
The connection specific details for a connection is specified in a centralized file. Please look at the page Configure Destinations for further details on this topic.
The table below shows the supported destination elements.
Element | Descriptions | Supports Additional Headers | Supports Retry Attempt |
---|---|---|---|
|
Useful during development to see the payload printed out to the system output. |
No |
No |
|
Sends the payload into a file. |
No |
No |
|
Sends the payload to a HTTP endpoint. |
Yes |
Yes |
|
Sends the payload to a SOAP service |
No |
Yes |
|
Sends the payload to a JMS queue or topic |
Yes |
Yes |
|
Sends the payload to an Rabbit MQ queue/topic (via AMQP) |
Yes |
Yes |
|
Sends the payload to a Native MQ destination |
Yes |
Yes |
|
Sends the payload to a email recipient |
No |
No |
|
Sends the payload to an Apache Kafka topic |
Yes |
No |
|
Defines a custom destination class. The class must extend from the base class |
- |
Yes (if class implements Retryable) |
The destination element(s) contains an "id" attribute, which refers to a corresponding destination definition within the However, the
|
Example
The example below will send the payload to:
-
the system-out (standard output)
-
file destination as configured with the ID "fileDest1" in the destinations.xml file
-
custom destination implemented by class "com.acme.integration.MyCustomDestination"
<Job>
<TransferData>
<Destinations>
<SysOut/>
<File id="fileDest1"/>
<SOAP id="webservice-test"/>
<Custom className="com.acme.integration.MyCustomDestination"/>
</Destinations>
</TransferData>
</Job>
Conditionally Include Destination
You may specify conditions that must be met in order to transfer the payload to a certain destination.
Within the job configuration, the destination supports two different attributes called if
and unless
.
Here you specify the name (or names) of a property/parameter that must be OR must not be present on the Job itself in order to transfer the data to the destination in question.
Below is an example of using a conditionally inclusion of a destination:
<Job>
...
<TransferData>
...
<Destinations>
<Http id="http-1" if="send.to.1" />
<Http id="http-2" unless="send.to.1" />
</Destinations>
</TransferData>
</Job>
JMS Destination
The subchapters below describes the configuration aspects for a JMS destination.
JMS Reply To
You can specify a destination to which replies are generated. This is done via the replyTo
attribute.
The value of this attribute must be the ID of an existing destination inside the destinations configuration file.
See chapter Configure Destinations for more information.
Below is an example how to configure the replyTo.
<JMS id="dest-id" replyTo="reply-dest-id" />
And within the destinations.xml
file:
<Destinations>
<JMS id="dest-id"
initialContextFactory="org.apache.activemq.jndi.ActiveMQInitialContextFactory"
providerURL="tcp://172.16.16.141:61616">
<Queue name="part.info" jndiKey="queue.%s"/>
</JMS>
<JMS id="reply-dest-id"
initialContextFactory="org.apache.activemq.jndi.ActiveMQInitialContextFactory"
providerURL="tcp://172.16.16.141:61616">
<Queue name="part.info.reply" jndiKey="queue.%s"/>
</JMS>
</Destinations>
When you have set the attribute replyTo, then you should setup a so called reply-handler that is able to update TIF when the reply arrives in order to track whether the integration job failed or not. If you do not set up such reply handler, your jobs will stay in the state "Awaiting Reply". See chapter this document for further details. |
JMS Correlation ID
The correlation id of a message is by default set to:
${tif.instance.id}|${job.id}|${destination.id}
This value contains macros, which during runtime are resolved to real values. The information in the correlation id is used for example by the reply handler to correlate a message back to its origin.
You may change this, but, please note that if you do then you must consider so in the reply-handler that you might use.
Below is an example how to configure the correlation id.
<JMS id="dest-id" correlationId="${job.id}/${destination.id}" />
The macros are described here. |
JMS Type
You may specify a value for the JMS type property as shown below.
<JMS type="something" />
<JMS type="${macro}" />
The macros are described here. |
JMS Priority
You may specify a value for the JMS message priority as shown below. The priority is a value between 0 and 9.
<JMS priority="4" />
JMS Delivery Mode
You may specify persistent or non-persistent delivery modes by specifying this
in the deliveryMode
attribute as shown below.
<JMS deliveryMode="non-persistent"/>
<JMS deliveryMode="persistent"/>
JMS Message Type
When seding data to a JMS destination, TIF will by default use a StreamMessage when sending the content.
If you for some reason want to change the type, you can do so as shown below.
<JMS id="the-id" messageType="text"/>
<JMS id="the-id" messageType="byte"/>
<JMS id="the-id" messageType="stream"/>
Additional Headers
Some destinations allow additional meta-data to be passed along with the payload itself. Such meta-data is called headers and consists of key/value pairs.
A header value can be static or dynamic. It is also possible to provide a custom class that provides the headers. These are exemplified below:
<Job>
<TransferData>
...
<Destinations>
<JMS id="...">
<!-- Static parameter -->
<Header name="test1" value="bar"/>
<!-- Dynamic parameter, value taken from RPE (ENOVIA Runtime Program Env) -->
<Header name="test2" value="${job.rpe.TYPE}"/> (1)
<!-- Dynamic parameter, value taken from the additional arguments -->
<!-- passed via the trigger program object in ENOVIA for this job -->
<Header name="test3" value="${paramName}"/>
<!-- Define a custom header provider -->
<!-- Such class must implement the interface: -->
<!-- com.technia.tif.enovia.job.destination.HeaderProvider -->
<HeaderProvider>com.acme.foo.MyHeaderProvider</HeaderProvider>
</JMS>
...
</Destinations>
</TransferData>
</Job>
1 | The macros are described here |
Rabbit MQ / AMQP
The subchapters below describes the configuration aspects for a Rabbit MQ destination.
You can control the following aspects of a Rabbit MQ message:
-
Reply To
-
Correlation ID
-
Routing Key
-
Type
-
Priority
-
Delivery Mode
-
User ID
-
Application ID
Reply To
You can specify an exchange to which replies are generated.
This is done via the replyTo
attribute.
The value of this attribute must be the name of an exchange in your Rabbit MQ broker.
Below is an example how to configure the replyTo.
<RabbitMQ id="rabbit-mq-dest-id" replyTo="NAME-OF-EXCHANGE" />
If a job is sent to a Rabbit MQ destination whose replyTo has been set, the job will after being processed get the status "Awaiting Reply."
When you have set the attribute replyTo, then you should setup a so called reply-handler that is able to update TIF when the reply arrives in order to track whether or not the integration job failed or not. If you do not set up such reply handler, your jobs will stay in the state "Awaiting Reply". See this chapter for further details. |
Routing Key
The routing key can be defined in the destinations.xml file, but a more flexible approach is to define the routing key per use-case.
The routing key can be defined as macro, allowing to resolve dynamic value based upon some parameter etc.
Example:
<RabbitMQ id="rabbit-mq-dest-id"
routingKey="${job.param.NAME-OF-PARAM}" />
The macros are described here |
Correlation ID
The correlation id of a message is by default set to:
${tif.instance.id}|${job.id}|${destination.id}
This value contains macros, which during runtime are resolved to real values. The information in the correlation id is used for example by the reply handler to correlate a message back to its origin.
You may change this, but, please note that if you do then you must consider so in the reply-handler that you might use.
Below is an example how to configure the correlation id.
<RabbitMQ id="rabbit-mq-dest-id"
replyTo="NAME-OF-EXCHANGE"
correlationId="${job.id}/${destination.id}" />
The macros are described here |
Type
You may specify a value for the type property as shown below.
<RabbitMQ type="something" />
<RabbitMQ type="${macro}" />
The macros are described here |
Priority
You may specify a value for the message priority as shown below. The priority is a value between 0 and 255.
<RabbitMQ priority="4" />
Delivery Mode
You may specify persistent or non-persistent delivery modes by specifying this
in the deliveryMode
attribute as shown below.
<RabbitMQ deliveryMode="non-persistent"/>
<RabbitMQ deliveryMode="persistent"/>
IBM MQ
The subchapters below describes the configuration aspects for a IBM MQ / Native MQ destination.
Reply To
You can specify a destination to which replies are generated. This is done via the replyTo
attribute.
The value of this attribute must be the ID of an existing native-mq destination inside the destinations configuration file.
See chapter Configure Destinations for more information.
Below is an example how to configure the replyTo.
<NativeMQ id="mq-dest-id" replyTo="mq-reply-dest-id" />
And within the destinations.xml
file:
<NativeMQ id="mq-dest-id"
queueManagerName="QM_technia_mq"
hostName="172.16.16.141"
port="1414"
characterSet="1208"
encoding="546"
channel="S_technia_mq"
connectOptions="">
<Queue name="partdata_req" options="INPUT_AS_Q_DEF,OUTPUT"/>
</NativeMQ>
<NativeMQ id="mq-reply-dest-id"
queueManagerName="QM_technia_mq"
hostName="172.16.16.141"
port="1414"
characterSet="1208"
encoding="546"
channel="S_technia_mq"
connectOptions="">
<Queue name="partdata_resp" options="INPUT_AS_Q_DEF,OUTPUT"/>
</NativeMQ>
When you have set the attribute replyTo, then you should setup a so called reply-handler that is able to update TIF when the reply arrives in order to track whether or not the integration job failed or not. If you do not set up such reply handler, your jobs will stay in the state "Awaiting Reply". See this chapter for further details. |
Correlation ID
The correlation id of a MQ message is by default set to and ID, which resolves to the ID of the transfer.
A correlation id of a MQ message can only contain maximum 24 bytes. |
This value accepts macros, which during runtime are resolved to real values. The information in the correlation id is used for example by the reply handler to correlate a message back to its origin.
You may change this, but, please note that if you do then you must consider so in the reply-handler that you might use.
Below is an example how to configure the correlation id.
<NativeMQ id="dest-id" correlationId="${some.macro}" />
See this section for more details regarding macros.
Group ID
A MQ Message sent from a TIF server contains per default the tif instance id as group id value. That information is used by reply handlers to only filter messages originating from a particular TIF instance.
If you only use one TIF instance, you may disable the use of group id’s.
To disable the use of group id’s, either specify this on the destination
element as shown below or globally in tif.custom.properties
using the
property nativeMQ.defaultUseGroupId = false
.
<NativeMQ id="..." setGroupId="false" />
Message Type
There are different strategies available how to send the message. You may choose between one of the following
- string
-
The message read into a string and is sent using the
writeString
on the MQMessage - utfstring
-
The message read into a string and is sent using the
writeUTF
on the MQMessage - byte
-
The message bytes are written using the
write
method on the MQMessage.
You configure the strategy as shown below
<NativeMQ id="..." type="string" />
The default strategy is byte
.
Additional Settings
On the destination definition you may specify character-set, encoding, priority and expiracy. See this section for more details.
However, you may override these on the <NativeMQ>
element per use case also.
Example:
<NativeMQ id="..."
encoding="546"
characterSet="1208"
priority="7"
expiracy="604800" />
File Destination
The File destination contains one configurable options, e.g. define whether or not the result is updated asynchronously.
Example:
<File id="file-1" asyncReply="true" />
If you utilize this feature you should configure a so called reply handler. You can read more about reply handlers here |
Kafka Destination
The Kafka destination references the dito kafka-destination from the file ${TIF_ROOT}/etc/destinations.xml
file using the id attribute.
See Configure Destinations for more details.
General Usage
In your job configuration, you use the element <Kafka>
to send data to a Kafka topic.
You can define the topic to transfer the data to using the topic
attribute as shown below.
If you omit this attribute, the Kafka handler will try to resolve the topic from the destination with the given id.
<Job>
<TransferData>
<Payload>...</Payload>
<Destinations>
<Kafka id="kafka-1" (1)
topic="TIF-JOB-TEST" /> (2)
</Destinations>
</TransferData>
</Job>
1 | Reference the destination via the id attribute |
2 | Here you define the topic to use. Note that you could define a topic in the core destination definition, and omit this one, or vice versa. The topic defined here takes precedence. |
Configure the Record Key
You can via the attribute keyMacro
specify a macro that will resolve what key the produced Kafka record will use. Note that if you omit the keyMacro, then no key will be associated with the record.
Example below:
<Job>
<TransferData>
<Payload>...</Payload>
<Destinations>
<Kafka id="kafka-1" topic="TIF-JOB-TEST" keyMacro="${job.source.name}" />
</Destinations>
</TransferData>
</Job>
This will use the name from the source object within ENOVIA, which the Job was created for.
Please see here for details around using macros.
Asynchronous Replies
If you want to wait to complete the job after you have received a reply from Kafka, using a reply handler, you need to specify that the Kafka destination will produce a reply later.
This is controller via the asyncReply
attribute.
<Kafka id="kafka-1" asyncReply="true" />
Additional Headers
You can add headers also with the Kafka record that is being created.
First of all, any header specified within the used destination of ${TIF_HOME}/etc/destinations.xml
will be added.
Additionally, you can specify extra headers per job configuration like shown below.
<Job>
<TransferData>
<Payload>...</Payload>
<Destinations>
<Kafka id="kafka-1" topic="TIF-JOB-TEST" keyMacro="${job.source.name}">
<Header name="replyTo" value="TIF-REPLY-TEST" />
<Header name="tifInstance" value="${tif.instance.id}" />
<Header name="jobId" value="${job.id}" />
<Header name="destinationId" value="${destination.id}" />
<Header name="sourceId" value="${job.source.id}" />
</Kafka>
</Destinations>
</TransferData>
</Job>
As illustrated, the headers also supports macros to allow passing in dynamic values.
Another example shown below:
<Job>
<TransferData>
...
<Destinations>
<Kafka>
<!-- Static parameter -->
<Header name="test1" value="bar"/>
<!-- Dynamic parameter, value taken from RPE (ENOVIA Runtime Program Env) -->
<Header name="test2" value="${job.rpe.TYPE}"/> (1)
<!-- Dynamic parameter, value taken from the additional arguments -->
<!-- passed via the trigger program object in ENOVIA for this job -->
<Header name="test3" value="${paramName}"/>
<!-- Define a custom header provider -->
<!-- Such class must implement the interface: -->
<!-- com.technia.tif.enovia.job.destination.HeaderProvider -->
<HeaderProvider>com.acme.foo.MyHeaderProvider</HeaderProvider>
</Kafka>
</Destinations>
...
</TransferData>
</Job>
1 | The macros are described here |
The Email destination will by default include the content of the payload in the email as is (e.g. attachPayload is by default false). You can configure to attach the payload as a file to the mail instead, this is shown below.
<Email id="email-dest-1" attachPayload="true" attachAs="payload"/>
The attachAs
value is by default the static text payload
, and you may change this to something more appropriate.
The value may be a macro that resolves to some dynamic string.
HTTP
The HTTP destination references the dito http-destination from the file ${TIF_ROOT}/etc/destinations.xml
file using the id attribute.
See this document for more details.
The subchapter below describes the configuration aspects for an HTTP destination.
Macros in the URLs
In the ${TIF_ROOT}/etc/destinations.xml
file, the url for the http destination is defined.
This URL may contain macros, which are resolved at runtime when the job is being executed.
The macro syntax is described in this document.
An example of this is shown below
<Destinations>
<Http id="http-1"
url="https://acme.org/z/service-a/${job.param.identifier}"
retryCount="1" retryDelay="1000" method="PUT">
</Destinations>
Multipart
When transfering the payload to a destination, you can configure the Http request to send the file as a multipart request.
<Job>
...
<Destinations>
<Http id="system-x" multiPart="true" />
</Destinations>
</Job>
Per default the file is send using the parameter file
but the attribute multiPartFileParam
can be used to change this to something else.
Also, the attribute multiPartFileName
may be used to specify a custom file name when sending the file.
Success Status Codes
When data is transferred to an HTTP destination, the HTTP response status code is evaluated. By default, status code "2xx" is considered as success status code (e.g. any code between 200-299 inclusive).
To override the default behavior, you can configure a custom list of success status codes via <SuccessStatusCodes>
configuration element. The element text must contain a comma separated list of codes.
Below is an example where "200 OK", "201 Created" and "202 Accepted" are considered as success status codes.
<Http id="my-http-dest">
<SuccessStatusCodes>200, 201, 202</SuccessStatusCodes>
</Http>
Custom Status Evaluator
You may implement a custom Java class that can evaluate the HTTP response. The class must implement the interface com.technia.tif.enovia.job.destination.HttpStatusEvaluator
.
The interface is defined as below:
package com.technia.tif.enovia.job.destination;
import com.technia.tif.core.annotation.API;
import com.technia.tif.core.io.http.HttpResponse;
import com.technia.tif.enovia.job.EnoviaJob;
@API
public interface HttpStatusEvaluator {
/**
* Evaluates the HTTP response.
*
* @param job Job object
* @param response HTTP response.
* @return Whether the status evaluates to true or false.
*/
boolean evaluate(EnoviaJob job, HttpResponse response);
}
An example configuration:
<Http id="my-http-dest">
<StatusEvaluator className="com.acme.tif.MyStatusEvaluator" />
</Http>
URL Provider
Sometimes it is not possible to configure the URL for HTTP destination in a static way, but it requires some input from the job being executed. To enable this, you may implement a custom Java class that can provide the URL.
The class must implement the interface com.technia.tif.enovia.job.destination.URLProvider
.
The interface is defined as below:
package com.technia.tif.enovia.job.destination;
import com.technia.tif.core.annotation.API;
import com.technia.tif.enovia.job.EnoviaJob;
@API
public interface URLProvider {
/**
* Provides URL. The implementation is responsible for possible URL
* encoding.
*
* @param job Job object
* @param destUrl URL configured in destination config
* @return URL
*/
String provide(EnoviaJob job, String destUrl);
}
An example configuration:
<Http id="my-http-dest">
<UrlProvider className="com.acme.tif.urlprovider.MyUrlProvider" />
</Http>
Custom Destination
The subchapter below describes the configuration aspects for a custom destination.
Transfer Properties
When tranferring data to a custom destination, it might be useful to store information related to the transfer and view it in Admin UI afterwards.
This is possible by annotating field(s) containing the information with the annotation com.technia.tif.core.transfer.TransferProperty
.
The overall size of the information to be stored in TIF database is limited to 128k. |
For example:
import com.technia.tif.core.transfer.TransferProperty;
import com.technia.tif.enovia.job.TransferHandler;
public class MyDestination extends TransferHandler {
@TransferProperty
private int SomeField = 12345;
@TransferProperty
private String OtherField = "Hello World";
...
}
In addition, you may also return an object containing transfer properties by overriding the method getTransferProperties()
in the class that implements the custom destination.
The returned object is also available in the method transfer()
via argument TransferHandlerContext
.
For example:
import com.technia.tif.enovia.job.EnoviaJob;
import com.technia.tif.enovia.job.TransferHandler;
import com.technia.tif.enovia.job.TransferHandlerContext;
public class MyDestination extends TransferHandler {
@Override
public Object getTransferProperties(EnoviaJob job) {
return new MyProperties();
}
}
and
import com.technia.tif.enovia.job.EnoviaJob;
import com.technia.tif.enovia.job.TransferHandler;
public class MyProperties {
@TransferProperty
final boolean AnotherField = true;
}
Retryable Destination
The destinations that supports being retryable if the transfer fails are listed in the table above. Such destination can be configured like shown below:
<Job>
<TransferData>
...
<Destinations>
<JMS>
<RetryAttempts>20</RetryAttempts>
<RetryDelay>1000,5000,10000</RetryDelay>
</JMS>
</Destinations>
</TransferData>
</Job>
The <RetryAttempts>
element defines how many times TIF will try to resend the payload to the destination.
The time to wait between two attempts is defined within the <RetryDelay>
element. This value contains comma separated integer values where each value defines a period in milliseconds.
In the example above, TIF will wait 1 second between the 1:st and 2:nd attempt; 5 seconds between the 2:nd and 3:rd attempt and 10 seconds between the 3:rd and all attempts up to the 20:th. If the destination fails after the max count has been exceeded, the job will be marked as failed.
The default values are:
Retry Attempts: 10
Retry Delays: 1000,5000,10000,20000,40000,60000
Transaction Type
During the execution of the integration job of type "transfer data", TIF will by default start a READ transaction inside the ENOVIA/3DEXPERIENCE database.
If you for some reason want to change this, you can do so like the example below illustrates.
<Job>
<TransferData>
<!-- used for read transaction (default) -->
<TransactionType>read</TransactionType>(1)
<!-- To start an update transaction -->
<TransactionType>update</TransactionType>(2)
<!-- To not start a transaction at all, use the value of none or inactive -->
<TransactionType>none</TransactionType>(3)
</TransferData>
</Job>
1 | Starts a read transaction |
2 | Starts an update transaction |
3 | Does not start a transaction |
With Context
The <WithContext>
element controls what context to be used for the operation.
Per default, if omitted, the system user context will be used for the operation.
In most situations, the system user is the person you want to run the use case as in order to not having access rights issues while reading the required data.
However, in some cases you may want to limit the access for some reason.
The <WithContext>
element supports the following attributes.
Attribute | Required | Description |
---|---|---|
user |
No |
If omitted, the super user will be used |
securityContext |
No |
Defines the security context to be used |
useDefaultSecurityContext |
No |
If set to true, the default security context for the user will be used. The used user must have a default security context otherwise an error will be raised. |