Troubleshooting / FAQ
Performance of the Source System Part of the Connector
There are many factors affecting the performance of content traversals on the source system side. Apart from the usual factors (especially the machine resources available to the connector, OpenText Content Server, the involved databases, etc. and the network connection to OpenText Content Server), several configuration options for the source system part of the connector directly affect the performance of the content traversals. They are listed below, structured according to the configuration groups in the connector UI.
An additional factor for the performance of content traversal is the order in which traversals are started. Both principal and content traversals need some information about principals/members, and for content traversals, this information is cached. The principal traversal does not use this cache, fetching the information directly from OpenText Content Server, but it does fill the cache. So if the configured cache expiration duration is longer than the duration of a principal traversal, starting content traversals directly after a principal traversal has finished improves the performance of the content traversal.
OpenText Content Server Connection Settings
As mentioned above, some data about principals/members are cached during content
traversals, and the time to live/cache expiration duration configured in this
section affects the performance as expected: a higher duration improves
performance at the cost of potentially putting stale data into the metadata of
items, especially the ACLs. So this value should primarily be set as required by
data freshness and only secondarily by performance needs. The data freshness is
not affected by how often the values are read from the cache, those reads do not
reset any timer. Due to the limited amount of data stored, the caches do not
have a size limit.
The configuration options HTTP timeout and maximum number of retries affect
performance in case of errors that cause the API call to be retried. Therefore,
these options should not be relevant for performance during normal operation and
should be set as required for the handling of transient errors.
The maximum REST API calls per second configuration option affects performance
of content traversals as expected (principal traversals do not use the REST
API), but only up to a certain point. This is because this option only
configures an upper limit on the rate of calls. All API calls are made
synchronously, and the connector only makes at most one API call per traversal
thread. If the rate of number of traversal threads divided by the average time
needed for an API call is lower than the configured API call rate limit, the
connector will not reach the limit and raising it will not improve performance.
OpenText Content Server Item Settings
Due to the OpenText Content Server APIs, each of the data activated/not set to zero in this section requires a separate call to OpenText Content Server per node. So the connector will make seven (7) extra calls per node when all options are activated, and with the exception of the call to fetch content, all calls will take about the same amount of time. So if some of the metadata are not needed, the corresponding option should be switched off to improve traversal performance.
The connector also honors the maximum content size limit as soon as possible.
That is, if the metadata of a node contain an accurate size of the content, and
that size is bigger than the configured limit, the connector will suppress
fetching the content, as it would be filtered out anyway. If the size is not
known in advance, the content is discarded directly after fetching about
configured limit + 1 bytes. The only exception to the preceding statement
happens when a limit of zero (0) bytes is configured. In this special case, the
connector does not rely on the content size from the metadata and always
suppresses fetching the content.
OpenText Content Server Debugging Settings
The JSON dumped with this configuration option is written to disk synchronously, directly affecting the performance of content traversals. So this option should be left unconfigured unless and until the output is actually needed, and the value should be returned to the default as soon as possible.
OpenText Content Server Content Traversal Settings
Apart from the expected effect (more content to traverse means more time needed to traverse the content), the root nodes configured in this section should also not overlap, as the connector may not be able to deduplicate the traversed nodes/items until they already have been fetched from OpenText Content Server and processed at least partially in the connector. This is especially true if multiple connector instances traverse the same OpenText Content Server and push the items to the same target system.
Delayed or Missing Updates with Automated Change Processing
While Automated Change Processing does get certain changes into the target system faster than scheduled incremental traversals, it still is not instantaneous and cannot handle certain types of changes at all, due to limitations of OpenText Content Server. Regardless of the approach used, there are a few general reasons why updates may be delayed or missing:
- Connection problems between the connector and OpenText Content Server / the connector database / the target system
-
Automated Change Processing (and the connector in general) relies on several external systems to work, and if any of these systems fails, changes cannot be processed. For some connection problems, processing the changes is retried indefinitely, thus only delaying the update; other problems may cause the update to be missing (until the node is changed again or the next incremental or full content traversal).
- Disabled Automated Change Processing
-
Automated Change Processing can be disabled by the administrators, which means that changes are not processed anymore. (This might be desired in certain situations.) When Automated Change Processing is re-enabled, the processing of changes continues at the point where the last run stopped.
- Connector shutdown
-
Automated Change Processing depends on the rest of the connector to work. So, just like with disabled Automated Change Processing, if the connector has been shut down, Automated Change Processing will not process changes anymore until the connector is started again (after which Automated Change Processing continues at the point where the last run stopped).
- Large number of changes
-
While Automated Change Processing does speed up getting changes into the target system, that speed-up comes from handling less nodes; the per-node overhead is largely the same. So, if for example a million changes happen per day, Automated Change Processing may not be able to keep up until the rate of changes goes down again.
- No access to the changed nodes for the technical user
-
As long as Automated Change Processing knows that the change(s) for a certain node include a permission change, it can handle losing access to that node (by deleting the corresponding item from the target system). But if it does not know about that (e.g., because the permission change is not [yet] part of the changes to process; cf. below for another reason), the deletion will be delayed or missing.
- Metadata overrides in the connector pipeline
-
If a changed metadata value is modified or overridden in the connector pipeline and the modification/override gives the same value for the new original metadata value as for the old original metadata value, the metadata value in the target system will stay the same.
When using the LiveReport approach to Automated Change Processing, possible additional causes for delayed or missing updates include:
- Mangled SQL query in the LiveReport
-
Automated Change Processing expects the LiveReport to return the audit log entries in a specific format. And while certain limited modifications to the query are supported, others may cause the results to not be understandable to Automated Change Processing anymore. In that case, Automated Change Processing cannot process changes anymore until the query is fixed, after which Automated Change Processing continues at the point where the last run stopped. (See setup-source-system.adoc#otcs_changes_livereport for the default query.)
- No access to the LiveReport for the technical user
-
When using the LiveReport approach, Automated Change Processing fetches the results of running that LiveReport through the SOAP API/CWS, which checks whether the connector user has permission to run the LiveReport. If the connector user does not have that permission, Automated Change Processing cannot process changes anymore until the permission is given again, after which Automated Change Processing continues at the point where the last run stopped.
- Additional filters in the SQL query
-
Automated Change Processing does not question the completeness of the results returned by the LiveReport used by the LiveReport approach, it simply assumes nothing is missing. So if you added additional filters to the LiveReport query, changes filtered out by those are not processed by Automated Change Processing and the updates for those will be missing, in most cases even if you remove the additional filters again.
- Audit types missing from the filter in the SQL query
-
As a special point to the “Additional filters in the SQL query” point above, Automated Change Processing only processes changes with an audit log entry type that is mentioned in the query. So if you removed some audit log entry type from the LiveReport query, changes with those types are not processed by Automated Change Processing and the updates for those will be missing, in most cases even if you add the removed types again.
The connector also has an internal ignore list with audit log entry types that do not correspond to changes (e.g.,
LiveReport,View, andZipAndDownload), so adding them to the query will not cause Automated Change Processing to process those non-changes or generate updates from them.
When using the Search approach to Automated Change Processing, possible additional causes for delayed or missing updates include:
- Disabled, stopped, or broken OpenText Content Server search (indexing)
-
The Search approach to Automated Change Processing basically asks OpenText Content Server’s internal search to search for all nodes changed since the last run. If that internal search is disabled, is broken, does not index the nodes to be indexed, or if its services are stopped, Automated Change Processing cannot process changes anymore until it is re-enabled, after which Automated Change Processing continues at the point where the last run stopped, even if the search API returned valid empty results in the meantime.
- Delay in the internal search indexing
-
The changed Modified Date needs to be picked up and indexed by OpenText Content Server’s internal search before it is available to Automated Change Processing. Depending on the configuration and resources available, this takes some time.
- Unchanged Modified Date
-
Certain changes, like modifying the Records Management metadata of nodes, do not change the Modified Date of that node, making them unavailable to the Search approach. In most cases, those changes can be picked up by the LiveReport approach; otherwise, incremental traversals are necessary.
- Deletions
-
The OpenText Content Server API used by the Search approach only returns nodes that exist outside the Recycle Bin and are visible to the connector user. This makes the Search approach fundamentally incapable of detecting and processing deletions. To delete the items corresponding to nodes that are no longer accessible, the LiveReport approach or incremental traversals are necessary.
- No access to the changed nodes for the technical user
-
The OpenText Content Server API used by the Search approach only returns nodes that are visible to the connector user. If a permission change causes the connector user to not see a node anymore, Automated Change Processing will not notice this with the Search approach, and the corresponding item will not be deleted from the target system. To delete the items corresponding to nodes that are no longer accessible, the LiveReport approach or incremental traversals are necessary.
External Group and Group Membership Limits in Microsoft Search
Raytion connectors reflect the custom security model of the source system in Microsoft Search. In order to do so, the connector creates external groups, which correspond to the groups from the source system.
The connector also links Microsoft Search users, i.e., users in Azure AD, to the correct external groups. This way, early-binding security trimming works.
External Groups Limit
However, there is a limit regarding the synchronization of external groups: All Microsoft Graph Connectors within a tenant can create up to 100,000 external groups. If this threshold is reached in your tenant, please contact the Microsoft support and check if they can extend the limit.