Pipeline

Concept

The connector crawls items from a source system and feeds these items into a target system through a pipeline. To adjust or process each item before it gets ingested, it is possible to provide custom stages for this pipeline. These stages are software modules provided by us which can be uploaded and configured via the UI.

How to upload and use stages

To upload a stage open the pipeline view:

pipeline view

Then click on the + symbol between Source System and Target System and the Select Pipeline Stage view pops up:

select stage

Here you can see stages which are already uploaded to the connector. To upload a new stage, drag and drop the respective software module or click on Drag file to upload or browse to navigate to and select the module.

Once the module is uploaded it is listed in the available stages where it can be selected. After selecting the stage it is shown as step in the pipeline where it can be configured:

configure stage

After completing the configuration of this single stage, it has to be saved by pressing SAVE. It is possible to set up and configure multiple custom stages in any order. To apply stages this setup of stages has to be validated and then saved by pressing the respective buttons. After saving the setup, the connector has to be restarted so that the pipeline stages are applied in the next traversals. This can be done by pressing RESTART CONNECTOR on the popped up message:

finish setup

Default packaged stages

The connector is packaged with pipeline stages which are delivered but not applied by default:

  • File Item Type Enricher Stage

  • Icon Assigner Stage

  • Metadata Assigner Stage

  • Metadata Drop Stage

  • Metadata Mapper Stage

  • Root and Parent Breadcrumbs Stage

  • Metadata Regular Expression Stage

File Item Type Enricher Stage

The File Item Type Enricher Stage maps Item Types based on the File Extension of a given Item.

Mappings

As the Stage replaces the Item Type of items based on their File Extension, in the table below you can see which File Extensions map to which Item Type.

File Extension Nice Item Type

7z

ZIP Archive

acm

ACM Audio

aspx

HTML Document

avi

AVI Video

bat

Batch Script

bmp

BMP Image

doc

Microsoft Word

docx

Microsoft Word

dot

Microsoft Word Template

dotx

Microsoft Word Template

eml

Microsoft Outlook

gif

GIF Image

htm

HTML Document

html

HTML Document

img

Disk Image

jpeg

JPEG Image

jpg

JPEG Image

mov

QuickTime Video

mp3

MP3 Audio

mp4

MPEG-4 Video

mpeg

MPEG Video

mpg

MPEG Video

msg

Microsoft Outlook

ods

OpenDocument Spreadsheet

one

Microsoft OneNote

odt

OpenDocument Text

pdf

Adobe PDF

php

HTML Document

png

PNG Image

pot

Microsoft PowerPoint Template

potx

Microsoft PowerPoint Template

ppt

Microsoft PowerPoint

pptx

Microsoft PowerPoint

ps1

PowerShell Script

rar

RAR Archive

smil

SMIL Presentation

swf

Shockwave Flash

tif

TIFF Image

tiff

TIFF Image

txt

Plain Text

wav

WAV Audio

xls

Microsoft Excel

xlsx

Microsoft Excel

xlt

Microsoft Excel Template

xltx

Microsoft Excel Template

xml

XML Document

zip

ZIP Archive

Configuration

Property Description

Enable Item Type Enrichment

Toggles the enrichment of the Item Type of items based on the respective File Extension. Specifically, this will set a display-friendly Item Type for any well-known File Extensions. e.g. the Item Type will be set to 'Microsoft Word' for any documents with the File Extension 'doc'.

Enable Default to File Extension

If enabled, the Item Type will be set to the File Extension suffixed with '<file extension> File' for any unknown File Extensions. Otherwise, the Item Type remains unchanged.

Icon Assigner Stage

The Icon Assigner Stage sets the Item’s Icon URL based on the Item’s File Extension and Item Type.

Configuration

Property Description

Item Types

The Icon will only be assigned if one of the Item Types matches. If none are defined, all Item Types will be accepted.

File Extensions

The Icon will only be assigned if one of the File Extensions matches. If none are defined, all File Extensions will be accepted.

Icon URL

The Icon URL that will be added to the item if it matches both an Item Type and a File Extension.

Icon Metadata Key

The Key under which the Icon will be added to the metadata.

Metadata Assigner Stage

The Metadata Assigner Stage alters or assigns a metadata value.

Configuration

Property Description

Metadata Key

Key of the target metadata.

Metadata Values

The list of values which should be assigned to the metadata field using the specified assigner type strategy.

Assigner Type

The strategy how the metadata value is appended (APPEND, REPLACE or IGNORE).

Metadata Drop Stage

Drops metadata entries of an item with the specified key.

Configuration

Property Description

Metadata Keys

Keys of metadata entries to drop.

Metadata Mapper Stage

The Metadata Mapper Stage copies the values from the source key into target key.

Configuration

Property Description

Metadata Source Key

Key of the source metadata.

Metadata Target Key

Key of the target metadata.

Metadata Regular Expression Stage

The Metadata Regular Expression Stage is used to manipulate a metadata field by replacing all occurrences of a regular expression with a configured value.

Configuration

Property Description

Metadata Key

Key of the target metadata.

Pattern

The pattern to match and replace with the value.

Metadata Value

The value to replace the matched string with

Prefix

A prefix to add in front of the current value

Suffix

A suffix to add behind the current value

Example

With the configuration

Property Value

Metadata Key

text

Pattern

\\s+

Metadata Value

-

Prefix

start_

Suffix

_end

the metadata
text=this is an example
becomes
text=start_this-is-an-example_end