Flat File Connector

Overview

The flat file connector is used to import and export files, most commonly delimited (CSV) files. It has the same functionality as the built-in management delimited text file but it also has support for full export. Since it is written in PowerShell, you also have the possibility to change the import/export code according to your needs. For example, you might need to do pre- or post-processing of the file.

The connector can be configured in simple and complex mode. In Simple mode, the connector supports a single object type, and a single import file. In complex mode, the connector supports multiple partitions/object types, configuration of reference attributes, and one input file per object type. All non-reference attributes are assumed to be strings.

Conceptual Architecture

The connector is using a data stream reader for reading the file. This allows it to read files of (almost) unlimited size. It is then using convertfrom-csv to process the data.

It is using sample CSV files for detecting the schema.

In complex mode, each partition is an object type following the same pattern as the Generic SQL connector from Microsoft. These objects might have different schema but could be same object in the metaverse. For example, one object type could be employee and another contractor.

Since it is an ECMA2 connector, it has the ability to do a full export. That is, during export all objects and attributes are exported. This is the most common export remote systems want. This is an advantage over the built-in management agents where only delta export is available.

Connector Capabilities

The connector has support for the following:

• Full Import
• Paging support for Import
• Export and Full Export
• Partitions (i.e. object types)

Attributes are by default configured as strings. When using the complex configuration, reference attributes are supported as well.

Pre-requisites

You need a sample (schema) file with column headings.

Implementation details

Due to limitations in the PowerShell connector for storing additional information detected during schema discovery, the complex mode requires reading the schema files at every import. These should be stored in the extensions folder.