Configure Data Uploader

Overview

The following section guides you on how to configure and use Data Uploader (DU) efficiently to get optimum performance.

For additional help, see the Data Uploader FAQ page.

Hardware, Software, and User Account Requirements for Installation

Installation Requirements

Important: Currently, CAM supports and uses TLS 1.2 by default, but allows TLS 1.0/1.1 if the Data Uploader is run on a Windows 2012 Server.

On June 28, 2023, AWS is dropping all support for TLS 1.0/1.1 and this could affect users using Windows Server 2012 or older with the use of Data Uploader.

Windows Server 2016 and above are the natively supported versions with 1.2.

Hardware Requirements

• A Windows PC or Server

  • Windows Server 2016+ is recommended for the note below.

Currently, CAM supports and uses TLS 1.2 by default, but allows TLS 1.0/1.1 if the Data Uploader is run on a Windows 2012 Server.

As of June 28, 2023, AWS dropped all support for TLS 1.0/1.1 and this could affect users using Windows Server 2012 or older with the use of Data Uploader.

Windows Server 2016 and above are the natively supported versions with 1.2.

• At least 1GB of free space for program installation and execution

• Internet access to upload the CSV to CAM; 50mbps or higher upload speed is recommended

Software Requirements

• .NET Framework 4.8 or higher*

*= 4.8 is required for the new version of Data Uploader since Sprint 23.

Date Format

• Format all date values as 

yyyy-MM-dd HH:mm:ss

To format the dates, go to Administration-> Settings->General tab-> Default Data Format.

 

Open

 

SQL Requirements

Firms are strongly recommended to use an on-premises SQL database (instead of SQLLite) when the firm is a medium to large-sized company or it has more than 5,000 active matters. This allows Delta syncs to perform more efficiently as only necessary workspaces and metadata will be updated.

If the firm is using an on-premises SQL database, these are the requirements:

• SQL Server 2012 Standard Edition or above.

• Windows Server 2016+ is recommended for the note below.

CAM Data Uploader does support SQL 2019 and 2022.

Important Note for Windows Server 2012

• A dedicated or Shared SQL server

• 8+GB of free RAM

• 4+CPU (cores/threads)

• 10+GB free hard disk space*

• SQL Maintenance plans setup as typically configured.

Firewall and Port Requirements

Whitelist the following ports.

• Port 443 should be open. CAM and Data Uploader utilize the HTTPS ports for requests. The API manages the requests, and no data is locally accessed or stored. If you’ve changed the default port for HTTPS, use the changed port. Check with your IT team if your firm has changed the port, and if so they need to open that default port on the corresponding HTTPS.

• Port 1433 should be open for SQL communication between the on-premises SQL and the API. Check with your IT team if your firm has changed the port, and if so they need to open that default port on the corresponding SQL server.

 

User Account Requirements

Litera recommends a service account be used to run Data Uploader. For more details on setting up a user account, see Create CAM User Accounts

This service account should have:

• Permission to download, install, and run the Data Uploader on the Desktop or server.

• Read/Write access to the designated Data Uploader application directory.

• Read access to the source database on the SQL server (to query the database and upload to CAM)

• Permission to the local system to edit/enable the Windows Task Scheduler.

• Read access to Active Directory.

• Must be a system admin

• If the user is using the CheckOut feature, the user will need at a minimum read/write access to the check-out path. If it is a path on the local PC, then that local PC will also need to be on the network.

Handling Bulk Jobs and Load Balancing within Data Uploader

Best Practices

• Set the number of jobs under the 50k limit for better performance, to ensure the jobs run. E.g. if undeclaring more than 50k records, batch into 40k amounts per day.

• Use Data Uploader if the bulk actions from Service Desk or Analytics can’t handle the number of records that need to be modified.

Data Uploader Limits

To Download Data Uploader

1. Navigate to Administration.

2. Click on Downloads.

3. Download the Data Uploader EXE installer.

4. Navigate to the download location and double-click the program to start the installer.

Updating the Configurations

The new Data Uploader application has replaced many of the previous configuration files with new config files after an upgrade happens. Since the following instructions for copying old configurations to the new files reference line numbers, we recommend using a program like Notepad++ or other tools that have line numbering available.

All configuration files are now in a directory under the install path called config located here:

C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config

If you have made any configuration changes to the following files, please follow the instructions below to copy your configurations to the new files. If you have not modified any of these files, you can skip this section

1. Data.config

2. UserPropertiesMapping.config

3. SourceFilesConfig

4. ADMapping File

5. AppSetting.config

SQL Scripts

1. Copy the scripts from the C:\Program Files (x86)\Prosperoware\Data Uploader\sql file to the new. C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\sql

2. Move any additional custom scripts to this SQL folder.

Testing new workspace provisioning scripts

To ensure the successful deployment of a new workspace provisioning script, testing is crucial. Follow these recommended steps for effective testing:

·       Validate the script in SQL and review the output to identify any issues with missing metadata, templates, or with script logic.

·       Update and revalidate the script to address any problems.

·       Use Data Uploader to generate a CSV and examine the data for accuracy.

Remove all but a few records from the CSV and submit a test job with this smaller dataset to ensure accuracy before full deployment.

Active Directory Synchronization Overview

Setting up AD Sync is critical for synchronizing users and groups to CAM from Active Directory.  The process is driven by a connection to each AD Organizational Unit (OU) where the users and groups to be synced are located, and configuring the ADMapping.config file.  All the OUs are filtered through this one file by the configuration of criteria filters and other attributes and properties that can be set. Ultimately, the result is a CSV of the applicable people which can be scheduled in Data Uploader to automatically be sent to CAM for processing.

 Here are the recommended steps for the setup and configuration of AD Sync:

1. Review and cleanup Active Directory Organizational Unit data.

2. Connect to Active Directory Organization Units via Data Uploader.

3. Configure ADMapping.config.

4. Update SourceFiles.config with Organization Units.

5. Test the AD Sync configuration.

6. Schedule the AD Sync.

1. Review and Cleanup Active Directory Organization Unit Data

AD data review and cleanup should focus on purging non-user entities such as printers, multi-function devices (MFDs), and other legacy connected devices from each OU to ensure only user data is synchronized, preventing unnecessary entries in the DMS that could complicate the data environment.

2. Connect to Active Directory Organization Units via Data Uploader

In Data Uploader’s Active Directory tab, use the Add button to add each Organization Unit that contains the users and group you wish to sync.  There is no limit to the OUs that can be added for synchronization.

For information on adding Organization Units, please see Configure Data Uploader | Active Directory tab Connect to to AD

For adding multiple OU’s see, Configure Data Uploader | Scheduling multiple OU or Container syncs

3. Configure ADMapping.config

Overview

The ADMapping.config file, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, is pivotal for configuring AD Sync. This configuration tool defines the user and group data that will be organized in the CSV sent to CAM for processing, serving as the blueprint for synchronization.

All connected Organizational Units are configured within this single ADMapping.config file. The file contains three Objects (UsersAndGroups, UsersOnly, and GroupsOnly) with criteria that act as filters. For each OU, multiple instances of these Objects can be created, and their criteria configured to determine the specific users and groups to be included in the AD Sync process.  Properties and values for these users and groups can also be directly specified in the file, affecting the information that will be included in the CSV.  Once the ADMapping.config file has been configured, a CSV reflecting the user and group data configuration can be produced and provided to CAM for processing.

Anatomy of the ADmapping.config File

Objects Section

Open

1. Three objects are available: UsersAndGroups, UsersOnly, and GroupsOnly.

2. Each Object contains a <criteria> and <values> section.

A.      The <criteria> section is used to configure the filter for the users and groups that will be included in the AD Sync process. The ad-property and value are used to set the criteria and value for the filter, and the operator facilitates logic operations for the filtering process.

B.      The <values> section provides the ability to set properties and values for the users being filtered by the criteria section.  See the CSV parameters for User and Group Provisioning section for more information on setting properties and values.

3. To configure multiple OUs for synchronization, copy Objects as many times as needed, then add the OU for the Object via the DC attribute.  

<default>, <fields>, and <groupfields> Sections

The <default> section is used to assign a default value to all the records being filtered above.  If for example, you are setting the same database for all the records, instead of setting this value in the values section for each Object, set the Database property in the Default section.

The Fields and Groupfields sections are used to map specific Active Directory attributes to a CSV property.  The AD attribute’s value will be added to the CSV in a column with the property name you specify.  For example, change the “mail” attribute in AD to “userid” for iManage.

CSV Parameters for User and Group Provisioning

Parameters are available for tasks such as creating, editing, deleting, and disabling users and groups, as well as removing users from MyMatters in iManage. For the complete User and Group Provisioning parameter list, visit our documentation.

These parameters are coded as properties in the ADMapping.config file to set specific values for the synchronization process.

Below are examples of setting properties:

·       Disabled. This property is used to enable or disable a user or group in CAM or another system.  Set to True/1:
property name="Disabled" values="1"

·       IsCAMUser and IsCAMGroup.  This property determines whether to create a CAM user or group, the people and groups who will use CAM, during the AD Sync process. Setting to 1 will trigger creation; Setting to 0 will prevent the creation.
property name="isCAMUser" values="1"
property name="isCAMGroup" values="1"

·       Database. Determines the specific library or libraries to which users and groups will be associated. This property is essential for ensuring that the AD Sync process targets the correct database.

o   The value should be the exact name for the target database. 

o   When associating multiple databases, separate each with a pipe character (|).

o   Use with the Preferred Database property when specifying multiple databases to indicate which database is primary.
property name="Database" values="HR|Active" appliesTo="iManage"
property name="PreferredDatabase" values="HR" appliesTo="iManage"

• SkipUpdateIfExists: This parameter can be used with an AD sync or without one based on where you want users and groups to be imported. Set this setting here in AD configurations if you want the user to come from the AD system. 0 will update the user or group in both CAM and the external system, 1 will skip the external system and update only in CAM.

For more information, see Configure Data Uploader | Mapping Active Directory (AD) properties and Configure Data Uploader | Filtering Active Directory Users/Groups

4. Update SourceFiles.config with Organization Units

Add an entry for all connected OUs in the AD Sync process to the SourceFiles.config.  See the Source Files section for more information on adding connected OUs to the SourceFiles.config.

The SourceFiles.config, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, should include an entry for each active SQL script and each connected Active Directory Organizational Unit. The file ensures Data Uploader only processes and sends new or modified jobs to CAM by enabling the JobDelta table in the CAM Local Database to track job submissions.

5. Test the AD Sync Configuration

Before scheduling AD Sync, use Data Uploader to generate a CSV, then submit a job with a small subset of the data to ensure accuracy.

On Data Uploader’s Active Directory tab, click Generate CSV for the OU being tested. Open the CSV and examine the data to verify the configuration has produced the right results.  Next, remove all but a few records from the CSV and submit a test job with this smaller dataset, then verify the results of the job.

When user and group records are not updating, check:

·       Does the user meet the filter conditions defined within the <criteria> section for the relevant Object in the ADMapping.config file?

·       Does the user exist in the external system, and is the “SkipUpdateIfExists” configuration set to True in the ADMapping.config file?

·       Does the user have an entry in the JobDelta table?  Is there a change to the user record? Existing users with no changes will be skipped.

·       Is the OU where the user is a member included in the scope of the ADMapping.config file?

6. Schedule AD Sync

The best strategy for scheduling AD Sync is to schedule each OU individually from Data Uploader’s Active Directory tab.  It is important to determine the frequency of changes applicable to not only each OU, but also scheduled SQL scripts to ensure jobs do not overlap.

When adding an OU, or editing a connected OU, use the Run option, in the Domain details screen, to set the synchronization frequency for that specific OU.

For adding multiple OU’s see, Configure Data Uploader | Scheduling multiple OU or Container syncs

Notes on SkipCSV

If after an ad-hoc sync or a scheduled Data Uploader sync, you see the following in the Data Uploader logs that a job has been skipped with the log: Skip Uploading CSVName.csv, this means that the csv was skipped because of an empty file, or that there is no new data to upload or update from that job (eg all rows are the same from source to target).

The CSV that was skipped will reside inside the folder path Prosperoware\CAM\DataUploader\skipcsv

External System Configuration

How does Data Uploader determine how to import

If the last modified date is in the SQL files, then the last modified date is used to check the data brought in. However, the Job Delta ensures the records from the sync before remain in the Staging tables. Only any new, or changed records get imported from there into CAM. Similarly, AD sync or other tabs will use the system configuration.

For example, if a user or group is imported, If the client has other conditions in the SQL Scripts to check if a duplicate record exists using if/then statements, these are used to bring over the initial records to staging.

Then the records are compared using the username or group name to see if it exists. The client can change these with the source mapping configurations by stating the unique ID columns exactly!

Lastly in the script or in the source mapping, if there is a last modified date inside, this is used to determine the date range to go back to check and pull data.

For Active Directory, all records are checked in Active Directory, unless a particular group or OU or container is defined.

Automating CSV Imports for Batch DU Jobs

1. Users could automatically generate CSV files of their import jobs using their program of choice.

2. Then place the files inside the newCSV folder.

3. Run a DU job that is configured, and the program picks up the generated CSVs and imports them.

Data Uploader Files and Compatiblity

In V2 of the Data Uploader, the Jobs submit JSON files instead of CSV files in the backend to avoid overloading the queue processor.

Old Version Compatibility

If a firm requires the older version of Data Uploader, V1, make the following changes to the appsettings.config file:

New Version Features

• In addition to the new submission in JSON, there is a new folder where those CSVs are stored when the JSONs are uploaded. This folder is called newCSV. The CSVs are created and then converted from these files in this folder to JSON. If a job fails, this would be the folder that contains the JSON files to troubleshoot.

• Generated CSVs still reside in the generatecsv folder for all jobs.

• Archived CSVs still reside in the archivecsv folder for any successfully completed jobs over time.