Configure Data Uploader

Overview

The following section guides you on how to configure and use Data Uploader (DU) efficiently to get optimum performance.

For additional help, see the Data Uploader FAQ page.

Installation Requirements

Important: Currently, CAM supports and uses TLS 1.2 by default, but allows TLS 1.0/1.1 if the Data Uploader is run on a Windows 2012 Server.

On June 28, 2023, AWS is dropping all support for TLS 1.0/1.1 and this could affect users using Windows Server 2012 or older with the use of Data Uploader.

Windows Server 2016 and above are the natively supported versions with 1.2.

Hardware Requirements

  • A Windows PC or Server

    • Windows Server 2016+ is recommended for the note below.

Currently, CAM supports and uses TLS 1.2 by default, but allows TLS 1.0/1.1 if the Data Uploader is run on a Windows 2012 Server.

As of June 28, 2023, AWS dropped all support for TLS 1.0/1.1 and this could affect users using Windows Server 2012 or older with the use of Data Uploader.

Windows Server 2016 and above are the natively supported versions with 1.2.

  • At least 1GB of free space for program installation and execution

  • Internet access to upload the CSV to CAM; 50mbps or higher upload speed is recommended

Software Requirements

  • .NET Framework 4.8 or higher*

*= 4.8 is required for the new version of Data Uploader since Sprint 23.

Date Format

  • Format all date values as 

yyyy-MM-dd HH:mm:ss

To format the dates, go to Administration-> Settings->General tab-> Default Data Format.

 

image-20240528-140832.png

 

SQL Requirements

Firms are strongly recommended to use an on-premises SQL database (instead of SQLLite) when the firm is a medium to large-sized company or it has more than 5,000 active matters. This allows Delta syncs to perform more efficiently as only necessary workspaces and metadata will be updated.

If the firm is using an on-premises SQL database, these are the requirements:

  • SQL Server 2012 Standard Edition or above.

  • Windows Server 2016+ is recommended for the note below.

CAM Data Uploader does support SQL 2019 and 2022.

Important Note for Windows Server 2012

  • A dedicated or Shared SQL server

  • 8+GB of free RAM

  • 4+CPU (cores/threads)

  • 10+GB free hard disk space*

Firewall and Port Requirements

Whitelist the following ports.

  • Port 443 should be open. CAM and Data Uploader utilize the HTTPS ports for requests. The API manages the requests, and no data is locally accessed or stored. If you’ve changed the default port for HTTPS, use the changed port. Check with your IT team if your firm has changed the port, and if so they need to open that default port on the corresponding HTTPS.

  • Port 1433 should be open for SQL communication between the on-premises SQL and the API. Check with your IT team if your firm has changed the port, and if so they need to open that default port on the corresponding SQL server.

 

User Account Requirements

Litera recommends a service account be used to run Data Uploader. For more details on setting up a user account, see

This service account should have:

  • Permission to download, install, and run the Data Uploader on the Desktop or server.

  • Read/Write access to the designated Data Uploader application directory.

  • Read access to the source database on the SQL server (to query the database and upload to CAM)

  • Permission to the local system to edit/enable the Windows Task Scheduler.

  • Read access to Active Directory.

  • Must be a system admin

  • If the user is using the CheckOut feature, the user will need at a minimum read/write access to the check-out path. If it is a path on the local PC, then that local PC will also need to be on the network.

Handling Bulk Jobs and Load Balancing within Data Uploader

Best Practices

  • Set the number of jobs under the 50k limit for better performance, to ensure the jobs run. E.g. if undeclaring more than 50k records, batch into 40k amounts per day.

  • Use Data Uploader if the bulk actions from Service Desk or Analytics can’t handle the number of records that need to be modified.

Data Uploader Limits

To Download Data Uploader

  1. Navigate to Administration.

  2. Click on Downloads.

  3. Download the Data Uploader EXE installer.

  4. Navigate to the download location and double-click the program to start the installer.

Installing Data Uploader

  1. Ensure the requirements are met. The requirements can be found here.

  2. In the Welcome to Data Uploader Setup Wizard dialog, click Next.

  3. If you would like to install the Data Uploader somewhere other than the default location displayed in the installer, click Browse and navigate to your desired installation folder; otherwise, leave the path unchanged.

  4. To install it for your access only, click Just me or select Everyone to allow access to anyone using your system.

  5. Click Next to continue.

  1. Click Close on the Installation Complete pop-up box once it notes DataUploader has been successfully installed.

  2. The Cloud Data Uploader is installed as a shortcut on your desktop. Double-click it to launch the application.

 

Upgrade Data Uploader

Installation

  1. Review the Hardware and Software Requirements for Data Uploader.

  2. You must create a backup of the existing DU database before the upgrade. If you are not utilizing a dedicated SQL database for DU, then skip this step.

  3. Make a backup copy of the current Data Uploader installation directory:

    • Prosperoware Data Uploader folder found at: C:\Program Files (x86)\Prosperoware\Prosperoware Data Uploader.

  1. Log into your CAM website.

  • If you are already logged in, sign out and sign in back in to ensure the latest update is available.

  1. Navigate to Administration

  2. Click on Downloads and download the latest version of Data Uploader onto the server where DU is installed.

  1. Run the installation wizard.

  2. Click Next. Keep the install folder path the same.

  3. Click Next.

  1. This version of the Data Uploader now requires that you enter a Windows Services domain, username, and password of the service account.

  2. Click Install to begin the installation.

  3. Once the installation is complete, click Close to close the wizard. Continue with Updating the Configurations section below.

Updating the Configurations

The new Data Uploader application has replaced many of the previous configuration files with new config files after an upgrade happens. Since the following instructions for copying old configurations to the new files reference line numbers, we recommend using a program like Notepad++ or other tools that have line numbering available.

All configuration files are now in a directory under the install path called config located here:

C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config

If you have made any configuration changes to the following files, please follow the instructions below to copy your configurations to the new files. If you have not modified any of these files, you can skip this section

  1. Data.config

  2. UserPropertiesMapping.config

  3. SourceFilesConfig

  4. ADMapping File

  5. AppSetting.config

Source Files Config

Source files set where the source files for Data Uploader and any mappings exist. The out-of-the-box configurations set uniqueids which are important unique identifiers in CAM and other systems used. This is a new configuration file that allows you to set a separate unique identifier for each of your provisioning scripts. Some content systems might support a matter index ID for a unique identifier, while others might only be able to support client and matter ID. This file configures the unique ID per script.

  1. Update the uniqueid line to the columns and source that makes a user unique to the firm. Most commonly it is the type of user, the userid, and the database of the source.

For example, uniqueid client id, matterid is pulled from CreateorUpdateWorkspace.sql

 

Last Modified Date in the Config

If the last modified date is in the config, then the last modified date is used to check the data brought in. However, the Job Delta ensures the records from the sync before remain in the Staging tables. Only any new, or changed records get imported from there into CAM. Similarly, the SQL files tab will use the last modified date in those SQL files.

Modifying Application Settings

The application configuration can be further edited from the AppSettings.config file in the Data Uploader directory.

To restore a prior configuration:

  1. First, backup the appsettings.config.

  2. Then, upgrade Data Uploader as you normally would.

  3. Finally, restore the config, or copy the configuration to another system.

 

 

 

SQL Scripts

  1. Copy the scripts from the C:\Program Files (x86)\Prosperoware\Data Uploader\sql file to the new. C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\sql

  1. Move any additional custom scripts to this SQL folder.

 

Testing new workspace provisioning scripts

To ensure the successful deployment of a new workspace provisioning script, testing is crucial. Follow these recommended steps for effective testing:

·       Validate the script in SQL and review the output to identify any issues with missing metadata, templates, or with script logic.

·       Update and revalidate the script to address any problems.

·       Use Data Uploader to generate a CSV and examine the data for accuracy.

Remove all but a few records from the CSV and submit a test job with this smaller dataset to ensure accuracy before full deployment.

Active Directory Synchronization Overview

Setting up AD Sync is critical for synchronizing users and groups to CAM from Active Directory.  The process is driven by a connection to each AD Organizational Unit (OU) where the users and groups to be synced are located, and configuring the ADMapping.config file.  All the OUs are filtered through this one file by the configuration of criteria filters and other attributes and properties that can be set. Ultimately, the result is a CSV of the applicable people which can be scheduled in Data Uploader to automatically be sent to CAM for processing.

 Here are the recommended steps for the setup and configuration of AD Sync:

  1. Review and cleanup Active Directory Organizational Unit data.

  2. Connect to Active Directory Organization Units via Data Uploader.

  3. Configure ADMapping.config.

  4. Update SourceFiles.config with Organization Units.

  5. Test the AD Sync configuration.

  6. Schedule the AD Sync.

1. Review and Cleanup Active Directory Organization Unit Data

AD data review and cleanup should focus on purging non-user entities such as printers, multi-function devices (MFDs), and other legacy connected devices from each OU to ensure only user data is synchronized, preventing unnecessary entries in the DMS that could complicate the data environment.

2. Connect to Active Directory Organization Units via Data Uploader

In Data Uploader’s Active Directory tab, use the Add button to add each Organization Unit that contains the users and group you wish to sync.  There is no limit to the OUs that can be added for synchronization.

For information on adding Organization Units, please see

For adding multiple OU’s see,

3. Configure ADMapping.config

Overview

The ADMapping.config file, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, is pivotal for configuring AD Sync. This configuration tool defines the user and group data that will be organized in the CSV sent to CAM for processing, serving as the blueprint for synchronization.

All connected Organizational Units are configured within this single ADMapping.config file. The file contains three Objects (UsersAndGroups, UsersOnly, and GroupsOnly) with criteria that act as filters. For each OU, multiple instances of these Objects can be created, and their criteria configured to determine the specific users and groups to be included in the AD Sync process.  Properties and values for these users and groups can also be directly specified in the file, affecting the information that will be included in the CSV.  Once the ADMapping.config file has been configured, a CSV reflecting the user and group data configuration can be produced and provided to CAM for processing.

Anatomy of the ADmapping.config File

Objects Section

 

image-20240816-115151.png

 

  1. Three objects are available: UsersAndGroups, UsersOnly, and GroupsOnly.

  2. Each Object contains a <criteria> and <values> section.

A.      The <criteria> section is used to configure the filter for the users and groups that will be included in the AD Sync process. The ad-property and value are used to set the criteria and value for the filter, and the operator facilitates logic operations for the filtering process.

B.      The <values> section provides the ability to set properties and values for the users being filtered by the criteria section.  See the CSV parameters for User and Group Provisioning section for more information on setting properties and values.

  1. To configure multiple OUs for synchronization, copy Objects as many times as needed, then add the OU for the Object via the DC attribute.  

<default>, <fields>, and <groupfields> Sections

The <default> section is used to assign a default value to all the records being filtered above.  If for example, you are setting the same database for all the records, instead of setting this value in the values section for each Object, set the Database property in the Default section.

The Fields and Groupfields sections are used to map specific Active Directory attributes to a CSV property.  The AD attribute’s value will be added to the CSV in a column with the property name you specify.  For example, change the “mail” attribute in AD to “userid” for iManage.

 

CSV Parameters for User and Group Provisioning

Parameters are available for tasks such as creating, editing, deleting, and disabling users and groups, as well as removing users from MyMatters in iManage. For the complete User and Group Provisioning parameter list, visit our documentation.

These parameters are coded as properties in the ADMapping.config file to set specific values for the synchronization process.

Below are examples of setting properties:

·       Disabled. This property is used to enable or disable a user or group in CAM or another system.  Set to True/1:
property name="Disabled" values="1"

·       IsCAMUser and IsCAMGroup.  This property determines whether to create a CAM user or group, the people and groups who will use CAM, during the AD Sync process. Setting to 1 will trigger creation; Setting to 0 will prevent the creation.
property name="isCAMUser" values="1"
property name="isCAMGroup" values="1"

·       Database. Determines the specific library or libraries to which users and groups will be associated. This property is essential for ensuring that the AD Sync process targets the correct database.

o   The value should be the exact name for the target database. 

o   When associating multiple databases, separate each with a pipe character (|).

o   Use with the Preferred Database property when specifying multiple databases to indicate which database is primary.
property name="Database" values="HR|Active" appliesTo="iManage"
property name="PreferredDatabase" values="HR" appliesTo="iManage"

 

For more information, see and

4. Update SourceFiles.config with Organization Units

Add an entry for all connected OUs in the AD Sync process to the SourceFiles.config.  See the Source Files section for more information on adding connected OUs to the SourceFiles.config.

The SourceFiles.config, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, should include an entry for each active SQL script and each connected Active Directory Organizational Unit. The file ensures Data Uploader only processes and sends new or modified jobs to CAM by enabling the JobDelta table in the CAM Local Database to track job submissions.

5. Test the AD Sync Configuration

Before scheduling AD Sync, use Data Uploader to generate a CSV, then submit a job with a small subset of the data to ensure accuracy.

On Data Uploader’s Active Directory tab, click Generate CSV for the OU being tested. Open the CSV and examine the data to verify the configuration has produced the right results.  Next, remove all but a few records from the CSV and submit a test job with this smaller dataset, then verify the results of the job.

When user and group records are not updating, check:

·       Does the user meet the filter conditions defined within the <criteria> section for the relevant Object in the ADMapping.config file?

·       Does the user exist in the external system, and is the “SkipUpdateIfExists” configuration set to True in the ADMapping.config file?

·       Does the user have an entry in the JobDelta table?  Is there a change to the user record? Existing users with no changes will be skipped.

·       Is the OU where the user is a member included in the scope of the ADMapping.config file?

6. Schedule AD Sync

The best strategy for scheduling AD Sync is to schedule each OU individually from Data Uploader’s Active Directory tab.  It is important to determine the frequency of changes applicable to not only each OU, but also scheduled SQL scripts to ensure jobs do not overlap.

When adding an OU, or editing a connected OU, use the Run option, in the Domain details screen, to set the synchronization frequency for that specific OU.

 

For adding multiple OU’s see,

 

Notes on SkipCSV

If after an ad-hoc sync or a scheduled Data Uploader sync, you see the following in the Data Uploader logs that a job has been skipped with the log: Skip Uploading CSVName.csv, this means that the csv was skipped because of an empty file, or that there is no new data to upload or update from that job (eg all rows are the same from source to target).

The CSV that was skipped will reside inside the folder path Prosperoware\CAM\DataUploader\skipcsv

 

External System Configuration

How does Data Uploader determine how to import

If the last modified date is in the SQL files, then the last modified date is used to check the data brought in. However, the Job Delta ensures the records from the sync before remain in the Staging tables. Only any new, or changed records get imported from there into CAM. Similarly, AD sync or other tabs will use the system configuration.

For example, if a user or group is imported, If the client has other conditions in the SQL Scripts to check if a duplicate record exists using if/then statements, these are used to bring over the initial records to staging.

Then the records are compared using the username or group name to see if it exists. The client can change these with the source mapping configurations by stating the unique ID columns exactly!

Lastly in the script or in the source mapping, if there is a last modified date inside, this is used to determine the date range to go back to check and pull data.

For Active Directory, all records are checked in Active Directory, unless a particular group or OU or container is defined.

Automating CSV Imports for Batch DU Jobs

  1. Users could automatically generate CSV files of their import jobs using their program of choice.

  2. Then place the files inside the newCSV folder.

  3. Run a DU job that is configured, and the program picks up the generated CSVs and imports them.

 

Data Uploader Files and Compatiblity

In V2 of the Data Uploader, the Jobs submit JSON files instead of CSV files in the backend to avoid overloading the queue processor.

Old Version Compatibility

If a firm requires the older version of Data Uploader, V1, make the following changes to the appsettings.config file:

\<setting name="v2Enabled" serializeAs="String"\> \<value\>False\</value\> \</setting\>

New Version Features

  • In addition to the new submission in JSON, there is a new folder where those CSVs are stored when the JSONs are uploaded. This folder is called newCSV. The CSVs are created and then converted from these files in this folder to JSON. If a job fails, this would be the folder that contains the JSON files to troubleshoot.

  • Generated CSVs still reside in the generatecsv folder for all jobs.

  • Archived CSVs still reside in the archivecsv folder for any successfully completed jobs over time.

 

 

Let's Connect📌

☎ +1 630.598.1100
☎ ‪+44 20 3880 1550‬
📧 support@litera.com
💻 https://www.litera.com/support/

📝 Support is available:
4 am - 8 pm US Eastern
(9 am - 1 am GMT/BST
7 pm - 11 am AET) on normal business days (excluding holidays)

© 2024 Litera