Configure Data Uploader
Overview
The following section guides you on how to configure and use Data Uploader (DU) efficiently to get optimum performance.
For additional help, see the Data Uploader FAQ page.
- 1 Installation Requirements
- 2 Handling Bulk Jobs and Load Balancing within Data Uploader
- 2.1 Best Practices
- 3 Data Uploader Limits
- 4 To Download Data Uploader
- 5 Installing Data Uploader
- 6 Upgrade Data Uploader
- 6.1 Installation
- 7 Updating the Configurations
- 8 Source Files Config
- 9 Modifying Application Settings
- 10 Filtering Active Directory Users/Groups
- 10.1.1 ADMapping.config
- 10.2 To set the config:
- 10.2.1 OU filtering
- 11 AppSettings
- 11.1.1 Available Settings
- 11.1.2 App Settings Implementation Checklist
- 11.1.2.1 Must Review or Add
- 11.1.2.2 Optional or Default System Settings
- 11.1.3 SQL Scripts
- 11.1.4 Testing new workspace provisioning scripts
- 11.2 1. Review and Cleanup Active Directory Organization Unit Data
- 11.3 2. Connect to Active Directory Organization Units via Data Uploader
- 11.4 3. Configure ADMapping.config
- 11.5 4. Update SourceFiles.config with Organization Units
- 11.6 5. Test the AD Sync Configuration
- 11.7 6. Schedule AD Sync
- 11.8 Notes on SkipCSV
- 12 External System Configuration
- 13 CAM Credentials tab
- 14 Connection String tab
- 15 Creating and Connecting to the CAM Local Database
- 16 Active Directory tab Connect to to AD
- 17 Additional AD Information
- 18 Scheduling multiple OU or Container syncs
- 19 Local Sync tab
- 20 SQL Files tab
- 21 Setup the Schedule button
- 21.1 Validate button
- 21.2 Run Now button
- 22 Task Scheduler Configurations
- 23 Logging of the Job
- 24 Sync with an On-Premises SQL Database
- 25 Mapping Active Directory (AD) properties
- 25.1 To set the property mapping
- 25.2 Passing Users via CSV
- 25.2.1 OS Type Ids
- 26 Configure Metadata names to Export the Document to Network Path
- 27 Automating CSV Imports for Batch DU Jobs
- 28 Data Uploader Files and Compatiblity
Handling Bulk Jobs and Load Balancing within Data Uploader
Best Practices
Set the number of jobs under the 50k limit for better performance, to ensure the jobs run. E.g. if undeclaring more than 50k records, batch into 40k amounts per day.
Use Data Uploader if the bulk actions from Service Desk or Analytics can’t handle the number of records that need to be modified.
Data Uploader Limits
To Download Data Uploader
Navigate to Administration.
Click on Downloads.
Download the Data Uploader EXE installer.
Navigate to the download location and double-click the program to start the installer.
Updating the Configurations
The new Data Uploader application has replaced many of the previous configuration files with new config files after an upgrade happens. Since the following instructions for copying old configurations to the new files reference line numbers, we recommend using a program like Notepad++ or other tools that have line numbering available.
All configuration files are now in a directory under the install path called config located here:
C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config
If you have made any configuration changes to the following files, please follow the instructions below to copy your configurations to the new files. If you have not modified any of these files, you can skip this section
Data.config
UserPropertiesMapping.config
SourceFilesConfig
ADMapping File
AppSetting.config
SQL Scripts
Copy the scripts from the C:\Program Files (x86)\Prosperoware\Data Uploader\sql file to the new. C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\sql
Move any additional custom scripts to this SQL folder.
Testing new workspace provisioning scripts
To ensure the successful deployment of a new workspace provisioning script, testing is crucial. Follow these recommended steps for effective testing:
· Validate the script in SQL and review the output to identify any issues with missing metadata, templates, or with script logic.
· Update and revalidate the script to address any problems.
· Use Data Uploader to generate a CSV and examine the data for accuracy.
Remove all but a few records from the CSV and submit a test job with this smaller dataset to ensure accuracy before full deployment.
Active Directory Synchronization Overview
Setting up AD Sync is critical for synchronizing users and groups to CAM from Active Directory. The process is driven by a connection to each AD Organizational Unit (OU) where the users and groups to be synced are located, and configuring the ADMapping.config file. All the OUs are filtered through this one file by the configuration of criteria filters and other attributes and properties that can be set. Ultimately, the result is a CSV of the applicable people which can be scheduled in Data Uploader to automatically be sent to CAM for processing.
Here are the recommended steps for the setup and configuration of AD Sync:
Review and cleanup Active Directory Organizational Unit data.
Connect to Active Directory Organization Units via Data Uploader.
Configure ADMapping.config.
Update SourceFiles.config with Organization Units.
Test the AD Sync configuration.
Schedule the AD Sync.
1. Review and Cleanup Active Directory Organization Unit Data
AD data review and cleanup should focus on purging non-user entities such as printers, multi-function devices (MFDs), and other legacy connected devices from each OU to ensure only user data is synchronized, preventing unnecessary entries in the DMS that could complicate the data environment.
2. Connect to Active Directory Organization Units via Data Uploader
In Data Uploader’s Active Directory tab, use the Add button to add each Organization Unit that contains the users and group you wish to sync. There is no limit to the OUs that can be added for synchronization.
For information on adding Organization Units, please see Configure Data Uploader | Active Directory tab Connect to to AD
For adding multiple OU’s see, Configure Data Uploader | Scheduling multiple OU or Container syncs
3. Configure ADMapping.config
Overview
The ADMapping.config file, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, is pivotal for configuring AD Sync. This configuration tool defines the user and group data that will be organized in the CSV sent to CAM for processing, serving as the blueprint for synchronization.
All connected Organizational Units are configured within this single ADMapping.config file. The file contains three Objects (UsersAndGroups, UsersOnly, and GroupsOnly) with criteria that act as filters. For each OU, multiple instances of these Objects can be created, and their criteria configured to determine the specific users and groups to be included in the AD Sync process. Properties and values for these users and groups can also be directly specified in the file, affecting the information that will be included in the CSV. Once the ADMapping.config file has been configured, a CSV reflecting the user and group data configuration can be produced and provided to CAM for processing.
Anatomy of the ADmapping.config File
Objects Section
Three objects are available: UsersAndGroups, UsersOnly, and GroupsOnly.
Each Object contains a <criteria> and <values> section.
A. The <criteria> section is used to configure the filter for the users and groups that will be included in the AD Sync process. The ad-property and value are used to set the criteria and value for the filter, and the operator facilitates logic operations for the filtering process.
B. The <values> section provides the ability to set properties and values for the users being filtered by the criteria section. See the CSV parameters for User and Group Provisioning section for more information on setting properties and values.
To configure multiple OUs for synchronization, copy Objects as many times as needed, then add the OU for the Object via the DC attribute.
<default>, <fields>, and <groupfields> Sections
The <default> section is used to assign a default value to all the records being filtered above. If for example, you are setting the same database for all the records, instead of setting this value in the values section for each Object, set the Database property in the Default section.
The Fields and Groupfields sections are used to map specific Active Directory attributes to a CSV property. The AD attribute’s value will be added to the CSV in a column with the property name you specify. For example, change the “mail” attribute in AD to “userid” for iManage.
CSV Parameters for User and Group Provisioning
Parameters are available for tasks such as creating, editing, deleting, and disabling users and groups, as well as removing users from MyMatters in iManage. For the complete User and Group Provisioning parameter list, visit our documentation.
These parameters are coded as properties in the ADMapping.config file to set specific values for the synchronization process.
Below are examples of setting properties:
· Disabled. This property is used to enable or disable a user or group in CAM or another system. Set to True/1:
property name="Disabled" values="1"
· IsCAMUser and IsCAMGroup. This property determines whether to create a CAM user or group, the people and groups who will use CAM, during the AD Sync process. Setting to 1 will trigger creation; Setting to 0 will prevent the creation.
property name="isCAMUser" values="1"
property name="isCAMGroup" values="1"
· Database. Determines the specific library or libraries to which users and groups will be associated. This property is essential for ensuring that the AD Sync process targets the correct database.
o The value should be the exact name for the target database.
o When associating multiple databases, separate each with a pipe character (|).
o Use with the Preferred Database property when specifying multiple databases to indicate which database is primary.
property name="Database" values="HR|Active" appliesTo="iManage"
property name="PreferredDatabase" values="HR" appliesTo="iManage"
For more information, see Configure Data Uploader | Mapping Active Directory (AD) properties and Configure Data Uploader | Filtering Active Directory Users/Groups
4. Update SourceFiles.config with Organization Units
Add an entry for all connected OUs in the AD Sync process to the SourceFiles.config. See the Source Files section for more information on adding connected OUs to the SourceFiles.config.
The SourceFiles.config, located in C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config, should include an entry for each active SQL script and each connected Active Directory Organizational Unit. The file ensures Data Uploader only processes and sends new or modified jobs to CAM by enabling the JobDelta table in the CAM Local Database to track job submissions.
5. Test the AD Sync Configuration
Before scheduling AD Sync, use Data Uploader to generate a CSV, then submit a job with a small subset of the data to ensure accuracy.
On Data Uploader’s Active Directory tab, click Generate CSV for the OU being tested. Open the CSV and examine the data to verify the configuration has produced the right results. Next, remove all but a few records from the CSV and submit a test job with this smaller dataset, then verify the results of the job.
When user and group records are not updating, check:
· Does the user meet the filter conditions defined within the <criteria> section for the relevant Object in the ADMapping.config file?
· Does the user exist in the external system, and is the “SkipUpdateIfExists” configuration set to True in the ADMapping.config file?
· Does the user have an entry in the JobDelta table? Is there a change to the user record? Existing users with no changes will be skipped.
· Is the OU where the user is a member included in the scope of the ADMapping.config file?
6. Schedule AD Sync
The best strategy for scheduling AD Sync is to schedule each OU individually from Data Uploader’s Active Directory tab. It is important to determine the frequency of changes applicable to not only each OU, but also scheduled SQL scripts to ensure jobs do not overlap.
When adding an OU, or editing a connected OU, use the Run option, in the Domain details screen, to set the synchronization frequency for that specific OU.
For adding multiple OU’s see, Configure Data Uploader | Scheduling multiple OU or Container syncs
Notes on SkipCSV
If after an ad-hoc sync or a scheduled Data Uploader sync, you see the following in the Data Uploader logs that a job has been skipped with the log: Skip Uploading CSVName.csv, this means that the csv was skipped because of an empty file, or that there is no new data to upload or update from that job (eg all rows are the same from source to target).
The CSV that was skipped will reside inside the folder path Prosperoware\CAM\DataUploader\skipcsv
External System Configuration
How does Data Uploader determine how to import
If the last modified date is in the SQL files, then the last modified date is used to check the data brought in. However, the Job Delta ensures the records from the sync before remain in the Staging tables. Only any new, or changed records get imported from there into CAM. Similarly, AD sync or other tabs will use the system configuration.
For example, if a user or group is imported, If the client has other conditions in the SQL Scripts to check if a duplicate record exists using if/then statements, these are used to bring over the initial records to staging.
Then the records are compared using the username or group name to see if it exists. The client can change these with the source mapping configurations by stating the unique ID columns exactly!
Lastly in the script or in the source mapping, if there is a last modified date inside, this is used to determine the date range to go back to check and pull data.
For Active Directory, all records are checked in Active Directory, unless a particular group or OU or container is defined.
Automating CSV Imports for Batch DU Jobs
Users could automatically generate CSV files of their import jobs using their program of choice.
Then place the files inside the newCSV folder.
Run a DU job that is configured, and the program picks up the generated CSVs and imports them.
Data Uploader Files and Compatiblity
In V2 of the Data Uploader, the Jobs submit JSON files instead of CSV files in the backend to avoid overloading the queue processor.
Old Version Compatibility
If a firm requires the older version of Data Uploader, V1, make the following changes to the appsettings.config file:
\<setting name="v2Enabled" serializeAs="String"\>
\<value\>False\</value\>
\</setting\>
New Version Features
In addition to the new submission in JSON, there is a new folder where those CSVs are stored when the JSONs are uploaded. This folder is called newCSV. The CSVs are created and then converted from these files in this folder to JSON. If a job fails, this would be the folder that contains the JSON files to troubleshoot.
Generated CSVs still reside in the generatecsv folder for all jobs.
Archived CSVs still reside in the archivecsv folder for any successfully completed jobs over time.
Let's Connect📌
☎ +1 630.598.1100
☎ +44 20 3880 1550
📧 support@litera.com
💻 https://www.litera.com/support/
📝 Support is available:
4 am - 8 pm US Eastern
(9 am - 1 am GMT/BST
7 pm - 11 am AET) on normal business days (excluding holidays)
© 2024 Litera