Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

Data Uploader is a CAM Windows utility to read data from your on-premises, document management, billing, NBI, or Active Directory system and upload the information to the CAM cloud for processing. Based on the rules, metadata properties, and configurations set up, CAM will create or update workspaces, users, groups, and group membership across all specified systems.

Info

Important: All Data Uploader users are encouraged to always run the latest build available for download, as that is always the most optimized. If you get a notification in Data Uploader itself, or from the CAM Administration Download tab that Data Uploader is outdated, you can find the latest version in your CAM Admin-> Downloads.

See the Data Uploader FAQ page for additional help.

Note: For each task added via the Data Uploader, a job is queued in the Jobs tab which will then be further processed by CAM. No changes are ever made to your on-premises database.

...

Data Uploader Process Workflow

Downloading Data Uploader from CAM

  1. Navigate to the Administration tab.

  2. Click on Downloads.

  3. Click on Data Uploader and save it to your machine.

Expand
titleInstall Data Uploader

Install Data Uploader

Info

Important: All Data Uploader users are encouraged to always run the latest build available for download (#552 as of Dec 2023), as that is always the most optimized. Issues like the validation of connection strings as ODBC in build #533 are no longer present in build #552.

This section describes the Connection Strings tab in Data Uploader and how Data Uploader is using, storing, and accessing data. In both the current and previous versions of Data Uploader, there exists a database called ClientApp2.sqlite. This database is used to store the scripts and results of the queries. This database is also used to run on-premises comparisons of the CSV files for matter provisioning and user and group management to improve performance and cut down on unnecessary transactions due to false positives.

An example of this might be that a matter property that is unrelated to the workspace was updated in the time & billing system. Because the modification date on the matter in that system was updated, it might feed through many matters into the provisioning CSV. However, since none of those matter changes are related to the workspace as provisioned by CAM, CAM still needs to compare each matter entry to make sure no data has changed that would need to be updated in the downstream system (iManage, NetDocuments, etc.). By running a comparison of the files by Data Uploader using an on-premises database, we can only submit matter jobs with true changes, thus improving overall performance.

Note: The minimum requirement for Data Uploader is a standard install of SQL.

Info

Important Information: if you are a larger organization, we might suggest you create an on-premises SQL database for this purpose. Sizing recommendations can be found at On-Premises Recommendation.

Using SQL

If you are using an SQL Database, it is recommended you recreate the connection strings to the source database for workspace provisioning. This will be outlined in the steps below.

  1. If you are using an on-premises SQL database, create a Database – suggested name = “CAMLocal”

  2. To create a connection string to this database, CAM will need an account that has dbo rights to this database so it can create the schema and manage the data in this database.

  3. The schema for this database will be created after you launch Data Uploader and create a connection string to this database outlined in the Create a Connection String section.

Running Data Uploader

Tip

Best Practice: When you first run the new Data Uploader, you must run it as an administrator.

  1. From the Windows Start menu, right-click on Prosperoware Data Uploader and Run as administrator.

  2. If Data Uploader prompts you to reauthenticate, put in the tenant URL then the user email and password for your CAM account.

  3. Click on the Connection Strings tab

  4. If using the SQLite database, and if you copied your connection strings configurations to the new ConnectionString.config, you should see your connection strings here.

    1. If not, then you can manually add the connection string by selecting Add, then selecting the “Select Connected System” as Source Database.

  5. If using an on-premises SQL database, you must first create the CAM Local Database connection string before creating any other connection strings.

Create a Connection String

  1. Click the Add button.

  2. Create a name for your connection string, e.g., CAM Local.

  3. From the “Select Connected system”, choose CAM Local Database.

  4. d. Select whether this is a direct SQL Connection or ODBC Connection.

    1. i. If ODBC, you will need to add a connection string.

    2. An example is: Data Source=100.10.200.220\MSSQL2017;Database=localdb;User=sa;Password=yourpassword.;Integrated Security=false

      1. Type in the name of the SQL Server or instance first. If in azure or a remote system include the ports first.

      2. Type in the name of the Database.

      3. Select login type and pass the correct credentials. Reminder: this account needs to have dbo rights to this database.

      4. Click Connect or Save

  5. You may now add all other desired connections to the data source for the workspace provisioning. Make sure to select Source Database from the “Select Connected System” drop-down.

Other Verifications

Click on the other tabs to verify that the data from the old configurations have been copied over correctly

SQL Files

  1. Click on the SQL Files tab and click the Refresh button if you do not see your SQL files. If you still do not see them, make sure they were copied to the correct directory.

  2. Next to each script, click the hamburger menu and select Edit.

    1. Select whether you want to enable this script and select the appropriate connection string against which this script will be executed. CAM supports multiple databases against which you can execute scripts.

  3. From the hamburger menu, you can also select the Validate option to test and validate your scripts before you run them in production.

Active Directory

Click on the Active Directory tab and check that your LDP configurations have been copied over correctly. If you do not see them, go back and verify you copied the data correctly from the old configuration file to the new one. Otherwise, you can recreate this by clicking the Add button.

CAM Agent Update

Data Uploader comes with a new CAM Agent. There is nothing you need to do to upgrade this – the old one will be uninstalled and a new one will be installed running under the service account credentials you provided during the installation.

If you did not enter the correct service account for the CAM Agent, then you can update this by doing the following:

  1. Go to the Windows Services

  2. Double-click on CAM Agent

  3. Click on the Log on tab. The CAM agent will be installed by the installer. The user has to select just Service Account and should have access to the CAM data uploader folder or accessible part.

Tip

Best Practice: If you are currently using the CAM Agent, just make sure it is running at some point during the upgrade.

Read here for complete instructions to configure and use the Cloud Data Uploader.

Note – CAM Agent service is a window service. By default, it’s Enabled, for verification make sure that it's running.

Tip

Tip: You can set multiple instances of the Data Uploader. For e.g. You can set an instance to run the SQL file to update the matter information based on a schedule and another instance to update the AD sync on a different schedule.

Info

Important: If an SQL File is disabled and the connection string is not set, the file will be skipped by default.

Expand
titleUpgrade Data Uploader

Upgrade Data Uploader

Note

Important: Please review all of these steps in detail before installing the new Data Uploader.

Installation

  1. Review the Hardware and Software Requirements for Data Uploader.

  2. You must create a backup of the existing DU database before the upgrade. If you are not utilizing a dedicated SQL database for DU, then skip this step.

  3. Make a backup copy of the current Data Uploader installation directory:

    • Prosperoware Data Uploader folder found at: C:\Program Files (x86)\Prosperoware\Prosperoware Data Uploader.

Info

Information: The installation path is selected programmatically while installing the Data Uploader.

Note: 

  1. Verify the logs so that no DU upload jobs are running. If there are jobs running, wait until they are done processing. This is a required step, which may result in data loss if jobs are running during the upgrade.

  2. End and disable DU Windows scheduled tasks (if applicable) to prevent them from running while the upgrade is happening.

  1. Log into your CAM website.

  • If you are already logged in, sign out and sign in back in to ensure the latest update is available.

  1. Navigate to Administration

  2. Click on Downloads and download the latest version of Data Uploader onto the server where DU is installed.

Note: Browsers typically block MSI and exe installers downloaded from the internet, which may interfere with the version upgrade. Unblock the DU msi installer by Right-click on the file properties.

  1. Run the installation wizard.

  2. Click Next. Keep the install folder path the same.

  3. Click Next.

Note: The installation directory has changed slightly to “C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\”

  1. This version of the Data Uploader now requires that you enter a Windows Services domain, username, and password of the service account.

  2. Click Install to begin the installation.

  3. Once the installation is complete, click Close to close the wizard.

Note

Important: Prior to launching Data Uploader, you must first complete all the steps below.

Updating the Configurations

The new Data Uploader application has replaced many of the previous configuration files with new config files. Since the following instructions for copying old configurations to the new files reference line numbers, we recommend using a program like Notepad++ or other tools that have line numbering available.

All configuration files are now in a directory under the install path called config located here:

C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\service\config

If you have made any configuration changes to the following files, please follow the instructions below to copy your configurations to the new files. If you have not modified any of these files, you can skip this section.

  1. ActiveDirectoryIntegration.config

  2. Data.config

  3. UserPropertiesMapping.config

  4. SourceFilesConfig

  5. ADMapping File

  6. AppSetting.config

ActiveDirectoryIntegration.config > ADMapping.config

If you did not make any changes to the ActiveDirectoryIntegration.config, you may skip to section 2.

  1. Open the backed-up ActiveDirectoryIntegration.exe.config file.

  2. Copy the lines between the <configuration> and </configuration> tags.

    1. Exclude lines

    Code Block
    <startup>
            <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
            </startup>
  3. Paste the configuration in the ADMapping.config file between lines 2-51.

  4. Use the commented lines 2-48 as mapping for AD. Samples are commented.

  5. Use commented lines 49-51 as fields for AD. Samples are commented.

Active Directory Filter

The organization units in Active Directory contain multiple groups with multiple users.
Now you can filter the user from a specific group in a single organization unit. The supported property is the "memberof." The "memberof" property takes the group name as a value and operators (%% - contains and !% - not contains) that are used to help in the filter. These parameters return the list of the users who are present or not present in the specific group.

The following are examples of supported operators:

Contain (%%)

Code Block
<ad-property name="memberof" value="testgroup" operator="%%" />   

Not Contain (!%)

Code Block
<ad-property name="memberof" value="testgroup" operator="!%" />

Wildcard

A wildcard is a character(s) that substitutes for another character or string of characters when searching a database.

Searching for values in the ad-property name property using a wildcard is now supported.

The following are examples of a supported wildcard:

Code Block
<ad-property name="company" value="tes" operator ="s%"/>

In the above example,

  • ad-property name - represents the search value or string.

  • Value – represents character(s). It must be present in the search value or string.

  • Operator – represents the searchable value. It must come either at the start of the string (s%) or at the end of the string (e%)

Data.config >ADMapping.config

If you did not make any changes to the Data.config, you may skip to section 3. If you applied filters for users and groups in the Data.config file, you will need to recreate these manually in the new ADMapping.config as there have been some changes. There are instructions in the ADMapping.config on how to apply the user and group filters.

  1. Open the backed-up Data.config file and note all the filters applied.

  2. Recreate the filter logic between the <configuration></configuration> tags.

User.config >ADMapping.config

Info

Note: If you did not make any changes to the Data.config, you may skip to this section, otherwise, continue.

Note

Warning: To exclude the user and group from AD sync to CAM, it should contain only one Organizational Unit.

  1. Open the backed-up User.config file

  2. Copy the fields between the <fields> and </fields> tags to the same tags in the ADMapping.config (located at the bottom of the file before the </configuration> tag.

Read more on ADMapping.config.

SourceFilesConfig

This is a new configuration file that allows you to set a separate unique identifier for each of your provisioning scripts. Some content systems might support a matter index ID for a unique identifier, while others might only be able to support client and matter ID. This file configures the unique ID per script.

  1. Open this file and update the uniqueId field. E.g. <uniqueId columns="clientId,matterid" source="CreateOrUpdateWorkspace.sql"/>

ADMapping.config

  • MultivaluedDatabasevalueMapping

    • It is a new database property value "MultivaluatedDatabase" which allows you to add multiple databases to the value field in AD-mapping that allows you to generate multi-input CSV or multi- delta CSV, respectively.

Sample Multivalueddatabase property value

Code Block
<property name="Database" values="firstDatabase|secondDatabase|thirdDatabase" appliesTo="iManage"/>

Note: Multivalued database property is applicable only for the iManage system.

  • SkipUpdateIfExists

    • While importing new AD users to iManage for the first time, the property name "SkipUpdateIfExists" with a value ="True" for iscamuser and iscamgroup has been added in the AD-Mapping that allows you to skip the update programmatically if it exists already.

Sample SkipUpdateIfExists property value - iManage

Code Block
<property name="SkipUpdateIfExists" values="true" appliesTo="iManage"/>

Sample SkipUpdateIfExists property value - M365

Code Block
<property name="SkipUpdateIfExists" values="TRUE" appliesTo="Office365"/>

Note: Specify "SkipUpdateIfExists" property value manually only for M365.

  • Preferred Database field

    • Add property value "PreferredDatabase", allows you to specify the user's Preferred Database on user creation

Sample PreferredDatabase value

Code Block
<property name="PreferredDatabase" values="WORKSITE2020" appliesTo="iManage"/>

Note: PreferredDatabase property is applicable only for the iManage external system.

Password flag Mapping

The property name “passwordExpired“ allows you to set the password expiration against user creation in iMange. Allows you to set password update on the first login after user creation in iManage using the property name “ForceUserToChangePassword“.

Note: passwordExpired and ForceUserToChangePassword properties are applicable only for iManage.

Sample Password flag Mapping

Code Block
<property name="PasswordExpired" values="True" appliesTo="iManage"/>
<property name="ForceUserToChangePassword" values="False" appliesTo="iManage"/>

SQL Scripts

  1. Copy the scripts from the C:\Program Files (x86)\Prosperoware\Data Uploader\sql file to the new. C:\Program Files (x86)\Prosperoware\CAM\Data Uploader\sql

Info

Please Note: Out of the box the CreateOrUpdateWorkspace.SQL script is included in the sql folder. You can delete this script if you already have scripts created. If you have not created any scripts, you can edit this script for use in provisioning.

  1. Move any additional custom scripts to this SQL folder.

Note: Users and groups can be created using two ways, one by uploading the API or CSV file and two by using the AD.

AppSetting.config

It provides a new export option to allow you to exclude duplicate documents like emails from the iManage and NetDocuments system during export.

The following code snippet allows you to create a folder where all duplicate emails are present:

Code Block
<"SkipDupeDetection" value="0" />
<"DupeDetectionType" value="1" />
<"DupeFolderPath" value="Duplicate" />
<"EmailProfileColumns" value=""MessageId#Name,DateTimeReceived,DateTimeSent" " />

The following code snippet allows you to skip all duplicate emails during export:

Code Block
<"SkipDupeDetection" value="0" />
<"DupeDetectionType" value="0" />
<"DupeFolderPath" value="foldername" />
<"EmailProfileColumns" value="MessageId#Name,DateTimeReceived,DateTimeSent" />

Note: 1. DupeFolderPath" value="foldername - Duplicate folder name value allows you to create folder name.

2. EmailProfileColumns" value="MessageId#Name,DateTimeReceived,DateTimeSent" - The Email profile columns value includes “# - indicates OR operation“ and “, - indicates AND operation“.

Data Uploader Limits

Info

Important: Currently, you can set the daily Throttle limit to a maximum of 50,000 jobs for optimum and error-free performance. Daily, you can upload the maximum jobs that are permitted as per the Throttle limit. After the daily limit is reached, you can find the Throttle limit warning message in the log.

Note

Important: CAM has a limitation of 492 characters in the group name when adding to the group path.

Required Permissions

...

Permissions

...

Allows User To

...

View Data Uploader

...

To download Data Uploader from the Administration tab->Downloads.

Tip: Read here for a complete list of available permissions and instructions to allow/deny CAM permissions to users/groups.

Related Topics

...

Child pages (Children Display)
alltrue
depth0
allChildrentrue
style
sortAndReverse
first0