Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2

AWS Client Stack

CAM does not store, nor take possession of, the organization’s documents at any point. CAM only stores metadata about the documents, workspaces, sites, Teams, channels and folders. Whenever document storage is required, either temporarily using ETL or for longer term with Business Continuity, it uses a cloud storage bucket controlled by the organization. Prosperoware provides a small piece of code to be configured on that cloud storage bucket to provide the integration needed for content sync or storage. Currently we support AWS S3 buckets as an option, but we plan to support Azure as an alternative.

CAM utilizes this feature set in 2 ways:

  1. Business Continuity: CAM can sync documents in the organization’s DMS to the AWS S3 bucket so if the organization’s cloud DMS goes down, users can access their documents securely through CAM using a one-time password.

  2. ETL (Extract Transform Load) Manager: Users can move documents stored in MS 365 Teams and Channels to the DMS, or from the DMS to Teams in order to facilitate easy access for the end-user to access documents when needed, while providing governance around that content so it ends up in the desired system of record for documents.

In both instances, all documents are stored in the organization’s AWS S3 bucket and the synching happens between AWS S3 and the DMS systems.

Pre-requisites

  1. Configure AWS at Client - To Access the client’s AWS account with access keys .

  2. Configure AWS on CAM - To Access AWS from CAM to Sync contents with access keys.

Configure AWS at Client

Create below resources manually

 1. Create SQS Queues

Create 2 SQS queues manually as follows:

  1. contentsync-prod-job-process-v1

Description

Standard Queue

Receive Message Wait Time: 0 seconds

Message Retention Period: 14 days

Maximum Message Size: 256 (kB)

Don't use redrive policy

Don't use SSE (Server-Side Encryption)

Delivery Delay: 0 seconds

Default Visibility Timeout: 16 minutes

contentsync-prod-etl-process-v1

Description

Standard Queue

Receive Message Wait Time: 0 seconds

Message Retention Period: 14 days

Maximum Message Size: 256 (kB)

Don't use redrive policy

Don't use SSE (Server-Side Encryption)

Delivery Delay: 0 seconds

Default Visibility Timeout: 16 minutes

 2. Create S3 Buckets

Create 2 buckets manually:

content-sync-configuration-$subdomain - This bucket will use to share AWS setup script to user.

$subdomain-prosperoware-io-encrypted-bucket - This bucket will use to store client's content. Set default encryption to AES-256

πŸ“— Note: $subdomain Specify your subdomain name.

 3. Create VPC
  • Prosperoware will share the scripts to create VPC.

  • Download the scripts will be available at content-sync-configuration-$subdomain/vpc-configuration/.

  • Configure AWS CLI. Follow instructions provided at AWS: AWS CLI.

  • Execute below command on the terminal to create VPC:

  • sh deploy.script create-stack --region us-east-1

πŸ“— Note: --region Specify which AWS Region to send this command's AWS request to.

 4. Create Security Group for Lambda
  1. Head over to VPC

  2. Under Security Group click Create Security Group:

  • Set the security group name to Lambda-SG

  • Select the VPC created from previous step.

  • Add a rule for inbound and outbound as All Trafic and Source 0.0.0.0/0 or Destination as 0.0.0.0/0

 5. Modify and upload the shared content to S3 bucket

Update below 2 files

  1. Open params-prod.yml available at S3Content/serverless/params/us-east-1

    • LambdaRole (update the AWSACCOUNTID to your AWSACCOUNTID)

    • SecurityGroup (Lambda-SG created in the previous step)

    • SubnetIds (please update the SubnetIds to the private subnet IDs of the VPC created earlier)

    • ContentSyncJobProcessQueue (update the AWSACCOUNTID value to your AWSACCOUNTID)

  2. Open appconfig-prod.yml available at S3Content/config/ymls

    • CS_JOB_QUEUE (Update the AWSACCOUNTID value to your AWSACCOUNTID

  3. Upload these files in S3 bucket content-sync-configuration-$subdomain

 6. Create CodeBuild project
  1. Go to CodeBuild Console

  2. Click Create build project.

  3. Setup a Project name and Description in section Project Configuration.

  4. Select the source provider to S3.

  5. Choose the bucket content-sync-configuration-$subdomain created in previous steps.

  6. Select s3 object key or folder type “serverless/

πŸ“— Note: This folder on S3 used for stack.

7. Select operating system Ubuntu as Environment Section

8. Set the Runtime (s) to standard. Set image to: aws/codebuild/standard:2.0. Image version to Always use the latest image for this runtime version. And environment type to Linux.

9. Leave the privileges section unchecked.

10. Select New service role and set the role name to contentsync_role.

11. Click on Additional Configuration.

12. Set VPC to created in the previous steps.

13. Set the Subnets to Private Subnet #1 and Private Subnet #2.

14. Set the Security Group to the Lambda Security Group created at Step #3.

15. Under BuildSpec, select insert build commands and insert the build spec file shared at S3Content/serverless/buildspec.yml.

πŸ“— Note: Update --region if it's other than us-east-1.

16.On the buildspec file there are some pre-configured commands that don’t need changes:

  • The runtime version. Installing serverless. Deploying the Stack to the desired AWS region.

17. No change to artifacts.

18. Enable CloudWatch logs option.

19. Set The group name and stream name to blank.

20. Click create build project.

 7. Create IAM Lambda Role

Name the Lambda role: content_sync_lambda_service_node

Provide following permissions to Lamda role:

  • AmazonEC2FullAccess

  • SecretsManagerReadWrite

  • AmazonSQSFullAccess

  • AmazonS3FullAccess

  • AmazonDynamoDBFullAccess

  • AmazonSESFullAccess

  • AmazonVPCFullAccess

  • AmazonSNSFullAccess

  • ContentSyncLambdaBasicExecution(Managed Policy)

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"logs:CreateLogStream",
				"logs:PutLogEvents"
			],
			"Resource": "arn:aws:logs:*:*:*"
		},
		{
			"Sid": "VisualEditor1",
			"Effect": "Allow",
			"Action": "logs:CreateLogGroup",
			"Resource": "arn:aws:logs:*:*:*"
		}
	]
}
	
  • KMSWriteLambda(Managed Policy)

{
	"Version": "2012-10-17",
	"Statement": [
	  {​​​​​
		"Sid": "VisualEditor0",
		"Effect": "Allow",
		"Action": [
			"kms:DescribeCustomKeyStores",
			"kms:Decrypt",
			"kms:Encrypt",
			"kms:GenerateDataKey",
			"kms:ReEncryptTo",
			"kms:GenerateDataKeyWithoutPlaintext",
			"kms:DescribeKey",
			"kms:GenerateDataKeyPairWithoutPlaintext",
			"kms:GenerateDataKeyPair",
			"kms:ReEncryptFrom"
		],
		"Resource": "*"
	  }​​​​​

 8. Edit IAM Role for Code build

On the IAM console go to Roles.

Click contentsync_role.

Under Permissions of role, attach the AdministratorAccess Policy.

πŸ“— Note: This policy is managed by AWS and permission is given to the CodeBuild service alone. It is required to create and update multiple services such as CloudFormation stack, Lambda function, DynamoDB table, etc.

 9. Deploy Stack
  1. Go to CodeBuild service console.

  2. Click the created Content Sync project under Build Projects.

  3. Click Start Build.

The data under Start Build section is predefined and do not be overwritten.

 10. Validate the Lambdas

Go to Lambda console

Validate following 8 lambda functions are available, which are required for Content Sync and ETL:

  1. contentsync-io-contentsync-worker-v1

  2. contentsync-io-contentsync-api-v1

  3. contentsync-io-contentsync-manager-v1

  4. contentsync-io-contentsync-etl-worker-v1

  5. contentsync-io-contentsync-etl-mapping-manager-v1

  6. contentsync-io-contentsync-etl-renewal-manager-v1

  7. contentsync-io-contentsync-etl-renewal-worker-v1

  8. contentsync-io-contentsync-etl-mapping-worker-v1

 11. Function code section

Click Upload under function package. Upload the jar shared at S3Content\lambda-functions-code folder, and click save on top right corner.

 Security Configuration and Validation

AWS operates under a shared responsibility model. AWS takes care of security ‘of’ the cloud while we (AWS customers) are responsible for security ‘in’ the cloud. Lets validate the security for each AWS service used.

Identity and Access Management (IAM)

  1. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.

  2. Do not access AWS with root administrator instead encourage to use user defined in IAM

SQS Queues

  • The queues use the default encryption provided by AWS.

S3 - Simple Storage Service

  1. Default encryption – The second bucket is where the content will be stored, is encrypted with AES-256. This helps securely stores the files residing in the bucket.

  2. Object lock – Prevents certain objects from being deleted.

  3. Bucket policy, ACLs, and CORS - Not be accessible to the open web, or even from other accounts within your AWS Organisation. To protect sensitive data, permissions can be set by configuring.

VPC

  • One VPC per your environment with 2 private subnets and 2 public subnets. The private subnets will be used when setting up the deployed resources so there’s no external access to the AWS resources provisioned.

Lambda Security Group

  1. The Lambda security group is created inside the VPC so there’s no outside access allowed to the resources using the security group.

  2. The inbound and outbound rules will allow access to all the services in the VPC.

  3. Create lambda role - The lambda role will be created to share access between Lambda and other resources; and other resources between themselves.

  4. Custom policies allows the lambda to create and write logs into CloudWatch

DynamoDB

  1. Data is redundantly stored on multiple devices across multiple facilities in an Amazon DynamoDB Region

  2. All user data stored in Amazon DynamoDB is fully encrypted at rest, it encrypts your data using 256-bit Advanced Encryption Standard (AES-256),

  3. DynamoDB encryption at rest provides an additional layer of data protection by securing your data in an encrypted table—including its primary key, local and global secondary indexes, backups.

  4. Use IAM roles to authenticate access to DynamoDB

    Use IAM policies for DynamoDB base authorization

Set up security monitoring

  1. API Gateway

    1. This AWS Service is for creating, publishing, maintaining, monitoring and securing APIs.

    2. In CAM we use REST APIs integrated with Lambda functions endpoints or other AWS Services to restrict unauthorized access.

    3. The data which is in transit is encrypted by default through API Gateway

  2. CloudTrails

    1. AWS Cloudtrail is another service we use as an integrated service with API Gateway

    2. Cloudtrail is a service that provides a record of events or actions taken by a user, role or AWS Service in API Gateway.

    3. Cloudtrail captures all REST API calls. This helps us strengthen security on API Gateway by auditing and monitoring internally the REST API calls being made from the API Gateway console and also code calls to the API Gateway service APIs.

  3. GuardDuty

    1. Amazon GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in AWS.

    2. GuardDuty analyzes tens of billions of events across multiple AWS data sources, such as AWS CloudTrail event logs, Amazon VPC Flow Logs, and DNS logs.

    3. We have enabled Amazon GuardDuty in all AWS Regions (in those which are not using either) as recommended by AWS for better security protection.

    4. The following list contains some types of attacks that GuardDuty can detect:

      • Denial Of Service

      • Brute Force

      • Tor Client

      • Attacks from different OS (Kali Linux, Parrot Linux, Pentoo Linux)

      • Phishing Domain Request

Configure ETL on CAM

Click here for details.

  • No labels