Skip to main content

File exchange​

Connect your storage systems to exchange files with Enigma. You can send data for batch enrichment and receive processed results delivered directly to your connected storage.

How file exchange works​

File exchange provides bidirectional data transfer between your systems and Enigma:

  1. Send files to Enigma - Upload data files for batch enrichment, matching, or processing
  2. Receive results from Enigma - Get enriched records, match results, and processed data delivered to your storage

This workflow supports large-scale data operations that go beyond individual API calls.

File path length limit

File paths within any connected source cannot exceed 110 characters. This includes the full path from the root of your bucket or SFTP directory.

Connection options​

TypeBest for
Enigma SFTP sourceGetting started quickly with Enigma-hosted infrastructure
Your SFTP sourceOrganizations with existing SFTP servers
Your AWS S3 sourceAWS-native workflows and large file volumes

Connect a storage source​

  1. Navigate to Settings > Account > Data connections.
  2. Click Connect new source.
  3. Select your source protocol from the dropdown.
  4. Enter a nickname to identify this connection in the Console.
  5. Click Confirm to create the source.
  6. Configure the source settings (see the following sections for each source type).

Enigma SFTP​

Enigma provides a dedicated SFTP server for your organization. This option requires minimal setup since Enigma manages the server infrastructure.

Setup​

  1. Select Enigma SFTP source when creating a new source.
  2. Enter a nickname for the connection.
  3. After creating the source, add your SSH / SSH2 key(s) (required). Enigma installs these keys on your dedicated server for authentication.

Connection details​

After setup, the Console displays your SFTP connection details under Enigma credentials:

  • Server name: The SFTP server address
  • User name: Your organization's username

Use these details with your SSH private key to connect from any SFTP client. The Console also displays Enigma's PGP public key, which you can use to encrypt files you send to Enigma.

Your SFTP server​

Connect Enigma to your own SFTP infrastructure when you need to use existing servers or maintain control over the SFTP environment.

Prerequisites​

  • An SFTP server accessible from the internet
  • Credentials for Enigma to authenticate (username with password or SSH key)

Setup​

  1. Select Your SFTP source when creating a new source.
  2. Enter a nickname for the connection.
  3. After creating the source, configure the following:
    • Server name (required): Your SFTP server hostname
    • Username (required): The account Enigma uses to connect
    • Authentication method: Choose Password or SSH private key, then provide the corresponding credential
  4. Optionally configure:
    • Folder path to read files from: Directory where Enigma reads your input files
    • Folder path to write files to: Directory where Enigma delivers results

When using SSH key authentication, install the Enigma host SSH key (displayed under Enigma credentials in the Console) on your SFTP server.

Your S3 bucket​

Connect an AWS S3 bucket for file exchange. S3 is ideal for large file volumes and integrates with AWS-native workflows.

Authentication options​

The configuration form accepts either access-key credentials or an IAM role ARN. Provide one of the two.

Option 1: Access keys​

  1. Create an IAM user in your AWS account.
  2. Attach the S3 permissions policy (see the following section).
  3. Generate an access key for the IAM user.
  4. In the Console, enter the AWS access key ID and AWS secret access key.

IAM roles provide temporary credentials and don't require managing long-term access keys.

  1. Create an IAM role in your AWS account.
  2. Attach the S3 permissions policy (see the following section).
  3. Add Enigma to the role's trust policy (see the following section).
  4. In the Console, enter the AWS role ARN.

Required S3 permissions​

Attach this policy to your IAM user or role. Replace YOUR-BUCKET-NAME with your actual bucket name.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
    }
  ]
}

Trust policy for IAM role​

If using IAM role authentication, add this statement to your role's trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::435097078665:role/enigma-file-manager-service-prod"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Enigma includes an external ID when assuming your role. To scope the trust policy more tightly, add an sts:ExternalId condition; contact support@enigma.com for your organization's external ID value.

S3 configuration options​

After creating the source, you can configure:

  • Bucket name: Your S3 bucket
  • Region name: AWS region where the bucket is located
  • Endpoint URL: Optional custom S3 endpoint
  • Folder path to read files from: Path prefix where Enigma reads your input files
  • Folder path to write files to: Path prefix where Enigma delivers results
  • PGP key (optional): Enigma encrypts all delivered files with your key

File encryption​

For additional security on S3 sources, provide your PGP public key in the source configuration. When a PGP key is configured, Enigma encrypts all files before delivering them to your storage. You decrypt the files using your corresponding PGP private key.

For all source types, the Console displays Enigma's PGP public key, which you can use to encrypt files you send to Enigma.

This encryption is in addition to the transport-layer security (TLS for S3, SSH for SFTP) already protecting data in transit.

Last updated on