Skip to main content

​
Prerequisites

  • By default, SFTP uses keypair authentication for access. You will need a provided public key to configure your destination. It will look roughly like this:
Public key format
1

Create a user on the SFTP server

Log in to the SFTP server and complete the steps below.
  1. Create group sftpwriter:
Create group
  1. Create user sftpwriter:
Create user
  1. Switch to the sftpwriter user:
Switch user
  1. Create the .ssh directory:
Create .ssh directory
  1. Set permissions:
Set directory permissions
  1. Navigate to the .ssh directory:
Navigate to .ssh
  1. Create the authorized_keys file:
Create authorized_keys file
  1. Set permissions:
Set file permissions
  1. Add the public key to the authorized_keys file. The key, including the “ssh-key” and comment, should be all on one line in the file, without linebreaks.
Add public key
2

Add your destination

Use the following details to complete the connection setup: host name, folder name, username, port and preferred delimiter character.
Write permissions at the SFTP root are requiredIn addition to write access within your configured <folder>, this destination writes per-transfer manifest files under a _manifests/ directory created at the root of the SFTP home/path. Ensure the SFTP user can create and write to _manifests at that root (even if your data lands under a subfolder). Manifests allow downstream systems to detect when a transfer is complete. See the FAQ below for how these files are organized.
Connection Test File Cleanup (Optional)By default, a connection test is performed against the SFTP destination during initial configuration. This test writes temporary artifacts under the prefix _test_connection/ (located at either the SFTP root, or inside the custom folder prefix configured on the destination, if present).If the SFTP user does not have permission to delete files, the connection test will still succeed, but these test files will remain on your SFTP server. To avoid clutter, you can optionally configure a custom periodic script (such as a cron job running on your SFTP host) to delete files/directories matching the _test_connection/ prefix (or <your_folder>/_test_connection/ if a custom folder was configured), or delete them manually.
Optional: PGP encryptionSFTP files are encrypted in transit, by virtue of the SFTP protocol. We offer an optional, additional layer of encryption at rest for SFTP files using PGP encryption.To enable PGP encryption, you will need to generate your own PGP public/private key pair, and provide the public key on a per-destination basis (one for each destination) when configuring each PGP-enabled destination. The public key must be provided in ASCII armored format, beginning with the header line -----BEGIN PGP PUBLIC KEY BLOCK-----, and ending with the tail line -----END PGP PUBLIC KEY BLOCK-----. For security reasons, only RSA and ECC keys are supported. RSA keys must have a key size of 2,048 bits or more.If PGP encryption is enabled, both the manifest files and landed data will be encrypted in the PGP binary format. They will have an additional .pgp file extension appended to their filename. For example, encrypted CSV files will have filenames like your_data.csv.pgp.It is not possible to enable or disable the PGP encryption settings of an existing destination. You must create a new destination from scratch with the new PGP configuration.

​
Permissions checklist

  • SFTP user created with SSH key-based authentication.
  • Provided public key added to ~/.ssh/authorized_keys for the SFTP user.
  • SFTP user has write access to the configured folder.
  • SFTP user can create and write to _manifests/ at the SFTP home/path root.
  • Firewall or network perimeter allows the service’s egress IP to connect on port 22.
  • (If using PGP encryption) PGP public key in ASCII armored format is ready to provide during destination configuration. RSA keys must be 2,048 bits or larger.

​
Frequently asked questions

The data will be loaded with the configured file format (Parquet, CSV, or JSON/JSONL) in a predictable folder structure that can be easily parsed by downstream systems.
sftpwriter_home_folder
some_provided_folder
some_table_a
dt=2024-01-01
0_20240101181004.csv
1_20240101184002.csv
dt=2024-01-02
0_20240102180123.csv
dt=2024-01-03
0_20240103182145.csv
some_table_b
dt=2024-01-01
0_20240101186004.csv
dt=2024-01-02
0_20240102185123.csv
dt=2024-01-03
0_20240103187145.csv
Use SSH key-based authentication for a dedicated, least-privileged SFTP user. Restrict access to only the required directories (e.g., chroot), and allowlist the service’s static egress IP at your network perimeter.
Parquet (default/recommended), CSV, and JSON/JSONL.
Each transfer writes a manifest JSON file per model under _manifests/ at the root. Files follow the pattern: _manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json. Use these manifests to trigger downstream processing.
File-based destinations are append-oriented. The change-detection process uses a lookback window to prevent missed records, which can create duplicates across adjacent transfers. Downstream pipelines can deduplicate by primary key prioritizing rows in the most recent transfer window.
We do not support providing your own public key for security reasons. The private key is securely generated and stored in our system and is never shared externally.
Yes, you can optionally provide your own PGP public key for us to encrypt the SFTP files with, on a per-destination basis (one for each destination). The public key must be provided in ASCII armored format.
No, the PGP key is not an alternative to the mandatory SSH/SFTP key. The PGP key is optional, while the SFTP key is mandatory. The two kinds of keys serve functionally distinct purposes. The SSH/SFTP key is used for authentication and encryption in transit. On the other hand, the PGP key is not used for authentication, and is instead used to add another layer of encryption of files in transit, and importantly, to ensure that the files remain encrypted at rest in the SFTP destination.
The PGP encryption is applied within our system, after the data is pulled from the source. The data remains PGP-encrypted in transit between our system and the SFTP server, and stays PGP-encrypted at rest in the SFTP destination.
Files are encrypted in the binary format only.
If PGP encryption is enabled, both the manifest files and landed data will be encrypted.
RSA key sizes must be 2,048 bits or more.