Skip to main content

​
Prerequisites

  • By default, SFTP uses keypair authentication for access. You will need a provided public key to configure your destination. It will look roughly like this:
Public key format
ssh-key <ssh_public_key_beginning_with_AAAA> some-comment
1

Create a user on the SFTP server

Login to the SFTP server and complete the steps below.
  1. Create group sftpwriter:
Create group
sudo groupadd sftpwriter
  1. Create user sftpwriter:
Create user
sudo useradd -m -g sftpwriter sftpwriter
  1. Switch to the sftpwriter user:
Switch user
sudo su - sftpwriter
  1. Create the .ssh directory:
Create .ssh directory
mkdir ~/.ssh
  1. Set permissions:
Set directory permissions
chmod 700 ~/.ssh
  1. Navigate to the .ssh directory:
Navigate to .ssh
cd ~/.ssh
  1. Create the authorized_keys file:
Create authorized_keys file
touch authorized_keys
  1. Set permissions:
Set file permissions
chmod 600 authorized_keys
  1. Add the public key to the authorized_keys file. The key, including the “ssh-key” and comment, should be all on one line in the file, without linebreaks.
Add public key
echo "ssh-key <ssh_public_key_beginning_with_AAAA> sftpwriter-public-key" > authorized_keys
2

Add your destination

Use the following details to complete the connection setup: host name, folder name, username, port and preferred delimiter character.
Write permissions at the SFTP root are requiredIn addition to write access within your configured <folder>, this destination writes per-transfer manifest files under a _manifests/ directory created at the root of the SFTP home/path. Ensure the SFTP user can create and write to _manifests at that root (even if your data lands under a subfolder). Manifests allow downstream systems to detect when a transfer is complete. See the FAQ below for how these files are organized.

​
Frequently asked questions

The data will be loaded with the configured file format (Parquet, CSV, or JSON/JSONL) in a predictable folder structure that can be easily parsed by downstream systems.
sftpwriter_home_folder
some_provided_folder
some_table_a
dt=2024-01-01
0_20240101181004.csv
1_20240101184002.csv
dt=2024-01-02
0_20240102180123.csv
dt=2024-01-03
0_20240103182145.csv
some_table_b
dt=2024-01-01
0_20240101186004.csv
dt=2024-01-02
0_20240102185123.csv
dt=2024-01-03
0_20240103187145.csv
Use SSH key-based authentication for a dedicated, least-privileged SFTP user. Restrict access to only the required directories (e.g., chroot), and allowlist the service’s static egress IP at your network perimeter.
Parquet (default/recommended), CSV, and JSON/JSONL.
Each transfer writes a manifest JSON file per model under _manifests/ at the root. Files follow the pattern: _manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json. Use these manifests to trigger downstream processing.
File-based destinations are append-oriented. The change-detection process uses a lookback window to prevent missed records, which can create duplicates across adjacent transfers. Downstream pipelines can deduplicate by primary key prioritizing rows in the most recent transfer window.
We do not support providing your own public key for security reasons. The private key is securely generated and stored in our system and is never shared externally.