DocumentationAPI Reference
Documentation

SFTP

Configuring your SFTP server to load data.

Prerequisites

  • By default, SFTP uses keypair authentication for access. You will need a provided public key to configure your destination. It will look roughly like this:
ssh-key <ssh_public_key_beginning_with_AAAA> some-comment

Step 1: Create a user on the SFTP server

Login to the SFTP server and complete the steps below.

  1. Create group sftpwriter:
sudo groupadd sftpwriter
  1. Create user sftpwriter:
sudo useradd -m -g sftpwriter sftpwriter
  1. Switch to the sftpwriter user:
sudo su - sftpwriter
  1. Create the .ssh directory:
mkdir ~/.ssh
  1. Set permissions:
chmod 700 ~/.ssh
  1. Navigate to the .ssh directory:
cd ~/.ssh
  1. Create the authorized_keys file:
touch authorized_keys
  1. Set permissions:
chmod 600 authorized_keys
  1. Add the public key to the authorized_keys file. The key -- including the "ssh-key" and comment -- should be all on one line in the file, without linebreaks.
echo "ssh-key <ssh_public_key_beginning_with_AAAA> sftpwriter-public-key" > authorized_keys

Step 2: Add your destination

Share your host name, folder name, username, port and preferred delimiter character with us to complete the connection.

Frequently Asked Questions

  • How will the data appear in my SFTP server?
    • The data will be loaded as CSV files, in a predictable folder structure that can be easily parsed by downstream systems.
      sftpwriter_home_folder/
      ├─ some_provided_folder/
      │  ├─ some_table_a/
      │  │  ├─ dt=2024-01-01/
      │  │  │  ├─ 0_20240101181004.csv
      │  │  │  ├─ 1_20240101184002.csv
      │  │  ├─ dt=2024-01-02/
      │  │  │  ├─ 0_20240102180123.csv
      │  │  ├─ dt=2024-01-03/
      │  │  │  ├─ 0_20240103182145.csv
      │  ├─ some_table_b/
      │  │  ├─ dt=2024-01-01/
      │  │  │  ├─ 0_20240101186004.csv
      │  │  ├─ dt=2024-01-02/
      │  │  │  ├─ 0_20240102185123.csv
      │  │  ├─ dt=2024-01-03/
      │  │  │  ├─ 0_20240103187145.csv