> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prequel.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Datalake

> Understand how Prequel Import uses your datalake for secure data caching and transfer

## What is the datalake?

Prequel Import makes use of a secure cache of the source data in your cloud storage bucket. This enables efficient change detection without the need to store any customer data on Prequel's core infrastructure.

## How is the datalake structured?

A Datalake configures one cloud storage bucket. Each Provider is assigned to exactly one Datalake at creation, and that assignment is immutable. The datalake can be single-tenant or multi-tenant:

* **Shared Datalake:** Many Providers point to the same Datalake. Prequel partitions the shared bucket internally by tenant, so data from separate Providers is never comingled.
* **Per-Provider Datalake:** Each Provider has a dedicated Datalake. Use this if you prefer bucket-level isolation.

You can mix the two patterns within a single environment, sharing Datalakes among some customers and pinning other customers to their own.

## Lifecycle responsibility

Prequel manages lifecycle for cached data under the `lakehouse/` prefix of your datalake. Staging data written to the `artifacts/` prefix is not managed by Prequel. You must configure a lifecycle policy on the `artifacts/` prefix (or `<bucket_prefix>/artifacts/` if you set a bucket prefix on the datalake) to expire it. The setup steps below include a lifecycle policy step for each vendor. We recommend matching the lifecycle policy duration to your datalake's `retention_window_days`.

## Configuring your datalake

<Tabs>
  <Tab title="AWS S3">
    <Steps>
      <Step title="Create an S3 bucket">
        In the AWS console, navigate to the **S3** service page and click **Create bucket**. Enter a **Bucket name** and choose an **AWS Region**. We recommend setting **Object Ownership** to "ACLs disabled" and **Block Public Access settings for this bucket** to "Block all public access". Make a note of the bucket name and region.
      </Step>

      <Step title="Create an IAM policy">
        Create an IAM policy with the following permissions, replacing `BUCKET_NAME` with the name of the bucket you created:

        ```json title="IAM policy" icon="brackets-curly" expandable theme={null}
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": "s3:ListBucket",
              "Resource": "arn:aws:s3:::BUCKET_NAME"
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
              ],
              "Resource": "arn:aws:s3:::BUCKET_NAME/*"
            }
          ]
        }
        ```
      </Step>

      <Step title="Create an IAM role">
        Create an IAM role with the trust policy below, then attach the policy from the previous step. Replace `<some_service_account_id>` with the service account ID from your deployment details.

        ```json title="Trust policy" icon="brackets-curly" expandable theme={null}
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Federated": "accounts.google.com"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "accounts.google.com:sub": "<some_service_account_id>"
                }
              }
            }
          ]
        }
        ```

        Once the role is created, make a note of its **ARN**.
      </Step>

      <Step title="Set a lifecycle policy on the artifacts prefix">
        On the bucket, navigate to **Management** > **Lifecycle rules** and click **Create lifecycle rule**. Limit the rule scope by prefix to `artifacts/` (or `<bucket_prefix>/artifacts/` if you set a bucket prefix on the datalake), then add an **Expire current versions of objects** action set to the same number of days as the datalake's `retention_window_days`.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Google Cloud Storage">
    <Steps>
      <Step title="Create a GCS bucket">
        In the GCP console, navigate to **Cloud Storage** and click **Create**. Enter a **bucket name** and choose a **region**. At the **Choose how to control access to objects** step, we recommend selecting **Enforce public access prevention on this bucket**. Make a note of the bucket name and region.
      </Step>

      <Step title="Create a service account">
        Navigate to **IAM & Admin** > **Service Accounts** and click **Create service account**. Name the service account and click **Done**. You will grant permissions in the next steps.
      </Step>

      <Step title="Grant bucket permissions to the service account">
        On the bucket you created, grant the service account the following permissions, either via a custom role or via the built-in **Storage Object Admin** role plus `storage.buckets.get`:

        * `storage.buckets.get`
        * `storage.objects.list`
        * `storage.objects.get`
        * `storage.objects.create`
        * `storage.objects.delete`
      </Step>

      <Step title="Grant the Prequel service account access to impersonate yours">
        Open the service account you created, select the **Permissions** tab, and click **Grant access**. Add the Prequel service account from your deployment details as a principal and assign the **Service Account Token Creator** role. Click **Save**.

        This allows Prequel to impersonate your service account via short-lived tokens. Prequel never handles your service account keys, and all operations appear in your GCP audit logs.
      </Step>

      <Step title="Set a lifecycle policy on the artifacts prefix">
        On the bucket, open the **Lifecycle** tab and click **Add a rule**. Set the action to **Delete object**, with the conditions **Object name matches prefix** = `artifacts/` (or `<bucket_prefix>/artifacts/` if you set a bucket prefix on the datalake) and **Age** equal to the datalake's `retention_window_days`.
      </Step>

      <Step title="Provide your service account email">
        Use the following details to complete the datalake setup: **bucket name**, **bucket region**, and the **service account email** you created.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Azure Blob Storage">
    <Steps>
      <Step title="Create or configure the storage account">
        In the Azure portal, navigate to **Storage accounts** and select an existing account or create a new one. In the account's **Configuration** settings, ensure **Storage account key access** is enabled and **Hierarchical namespace** is enabled.
      </Step>

      <Step title="Create a container">
        In the storage account, navigate to **Containers** under **Data storage** and click **+ Container**. Choose a name and click **Create**.
      </Step>

      <Step title="Retrieve the storage account access key">
        In the storage account, navigate to **Security + networking** > **Access keys**. Reveal one of the two keys and make a note of it. This key grants full access to the storage account, so store it securely.
      </Step>

      <Step title="Set a lifecycle policy on the artifacts prefix">
        In the storage account, navigate to **Data management** > **Lifecycle management** and click **Add a rule**. Scope the rule to **Limit blobs with filters** with the prefix `<container_name>/artifacts/` (or `<container_name>/<bucket_prefix>/artifacts/` if you set a bucket prefix on the datalake), and set **Delete the blob** to the same number of days as the datalake's `retention_window_days`.
      </Step>

      <Step title="Provide your storage account details">
        Use the following details to complete the datalake setup: **storage account name**, **container name**, and the **storage account access key**.
      </Step>
    </Steps>
  </Tab>
</Tabs>