Docs/Fivetran

Use DataContractGate with Fivetran

For DataContractGate and AISchemaGate users who want events that passed their contract in their own warehouse. Both products share one engine. The connector reads only those passing records and writes one table per contract.

Plans and self-hosting

Connect Fivetran is available on paid hosted plans and on any self-hosted instance with CONTRACTGATE_SELF_HOSTED=true.

Hosted free plans get 403. See Self-hosting.

Paid hosted plans are growth, aisg_pro, and enterprise (Pro and Enterprise in the dashboard). Hosted free and aisg_free receive that 403 when CONTRACTGATE_SELF_HOSTED is unset. The API error names those three plan ids. The Contracts form reads paid_features from GET /v1/me. A hosted free org sees an upgrade prompt. If that field is missing, the form follows the stored plan.

Fivetran API key and destination group

Create an API key in your Fivetran account. Fivetran gives you a key and a secret. The Connect Fivetran form has a field for each: Fivetran API key and Fivetran API secret. Paste them as Fivetran showed them. Do not Base64-encode them for the button. DataContractGate uses the pair once to upload the connector and create the connection, then drops them. They are not stored and are redacted from logs.

Destination group id is the id of the Fivetran destination (the warehouse group) the tables should land in. Create a destination in Fivetran first if you do not have one. List groups with the same key and secret; the id is what the form wants. The name is what the manual fivetran deploy --destination flag wants.

curl -sS -u "$FIVETRAN_KEY:$FIVETRAN_SECRET" \
  https://api.fivetran.com/v1/groups

An unknown group returns 400: “Invalid request: Fivetran group <id> was not found”. No DataContractGate key is minted in that case.

Connect Fivetran

  1. Open Contracts.
  2. Click Connect Fivetran.
  3. Paste the Fivetran API key, the Fivetran API secret, and the Destination group id.
  4. Under Contracts to sync, check each contract to include. Up to 32. Each checked row has an optional table name. Leave it blank to use the default name below.
  5. Click Deploy connector. The button reads Connecting… while the request runs.
  6. The connection is created paused. The panel shows the connection id, the status, and Open in Fivetran. Start the first sync from Fivetran after you review it.

DataContractGate mints a new API key named Fivetran connector, scoped to your org and the selected contracts, and sends it to Fivetran as a secret. Your dashboard key is not reused. The raw key is not shown in the browser. The same call is POST /v1/integrations/fivetran/connect. The scheduled sync is every 6 hours once you unpause it.

Connect request

POST /v1/integrations/fivetran/connect takes a JSON object. Send the DataContractGate API key in x-api-key. The body max is 64 KiB.

curl -sS -X POST "$API/v1/integrations/fivetran/connect" \
  -H "x-api-key: $CONTRACTGATE_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "fivetran_api_key": "your_fivetran_key",
    "fivetran_api_secret": "your_fivetran_secret",
    "group_id": "projected_sickle",
    "contract_ids": ["427c3304-ead1-4ba7-b7e0-123456789abc"],
    "table_names": "427c3304-ead1-4ba7-b7e0-123456789abc=user_events_427c3304"
  }'

Connect response

A created connection returns HTTP 200. setup_url is null when the connection id is not safe to put in a URL. status is paused until you start the sync. It is setup_failed when a setup test reports FAILED or JOB_FAILED.

{
  "connection_id": "conn_id",
  "setup_url": "https://fivetran.com/dashboard/connections/conn_id/status",
  "status": "paused"
}

Errors use {"error","status"}.

A later Fivetran failure is HTTP 502, Fivetran request failed (<status>). The key minted for that attempt is revoked, and an uploaded package that never became a connection is deleted.

Destination schema

The Connect Fivetran form does not ask for a schema. When the request omits destination_schema, DataContractGate picks a new schema named contractgate_ plus 8 hex digits, for example contractgate_a1b2c3d4. Fivetran creates that schema in the destination on setup. The warehouse user on the destination must be allowed to create schemas, or that setup fails.

API callers can set destination_schema to a schema that already exists. Use that when the destination user cannot create schemas. The value must start with a lowercase letter and then use only lowercase letters, digits, and underscores, at most 63 characters. Anything else is HTTP 400: Invalid request: destination_schema must start with a lowercase letter and contain only lowercase letters, digits, and underscores, at most 63 characters. Fivetran still needs permission to create tables in that schema.

Names are stored with Fivetran SOURCE_NAMING, so the schema and table names above are not rewritten. FIVETRAN_NAMING would split letter and digit boundaries (user_events_427c3304 would become user_events_427_c_3304).

What lands in the warehouse

One table per contract. You can write the default name before the first sync. It is <slug>_ plus the first 8 hex characters of the contract id, which are the eight characters before the first hyphen, in lowercase. Contract User Events and id 427c3304-ead1-4ba7-b7e0-123456789abc becomes user_events_427c3304.

The slug keeps ASCII letters and digits only, lowercased. Each run of other characters becomes one underscore. Leading digits and underscores are dropped, so 123 Orders with that same id is orders_427c3304. Café becomes caf_<8hex>. The slug stops at 54 characters so the full name stays within 63. A name with no ASCII letters left, such as !!!, becomes passed_ plus those same 8 hex characters: passed_427c3304. An optional table name on the form, or a table_names pair, replaces that default for that contract only. Names must be unique across the contracts you selected.

The primary key is id. Columns:

payload is null unless event payload storage is on. Storage is on only for a paid plan when Event Payload Storage on Account is on and Payload storage for that contract (on the Quarantine tab) is on. A null payload is still a passed event; the body was not stored. Kafka and Kinesis consumers do not write this feed. HTTP ingest, egress, and quarantine replay do.

Sync is incremental. The connector calls GET /v1/contracts/{id}/passed with the saved cursor, oldest first, at most 1000 events per page, and checkpoints after every page. Rows newer than 5 seconds are left out, so an insert still in flight is not skipped. Events show up about 5 seconds late. The next sync resumes from the saved cursor.

Troubleshooting

Cleanup and another connect

Each successful connect inserts one api_keys row named Fivetran connector, limited to the selected contracts, and creates one new paused Fivetran connection. The raw key is not returned. Connect does not look for an existing connection and does not update one. A second click mints a second key and a second connection.

Run connect once per set of contracts. To change the contracts, the table names, or the schema, delete the Fivetran connection and revoke the key, then connect again. Leaving the old connection in place keeps it syncing on the old key.

Revoke the key on Account → API Keys. The button calls DELETE /api/keys with {"id":"<api key uuid>"} and sets revoked_at. On the compose database the same update is:

UPDATE api_keys
SET revoked_at = now()
WHERE org_id = '<org uuid>'
  AND name = 'Fivetran connector'
  AND revoked_at IS NULL
  AND deleted_at IS NULL;

The gateway cache keeps a revoked key working for up to 60 seconds. Delete the connection in Fivetran from the setup_url, or with the same key and secret:

curl -sS -u "$FIVETRAN_KEY:$FIVETRAN_SECRET" -X DELETE \
  "https://api.fivetran.com/v1/connections/$CONNECTION_ID"

$CONNECTION_ID is connection_id from the 200 response. DataContractGate does not store it. A connect that fails before the key is minted (400, 401, 403, 404, 413, 500) leaves nothing to revoke. A connect that fails after the key is minted revokes that key itself.

Self-hosting

Self-hosted orgs default to free. Set CONTRACTGATE_SELF_HOSTED=true on the gateway. Every paid gate then opens, including Connect Fivetran, passed-event export, and payload storage. The monthly event cap is unlimited, the default rate limit is the paid limit unless the key overrides it, and the passed-event window is 365 days. The stored plan stays free. The connector key is a normal API key for the org, so it works against GET /v1/contracts/{id}/passed. Hosted free and aisg_free still receive 403. Leave the variable unset on hosted production and staging.

Keep auth on for any instance Fivetran can reach. Fivetran's cloud must call the API over public HTTPS. An auth-off gateway on that URL would expose your data to anyone who can reach it. Set CANONICAL_API_ORIGIN to that public URL. Connect returns 500 and mints no key when it is unset.

CONTRACTGATE_DEV_NO_AUTH is only for local or private use. Fivetran cloud cannot call that process. Use the manual connector path from github.com/nightmoose/nmb-contractgate-fivetran when the gateway stays private. The commands are in the next section.

docker-compose publishes the gateway on localhost and runs with auth off. That is the local stack. To let Fivetran reach it, put a tunnel or a reverse proxy with TLS in front, turn auth on, set CONTRACTGATE_SELF_HOSTED=true, and set CANONICAL_API_ORIGIN to the proxy's https URL.

A fresh clone with a real dashboard login, from a machine that has not already created the compose volume:

cp .env.selfhost.example .env
docker compose -f docker-compose.yml -f docker-compose.selfhost.yml up --build

Sign up at http://localhost:3000/auth/signup, then sign in at http://localhost:3000/auth/login. Confirmation mail is not sent. The account is confirmed automatically. Do not add --profile demo. That profile is the zero-auth dashboard. If this machine already ran make demo, run docker compose -f docker-compose.yml -f docker-compose.selfhost.yml down -v before the first up. The steps and the placeholder keys are in docs/self-host-reference.md.

Signup inserts auth.users, which fires handle_new_user(). That function inserts user_profiles, an orgs row, and an owner org_memberships row. The plan is free unless signup metadata has product = aischemagate, which starts the org on aisg_free.

GET /v1/me still reports plan free. paid_features and self_hosted are true, monthly_event_limit is null, and the default rate limit is the paid limit. Issue a key from the dashboard and use it as x-api-key for connect, or as api_key in a manual configuration.json. Set CANONICAL_API_ORIGIN to the public https origin before Fivetran cloud calls the gateway.

Deploy the connector yourself

If you do not use the button, install the public connector with the Fivetran Connector SDK CLI. The source is github.com/nightmoose/nmb-contractgate-fivetran. From that directory, with Python 3.10–3.14:

python -m venv .venv
source .venv/bin/activate
pip install fivetran-connector-sdk

# FIVETRAN_API_KEY is Base64 of key:secret, not the DataContractGate key.
fivetran deploy \
  --api-key "$FIVETRAN_API_KEY" \
  --destination "$FIVETRAN_DESTINATION_NAME" \
  --connection contractgate_passed \
  --configuration configuration.json \
  --python 3.12

configuration.json is the object the connector reads. api_key and contract_ids are required. base_url and table_names are optional. Production base URL is https://api.datacontractgate.com. The connection is created paused until you start the initial sync in Fivetran.

{
  "api_key": "cg_live_0123456789abcdef0123456789abcdef0123456789abcdef",
  "contract_ids": "427c3304-ead1-4ba7-b7e0-123456789abc",
  "base_url": "https://api.datacontractgate.com",
  "table_names": "427c3304-ead1-4ba7-b7e0-123456789abc=user_events_427c3304"
}

A manual deploy that leaves table_names empty uses the connector default passed_<id> (the contract UUID, hyphens replaced by underscores), not the slug name the button sends. Set table_names to contract_id=name pairs when you want the same readable names. The setup test is named Configuration check.