Onboarding a Database

Icepack discovers Iceberg tables through a PyIceberg catalog and, in opt-out mode (the default), maintains every table in an allowlisted database unless a table opts out. Onboarding a new database is primarily a one-step process: register the database in the Helm chart so the orchestrator is allowed to automate it. Optionally opt individual tables out.

Step 1 — Add the database to Helm values

Open services/icepack/charts/icepack/values-dev.yaml (or the values file for the target environment) and add the database name to the single databases value. This list is the orchestrator’s maintenance allowlist. It is a comma-separated string:

databases: "offer_service,points_service,<new_database>"

Once the database is allowlisted and deployed, every table in it is maintained automatically — there is no per-table opt-in step.

An allowlist is required in opt-out mode

In opt-out mode the orchestrator maintains every table it is shown. An empty databases value means no database filter, so the orchestrator would consider every table in the catalog. Always set an explicit allowlist.

Table cache and fleet health

The /tables inventory is populated by the API’s TableCacheSyncWorker. The single databases value controls which databases the orchestrator maintains. Fleet-wide cached status and health are collected separately by IceLens (DL-515); the recommendation endpoint reads IceLens cached status and falls back to a live status collection when it is unavailable.

Step 2 — (Optional) Opt out or tune individual tables

Tables in an allowlisted database are maintained by default. To exclude a specific table, opt it out via Spark SQL (or Kyuubi):

ALTER TABLE lakehouse_dev.<database>.<table>
  SET TBLPROPERTIES ('icepack.maintenance_enabled' = 'false');

Use compaction_skip = 'true' instead to hard-exclude a table from all maintenance regardless of mode (for example, during a migration).

Per-table cadence override. By default the orchestrator respects the cluster-wide cadenceHours value (24 hours). To override the cadence for a specific table, set the icepack.maintenance_cadence_hours property:

ALTER TABLE lakehouse_dev.<database>.<table>
  SET TBLPROPERTIES ('icepack.maintenance_cadence_hours' = '12');

Per-table Spark SQL overrides. For tables where Iceberg maintenance hits Spark join-planning issues (e.g., broadcast joins exceeding memory on large remove_orphan_files), set allowlisted icepack.spark.sql.* properties. Icepack applies these as session-scoped SET statements before each action:

ALTER TABLE lakehouse_<env>.<database>.<table>
  SET TBLPROPERTIES (
    'icepack.spark.sql.autoBroadcastJoinThreshold' = '-1',
    'icepack.spark.sql.adaptive.autoBroadcastJoinThreshold' = '-1'
  );

See the table properties reference for the full allowlist and rollback instructions.

Step 3 — Deploy via Terraform

All Icepack infrastructure changes are deployed through Terraform. Never run helm install or helm upgrade directly — Terraform owns the Helm release and direct Helm commands cause state drift.

terraform apply

If you modified any files under services/icepack/charts/icepack/templates/, remember to bump the version field in services/icepack/charts/icepack/Chart.yaml as well. Terraform detects chart changes by comparing the chart version; template-only edits without a version bump are invisible to the plan. (Editing a values-*.yaml file does not require a version bump — Terraform reads the values file directly.)

Step 4 — Verify

After Terraform applies successfully, wait for the next CronJob firing. In the dev environment the orchestrator runs hourly at :30 past the hour.

Check recent orchestrator runs to confirm the new database’s tables were assessed:

curl -s https://<icepack-host>/orchestrator/runs | jq '.[0]'

A successful run includes tables_assessed, jobs_submitted, and jobs_completed counts. If the new tables do not appear, verify that:

1. The table appears in the API table cache (GET /tables).
2. The database is listed in the databases value in the deployed values.
3. The table has not opted out (icepack.maintenance_enabled is not false and compaction_skip is not true).

You can also confirm a specific table is visible in the cache:

curl -s https://<icepack-host>/tables?database=<new_database>

This returns the list of tables Icepack knows about for that database. Each entry includes in_maintenance_allowlist: true/false indicating whether the table’s database is in the databases allowlist. If the list is empty, the API table-cache sync has not discovered the database yet or the catalog cannot list it.

In the web UI (/ui), databases outside the maintenance allowlist appear greyed out with their tables disabled. After a successful onboard, the database should no longer be greyed.