System Overview

Icepack is a control plane for Apache Iceberg table maintenance. It discovers tables through a PyIceberg catalog, inspects Iceberg metadata, derives table health, and runs maintenance operations (rewrite, compact, expire, cleanup) through Spark via Kyuubi. Named deployed environment values use the bundled iceberg-go helper for metadata inspection; the base chart and app config still default to PyIceberg. All state lives in Postgres — there is no Redis, no in-memory queue, no sidecar cache.

Architecture diagram

Icepack architecture

High-level topology

flowchart TD
    UI["Web UI<br/>/ui Alpine.js SPA"] -->|HTTPS| API["FastAPI API<br/>icepack-api, N=2<br/>/readyz /healthz"]
    API -->|SQL| PG["Postgres<br/>jobs, job_queue, table_locks,<br/>job_actions, table_cache_entries"]
    API -->|"TableCacheSyncWorker<br/>table list"| Polaris["Polaris REST<br/>Catalog"]
    API -->|"live status/health<br/>metadata"| Glue["AWS Glue + S3<br/>metadata inspector"]

    PG -->|"polled every 30s<br/>by KEDA postgresql"| KEDA["KEDA ScaledJob<br/>max replicas 5 default<br/>3 dev/stage/preprod, 16 prod"]
    KEDA -->|spawns| Worker["Worker Pod<br/>one job per pod"]
    Worker -->|"load table metadata"| Glue
    Worker -->|"Spark SQL<br/>actions"| Spark["Spark / Kyuubi<br/>Thrift JDBC"]
    Spark --> Iceberg["Apache Iceberg<br/>tables on S3"]

    Polaris -.->|"table inventory"| Iceberg
    Glue -.->|metadata files| Iceberg

    Worker -->|"job<br/>history"| PG

    Orchestrator["Orchestrator CronJob<br/>default: every 2h<br/>dev/stage/preprod/prod: hourly at :30"] -->|calls API| API

The standalone CLI (icepack ...) bypasses the API queue and runs directly against the configured PyIceberg catalog and Spark / Kyuubi endpoint.

The API, its /docs Swagger UI, and the /ui console are served per environment at icepack-api.data-platform.us-east-1.{env}-dataops.fetchrewards.com (canonical for the API and Swagger). The web UI is also served at the LakePilot-branded lakepilot-api.{env}…/ui, an alias on the same Service that is the canonical UI URL and is reserved to evolve into the control-plane front door (DL-140). Both hostnames currently resolve to the same Icepack service; a dedicated LakePilot UI (DL-730) will later let lakepilot-api front its own app.

The orchestrator.schedule controls how often the CronJob checks tables. orchestrator.cadenceHours separately throttles how often a given table may be maintained. Current Helm values run the default chart every two hours (0 */2 * * *) and the dev/stage/preprod/prod environments hourly at :30 (30 * * * *); all keep the per-table cadence at 24 hours.

Fleet health collection lives in IceLens (DL-515); the recommendation endpoint reads IceLens cached status through the generated icelens-client with a live-inspection fallback.

Component inventory

Shared library (`libs/iceberg-observability/`)

Policy-independent Iceberg observability primitives — catalog discovery, metadata inspection (PyIceberg and iceberg-go backends), raw table status, and health assessment — shared across LakePilot services. Has no dependency on Icepack service code. See Project Structure for the full module map.

Application (Python, `services/icepack/icepack/`)

Module	Responsibility
`api.py`	FastAPI app — all HTTP endpoints + lifespan
`cli.py`	Standalone `icepack` CLI commands; bypasses the API queue
`worker.py`	KEDA-invoked maintenance worker (one job/pod)
`jobs.py`	`JobStore` — Postgres-backed queue + state (DL-197 fence)
`locks.py`	`TableLock` — per-table ownership-checked lock
`table_cache.py`	`TableCache` + `TableCacheSyncWorker` (atomic-swap refresh)
`history.py`	`HistoryStore` — schema management + persistent reads
`backend.py`	`select_job_store` / `select_table_cache` factory
`metrics.py`	OTel/Prometheus gauges (queue depth, workers, etc.)
`config.py`	Pydantic `CompactionConfig` — env-driven
`discovery.py`	Icepack config-aware wrapper around shared catalog discovery
`catalog.py`	PyIceberg catalog factories (Polaris + Glue)
`status.py`	Compatibility import surface for shared raw table status contracts
`health.py`	Compatibility import surface for shared policy-independent health assessment contracts
`recommendation.py`	Derives policy-aware maintenance action intent from status, table metadata, and history
`inspector.py`	Icepack metadata inspector factory and threshold adapter for the shared inspector contracts
`observability.py`	Converts Icepack threshold config into shared observability thresholds
`maintenance.py`	`MaintenanceRunner` — executes one action via Spark
`action_impact.py`	Derives nullable per-action impact summaries from procedure output or before/after metadata metrics
`spark_sql_overrides.py`	Resolves allowlisted `icepack.spark.sql.*` table properties into session-scoped Spark SQL `SET` statements
`spark.py`	`SparkQueryEngine` — Thrift/Kyuubi wrapper
`service.py`	`CompactionService` — request-scoped service composition
`orchestrator.py`	Auto-submit maintenance based on API recommendations
`icelens_source.py`	Wraps the generated `icelens-client` to source recommendation cached status from IceLens (DL-515)

Infrastructure (Helm chart, `services/icepack/charts/icepack/`)

Template	Resource	Notes
`api-deployment.yaml`	API pods (N=2)	Startup/Liveness/Readiness probes with 5s timeout
`api-service.yaml`	LoadBalancer -> API	NLB with ACM-terminated TLS; LakePilot API hostname
`worker-scaledjob.yaml`	KEDA ScaledJob	postgresql trigger, 30s polling, max 5 replicas by default; 3 in dev, stage, and preprod; 16 in prod
`keda-postgres-auth.yaml`	TriggerAuthentication	References the postgres secret
`postgres-secret.yaml`	Secret	host/user/password (materialized from values)
`postgres-deployment.yaml` + `postgres-pvc.yaml` + `postgres-nlb.yaml`	Internal Postgres	Used when `postgres.internal.enabled=true`
`orchestrator-cronjob.yaml`	CronJob	Default every 2h; dev/stage/preprod/prod hourly at `:30`; calls API to submit jobs
`irsa.yaml`	ServiceAccount + IRSA	IAM role for S3 / Glue access
`_helpers.tpl`	Helpers	`icepack.postgresDatabase`, `icepack.postgresSslMode`

Provisioning (Terraform, `services/icepack/terraform/icepack-api/`)

Each environment has its own Terraform root under services/icepack/terraform/icepack-api/env/{dev,stage,preprod,prod}/main.tf. Each root provisions what Helm cannot: the AWS IAM role (IRSA), the Secrets Manager entry for the Postgres password, and the helm_release resource that applies services/icepack/charts/icepack with the corresponding values-{env}.yaml. The chart version + values are the trigger for Terraform to re-apply.

Authentication and secrets

Two secrets are managed in AWS Secrets Manager and injected into pods via Helm set_sensitive blocks — never hardcoded in values files:

Secret	Secrets Manager ID	Consumer
Polaris service principal	`{env}/polaris/icepack-principal` (JSON: `client_id`, `client_secret`)	API table-cache discovery; worker pods may receive the env vars, but maintenance currently loads metadata through Glue/S3
Internal Postgres password	generated by Terraform (`random_password`)	API, workers, and KEDA trigger authentication

Polaris OAuth2 flow

Icepack uses PyIceberg’s REST catalog with OAuth2 client credentials — credential = "{client_id}:{client_secret}" and scope = PRINCIPAL_ROLE:ALL. PyIceberg handles the token exchange and caching. The PolarisConfig validator rejects half-configured deploys: if uri is set, both credential fields must also be set.

Injection path: Secrets Manager -> Terraform data.aws_secretsmanager_secret_version -> helm_release.set_sensitive -> ICEPACK_POLARIS_* env vars on pods -> Pydantic PolarisConfig. When polaris.uri is empty, create_iceberg_catalog falls back to Glue.

IRSA for Glue/S3

AWS access (Glue, S3) uses IRSA — the service account assumes aws_iam_role.icepack via OIDC, no static AWS credentials.

Current catalog/metadata split

TableCacheSyncWorker calls create_iceberg_catalog; in dev this uses Polaris REST because Terraform sets polaris.uri and credentials on API pods.
/tables/{db}/{table}/status and /tables/{db}/{table}/health inspect Iceberg metadata through Glue/S3 using the configured metadata inspector. The base chart and app config default to ICEPACK_METADATA_INSPECTOR=pyiceberg; named deployed environment values set ICEPACK_METADATA_INSPECTOR=iceberg-go, which shells out to the bundled helper for metadata reads.
/tables/{db}/{table}/maintenance/recommendation consumes status plus table policy/history to decide action intent. The orchestrator calls this endpoint instead of parsing /health. It prefers IceLens cached status through the generated icelens-client (DL-515), falling back to live inspection when IceLens is unavailable.

For Iceberg v3 tables with schema features not yet supported by PyIceberg, Icepack can use the bundled icepack-iceberg-inspector helper. The helper loads Iceberg metadata through Glue/S3 and streams the current snapshot’s manifest entries with iceberg-go. Active table health metrics are current snapshot metrics: live data files, live delete files, live data-file size, small-file count, and manifest count. Snapshot count and oldest snapshot age remain retention/history metrics across all retained snapshots.