Project Structure

LakePilot is organized as a root workspace with product-owned docs, UI, and monitoring surfaces. Shared observability primitives live under libs/iceberg-observability/. The workspace has two service members:

Icepack (services/icepack/) — Iceberg table maintenance and compaction.
IceLens (services/icelens/) — Iceberg table-health observability: scheduled collection (DL-513), health storage and read API (DL-514), and metrics/dashboards/alerting (DL-516). IceLens is the source of truth for table health; Icepack consumes it via the generated icelens-client (DL-515).

This page maps the Icepack modules and explains how the service layers fit together. For IceLens architecture, see IceLens Service.

Shared libraries and tools

Package	Responsibility
`libs/iceberg-observability/`	Policy-independent Iceberg table discovery, metadata inspection, raw table status, and health assessment primitives shared by Icepack and IceLens.
`libs/icelens-client/`	Generated typed client for the IceLens read API (DL-515). Consumed by Icepack; regenerate with `scripts/generate_icelens_client.py`.
`tools/iceberg-inspector/`	Go CLI that reads Iceberg metadata from S3 via the Glue catalog. Compiled into both service images under service-specific binary names (`icepack-iceberg-inspector`, `icelens-iceberg-inspector`); handles V3 `variant` schemas PyIceberg cannot parse.

Module map

Module	Responsibility
`api.py`	FastAPI app — all HTTP endpoints + lifespan.
`cli.py`	Standalone `icepack` CLI commands; bypasses the API queue.
`worker.py`	KEDA-invoked maintenance worker (one job per pod).
`jobs.py`	`JobStore` — Postgres-backed queue + state (DL-197 fence).
`locks.py`	`TableLock` — per-table ownership-checked lock.
`table_cache.py`	`TableCache` + `TableCacheSyncWorker` (atomic-swap refresh).
`history.py`	`HistoryStore` — schema management + persistent reads.
`backend.py`	`select_job_store` / `select_table_cache` factory.
`metrics.py`	OTel/Prometheus gauges (queue depth, workers, etc.).
`config.py`	Pydantic `CompactionConfig` — env-driven configuration.
`discovery.py`	Icepack config-aware wrapper around shared catalog discovery.
`catalog.py`	PyIceberg catalog factories (Polaris + Glue).
`status.py`	Compatibility import surface for shared raw table status contracts.
`health.py`	Compatibility import surface for shared policy-independent health assessment contracts.
`recommendation.py`	Derives policy-aware maintenance recommendations from status, table metadata, and history.
`inspector.py`	Icepack metadata inspector factory and threshold adapter for the shared inspector contracts.
`observability.py`	Converts Icepack threshold config into shared observability thresholds.
`maintenance.py`	`MaintenanceRunner` — executes one action via Spark.
`spark.py`	`SparkQueryEngine` — Thrift/Kyuubi wrapper.
`service.py`	`CompactionService` — request-scoped service composition.
`orchestrator.py`	Auto-submit maintenance based on API recommendations.
`icelens_source.py`	Wraps the generated `icelens-client` to source recommendation cached status from IceLens (DL-515).

Layered architecture

Icepack follows a strict layered architecture. Each layer only calls into the layer directly below it, keeping concerns separated and individual components testable in isolation.

Entry points

There are three ways to interact with Icepack:

CLI (cli.py) — Click commands for ad-hoc operations. The CLI talks directly to the service layer; it does not go through the API.
REST API (api.py) — FastAPI endpoints for programmatic access and the orchestrator. All job submission and health queries flow through here.
Web UI — An Alpine.js single-page application served by the API. The UI is a pure API consumer and introduces no additional backend logic.

Service layer

The CLI uses CompactionService (service.py) directly. The API uses the same lower-level discovery/inspector/runner components where needed, but job submission itself is a Postgres queue operation. CompactionService composes:

Discovery (discovery.py) — lists tables from the configured PyIceberg catalog (Polaris REST when configured, Glue fallback otherwise).
Metadata inspector (inspector.py) — loads Iceberg table metadata through the configured backend: PyIceberg by chart/config default, or the bundled iceberg-go helper in named deployed environment values.
MaintenanceRunner (maintenance.py) — executes a single maintenance action (compaction, snapshot expiry, etc.) by issuing Spark SQL.

Status and health are shared observability contracts supplied by iceberg-observability. Icepack combines those contracts with recommendation.py, which remains Icepack-owned because it uses maintenance policy, operation history, and action ordering.

Query engine

At the bottom of the stack sits the QueryEngine protocol (spark.py). This is a minimal, backend-agnostic interface that the service layer uses to run SQL. The concrete implementation, SparkQueryEngine, wraps a PyHive/Thrift connection to Spark Thrift Server (or Kyuubi).

Data flow summary

CLI (Click)              REST API / Web UI
     |                         |
CompactionService        Postgres queue/cache + service components
     |                         |
Discovery / Inspector / Status / Health / Recommendation / MaintenanceRunner
(PyIceberg catalog)       (metadata inspector + Spark SQL)
     |                         |
QueryEngine Protocol -> SparkQueryEngine (PyHive / Thrift)

State management — job queue, table locks, table inventory cache, and run history — is handled entirely through Postgres via the jobs.py, locks.py, table_cache.py, and history.py modules.