IceLens Service

IceLens is LakePilot’s Iceberg table-health observability service. DL-513 added scheduled collection: a CronJob discovers the lakehouse catalog, inspects every table’s Iceberg metadata through the shared iceberg-observability library, and persists status and health snapshots to an IceLens-owned Postgres. Read APIs (DL-514) expose per-table health, catalog browsing, and collection operations over that data; table-health metrics, dashboards, and alerting (DL-516) land next.

Service boundary

• Lives at services/icelens/ as a uv workspace member.
• Depends on the shared libs/iceberg-observability/ package — never on the icepack package.
• Owns its own Docker image (icelens-api ECR repository), Helm chart (services/icelens/charts/icelens/), Kubernetes namespace (icelens), chart-managed collection-store Postgres, Terraform roots (services/icelens/terraform/icelens-api/env/), and read-only icelens-<environment> IRSA role for metadata collection (Glue read, lakehouse S3 read, lakeformation:GetDataAccess).
• Shares the LakePilot root docs site and root UI — no service-specific docs or UI surface.

Runtime contracts

Endpoint	Behavior
GET /healthz	Liveness — 200 {"status": "ok"}, no external dependencies
GET /readyz	Readiness — 200 once validated configuration and startup complete, 503 before; never depends on Postgres or Glue
GET /openapi.json	OpenAPI document titled IceLens with the service version
GET /tables/{database}/{table}/status	Live Iceberg status via the shared inspector against Glue; never persists
GET /tables/{database}/{table}/status/cached	Latest stored status snapshot; recomputed staleness; last_error when a newer collection failure exists
GET /tables/{database}/{table}/health	Live health assessment (library-default thresholds, same as collection); never persists
GET /tables/{database}/{table}/health/cached	Latest stored health assessment; recomputed staleness
GET /tables/{database}/{table}/health/history	Compact health trend within the retention window
GET /databases	Per-database rollups: table counts, health breakdown, staleness, storage totals
GET /tables	Flat per-table summaries with database, health_status, and stale filters
GET /cycles	Recent collection cycles with completeness (finished_at IS NULL = incomplete)
GET /cycles/{cycle_id}/failures	Per-cycle collection failure detail

Cached, browsing, and ops reads serve from the collection store and return 503 while Postgres is unreachable. Live reads inspect through the same inspector configuration as collection (Glue + IRSA; no Polaris) and are side-effect-free. Staleness keys off per-table snapshot age (ICELENS_STALE_AFTER_SECONDS, default 1800 = 2x the collection cadence), never cycle membership.

Consumers — the generated client (DL-515)

IceLens is the source of truth for fleet table health. Consumers integrate through the generated typed client at libs/icelens-client/ — generated from this service’s OpenAPI schema by scripts/generate_icelens_client.py, committed, and held current by the client-drift.yml check plus a schema-equality test in the IceLens suite. Services depend on the generated client, never on each other’s source.

Icepack’s maintenance-recommendation endpoint reads GET /tables/{database}/{table}/status/cached through the client (ICEPACK_ICELENS_URL; empty disables). An unusable answer — 404, 503, timeout, stale snapshot, or a newer collection failure — falls back to Icepack’s own live inspection, so an IceLens outage degrades maintenance to “slower”, never “broken”.

Scheduled collection

Every 15 minutes a collection CronJob (python -m icelens.collection_job, concurrencyPolicy: Forbid) runs one cycle:

1. Discover the table worklist — Polaris REST catalog primary, with an automatic Glue-catalog fallback if Polaris is unavailable. Fallback cycles log discovery_fallback_engaged and persist discovery_source = "glue".
2. Inspect each table concurrently (bounded by ICELENS_COLLECTION_CONCURRENCY) against Glue, reading Iceberg metadata from S3 with the pod’s IRSA credentials. Deployed environments use the bundled iceberg-go inspector (ICELENS_METADATA_INSPECTOR: iceberg-go, same helper icepack ships), which also handles V3 variant schemas that PyIceberg cannot parse yet; pyiceberg remains the local-dev default.
3. Persist status snapshots, health assessments, per-table failures, and per-cycle metadata to the internal chart-managed Postgres (collection_cycles, table_status_snapshots, table_health_snapshots, collection_failures), pruned to ICELENS_RETENTION_DAYS (14 days).

Per-table failures never abort a cycle; a cycle that cannot persist (both discovery paths down, or Postgres unavailable) exits non-zero so the Kubernetes Job records a failure. The API pods are not involved in collection; /readyz is unchanged.

Current-state RPO is one cycle: every row re-derives from Iceberg metadata in S3, so losing the database loses at most the retained history window. Durability decision (DL-515): IceLens keeps the chart-managed Postgres. The data is fully re-derivable within one 15-minute cycle, and the one consumer on a critical path — Icepack’s maintenance recommendation — degrades to live inspection when IceLens is unavailable, so a managed database would buy little. The chart’s postgres.external.host switch remains available; revisit if IceLens ever accumulates data that does not re-derive (long-horizon history, annotations).

Deployment topology

IceLens deploys through the shared path-scoped CI/CD (see Path-Scoped CI/CD): merges to main that affect IceLens deploy dev automatically; stage, preprod, and prod are manual Deploy IceLens workflow dispatches, one environment at a time, with prod protected by the main-deploy environment.

Runtime OpenAPI URLs:

• https://icelens-api.data-platform.us-east-1.dev-dataops.fetchrewards.com/openapi.json
• https://icelens-api.data-platform.us-east-1.stage-dataops.fetchrewards.com/openapi.json
• https://icelens-api.data-platform.us-east-1.preprod-dataops.fetchrewards.com/openapi.json
• https://icelens-api.data-platform.us-east-1.prod-dataops.fetchrewards.com/openapi.json