ML Feature Store

Training-serving skew — where the features used during model training differ from those available at inference time — is the most insidious source of ML model degradation in production. Feature stores solve this by providing a single source of truth for feature computation, storage, and serving. In this challenge, you will design a feature store architecture using Amazon SageMaker Feature Store that supports both batch feature ingestion for training and low-latency online feature retrieval for real-time inference. The offline store uses S3 with Parquet format and Glue Data Catalog integration, enabling Athena queries for training dataset construction with point-in-time correctness — crucial for preventing data leakage in time-series features. The online store uses SageMaker Feature Store's built-in low-latency storage backed by DynamoDB, providing single-digit-millisecond feature retrieval during inference. Feature ingestion runs on two paths: batch pipelines using Glue jobs that compute features from raw data in S3 and ingest them on a schedule, and streaming pipelines using Kinesis Data Streams with Lambda consumers that compute real-time features (like rolling averages, session counts) and ingest them immediately. You will design the feature group schema strategy — organizing features by entity (user features, product features, interaction features) with a consistent naming convention and version tracking. Feature transformations use SageMaker Processing jobs for batch and Lambda for streaming, with shared transformation code packaged as a Lambda layer to ensure consistency. The architecture includes a feature freshness monitoring system using CloudWatch metrics that track ingestion lag per feature group and alert when features become stale. Data quality validation runs on every batch ingestion using Great Expectations-style checks implemented in Lambda, blocking ingestion of features that fail schema or statistical distribution checks. Access control uses IAM policies scoped to feature groups, so ML teams can only read features relevant to their models. This challenge teaches feature store architecture, training-serving consistency, and the data engineering patterns that make ML systems reliable.