Customer Profile
- Industry — Financial Services
- Location — India
- Employees — 10,000+
- Challenge — Build a scalable data lake on Amazon S3 to ingest, process, and store financial data from an external source with a governed multi-tier architecture.
Business Challenge
- Ingest data files arriving from an external source into AWS reliably and securely.
- Eliminate manual handling of compressed source files and ad-hoc data processing.
- Establish a single, governed storage layer for raw, curated, and business-ready data.
- Ensure data quality through validation, cleansing, and deduplication before consumption.
- Manage growing data volumes cost-effectively with automated archival.
AWS Services Used
- Amazon S3
- AWS Glue (ETL jobs)
- Apache Iceberg on Amazon S3
- AWS KMS
- AWS Step Functions
- Lambda
Solution Design
- Source files are pulled from the external source and landed in the raw prefix of the Amazon S3 data lake bucket.
- Compressed archives are automatically expanded into processable files in S3.
- Raw files are validated and written to bronze-tier Apache Iceberg tables on S3.
- Bronze data is cleansed, deduplicated, and standardised into silver-tier tables.
- Silver data is aggregated into gold-tier, business-ready tables for analytics and reporting.
- Processed source files are moved to an S3 archive location, with lifecycle policies for cost-optimised long-term storage. All tiers — raw, bronze, silver, gold, and archive — are stored in Amazon S3, with encryption managed via AWS KMS.
Business Outcomes
- Fully automated, end-to-end ingestion pipeline from external source to business-ready data, all on Amazon S3.
- Processed and managing 8TB of data across the medallion (bronze/silver/gold) architecture.
- Improved data quality and consistency through automated validation, cleansing, and deduplication.
- Reduced storage costs via tiered S3 storage and automated archival of processed files.
- Scalable, secure foundation supporting growing data volumes and downstream analytics.