top of page

AWS Airline Batch

Problem:

  • Airline flight data needs to be processed and stored efficiently in batches to provide historical insights and analytics.

​​

Solution:

  • Data Ingestion - Stored airline data in AWS S3.

  • Data Processing - Used EventBridge, Step Functions, and AWS Glue ETL for processing.

  • Data Storage - Stored the processed data in AWS Redshift.

  • Data Monitor - Used SNS for notifications upon insertion success or failure.

images (2).png

System Architecture

Data:

  • Dimension - contains Airport info like airport_id, city, state, name

  • Fact - CSV airline flight data

    • Carrier, Arrival and Destination Airports

system_architecture.png

Video Walkthrough

In this video, I demonstrate the execution of a batch data pipeline for airline flight data. The pipeline utilizes AWS services such as S3, EventBridge, Step Functions, Data Crawler, Glue ETL, and Redshift to process and analyze flight data stored in CSV format. This project highlights the integration of various AWS components to streamline data processing workflows and ensure efficient data ingestion, transformation, and storage.​

​​

Conclusion

Working on the AWS Airline Batch Project allowed me to deepen my understanding of AWS services and their practical applications in building complex data pipelines. By integrating S3, EventBridge, Step Functions, Data Crawler, Glue ETL, and Redshift, I honed my skills in orchestrating various AWS components to create a seamless data processing workflow. This project enhanced my ability to ensure data quality and manage large datasets efficiently, reinforcing my confidence in designing scalable solutions for batch data processing.

Other Batch Projects!

AWS Sales Events

Designed a batch pipeline to handle sales data

bottom of page