Serverless AWS ETL Migration Pipeline

Заказчик: AI | Опубликовано: 08.12.2025

I need to move an existing Oracle-based knowledge base into eGain and keep the two systems in sync afterward. The solution must be completely serverless on AWS, built and deployed through CloudFormation stacks so I can reproduce every component in any account. Pipeline outline • Extract: pull articles, metadata, and attachments from Oracle on a recurring schedule, catching only new or updated records after the first full load. • Transform: map fields, normalise text, and prepare payloads for eGain’s REST APIs. • Load: push the clean data into eGain, confirming success for each record before advancing the checkpoint. Quality safeguards • Schema validation and data-integrity checks run automatically after each extract and again just before the load step. • A retry mechanism handles transient API or network glitches. • Errors are logged to CloudWatch, alerts are sent via SNS, and any failed records are copied to a dedicated S3 bucket for easy reprocessing. Deliverables 1. CloudFormation templates (YAML) that create every AWS resource—Lambda, Step Functions, EventBridge rules, S3 buckets, IAM roles, CloudWatch alarms, and any Glue or DynamoDB components you choose. 2. Parameterised configuration for incremental update frequency, batch size, and retention periods. 3. Well-commented source code for all Lambdas (Python preferred, but any language on the AWS runtime is acceptable). 4. A README that walks me through deployment, rollback, and day-two operations, plus a short handover call. 5. Validation and error-handling logic demonstrated end-to-end in my test account. Acceptance criteria • A full migration completes with 100 % of records present in eGain and passing both schema and data-integrity checks. • Incremental runs finish without manual intervention and process only changed data. • Failure of any single record does not halt the pipeline; the record appears in the “failed-records” S3 bucket, an alert fires, and the rest of the batch continues. • All infrastructure can be torn down and redeployed from CloudFormation with identical results. AWS architecture, ETL design, API integration, and clear documentation are mandatory; prior eGain experience is a welcome bonus but not essential as long as you are comfortable with REST integrations.