Senior Engineering Manager- Cloud Operations and Architecture - Data and AI/ML
- Successfully led the migration of legacy EDW and RMDBS systems into Snowflake (AWS), resulting in a cost savings of over eight (8) million dollars over a three-year period. Additionally, improved security landscape and time to market by 60% compared to legacy systems.
- Spearheaded the migration of the Data Science platform from AWS (Spark EMR) to Snowpark ML, enabling the deployment of over ten (10) ML models for near real-time inference. This initiative generated over 3.5 million dollars in additional revenue in two years. Real-time models were deployed to AWS Sagemaker.
- Conducted formal POCs to select the tech stack for ETL needs, resulting in the adoption of DBT for transformation and Data Vault modeling, Apache Airflow (Astronomer) for orchestration and scheduling, FiveTran and Confluent Kafka for data movement, and Snowflake for the Data Warehouse.
- Transformed existing Data Engineering and Platform teams into a DevOps/DataOps model, providing training sessions on GitOps and migrating all pipelines into GitLab. This reduced reliance on external DevOps and support teams by over 90% and increased the velocity of deliverables by over 75%.
- Assisted the Data Governance team in rolling out Data Governance (Collibra) and Data Quality (Soda Cloud) tools using AWS EKS edge components developed by our team. These pipelines leveraged CICD and API integration, allowing for a deeper understanding of the customer base and improving deliverables for the organization.
- Implemented tooling for AIOps/ MLOps using Apache Airflow , Snowpark ML API’s, DBT, and Snowflake Cortex models to roll out near real time capabilities for call center sentiment analysis. This effort was a 600K cost savings effort for the organization.
- Was primary architect for setting up reference architecture for LendingTree’s front end lead generation offers platform , and the introduction of ML algorithms into the platform. This was an extensive design review that utilized many changes on the web platform itself, and exposed additional API’s that allowed MLOps pipelines to interact with the platform. The initial lift for the organization was 1.3 million dollars in additional revenue for the first quarter this was instrumented.
- Developed custom Snowflake RBAC model using Terraform, Azure AD SCIM API’s , and Snowflake REST API’s to improve our security posture by having all roles automated and Auto Classification tags applied on the fly. This effort allowed moving our 3 FTE’s who was doing this as manual work and re-train them into DataOps engineers to work on MLOps and Data Engineering pipelines.