Brad Chapin

Technical lead and key contributor for a team of 3-6 responsible for Site Reliability, Configuration Management, Deployment, and Automated Failure Recovery for Lumen CDN managing 20K servers, VM's and containerized work flows. CDN produced $100M+ in yearly recurring revenue while delivering up to 100 Tbps.

CMAPI - Architected and Designed an authenticated RESTful API (FastAPI) to aggregate and cache CM data from 10+ systems. Currently used by monitoring, config management and repair automation as source of truth for network disposition. This system is constantly evolving as we add new data sources and have new requests for custom aggregations for both internal and external uses. It is the foundation and source of record for all projects listed below.

• Python, FastAPI, REST, TLS, CMDB, GitLab, CI/CD, Gunicorn, Nginx, Test-Kitchen, Postgres, mySQL, High Level Architecture, Software Design

Trial ROD - Implemented a GitOps model using Rest API's through GitLab CI/CD to identify clusters/servers for upgrade and apply new trial configurations through CMAPI interfaces and utilize Release On Demand to deploy new applications/configurations to the network (bare metal and VMS). The resulting project gave ownership of candidate releases to the engineers responsible for validating performance and expedited candidate deployments from monthly to 1-5 per week.

• Python, CMDB, Gitlab, GitOps, CI/CD, Salt-Cloud, Bare Metal Imaging, Test-Kitchen, Networking, Staged Deployment

Release On Demand - Architected and Implemented staged deployment for 20K operational CDN servers based on the disposition of the network. RoD removed the onus for generating a upgrade schedule from operations, while taking in to account traffic levels, metros, server health and network blackout dates. It increased release cadence for primary distribution network from 1-2 GA releases per year, to our targeted goal of once per month.

• Python, FastAPI, REST, TLS, CMDB, Staged Deployment, GitLab, CI/CD, Salt-Cloud, Bare Metal Imaging, Networking, Test-Kitchen

Salt Versioned Formulas - Extended the Salt concept of environments to include supporting multiple versioned CM code based on "Release States" and server role by rebuilding them on the fly prior to running a job. One control file defines which role and release state get which versions of code, allowing us to trial new formulas on a targeted network segment while knowing we would not affect other segments. Presented our results at 2019 Salt Conf.

• Salt, Salt-API, Salt-Cloud, Staged Deploy, Python, GitLab, CI/CD, Test-Kitchen, High Level Architecture, Software Design, Firewalld

RA/CM Replacement - Documented existing Architecture and Defined/Implemented next-gen architecture for migration from a wholly home-grown network automation system to Hybrid Salt solution. Incorporating Salt allowed us to more consistently and quickly incorporate changes to our growing network including but not limited to new OS's, containers and virtualization.

• High Level Architecture, Software Design, Salt, Staged Deploy, Python, GitLab, CI/CD, Test-Kitchen, Postgres, mySQL, RedHat Anaconda, REST

CI/CD - Incorporated and mandated the concept of CI/CD within the network automation code base. Provided templates and frameworks for consistent automated testing utilizing VM's and containers with the pytest and Test-Kitchen frameworks. Each new GitLab project is created with a set of configurations for virtual env's (Poetry, RVM), testing virtualization, linting, and deployment models. The developer just needs to add test either from scratch utilizing shared custom pytest libraries with common fixtures. This initiative has minimized the need for regression testing because tests grow naturally as bugs are found fixed and validated.

• High Level Architecture, Software Design, Salt, Staged Deploy, Python, GitLab, CI/CD, Test-Kitchen, Vagrant, Docker

VM/Container Integration - Architected, Designed, and Implemented frameworks for deploying and configuring VM's (VMware) and Containers (Docker/Kubernetes) in production environment. Containers allow us to abstract away the OS layer and maintain some of our legacy applications while still addressing security concerns.
Virtualization has reduced the time to grow and shrink our network from weeks to purchase and allocate hardware to minutes to define our new server in inventory and use automation to stand up and configure.

• High Level Architecture, Software Design, Salt-Cloud, Salt, Staged Deploy, Python, GitLab, CI/CD, Test-Kitchen, VMWare

Edge Portal - Implemented container based deployment of a secure publicly available interface utilizing Traefik with TLS certificates, custom Golang/Node.js applications, KrakenD, Postgres DB's, and ElaticSearch/Kibana monitoring using FileBeat, Metricbeat.

• Docker, Salt, Gitlab, CI/CD, Traefik, Elastic Stack, Test-Kitchen, TLS

Technical lead and key contributor that Architected, Designed, Implemented, and Supported an 60PB multi-site origin storage product with $20M in yearly recurring revenue supporting 3000+ customers with over 30 billion objects, achieving egress delivery in excess of 150 Gbps across four gateway locations.

Live Storage PSVue - Architected, Designed, and Implemented storage for live ingest and cloud DVR offering for short-lived OTT Sony PSVue offering. Responsible for multi bitrate live ingest in 5 second chunks and on-demand API requests for later DVR viewing. Utilized a unique A/B segment combination to uniquely identify each stream to adhere with the broadcast ownership legal ruling while still minimizing storage.

• Perl, Python, Ruby/Rails, Bare-Metal Imaging, SNMP, FTP, REST, Java, Linux Chrooting, IPTables, NFS, Postgres

OSP 3.0 Auditing - Designed and implemented dynamic decision tree functionality for auditing and recording 10K+ distinct values on 3K+ live servers across the globe utilizing distinguishing characteristics like region and role to identify any deviations from the expected configuration.

• REDIS, Perl, Python, Ruby/Rails, SNMP, REST, NFS, Postgres, SNMP

OSP 3.0 - Architected, Designed and Deployed with a team of 5 a 60PB geo redundant storage solution with 100% durability and 99.999% ingest SLAs utilizing GPFS (IBM) on DDN hardware. Defined and implemented a migration strategy to perform a live cutover of 1500 customers (including industry leaders like Hulu, Sony and Microsoft) and 1B files without service impact.

• GPFS Clustered Filesystem, Perl, Python, Ruby/Rails, Bare-Metal Imaging, SNMP, FTP, REST, IPTables, Postgres, SNMP

OSP 2.0 - Architected, Designed and Deployed with a team of 7 a 20PB geo-redundant storage built on Lustre filesystem with bi-direction continuous synchronization cross-site. Our team was responsible for all product systems including but not limited to secure shared ingress, secure egress, API driven customer service management, billing, inventory and data security, reliability and durability.

• Lustre Clustered Filesystem, Perl, Python, PHP, Ruby/Rails, Bare-Metal Imaging, SNMP, FTP, SOAP, IPTables, Postgres, SNMP, FTP, rsync

Key individual contributor for both infrastructure and signaling proxy for a VoIP softswitch producing 430M in yearly recurring revenue.

Standard Lib - Implemented STL extensions within C++ for standardized development framework across multiple applications including parsing, communications, logging and alerting. Also worked with a team to use a network sniffer to process and visually represent call tracing per call leg with decoded SS7 and SIP messaging for each network hop.

• C/C++, STL, SIP

Signaling Proxy - Implemented communications proxy parsing SS7, SIP and internal control plane messages to support Managed Modem and VoIP products. Specifically tasked with implementing features like SIP-T parsing, Local Number Portability, and E911.

• C/C++, STL, SIP, SS7, Automated Testing

Individual contributor on DoD project responsible for providing infrastructure libraries for a development team of ~50 developers.

Infrastructure Team - Implemented C++ tooling for standard messaging, logging, alerting, and signal handling libraries. Required Top-Secret clearance.

• C/C++, STL, Object Oriented Programming