Ep193: The Conductor Behind Your Data Orchestra: Astronomer's Approach to AI Pipeline Management
Failed to add items
Sorry, we are unable to add the item because your shopping cart is already at capacity.
Add to basket failed.
Please try again later
Add to Wish List failed.
Please try again later
Remove from Wish List failed.
Please try again later
Follow podcast failed
Unfollow podcast failed
Ep193: The Conductor Behind Your Data Orchestra: Astronomer's Approach to AI Pipeline Management
-
Narrated by:
-
By:
About this listen
Astronomer's Steven Hillion reveals how OpenAI, Anthropic, Uber, and Lyft use Apache Airflow to orchestrate AI and machine learning pipelines at scale on AWS.
Topics Include:
- Steven Hillion leads data and AI at Astronomer
- Apache Airflow surpassed Spark and Kafka in community metrics
- Astronomer coordinates data flow like conductor orchestrating instrumental platforms
- Organizations with data engineering teams use Airflow at scale
- Customers already used Airflow for ML before official promotion
- Uber and Lyft orchestrate pricing models using Airflow
- Astronomer runs on AWS with close integration partnerships
- OpenAI Anthropic and GitHub Copilot use Airflow for operations
- Internal data team uses Airflow creating feedback loops
- Evolved from constrained AI reports to agentic workflows
- Platform monitors generative AI output quality at user interactions
- Metadata and context increasingly critical for AI applications
- Learn more at Astronomer’s Data FlowCast podcast
Participants:
- Steven Hillion – SVP, Data and AI, Astronomer
See how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/
No reviews yet
In the spirit of reconciliation, Audible acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.