How Slack Used Generative AI on AWS to Optimize Apache Spark on Amazon EMR

Slack engineers transformed Spark performance and cost efficiency on Amazon EMR by building a custom metrics pipeline paired with generative AI-assisted tuning. Faced with slow jobs and escalating costs, Slack developed a detailed Spark monitoring framework that captures over 40 operational metrics and integrates with AI tools via Amazon Bedrock and other AWS services. This approach delivered up to 50% lower compute costs, 40% faster jobs, and 90% less manual tuning time, helping teams focus on insight rather than troubleshooting.

https://aws.amazon.com/blogs/big-data/how-slack-achieved-operational-excellence-for-spark-on-amazon-emr-using-generative-ai/