During the Neotys Jurassic PAC 2020 virtual conference, Luca Cavazzana (Software Engineer at Akamas) explained how Akamas uses AI techniques to find the optimal configuration for Spark applications. He presented a study where Akamas tuned both Apache Spark parameters and EC2 cluster size, finding an optimal trade-off between resource allocation and execution time that minimizes the overall cost.


The Optimal newsletter