Hopi – Google Cloud Data Analytics Success Story
Hopi scales its data analytics capabilities and personalizes its offerings with Google Cloud Data Analytics.
Results -Higher performance due to managed services -Scales at speed to make real-time data analytics possible using BigQuery, Dataflow, and Google Kubernetes Engine -Decrease in operational headcount costs -40% cost reduction with pre-purchasing BigQuery slots -Increase in data processing speed
With Google Cloud, Hopi has dramatically improved its data analytics in terms of scale and performance and gained end-to-end visibility.
Hopi currently runs its microservices-based infrastructure using Google Kubernetes Engine, Compute Engine, and Cloud SQL and uses Pub/Sub to manage app events. Since the Hopi app provides personalized offers based on user actions within the app and user location, they need to process 15 million events every day. They state that even when they have an inrush of users or transactions, their system continues to be functional without any congestion.
Since Hopi uses Google Cloud managed services for various cases, their workload is reduced. Changes made during development are pushed to the production environment automatically - triggered with changes - through a CI/CD pipeline using Cloud Build. Events in Pub/Sub are transferred directly to BigQuery without creating a maintenance load on Hopi, making Hopi's operational work much easier. The Hopi data team can focus on more critical tasks as their operational work is much less. The team states that they are very satisfied with BigQuery performance.
The data Hopi collects builds the infrastructure for the Data Science team. Hopi Data Science team creates a segmentation based on collected data, and this segmentation is used by the marketing team for advertising. Hopi also uses monitoring and alarm features for their jobs on Dataproc, getting notified with every user right change in their project. Dataproc cluster is not up all the time; it is automated to run during certain hours for cost optimization.
As Kartaca, we have maintained cost optimization with slot reservation on BigQuery, buying slots in advance instead of on-demand to reduce the costs of nighttime jobs. We achieved a 40% cost reduction without compromising performance by making an annual slot reservation and commitment to BigQuery.
We updated the GKE version and optimized the instance types for price and performance, decreasing the resource usage in deployments and switching to instances with smaller resources (CPU, memory).