"We provided expert guidance to help the client improve the overall reliability and stability of their system. This will ensure that their system is always dependable for their clients. "
Case Study: Cloud Optimization to Achieve the Ability to Sell 20k Tickets in 5 Minutes
Our cloud experts reviewed a client’s ticketing system and provided actionable recommendations to help them achieve their goal of selling 20k tickets in 5 minutes.
ABOUT THE CLIENT:
SERVICES USED:
SERVICES USED:
The Challenge
Our client sought a team of Cloud Consulting experts to optimize performance, address inefficient scaling, and achieve their goal of selling 20,000 tickets in five minutes.
The Solution
An action plan was prepared to help our client sell 20,000 tickets in 5 minutes, including reviewing the production environment, optimizing database queries, addressing performance issues, implementing data partitions, utilizing system insights for auto-scaling, blocking unproductive bot traffic, conducting load tests and prioritizing action items for performance enhancement.
The Result
We delivered a prioritized action plan with guidance for the client to:
- Optimize database queries
- Improve system efficiency
- Enhance autoscaling strategy
- Create a more responsive user experience
- Reduce costs and waste
The Need
Our client is a prominent events booking and discovery platform in Europe that came to us with a bold objective: They wanted to have the ability to sell 20,000 tickets in five minutes.
To meet this objective, they sought out Softjourn's expertise in Cloud Consulting, as it would take cloud optimization to potentially boost their performance to the desired level.
Our client wanted to receive cloud-related recommendations for performance efficiency, monitoring and alerts, and operational excellence, following an infrastructure assessment and database review.
Additionally, we wanted to help our client optimize costs, as they faced a significant challenge with their infrastructure maintenance costs due to inefficient scaling. They had a database and multiple web application servers, but the scaling was focused on the application servers based on response time, even though the root issue lay in the database and problematic queries.
When each application server attempted to retrieve information from the database, the execution time was significantly prolonged.
Softjourn Project Manager
The scaling approach taken was to increase the number of instances in response to the long response time. However, this only exacerbated the problem as more instances were added, leading to a higher demand for the database.
Consequently, our client ended up paying for numerous running instances without experiencing any significant benefits. The core issue was not with the application servers' performance, but rather with the code responsible for handling data in the database.
The client also employed a less commonly-used Cloud Provider, Heroku, due to its many features that are pre-made for Ruby deployment. This added an additional challenge when searching for a qualified cloud consultant, as they would need someone with ticketing industry expertise, experience with Ruby, and the flexibility to learn about Heroku.
The Solution
In order to evaluate the client's objective of selling 20k seats in 5 minutes, the following action plan was prepared:
- Production environment review
- High-level Database review
- Google Analytics review
- Performance evaluation of the current production environment
- Action Items Based on Greatest Impact on Performance
Production Environment Review
Improvements to Autoscaling
To make autoscaling more efficient, we took advantage of the dynamic nature of the Heroku platform, suggesting that the client pivots toward a more optimized autoscaling strategy that takes into account resource utilization, including CPU and memory, rather than focusing solely on response time. This progressive approach should resolve any response time challenges and better distribute resources, reducing wasteful spending and optimizing performance.
Rollbar Issues
We found several active issues that were reported in Rollbar. We identified urgent issues such as problems with canceling bookings, missing actions, and timeout exceptions.
High-Level Database Review
Optimization of User Profile Query Analysis
As part of our optimization efforts, we focused on improving the query performance. We introduced a user profile query analysis process that yielded remarkable results.
We identified a critical issue in this analysis, related to the selection of purchased tickets on the user profile page. Currently, this query takes more than 5 seconds to execute and frequently triggers autoscaling events.
Fortunately, with our suggested improvements, the execution time of this query can be significantly reduced to a mere 53 milliseconds, making it approximately 145 times faster than the original query. This enhancement ensures a much more efficient and responsive user experience on the user profile page.
In our database review, we also found that the client's Heroku database management console had recorded a couple of slow queries that must be resolved before applying any autoscaling changes.
Database Cleanup
We initiated a database cleanup process, that would remove any unnecessary records - including nearly 10 million phantom tickets - thereby reducing the amount of data stored in the database. This cleanup will not only enhance data integrity but also contribute to an improved query execution speed, leading to overall system performance optimization.
Additionally, to enhance query execution and improve overall performance, we advised the client to implement data partitions. By utilizing data partitions, the size of data processed during query execution can be significantly reduced, resulting in more efficient and streamlined operations. This approach would allow the client to better utilize resources and enable faster retrieval of relevant information from the database.
Google Analytics Review
System Insights
Google Analytics insights revealed a consistent pattern in system utilization throughout the day, with three distinct usage periods. These patterns offer valuable guidance for optimizing the auto-scaling strategy, enabling dynamic adjustments based on anticipated system loads during specific timeframes. This responsive approach will enhance resource allocation, ensuring efficient system performance in alignment with observed user activity trends.
Slowest Loading Pages
We identified pages with loading times exceeding 20 seconds and provided recommendations to improve load times, with the general target being less than 5 seconds, and ideally less than 2 seconds - with the ideal server response time should be less than 0.6 seconds.
To achieve this, we suggested that they utilize in-memory and file caching, optimize database queries, and improve the application codebase.
Usage Geography
We found that approximately 5% of the overall web traffic originated from a country that is not contributing to any transactions. This suggests the presence of bot activities from this country. To prevent unnecessary processing of invaluable traffic, we recommended that the client update their robotos.txt file to block unwanted bot activities.
Production Performance Assessment
To assess the client's production performance, we suggested an automated load test to determine the maximum number of users who can buy tickets in 5 minutes. We offered several load testing options to establish a tailored baseline for their specific business needs.
Proposed Action Items
We prepared a prioritized list of action items for the client related to the findings from our reviews and assessments of their production environment, database, Google Analytics, and performance.
These actions were prioritized based on the impact they would have on the client's aim to enhance performance and address the objective of selling 20,000 tickets within a 5-minute timeframe. By following these recommendations, our client can effectively evaluate and improve the performance of their production environment, ensuring it can handle the desired ticket sales volume efficiently.
The Benefits
By following Softjourn's cloud consultation recommendations, our client will gain the following advantages:
1. Enhanced Performance
Softjourn conducted a thorough review of their client's production environment, database, and Google Analytics. By identifying and addressing issues such as inefficient database queries, slow-loading pages, and unnecessary data records, Softjourn helped them understand how to optimize their system's performance. This would result in a significant improvement in the execution time of critical queries, making the user experience more efficient and responsive.
2. Cost Optimization
Softjourn identified ways to make the client's infrastructure scaling strategy more efficient to avoid unnecessary spending.
By suggesting improvements to the autoscaling strategy based on resource utilization rather than response time, Softjourn's recommendation would help them optimize costs. Additionally, the database cleanup and implementation of data partitions can reduce the amount of data stored, further optimizing resource usage.
3. Expertise in the Ticketing Industry and Ruby Deployment
The client faced the unique challenge of finding a cloud consultant with expertise in the ticketing industry, experience with Ruby, and the flexibility to work with the Heroku cloud platform. Softjourn demonstrated our expertise in these areas, providing tailored recommendations and solutions to meet their specific requirements. Our team worked hard to familiarize ourselves with the inner workings of Heroku to make sure the client would take advantage of everything it offers.
4. Recommendations for System Improvements
We provided expert guidance to help the client improve the overall reliability and stability of their system. This will ensure that their system is always dependable for their clients.
5. Performance Assessment and Load Testing
Softjourn proposed conducting an automation load test to assess the client's production environment's capabilities.
We helped them understand how to reach and test their objective, by using load testing to determine the maximum number of users who can successfully purchase tickets within a 5-minute window. These will serve as valuable insights for capacity planning and performance optimization.
6. Prioritized Action Plan
Softjourn provided the client with a prioritized list of action items based on their findings and assessments. This will help them focus on the most impactful recommendations to enhance performance to achieve their objective of selling 20,000 tickets within a 5-minute timeframe.
Additionally, Softjourn made sure to provide multiple options for action items, giving the client a guided choice on which path to take.
7. Other Consulting Services
During our time working with this client, we advised them on other areas outside of cloud optimization. We also provided consultation to our client on the following:
- Improving their Reserved Seating process to reduce manual work and define seats for their customers during ticket purchasing;
- Enabling Internationalization Support for the promotion of events in different countries through UI translation, local events discovery and events promotion, timezone specification, and multi-currency support with Stripe;
- Implementing a Recommendation Module to promote the most relevant events to each site visitor through the use of a recommendations engine, enhancements to their website for better events recommendation, an Events Promotion Campaign, and customer data research.
Conclusion
We enjoyed working with this client on such an exciting project and helping them reach their goal. The CEO and Founder appreciated Softjourn's knowledge of ticketing and said, "The work that we did with your team was really great."
Our Solutions Architect said that he found the cloud consultation for the client both complex and interesting and enjoyed brainstorming methods to optimize performance and costs to meet the client's needs.
We know that if applied carefully, our FinOps consulting services will help the client reach their goal of their ticketing system being able to sell 20,000 tickets in five minutes.
Partnership & Recognition
