Enhancing Web Uptime and Performance: A Collaboration Between Compound Partners and Divio

Compound Partners is a digital innovation consultancy that helps professional services firms develop a deep understanding of their audience's needs and architect high-performing, bespoke website experiences. They assist their clients to build an audience, articulate their value, architect complex user experiences, generate high-quality leads, and measure success.

Since October 2017, Compound Partners has leveraged Divio’s expertise in cloud hosting, app orchestration and management, application security—including monitoring and automatic backups—receiving cutting-edge cloud advice and optimizations to achieve these goals.

Compound Partners and Divio worked together to resolve a complex availability issue for an end client whose site experienced large traffic peaks and was sometimes targeted by bots. These challenges caused site outages, which were addressed with an innovative caching solution designed by both companies. However, maintaining the site's high dynamism and allowing real-time content edits added layers of complexity to the solution.

Compound Partners, always pioneering new technologies, was the first to adopt the integrated Divio and Cloudflare solution, enhancing their capabilities in tackling such challenges.

The partnership effectively addressed 4 several critical issues:

1. Downtime Issues due to Micro-outages

2. Slow Page Load Times

3. Accommodating Large Amounts of Dynamic Content

4. Lagging Content Edits Caused by Caching

The impact of outages across the industry is escalating significantly. The proportion of outages incurring costs over $100,000 has grown annually—from 39% in 2019 to more than 60% in 2022. Similarly, outages costing more than $1 million have risen to 15% this year, highlighting the importance of effective website uptime monitoring.

Furthermore, the duration of publicly reported outages has also extended, with nearly 30% in 2021 lasting over 24 hours, compared to only 8% in 2017. Remarkably, 80% of data center managers and operators have faced outages within the past three years, underlining the ubiquity of this challenge.

In this context, Compound Partners approached Divio to address a recurring issue that was affecting the uptime and causing brief, yet disruptive outages for one of their clients. By minimizing downtime costs and improving uptime monitoring, Divio was ready to assist in tackling this challenge and mitigating slow page load times.

Downtime Issues due to Micro-outages

The Challenge

Compound Partner’s client operates a substantial and dynamically generated site featuring over 2,500 articles, events, and intricately designed landing pages. This complexity made the site particularly susceptible to malicious bot activity, which significantly increased page requests, causing the site to momentarily lag or become unresponsive. Additionally, the site's high-traffic email newsletter campaigns generating high volumes of traffic further stressed the system, contributing to periodic slowdowns and slow page load times.

These conditions led to "micro outages" that, while brief—typically under 30 seconds—occurred multiple times per week, adversely affecting the website uptime and user experience.

The Resolution

In response, Compound Partners activated a combined solution from Divio and Cloudflare in mid-August, which has since delivered results beyond expectations. Cloudflare works by channeling traffic through its network; it both optimizes content delivery and can put custom safeguards in place for the destination site. Divio meticulously tailors the Cloudflare configuration for each client, ensuring an optimal, customized setup is implemented to improve website uptime monitoring and mitigate downtime costs.

Since launching the solution, micro outages have been completely eradicated, achieving 100% uptime. Initially, the site suffered from 157 micro-outages, which have now been reduced to none. Moreover, server response times have significantly decreased, and on average, the solution blocks 532,400 malicious requests each month, bolstering site security and uptime and performance monitoring.

Slow Page Load Times

The Challenge

The complexity and dynamic nature of the site’s pages, coupled with high traffic volumes, initially posed significant challenges to user experience, reflected in a lower-than-desired Apdex score. To monitor and improve this, Compound Partners employed Pingdom Real User Monitoring (RUM), which measures and records page load times for each pageview, assisting with ongoing uptime monitoring.

The Apdex score is calculated based on user satisfaction levels, categorized as Satisfied (less than 4 seconds), Tolerating (4 to 16 seconds) and Frustrated (more than 16 seconds).

Before implementation, the results were:

• 73% Satisfied

• 24% Tolerating

• 3% Frustrated 

This resulted in an Apdex score of 0.85.

The Resolution

Post-implementation, the site witnessed dramatic improvements in load times, significantly enhancing user experience metrics. The average response time was reduced from about 1.5 seconds to just 0.5 seconds. 

The updated user satisfaction rates were:

• 85% satisfied

• 14% tolerating

• 1% frustrated 

This improvement adjusted the Apdex score to 0.92, nearing the threshold for excellence.

Moving forward, Compound Partners plans to further enhance the Apdex score above 0.93, targeting "Excellent" by optimizing caching strategies for pages containing forms and listings. These changes aim not just to eliminate outages but to elevate the overall user satisfaction and site performance.

Complicating Factors and The Problem-solving Process 

Analysis and Setup

Before initiating any solutions, Divio, in collaboration with Compound Partners, conducted thorough fact-finding. A test environment that mirrored the live site was created using Divio’s Web App Mirroring, and Cloudflare was enabled to get a better understanding of what needed to be done.

The Challenges

Editorial Control

A significant issue was ensuring rapid content management to correct or remove sensitive material swiftly. For instance, if an error in commercially sensitive content occurs, it must be rectifiable immediately—even if the content is cached.

Dynamic Content

Another critical challenge stemmed from the site's architecture, which heavily relies on dynamic content. When new content, such as articles or events, is published, it appears in multiple locations across the site. Dynamic content presents a special issue because the solution to the micro-outages relies on caching, as well as WAF (Web Application Firewall, a constantly updated protection against hacking that rolls out every few seconds).  Cloudflare caches at a page level, therefore caching all the “related” content as part of the page meant that new content would not appear immediately across the site.  Since new content must appear immediately across the site, any caching solution would need to accommodate daily content updates throughout the site.

The Resolutions

Editorial CMS Updates

To address these editorial concerns, Divio and Compound Partners prioritized establishing robust editorial controls. The CMS was enhanced to signal Cloudflare to re-cache a page whenever modifications are made, ensuring updates are reflected instantly. This setup is crucial not only for immediate corrections but also for the frequent minor edits that articles typically undergo within the first week of publication—common practice among content teams to correct small errors swiftly. 

Moreover, publishing or unpublishing a page also triggers Cloudflare to cache or uncache that specific URL, providing editors the flexibility to rapidly remove content as needed. Additionally, to support ongoing uptime monitoring, a new functionality was integrated into the CMS admin to manually clear cached content at both individual and site-wide levels, offering editors comprehensive automated and manual control over content visibility.

Dynamic Content Updates

With editorial controls in place, the focus shifted to managing dynamic content effectively. Historically, this was generated via a call to the CMS to populate the latest content as part of the page load. The approach involved refactoring the code within the “related” content plugins. By leveraging JavaScript to dynamically load content after the initial page load, the core content could be cached and delivered via Cloudflare, with dynamic elements populating immediately afterward.

This strategic combination of technological adjustments and innovative problem-solving ensured that the site could handle both the high demands of dynamic content management and stringent editorial requirements efficiently.

Other Considerations

One persistent issue was related to pages containing forms that generate CSRF tokens, which are secure random tokens essential for security. These tokens typically prevent pages from being cached effectively by Cloudflare. This was particularly problematic for campaign pages, which not only contain forms but also tend to attract high volumes of traffic due to successful email marketing campaigns. The need to ensure these pages functioned seamlessly was paramount.

To ingeniously solve this, Divio restructured the form handling by separating the form itself onto a different page and then integrating it into the main campaign page via an iframe. This adjustment allowed the main page to be cached while still maintaining the form’s functionality and security.

Additionally, modifications were made to the Google TagManager setup to ensure that form submissions were accurately attributed to the main campaign page, not the iframe source. This solution effectively balanced high traffic management with precise tracking requirements, ensuring campaign pages performed flawlessly under peak conditions.

The journey to optimize Compound Partner’s client's site setup continues, with plans already underway to enhance the caching capabilities for listing pages. This development is crucial for mitigating the impact of rogue bots that systematically generate excessive pageviews by navigating through each listing filtered by multiple criteria. Implementing this change will not only fortify the site against bot traffic but is also expected to further improve the site's Apdex score, pushing it towards even higher standards of user satisfaction.

The collaborative efforts between Divio and Compound Partners have yielded outstanding results:

• Zero Downtime: Achieved complete elimination of downtime, ensuring uninterrupted service.

• Enhanced Security: Over half a million malicious requests are successfully blocked each month, safeguarding the end client’s site.

• Improved User Experience: The Apdex score has impressively increased from 0.85 to 0.92, indicating enhanced user satisfaction.

• Faster Response Times: Average server response times have been halved, dropping from 1.5 seconds to just 0.5 seconds.

Both Compound Partners and Divio had high expectations for this project, and the outcomes have surpassed these. Given the success, this initiative is unlikely to be the last of its kind. The partnership is poised for future collaborations, further leveraging Cloudflare innovations to enhance client experiences and operational efficiency.

Explore more case studies to see how Divio has helped other clients achieve similar successes.