We’ve often heard various assumptions or misconceptions about outsourcing A/B test development services. Our CEO, Abdullah Mashuk, shared his experience and broke down some of these misconceptions as one of the panelists on a Kameleoon webinar entitled “No developers for your A/B testing program? No problem. Learn why.”
Thank you to Kameleoon for the opportunity!
Mature experimentation programs often require more resources than organizations can give to their CROs — and building the “build,” or developing the tests, is a common roadblock. This is often a challenge for getting new programs started, too.
While A/B development outsourcing companies exist to solve this problem, misconceptions about outsourcing A/B test development stop many CROs from running the experiments they want.
For example:
- Is outsourcing A/B test development unsafe?
- Will an outside development agency kill my organization’s culture of experimentation?
- Am I essentially just paying for code?
Brillmark CEO, Abdullah Mashuk, joined a Kameleoon Live Session, along with Siobhan Solberg (Founder at Raze) and Tasin Reza (CEO at Echologyx) to explore the do’s and don’ts of outsourcing A/B test development service.
Here are the highlights.
1. Agency Developers Need Not Threaten Your Culture of Experimentation
Executing a test program is a collaborative effort.
Hiring external development talent provides stability.
During his time working as Red Eye’s CRO Director, Tasin Reza struggled to find good A/B testing engineers. It’s a constant challenge for most CRO programs because not only are specialized A/B test developers a rare lot, but they’re expensive, too. Tasin tried building an in-house development team by offering all the tools and training, only to lose his trained developers to full-stack development jobs at other places. That’s when he felt the need for such a service — one that assisted with building the test. The need is real!
Outsourcing can improve your culture of experimentation.
While in-house engineering teams don’t contribute to building the tests when outsourcing, they aren’t left out of the experimentation program.
Siobhan Solberg’s Raze makes sure “to keep the whole team involved.“ The in-house developers are, in fact, the ones implementing the winning tests. So they’re very much a part of the optimization program. “We make sure to let them know why we think [a winning version] won, what the analysis was, etc.“
Siobhan also tells how in-house development teams are the source of some of the best testing ideas and that they “really appreciate the fact that they can focus on their development projects, but they’re still kept in the loop.”
“It’s a partnership,” adds Abdullah Mashuk of Brillmark. “We often advise clients when they believe a test idea can negatively impact their conversions or produce no conclusive results. We work like a partner trying to get our clients the most value from their experimentation programs.”
Abdullah Mashuk – CEO of BrillMark
Tasin shares that there have even been cases where Echologyx developed winning tests, and the clients’ developers asked them to supply the code to use on the live website/product.
The process is crucial.
Powering a collaborative test program needs a great process. From establishing access levels and communication channels to setting expectations and everything in between, it’s the process that holds the key to successfully outsourcing test development.
2. Outsourcing Test Development Is SAFE
You’re always the one to sign off.
You control the access levels.
At Echologyx, Tasin explains, “We don’t push the tests live … It’s always the client. Everyone knows what they can and can’t do on the testing tool.”
While at Raze, it’s Siobhan or the consultant in charge who pushes a test live after it has passed all the QA processes. “We have rules. Developers don’t set a test live. They aren’t allowed to.”
Whereas at Brillmark, only the most experienced and senior-most developers get access to the testing tool.
In all, it’s usually you who is the one signing off on the experiment. You can opt to add a round of internal QA’ing to your test builds if you want to be doubly sure. Besides, if something still goes wrong, you can always just stop the test.
Nobody needs to touch your code unless you want them to.
Depending on the testing platform or ab testing tools you use, you can give your developers the right level of access to it and never to your live code. So it’s safe, notes Siobhan: “They’re not touching the code of the site. They aren’t messing anything up.”
You can avoid accidents.
Development agencies do their bit to keep your testing programs running smoothly. At Brillmark, Abdullah and his team take extra caution to ensure that the clients’ testing programs work just fine. To “avoid accidents,” they build the tests locally, and it’s only when the code is ready that they move it to the testing platform. “We don’t start the development directly on the testing platform. We have built some open source tools that allow us to build the test outside of the testing platform.” This way, access to the testing platform is restricted to just the experts.
3. A/B Development Agencies Do More Than Code
SPAs, server-side, in-app test development — you can outsource it all. Optimization development agencies are full stack in the real sense. They always have someone on their team to take your React- or Angular-based single page application and run that experiment you hypothesized. The same goes for complex server-side and in-app testing. For more involved testing like this, your external developers “need to actually get into the roadmap of the engineers and on the pipeline of their full stack development cycle,” explains Tasin.
It’s more than just outsourcing code. A/B test developers don’t just develop your build. They:
- Optimize it for speed (experimentation tools — all of them — cause lags).
- Test it across the many browsers and devices your target traffic uses (poor rendering on even one browser can skew an experiment’s result).
- Make it flicker free (not all experimentation tools come with built-in flicker protection).
Your in-house developers only work on what they know how to do. If you had unlimited development bandwidth, your in-house developers could pursue every single hypothesis. But that’s never the case. Given that 80-90% of the time, a test doesn’t produce a winner, an agency saves you hundreds of in-house dev hours. Plus, winning experiments validate ideas, so your developers only work on ones that will positively impact your business’s bottom line.
About BrillMark:
We are CRO development experts with years of experience building A/B, multivariate, and personalization experiments across the entire customer journey. With over 10,000 campaigns under our belt — the device, application, platform, or complexity doesn’t matter — we’ve got it covered!
You provide the strategy. We’ll provide the technical expertise and deliver flawless campaigns at competitive prices every time.
Our management team is made up of CRO veterans who recognized two needs:
- The need for deep technical expertise and reliable, affordable testing in the CRO space.
- The need for hands-on technical training and competitive employment for a new generation of developers.
As a result, our goal at BrillMark is twofold. We provide reliable, affordable testing for CRO agencies and in-house campaigns, and our team of A/B test developers is continuously training and employing talented developers around the world.
If you’re passionate about what you do, we’d love to connect and help you meet your goals on ab test development for shopify or others.