How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2%

by Derek Halpern · 87 comments

increase conversion rate

I’ll admit it… when I design opt-in forms, I’m confident that my form converts well.

It’s simple really. Make a compelling offer, demonstrate social proof, and ask for action, and bam! Instant subscribers.

Well, the other day, I discovered that my system was flawed.

When Opt-In Form Best Practices Fail

Social proof, as coined by Robert Cialdini, is persuasive because people often look to others before pulling the trigger on a decision.

So, when you create an opt-in form, offering up social proof is a best-practice because people assume “other people did it, I might as well too.”

However, when we tested our sidebar opt-in form with split-testing software, we noticed something strange… Social proof hurt our conversion rate.

As far as I knew, Social Proof was a sure-fire persuasion tactic for everything other than romantic desire. I was wrong.

Warning: I’m going to take you behind the scenes of the DIYthemes split-test. The rest of this article is different from what you’re used to reading here on the Thesis Blog, but I’m sure you’ll find it interesting.

The DIYthemes Opt-In Form Split-Test

To test these different forms, I used Visual Website Optimizer split testing software (aff link). It’s great split-testing software, and extremely easy to use. Now on to the results.

We used three different variations of our sidebar opt-in form. See them below, and click each one to enlarge:

As you can see, the control (the left), variation 2 (the middle), and variation 1 (the right) are all very similar, but they had markedly different results.

Figure 1: Control, Variation 2, and Variation 1 respectively. Click the image to enlarge.

Before we hop into the data, let me share my thought process behind these small changes.

In both variation 1 and variation 2, I changed “Get Free Email Updates” to “Get Email Updates (it’s Free)” because I figured that including (it’s Free) at the end would make it more visible.

Then, on variation 2, I decided to cut the social proof text completely, whereas on variation 1, I left it in tact.

I should have tested an opt-in form with the social proof below the submit button, instead of above the opt-in form. We’ll do that another time, so be sure to sign up free email updates here.

The Results of the DIYthemes Opt-In Form Split-Test

As you can see in the chart below, both variation 1 and the control performed rather poorly. However, variation 2 did extremely well.

Opt-in-form conversion graphFigure 2: See how Variation 2 improved over time?

When you split-test conversion rates, there’s always that chance that luck played a factor in your test. To prevent that, I tested this with 2,068 visits, and the split was as follows:

  • Control – 793
  • Variation 1 – 549 (I disabled this early because it performed so poorly)
  • Variation 2 – 736

In the end, Visual Website Optimizer predicted that Variation 2 would beat the control 95% of the time, and the final conversion rates were as follows:

  • Control – 1%
  • Variation 1 – 0.9%
  • Variation 2 – 2%

Yes, you’re seeing that right. Removing Social Proof increased the conversion rate of our sidebar opt-in form by 102.2%. (it’s greater than 100% because of a decimal)

Thinks about it… Sure, we may have only increased our conversion rate by 1%. However, when you’re working with hundreds of thousands of visitors, that’s potentially 1,000s of emails.

Why Did Social Proof Fail Here?

That’s a tough question, and I can’t answer that with certainty until I conduct more tests. However, my gut suggests that including a social proof message above the email sign-up form hurts conversions because it interrupts your visitors.

Additionally, the social proof may give readers a reason not to subscribe. Even though our numbers are respectable (almost 15,000), that may not be compelling enough for people.

Going forward, I’m going to conduct another test where I show the social proof below the submit button and see what happens.

What do you think? Why did social proof hurt conversions? Do you have any ideas on what I should test next?

About the Author: Derek Halpern ran marketing at DIYthemes, and is the founder of Social Triggers. To get more tips on how to be confident, sign up to his list here.

If you enjoyed this article, enter your email below to get free updates!

{ 87 comments }

Naomi Niles

My first instinct is to say that the text caused a distraction. Somehow the second option just looks easier even though it’s not. It could be that you have some very busy people coming in who are scanning quickly.

My question for you is: Do you think it’s worth the hit in conversions to increase the quality of your subscribers?

What I mean is, if the above were true, it could mean that the people who subscribed with the text there were more motivated to receive the updates. Later on, they may be more likely to actually read and click on the updates, which would make them better subscribers for you.

Interesting stuff. Thanks for sharing, Derek!

Derek Halpern

I think it’s much simpler than that. I think people already know about DIYthemes and Thesis, so that’s social proof enough. The extra social proof was an unnecessary distraction, but I’m going to test it… obviously. :-D

And you’re right. I would rather have a higher quality subscriber who’s motivated for email updates. One of the reasons we don’t promote the ebook download in the sidebar right now is because I don’t want freebie seekers.

Tim

Interesting theory…
You have a variable to test: those that know Thesis, and those that don’t.

You also have a parameter that would be pretty hard to pin down: what goodwill is inherent in the Thesis brand?

For me, the social proof text is neither a distraction nor an influencer. The reason I’m at the sign up form is because I have already read a bit on the site, want to know more, and like what I see…

So, I wonder if there’s another explanation?

Cynthia LaLuna

Hi Derek,

I love seeing your posts come into my email. I literally can’t remember when was the last time I opened Google Reader. I’ve gone old-school again in terms of blog reading.

Anyway, I digress, as usual. Thank you for sharing the results, and I’m definitely going to test the split-test manager you used. I hate Google’s product – I find it requires way too much effort to set up – so we’ll see if this one is easier.

I think it’s important to remember, too, that one site’s split tests don’t necessarily apply to another site – everyone needs to do their own testing to be sure, because the sample of readers is always going to be different. I would have made the same assumption about the social proof.

I agree that putting it under the Submit button could yield some interesting results as well.

Looking forward to the next installment!

Derek Halpern

I think it works for DIYthemes, but other sites should test it also. DIYthemes is different than most sites because it’s already a popular website. People already trust DIYthemes and Thesis. If you’re just getting started though, Social Proof may be necessary.

Bob Sutton

I can see an argument that split-testing social proof copy before the counter has acquired much credibility would influence results.

If I encountered this offer and the counter advised that I’d be just the “204th person” to subscribe, I might regard the offer as dodgy. I’d wonder “Why don’t more people care about this?”

But in your test, there were already, what?, 13,000 prior guinea pigs? Will be interested to see what your further tests reveal.

Tim

Good point. Is the number subjective?

Depends on the market? If I had a site about African southern right nosed bullfrogs and I saw 500, well, that’s fairly handy.

If it was the Miranda Kerr fan club and I was the 501st, I’d be worried.

Mark

Derek,
I am going to wager that the average visitor just doesn’t believe it.
Social proof is HUGE…don’t misunderstand me.
I can tell you when I first saw it, even though I trust you,
“Oh yea…sure…14,000 people. This is just another marketer trying to email spam me.”
I bet you if you added a couple pictures of real people (in your site demographics) recommending you right below or sandwiched in between the optin box as your social proof, you’d have a different story.

Derek Halpern

Could be possible

Matthew Hartley

Derek,

I haven’t read the other posts…but have you ever considered that the reason why your conversion rates where higher had NOTHING to do with the social proof…but rather…had to do with the fact that the form was easier for the customer to digest…with less text. I believe the social proof muddied up the form and people said screw this…when you remove the social proof text the FREE stands out more to the eye…and BAM…conversion rates go up…

that’s what I think happened…

Matt

Derek Halpern

Probably true. I think less text made it easier to sign up.

Cindy Sahli

Could it be that while people do look to others; they also want to know they are special…and can you really be special if you are 1 out of 14,000! So maybe there is a tipping point where being one of 10,000 – or whatever the nuber might be – discourages signing up! Something to think about.

Derek Halpern

Yea, people love feeling special. That’s why, when you’re selling a product that targets romantic desire, you should emphasize that. I wrote about that here:

http://socialtriggers.com/selling-fear-love/

However, the Thesis Blog helps you improve your business. I didn’t think that social proof would have a negative impact here, but it did.

Tim

Great point.

“Wow, so I will be one of 15,000? How will I feel special? Will the offers be specific and relevant to me?”

No numbers = sign up and see. Bit of merge tag for first name (Derek have you thought about collecting the first name) and voila.

I think personalised mail outs are a bit twee these days though, because there is no mystery around “how did they get my name?”.

Nope, I think the gold will be in analysing which readers open what, over time, then tailoring – curating – (manually or automatically) content just for them.

Mail Chimp needs to make that happen.

Peter

No data to support it, but I’m with this line of thinking. Folks are looking for an edge, a differentiator, and when there’s 15,000 people already getting the “tips” they sense the opportunity is already triaged.

Melissa Giovagnoli

I go with the belief that people questions the numbers from Social Proof. I think most people aren’t that savvy when it comes to technology and might have seen the numbers as unbelievable.

I also was recommended to put a pop-up for email signups on my blog and told that would increase signups. So far it has done well.

Derek Halpern

I’ve seen the popups and I know it does well. The problem I have with it is that I find it extremely annoying. Also, just because you’re getting more signups, are they the right kind of signups?

Tim

Over time, if they’re not, attrition will leave those that are right…?

So, I’d rather increase my sign ups by 300%, then lose 1/3 of them continually, than increase by 5% a month…

Jennifer Fontaine

Yeah! Another person proving the point I am always trying to make. It doesn’t matter what a business owner thinks, it doesn’t matter what their marketing person thinks, and it doesn’t matter what the designer thinks. In business, online or off, it matters what the customer (or in your case the user) thinks. Sometimes I am amazed at which things perform better online. I have spent years testing and adjusting sites, even back before the cool tools to help with that. I love this stuff, it geeks me out!

Thanks for sharing!

Derek Halpern

Thanks for stopping by Jennifer. Testing always shows the real results. And you know what, sometimes you have to run the test a few different times to make sure it’s the real results, too.

Alan @ Elliptical Source

This may not apply to the masses, but personally, I don’t like being a needle in a haystack – I know that opposes the herd theory marketing philosophy. However, I can’t say if that would detract me from something as simple as opting into a website on a subconscious level or not. Maybe your audience is more sophisticated and less susceptible to crowd behavior?

Derek Halpern

Most people don’t like being a needle in a haystack. However, when you’re learning how to build a business, wouldn’t you prefer the tried and true advice? Maybe…

I do have an interesting theory though. Maybe people think popular business advice is too popular to work for them. What do you think ?

Yak

You’re talking about a tight market that is very competitive. Why would someone want the same advice everyone else is getting? There’s no advantage.

You need to appeal to scarcity.

Tim

Disagree.

Marketing scarcity (whilst marketing to fear) is over blown, in my opinion.

Seth Godin and Chris Guillebeau are excellent proof of this.

Michael Blanchard

I’m with Alan. I think there’s another psychological factor at play here, which I don’t know the proper term for, but I’ll call it exclusivity. If 15,000 people already get these emails, why should I bother? How much of an advantage will I really gain? I’m more interested in something not a lot of people know, that gives me an edge, than in something lots of people already have.

That said, I also think the number is just distracting. I can easily decide if I want to get emails from you. Throw a number at me, and suddenly I’m thinking, “15,000 … is that a lot or a little? Do I want to be part of that group? Who are these people? Is my mom on this list? Did I call mom? Crap, I forgot to call mom!”

… and you’ve lost me.

Michael

ben

Hello! I’m very surprised no one has brought this up!

Wats the #1 rule of copywriting…the headline!

You changed the headline from the control, this could literally mean 2 things…

1- ur data is showing the effect in headlines
2 -both

But you don’t know which 1 because the variables weren’t controlled…

Try it with the same headline and also do that again with the placement of where the social proof is put!

Ben

Derek Halpern

Hey Ben,

In variation 1, I changed the headline only. In variation 2, I left the changed headline and deleted social proof.

Quite frankly, I don’t think the headline had an impact on the results. If you see the results between the control and variation 1, they were similar (1% vs 0.9%). The real magic happened when I changed the headline and removed the social proof.

Pashmina

I was thinking the same thing as Ben. I’m surprised you didn’t include a variation where you leave the headline exactly the same, and only remove the social proof. Let’s call this variation 4.

Given that Variation 1 (which only had a headline change) performed worse than your Control, one could deduce that it becomes probable that Variation 4 would have beat out Variation 2.

As for Social Proof itself and why it failed. Did it really fail? Or was it the readability? The text is set in Georgia, making the numbers hard to read. And the text takes up two lines with the last word being orphaned. And lastly you mentioned the position. It may work better on the bottom after the button.

Kevin

Yeah, I’m with Ben and Pashmina on this one Derek, sorry. When you’re attempting to accurately split test something, you can only change 1(one) thing/item/whatever, not 2 or more things; like you say just above ‘The real magic happened when I changed the headline and removed the social proof’…you don’t really know and/or can say with 100% certainty whether it was the headline change or the social proof now, can you? And really, that’s the only way you’ll know for-absolute-100%-sure what it was that affected the end results…not your biased personal beliefs or thought patterns. And believe me…whether you agree or not…your thoughts and beliefs are always biased, just like mine are…see? But accurate split testing gives you undeniable proof.

Cynthia LaLuna

Interesting that so many people see 15,000 as a lot. I subscribe to a lot of blogs that have numbers in the six figures – so to me 15,000 is very middle-of-the-road. A broadcast email is a broadcast email – I have no illusion that the email software thinks I’m special!

Of course, that is an insider’s point of view – which I think can always muddy perception of what the target audience will think.

Derek Halpern

You’re right. 15,000 isn’t that much, but give me a few months :-D

lou

Derek,

Great post! I might have missed it but how long (duration) was your test for? Thanks!

Lou

Derek Halpern

The test was conducted over 2,068 unique visits.

Andy

As a reader, I don’t care how many people subscribe to your mailing list. I’m much more concerned with the quality of information I’ll be receiving, and when I’m presented with what I perceive as unrelated information (in this case, your number of subscribers), I feel like two things are happening:

1. You’re trying to sell me on subscribing. If I wanted to hear a sales pitch, I’d go to the used car lot.
2. You’re trying to convince me that your number of subscribers has anything to do with the quality of your content.

Derek Halpern

Andy, I’m not selling anything. I’m sharing information about how we increased our conversion rates. I can’t tell you this same information will increase your conversion rates, but you should consider testing it yourself.

The going knowledge is this: use social proof to persuade people to sign up. However, that simply wasn’t true for DIYthemes. This is an insight which you should be able to use in your business.

Andy

Hi Derek,

I understand that you’re not selling anything, but including the number of subscribers feels like a micro sales pitch, which is off putting for me. Removing the element which people may react negatively — or at least, not positively– to may partially explain the increase in conversion.

Derek Halpern

Good points!

simon davies

Maybe you can have too much social proof? Surely some folks sign-up because they think they’re getting insider info that’s only available or accessed by a few savvy Internet users. By promoting the fact your newsletter is read by thousands dilutes the offer. So you get a situation where some people think, “Everyone reads it, so why should I bother.”

Connie DeLaVergne

I’m with a lot of your previous commentators . . . why would anyone want to feel like a cow in the herd? Especially with this niche where many of your readers know what you’re doing and aren’t interested in regurgitated information you can get from any number of online marketing gurus.

David C.

With this test it appears that the “social proof” information created more friction than incentive. Why?

My thoughts go to context. This opt-in form is for DIYthemes members who likely have a high opinion of the site. Making them stop to read and think likely interrupted an action (form completion) that they were already committed to, causing some to lose their train of thought and abandon.

Thoughts for future tests since you’ve established that less is more with your audience.
(1) Try a shorter call to action such as “Get Updates” or “Get Updates (It’s Free!)”; I say this because people like “Updates” but not necessarily “Email”.
(2) Since pop-overs might be too intrusive, try some mild, two-second animation with the” Join!” button, just enough of an effect so that the User looks at it once. You need attention in order to generate engagement and then conversion.

Derek Halpern

Good recommendations. I’ll try it out.

Bob Foster

I agree that the social proof is offputting. It’s like, make Sarah Palin your Facebook friend because 300,000 other people have. Not a good reason. Also, how do I know that you didn’t just make that number up? Whoops, you made me think. For me to sign up to receive spam, it’s absolutely essential I don’t think about it too hard.

Bob Foster

Good article, though! Bookmarked it. Seems obvious that you should test alternative sign-up appeals, or any click-through, really, now that you’ve illustrated it, but I never thought of it before.

Ana

I am a big fan of improving conversions and you tests, Derek, just go to show how drastically we can improve our results with just a few tweaks.

Just a personal opinion: being one of 15K would make me feel less “special”. What if I want to ask you a question? Would you answer me? After all you have such a huge number of people to stay in contact with…
That’s my reason to prefer your Variation 2.

Ana Hoffman

Ethan

I’m not convinced it’s the social proof.
Perhaps it’s the spacing between the Headline and where you enter your email address. Actually it could be anything.

Personally I think a few more things have to be tested, like:
1) moving the social proof under the button like you said
2) test adding another line of text, the same length as the social proof but, one that is not social proof such as a benefit of subscribing. Maybe this would test the spacing between the headline and where to enter your email address.

I believe the numbers you got, but shouldn’t the test have gone longer than just over 2000 visitors? Especially when dealing with around a 1 or 2% conversion rate.

If you are sampling around 700 visitors, just going from 7 to 10 signups can be counted as around a 43% increase. The result could be marketed as “Increase your signups by 43%” but the real difference with this sample size is still only 3 more signups. With such a small number changing the percentage so much, it just seems like a larger sample is needed.

Derek Halpern

I agree. We’ll have to run the test longer to be completely convinced, I suppose.

Dave Doolin

Ethan, I have to agree with you on this: there is no way to really know the reason. The very best marketing people I’ve read are unanimous: the reason is not only unknowable, it doesn’t actually matter.

Only the results matter.

Looking forward to the results from the next test!

Jeremy Reeves

Hey guys,

That’s interesting – however it’s not valid even in the slightest…

Ok… I see Ethan pointed out the same thing.

Personally I’d recommend having a MINIMUM of 30-50 conversions on each of the 3 before even looking at the results.

Definitely do another post when you’re done though, as that’s an interesting idea to test! :)

Jeremy Reeves

Derek Halpern

I appreciate the feedback. This was the first split test we ran on our site, but now that we have the right software, we’re going to be testing everything. So definitely look for us to improve our testing :-D

Pashmina

Btw, what software are you using?

Derek Halpern

Hi Pashmina. I linked to it in the article. It’s called Visual Website Optimizer.

Charl

My guess is that people want to feel special when doing stuff. They want to feel like they’re the only one being privy to some cool stuff. Even though they know they’re not. The numbers might just be a reminder that what they’ve just discovered isn’t that much of a secret?

Derek Halpern

I agree there, for sure.

Duane

In this context, I think most visitors will already have some trust or brand recognition or familiarity with DIY, and so the social proof you were displaying may have worked against other factors like Scarcity/Exclusivity. If they know 14,000 people already get the updates, then how much value can they bring me?

Derek Halpern

I think that’s the general consensus in the comments. People are already familiar with us, so additional social proof isn’t needed. Instead, they want exclusive / scarce information. To test this, I might have to try some more stuff out :-D

Dave Doolin

This email has been sitting in my inbox for a couple of days now, finally time to click through.

The take home I’m getting from this is two-fold:

1. It reinforces my current practice of NOT offering my opinion disguised as a fact of “what everyone knows is true.” This sort of behavior is rampant in Blogistan. I call it the Echo Chamber.

2. If someone else renders their opinion, and they don’t back it with measured results, I have to regard it as worthless. That is, of no worth. There are a fair number of fairly well-regarded bloggers I know now in this category. But I don’t treat as something personal. The next time someone such as that offers measured results, I’m happy to consider those results.

These two attitudes have long term consequences. The most significant of which is I’m no longer blogging much, a situation which will remain until I dig much deeper into testing to figure out what works and what doesn’t.

Thanks for making these results public. An inspiration to get the hustle on.

Jochen

Hi Derek,

I don’t want to go too much into the social proof argument, even though I wonder why it didn’t work well… I’d like to talk about your test traffic. If I get you right you had about 700 visits per Test, with 1-2% conversion. So that means we’re talking about 7-15 leads per variation. In my opinion, these are WAY to few for being able taking a sound decision. I run a lot of tests by myselft and as long as the leads per variation are below 100 I see tremendous results, but as soon as you get in the area 500+ (which takes a little time I have to admit), the results change heavily and I’m happy if I can keep a 20% uplift…
Would be interesting getting your ideas about that!

Cheers,
Jochen

Sean Weigold Ferguson

I agree with Jochen. Using the numbers you provided, I ran the tests myself. There isn’t a significant difference if you include all three variations in the model, nor is there one if you use a two-tailed test of significance.

I think it’s wide open for debate whether or not there is actually a statistically significant difference between them.

Dave

When I see 14,000 others, that makes me just another trophy in your collection and not all that important at all. In fact, I’m the center of the universe. Treat me like that!

Rob McCance

Derek,

Great post, thanks for sharing and definitely somewhat surprising.

But if I ignore all the current thinking and “best practices” on social proof and just analyze what I do personally, it’s not so surprising.

Personally, when I see things like “join the other 75,000 RSS Subscribers” and the like, to me it makes the entire offer seem LESS unique and special.

Most of the times, it’s just a neutral and if I like the content, or offer, I opt in or subscribe regardless of the social proof.

But sometimes it’s a negative.

So you go from neutral-no-bias to neutral-negative-bias and you can swing the statistics on large samples.

Aled

Great article. I’m just getting to grips with the whole world of split testing, so for me this was really helpful to reinforce the benefits of continually testing.

I also love the thesis theme and am in the process of rebuilding my website. Thesis will be on my shopping list.

Thanks

Aled

Mark Dowling

When I see highlighted specific numbers, eg 14,752 – I think of those spam emails that scream “Make $123,456.78 In 7.85 Hours! In Your Pyjamas! The Shocking Truth Revealed”. In other words, something in my brain equates specificity with BS. Maybe I’m jaded because I’ve seen it so many times? Control 2 seems so much cleaner & authoritative, & I for one would feel more comfortable clicking on that. Just my gut feel.

Vivek Parmar

Great post, surprised by seeing the results

Rob

It’s a bit strange that at the end of this article you use a similar opt in box to the one you said didn’t work? What’s going on?

“If you enjoyed this article, join 14,864 others and get free email updates!”

Derek Halpern

I didn’t split-test that box yet. Just because it didn’t work on top, doesn’t mean it doesn’t work on the bottom.

Patrick Donovan

Your mind is a labyrinth, Derek, a labyrinth! And as such, I know you know that statement to be a compliment. Thank you, Derek – always a pleasure reading your work.

“Just because it didn’t work on top, doesn’t mean it doesn’t work on the bottom.”

I think that belongs on a motivational poster.

Derek Halpern

People, including me, make too many assumptions. I assumed social proof in our sidebar helped our conversions. I tested it and proved otherwise.

If I then assumed that social proof didn’t work for our after-post conversion form, I’d be going against the whole testing mindset I’m trying to develop. Heh.

jonathan soroko

I think that with a paid blog theme – or any app, paid or FOSS – there’s an economy of scale that people are aware of intuitively. A sizable community means (1) the developers are doing well enough to deal with bugs; (2) have incentive to do better in the future; (3) there’s a community of other users who may be of assistance, whose mistakes one can learn from, etc. It’s a network effect – described by Paul Baran in his essays on distributed networks for RAND in the 1960′s leading up to the implementation of DARPANet. If anyone’s interested, I can find the essay (nine essays on the subjects; eight are publicly available, the ninth is still classified).
In any case – very interesting post and discussion.
Jon

Juho Tunkelo

I think the reason for the result is pretty clear. The ‘join XXX others’ is not social proof at all, it’s just data – if you’d said what benefit they’d derived, it might’ve worked. Also, ‘get free updates’ is equally meaningless. It just takes away instead of adding anything. No wonder the clean cut one won over.

Derek Halpern

What’s your definition of social proof?

Kala

The middle one was my fav-I’m an artist and intuitively it was just the easiest read, smoothest and quickest, and the free part hit me because it was the last word. It’s the one I would have picked without testing as I find that intimidating to try and do!

Amanda

I think the simplicity factor helped Variation #2, as well. Less to read, less advertising-y, more temptation to click. Sometimes, when I view such numbers that you’ve dubbed as “social proof,” I immediately think of impending spam. Not sure why. Also, that aggregate number also technically contains a piece of my data (my unit as one person)–seeing the “social proof” might make me subconsciously believe that the site would share further actual data about me, publicly.

– Layman’s point of view. :)

Databoy

I think there are a few things at play.
1) Simplicity – less is often more, especially in context. When the call-to-action is split (more than one close by) the importance of simplicity increases.
2) There may be other external factors at play. i.e. Perhaps social proof is more important in the summer? Perhaps social proof converts at 10% at night, .01% during the day? People using a certain browser prefer social proof? I think your sample size is OK, but it’s possible more tests might reveal more insights.
3) As some people commented, adding social proof might decrease the exclusivity of the offer. Testing, as you did, is the only way to get the best results. ABT – Always Be Testing

Stephen

It’s way too early to be analysing the results – there’s just not enough data yet.

As Jochen and Sean say above, the data breaks down like this:

Control 793 visitors, 1% CR, 8 conversions
v1 549, 0.9%, 5
v2 736, 2%, 15

The difference is between 8 and 15 conversions. According to VWO’s own significance calculator – http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ – it’s not a significant result yet.

I’d strongly recommend running the test a while longer. And keep v1 too – the test didn’t have enough data to be significant, so I wouldn’t recommend pausing a variation even earlier. It’ll also help indicate which of the two changes (title and social proof) is affecting the CR.

It’s an great test to run – I love that you’re not accepting anything as “best practice” and are testing commonly-held beliefs – but it’s just way too early to be discussing the results.

Bill Scully

Social proof in the form of just numbers on the web cannot be trusted, you can blame the get rich quick spammers for that; and alone may have an adverse effect.

I would try testing testimonials from real people (Using full names) beneath the submit button.

Chris von Nieda

I’m not gong to read every comment on here to see if this was already said but it appears Bill just said it. The number is big. 14k plus may be an unbelievable number and hurt credibility.

Vincent Polisi

Derek:

Great test and great information. I think the thing that is missing that doesn’t seem to have been touched upon is the true quality of the conversions. Conversions are great but conversions that translate to monetization are what everyone is really after.

I think what everyone needs to do is the split test and then tracking of how each segment translates to real revenue, un-subscribes, spam complaints, etc. In other words, if one segment of the split produces twice as many initial subscribers that don’t stick around and don’t translate to revenue, there is no inherent value in the conversion rate. Conversely, if the segment of the split that has a lower conversion rate produces stickier subscribers and greater documented revenue, though the number of subscribers isn’t nearly as appealing to the ego, the additional revenue more than makes up for it.

Sharon

I think the social proof hurt because people want a stealth insider secret….they don’t want to feel that 15000 others are using the same tactics! Irrational ego rules over logic any day :-)

Dwight

Derek:

When I started looking into split testing my sites, I found out there are 31 variables to test in a headline; All caps, color, font, ending, context, emotional plea etc. and the list goes on.

I also looked into a system called Taguchi testing, or scientific testing, where you can test over 14,000 different combinations using 18 variations. I found talk about it in Adwords for Dummies 2007 Howie Jacobson,PhD. The technical editor for this book knows more. His name is David Bullock.

I personally have a pretty good understanding because I like math and I have been using it for a few months now and it grows on you.

Bottom line, testing has changed my site and my income. I use adwords to complement my sites visitors and I had the same thoughts you had before I understood the consequences of not testing. Before testing, I had a .07% click through rate on my ads to my site.

I now enjoy upwards of 1.37% click through rate on all three networks combined per month. Some of my ads are pulling from 3.4% up to over 7%, most are around 5%.

I found this combination through testing and I will never go back to thinking that I know what my audience wants. I now ask them, “do you like this?”, or “do you like this better?” and let them tell me what they want. Using the data to find out what your customer really wants and will react to is the basis for success.

Keep up your good work, and if you want to shorten your testing time and ability to see what influences what, take it to the next level.

Brett

I tried Visual Website Optimizer and love the technology, but it only allows you to modify one page at a time.

I was wanting to test a banner I have across the top of every page encouraging an optin, but VWO only allowed me to change one page. Did you test your sidebar on every page? And if so, how did you get VWO to do that?

Cheers and thanks for the great info. This tool is really good!

Heluo

Derek, I can only speak for myself, but I would subscribe based on the author, the content and the subject if it would have my sustained interest, leaving all other factors out. That’s it. That’s three magnitudes that would make me hit the subscribe button, whereas actually, the chance of me subscribing would decrease by multitudes with each deviation, until each such distraction could have the three factors mentioned earlier vanish altogether and I might click away.

Anyone’s mileage may vary, but chances are, with the more testimonials, the more ‘subscribers’ or any other ‘sales’ proof, it wouldn’t be easy to keep me there (it is possible, but it is one of those indicators to set off a good-bye).

If you address me as one of 14,752, so that I am #14,752, it would be like I am in a soccer stadium and you are talking through a loud announcer. Whereas, you could be still talking to same amount of people, but if you make the ‘catchy phrase’ into a ‘personal message’ or some humoristic statement, I would feel it is just you an me, even though everyone knows I am surrounded by audience.

Actually, seeing with how many others one would have to share ‘secret of the gadget’ would increase the chance that I would not subscribe.

But then again, I am easily one of those saying they’d never be caught on candid camera and then one day I ended up in one of those television programs, never once having noticed there had been six cameras on my face :o)

Heluo

Jean-Luc

Well I’d rather say it’s a simple question of rythm, and not social proof
On variation 2, it says “1, 2, 3″, it is simple and efficient
On variation 1, it says “1, 1, 2, 3″, it is complicated and you say twice the same thing “Get Free Update”, by doing that you indirectly and unconsciously say to the reader that he’s dumb so you need to repeat, so you need to repeat (see? see? ^^)
If you want social proof, I’d put variation 2, and under the button, a simple text: “14000 others”, it’s smart and simple

my 2 cents ^^ (and maybe I’m totally wrong)

Chase

Hey Derek,

Loved seeing Cialdini cited here. Did you ever run that second test (with social proof under the opt in line?) I’m curious of the results.

Charles Tutt

Great article. Red meat!

Personally when I subscribe to an email list or make a purchase online, I’m turned off by testimonials or other “social” because it’s just part of the ‘sales pitch’ as far as I’m concerned. I do login to the forum, read other articles and in the case of purchase, check other websites for the same product and then make up my own mind.

rob aubrey

The reason social proof does poorly on this type of site is because the way people make decisions.

There are two types of processors for decision making, internal and external.

Internal processors like to find out things on their own and don’t rely on what other people think, do or say. In fact it repels them. They research and then decide.

External processors care more about what other people think. They ask friends, family members co-workers what they think and then go with the majority.

Neither is right or wrong- they just are.

So here’s the thing, most people that are reading sites like this are researchers and most likely fall into the internal category.

Because internal processors are actually repelled by “other people” it could actually hurt you.

Rob

Maria Peagler

Derek – My first thought was that with numbers in the 15,000s for subscribers, people might think it’s junk. Certainly that’s not the case, but touting the number of subscribers wouldn’t be as convincing as a real testimonial from just one of them.

bo

My immeadiate reaction was: 15.000 subscribers, a lot of mail, spam, mass, full inbox, don’t want it. So I associated it with getting a lot of mail.

Then another reaction. I do not want to see what the masses are subscribing to but more specific stuff. Even if that is not a valid point.