I think the priming effect is more subtle though. What people express out loud might be primed, but their actual feelings may not change. I wish we had kept the data on this, but our search experiment at reddit is a good counterpoint:
At one point we measured how search was doing, so we added a button to the top of the search results that said "did you find what you're looking for?". 70% clicked yes. Not great, not awful.
Then we upgraded the search engine, but didn't tell anyone. Suddenly that stat jumped to 90%+. But people would still complain just as much about how bad search is. Many months later, we finally announced that we had changed the search engine.
The stats on the button didn't change, but the public narrative did. So what people say they perceive and how they actually feel may not necessarily match.
Such things are very tricky, because negative experiences are remembered vividly. Search working is expected, search not working is a huge problem. Also, search is a hard thing to get right.
When it goes wrong on reddit, it goes annoyingly wrong. For instance, my main issue is that some searches return a flood of irrelevant content. Searching for some games brings a flood from r/GameSwap or some such place. Or, trying to search about Nikola Corporation will bring up a whole lot of sports personalities.
That makes sense in that it's a tough problem to solve, but what's annoying is that it has to be dealt by hand every time. I can write a filter, but what I'd really like is a permanent setting: "I'm not ever interested in results from /r/GameSwap or /r/SportsSubreddit". Also it might be helpful to be able to set a limit how much stuff can come from a single subreddit, because some contain very repetitive content that drowns out all useful results.
Edit: Also, search should parse youtube URLs and ignore HTTP vs HTTPS, youtube.com vs youtu.be and the ?feature=share junk at the end. I can't be the only one who thinks "This must have been discussed on Reddit, and the discussion has to be more useful over there", but Reddit comparing the URLs literally makes it annoying to find matches.
I am guessing that most people wouldn't use reddit search because of its reputation, so the 90% of people saying they found what they were looking for were a small % of users. When you posted that you updated search, a lot of people who had given up on it might have tested it out again and changed their opinions.
Besides the rare "OMG! This is AMAZING!", people who are upset by something are much more likely to make noise than people who are content with it. It's just how we are wired.
Odds are, no changes you make will completely silence the loud few, and even if it does, it'll trigger others. You can track engagement though. If the majority of people silently but demonstrably show that they enjoy how things are, you have a solid foundation to build on. The numbers will speak for themselves.
This is something I learned as a subreddit moderator. People were quite vocal about certain things, but when we polled the community, they turned out to be a minority.
For example, people constantly complain about pictures of the same few things filling the sub, but those pictures get massively upvoted, and several polls confirmed that people want them to stay. Yet we go over this topic every damn week.
Another was my own behaviour. I often post links to my own website in answer to questions. Some users were quite vocal about it being spammy, but every time we polled the community, they said it's alright. My comments get upvoted, OP gets his answer, and mostly everyone is happy. The mods also gave their approval.
There's a point where you can say "haters gonna hate", and rest easy knowing that most of your users are happy.
The fact that the search engine changed though is information. If the search engine is the same, maybe they had 490 mediocre experiences with the old one, and then 10 better ones. Since it's the same search engine, they're going to average all of those together and say it sucks.
If you tell them that the search engine is brand new, they'll reset their expectations and only look at the new data to make a judgment.
I hate that in the new design, when I'm IN a specific subreddit, there's no way to limit search to that specific subreddit. After the search is done, it'll show a link do only display results from inside that subreddit, but 100% of the time I'm in a subreddit, I want results from that subreddit.
Edit: adding that because of the search, and lack of managing multi-reddits, I'm still using the old reddit with the RES extension
I’ve been using reddit since around 2012 and throughout that time I rarely used Reddit’s search mainly because it didn’t search through comments in a post to score relevance. The only times I would use Reddit’s search was if I remembered some words or phrases in the posts title and I had a specific post in mind I was looking for. I’m also pretty sure that back then the relevance of search results in general using Reddit’s search was far inferior to site:reddit.com specifically in query expansion (synonyms & misspellings in particular).
I only started using Reddit’s search recently because of changes to google that make reddit search results have incorrect times.
People remember two things - when things don't work, and when they start working.
The first is why people will complain something is "crappy" if they had one bad experience with it, and the second is why the "new" thing is often perceived as better EVEN if it has more problems than the old - as long as it doesn't have the same problems.
After awhile the "new" wears off and it's crappy again.
Is this referring to Google. I never click any buttons or participate in surveys to send more data to companies like Google. Whether I found what I was looking for, etc. is none of their business. As such, any conclusions made from users (or bots) who do click such buttons is ignoring all the users who don't.
At the same time, I do have opinions about these companies and what they have done to the concept of a "www search engine". I may share these elsewhere, such as on HN. Any conclusions based solely on clicking buttons on a company website would be ignoring user comments elsewhere. One could argue any results are only potentially applicable to the type of user that clicks buttons asking for feedback, which may be a small subsection of total users.
"Priming" is an interesting idea. One of the earliest, most cited studies was performed by one of Robert Zajonc's PhD students who joined the faculty at NYU and is now at Yale -- John Bargh. However those foundational studies and subsequent ones by Bargh, as well as countless social psychology research that relies on them, were called into question about ten years ago when other labs found the results could not be replicated.^1 When one of these labs in Belgium published about the failure to replicate, Bargh went bananas. He attacked the investigator's paper but failed to address the issue by trying to replicate the original study himself. Daniel Kahnemman, whose popular science books which often rely on these studies, acknowledged the problem and called for more replication studies in social pscychology.^2
At one point we measured how search was doing, so we added a button to the top of the search results that said "did you find what you're looking for?". 70% clicked yes. Not great, not awful.
Then we upgraded the search engine, but didn't tell anyone. Suddenly that stat jumped to 90%+. But people would still complain just as much about how bad search is. Many months later, we finally announced that we had changed the search engine.
The stats on the button didn't change, but the public narrative did. So what people say they perceive and how they actually feel may not necessarily match.