Algorithm / Formula for Question Effectiveness

Forums: Science And Math
Email this Topic • Print this Page

Thought I would pose this, just because so many users on this board demonstrate very logical thinking - which is what this question ultimately requires.

We have a series of web-based Q&A's that are posted on our website, to allow employees conduct self-service searches for FAQ and answers.

We know how frequently, over any period of time, those questions are "hit", and are assumed to be read by our employees. The questions also have a 3-level rating scale (Helpful, Somewhat Helpful, Not Helpful), that employees will sometimes use to provide us with feedback on the questions - but not very consistently (unfortunately).

We're trying to ascertain a formula/algorithm to gauge the effectiveness of the Q&A's, using the "hits" frequency and the limited ratings we receive, in order to calculate a composite Q&A score or index value, that will indicate which questions are effective or ineffective, and which may benefit from revision or simply deletion - and where (approximately) is that dividing line, from a standpoint of (possibly) statistical significance?

We're trying to wrestle with how to account for the influence of questions with low frequency hits, and no ratings, or high frequency hits with both Helpful and Not Helpful ratings, or high frequency and some Not Helpful ratings (a highly popular, ineffective question?). And should the ratings be weighted as to time since the Q&A was originally posted - since the number hits is higher for new questions, versus Q&A's that are "older".

Any thoughts on this would be greatly appreciated.

Topic Stats
Top Replies
Link to this Topic

Type: Discussion • Score: 1 • Views: 743 • Replies: 6

No top replies

Assuming that a selection is made based solely on a posed question (need to select to see the answer), I'd say:

1. low frequency implies unnecessary (either they don't need to know, or they already know) - delete
2. high frequency implies necessary - keep
2.a. helpful rating implies effective - leave alone
2.b. not helpful rating implies ineffective - revise
2.c. mixed rating implies effective for some, not for others - supplement

I'd factor in time particularly with revised/supplemented answers.

0 Replies

Okay, some good initial feedback on this, and some of which we had already reasoned out for ourselves.

Unfortunately, quantifying this in some sort of algorithm or formula, that results in a question score or index value, is maybe not as easy as what markr suggests.

How should we weight the relative influence of questions that have a high frequency of hits, but receive a small, but predominant number of ratings that indicate they are Not Helpful? So, if a question gets 3,000 hits in a month, and receives 3 ratings of Not Helpful - the question should be either revised or deleted? My intuition tells me, probably not.

That's some of the dilemma. Any other thoughts on some of this given markr's initial feedback and some of the potential incongruity arises from what I mention above?

Thanks

0 Replies

CDobyns wrote:

How should we weight the relative influence of questions that have a high frequency of hits, but receive a small, but predominant number of ratings that indicate they are Not Helpful? So, if a question gets 3,000 hits in a month, and receives 3 ratings of Not Helpful - the question should be either revised or deleted? My intuition tells me, probably not.

Absolutely not, and that's not what I suggested. I would put this in category 2 (not 1, or 2b). I probably should have added:
2.d. little or no feedback, assume effective - leave alone

You're right, I didn't quantify. It's probably tough to do without seeing your data. Seems to me, to impement my sugesstions, you just need to determine some thresholds:
- low frequency threshold (when to delete)
- not helpful threshold (when to revise or supplement)
- helpful threshold when not helpful threshold is exceeded (to decide whether to revise or supplement)

I'd make the (not) helpful thresholds a percentage of hits, not a percentage of feedback. So, for your example, 0.1% not helpful ratings is not something I'd act on.

0 Replies

C-like algorithm
Less than and greater than symbols cause problems in the post; so I spelled them out.

Code:

Delete_Threshold = <some_constant>

Not_Helpful_Threshold = <some_constant>

Helpful_Threshold = <some_constant>



Not_Helpful_Rate = Not_Helpful_Count / Hits

Helpful_Rate = Helpful_Count / Hits



if (Hits less than Delete_Threshold) {

   Delete

}

else {

   if (Not_Helpful_Rate greater than Not_Helpful_Threshold) {

      if (Helpful_Rate greater than Helpful_Threshold) {

         Supplement

      }

      else {

         Revise

      }

   }

}

Revise is a wholesale revision.
Supplement is adding material to make the FAQ more helpful.

0 Replies

IMO, you have a few different things that need to be looked at seperately. Using "Hits" in your formula seems pretty useless to me. I think you are making a bad assumption in there by thinking that a "hit" means that the article was always applicable to their issue.

Does the page also allow the user to indicate if the page is applicable or not?

Generally a FAQ will list a question with an answer. But the specific question may come up in a search (i.e. be a "hit") and not be applicable to what the user was looking for. If they don't have the option of IDing the search result as "Not Applicable" to their problem they may be marking it as "Not Helpful" even though the answer answers the retreived question perfectly.

So you need to be able to tell is the FAQ is "Not Helpful" (or "Helpful") because it contains outdated info or if it just wasn't applicable to begin with. The first indicates a need for revision of the FAQ, the second indicates a need for better search optimization.

After that I'd look at the percentage of responses that were "Applicable" AND "Not helpful". If you have a high number there it would be an indicator that that section of the FAQ needs revision.

Your results with lots of hits and low (or negative) responses on helpfulness are probably items that actually aren't applicable to the original problem.

0 Replies

Okay, more thanks goes out to markr for his feedback and nifty conditional IF/ELSE decision tree illustration. That was helpful and got us thinking about what exactly those value constants are that we might want to consider that represent the "thresholds" piece of the puzzle.

For fishin, I can tell you there is no input that allows the user to indicate Applicability. On the other hand the we think the "hit" frequency probably is a pretty good indicator (of something). The reason is that our FAQs are actually set-up with a preliminary topic/subject line, which is usually 12-15 words, and which we think provides enough initial information to the user that allows them to make an informed selection with a high probability of the question and answer to provide the information specific to the user. So the hit frequency actually gets counted only after the user selects the FAQs after a preliminary review of the subject/topic.

Would welcome any additional input from the collective audience, and thanks.

0 Replies

Algorithm / Formula for Question Effectiveness

Related Topics

Quick Links

My Account

able2know