The story I am going to tell pretty much nailed that point.
On 28 August Singapore Time at 9am, while running a routine check of our discovery service Summon, to sample test the linking reliability of a new content provider/database we just turned on in Summon, I noticed to my horror that the number of results I was getting in Summon was half essentially by half! (This seemed to be affecting other institutions on Summon as well )
For those not familiar with Summon, you can do a "blank" search - http://nus.summon.serialssolutions.com/search?s.q= and it will show all the records you have turned on. I do a routine check at least once a week, so on our case, I knew we should have roughly 330 million results if not more.
But on 28 August, we were showing about 169 million results half of what was expected. Based on my last recorded content type break-down, a lot of the missing content was journal articles so it wasn't just inconsequential newspaper articles. For example, some major economics journals were now missing.
A check shows that Summon just wasn't registering a lot of our holdings so it wasn't showing those articles as available online.
Panic modeMy first thought was, we are going to be in big trouble. It was week 3 of the term, the academic year was in full swing. Librarians were teaching classes, we just got over the bump where it was mostly searches for class readings and we were starting to get requests for advisories sessions for thesis, dissertations & assignments.
I knew there were at least 4 classes in my library alone, and I myself was scheduled to do one the very next day, a starter class to assist honours year students who were planning their thesis next term.
It was panic mode time or so I thought. I quickly informed colleagues I knew who were doing classes, warning them that their canned search might give different results. In particular, I was worried that librarians extorting the power of Summon by demoing a known article title search (a very popular strategy) would be embarrassed if they suddenly found no results.
I was also worried about users. Would they be angry? Disappointed the results were now so poor because of the relatively poverty of the index now? Would we get a lot more futile searches? More document delivery requests for items we have access?
Reaction was muted
It turns out, none of these came to past, except for one librarian who was doing a class in the morning (before I had time to warn her) noticing her known article title search did not bring out the link to the full text.
As far as I know, no user even noticed the index was halved. We did not get any complaints, reference transactions and document services seemed to be at normal levels.
Because I sent out a quick mass email to all our librarians, we would never know how many of our librarians would have noticed this on their own. I am guessing short of looking at their own canned searches they prepared before-hand or going to looking for articles they know are covered probably not.
But what did the aggregate statistics say about user behavior? We might expect number of visits not to be affected but search per visits should be higher as people try harder to find what they want.
This issue lasted from roughly 28 Aug, Wednesday and I noticed it was fixed at around 3pm 30 Aug Friday.
- Page per visits based on Google analytics rose from 3.93 (Tuesday) to 4.24 (Wednesday) and 3.97 (Thursday) and rose even further to 4.55 (Friday)
- Curiously Summon's own native statistics show a fall in searches per visit
How much does index size matter?
One would expect EDS to do substantially better than EBSCO traditional, being a newer product, with many enhancements and more coverage, from the same vendor. Yet, this did not happen. EBSCO ‘traditional’ in fact was preferred substantially more than EDS — 13 to 5, not a statistically significant level in part due to small sample size, but striking nonetheless."