Whoever did it put it back the way it was!

The way it is setup now, it doesn’t work?! This new search functionality needs to be

di652 (BOB member since 2005-02-24)

I haven’t seen a change in functionality. What do you think was changed?

BOB has always ignored “stop words” – now the search screen tells you which words were ignored. Is that what you mean?

Anita Craig (BOB member since 2002-06-17)

If you can provide a specific issue, it can be discussed and perhaps addressed. Thanks.

Dave Rathbun (BOB member since 2002-06-06)

Thought I’d post this here rather than opening a new topic for it, but I know I’ve mentioned this before.

The problem with “stop words” is that they can make a search return too many irrelevant results. For instance, I now did a search on “object security” but the word “object” was removed, thus giving me some 6450 results for “security”. When I’m looking explicitly for “object security”, that result was unfortunately pretty useless.

I worked round the issue by going to Google and giving the following instruction:

site:forumtopics.com "object security"

and got some useful information that way. As forumtopics.com appears to only be hosting 2 fora at this time (this one and a digital photography one) where there is little to no conflict of terminology, that Google search works just fine.

Angoid (BOB member since 2004-11-30)

The intent of stop words is to prevent a server overload. I’ve written this up before, but I’ll repeat a summary here. The way the phpbb search works is that it retrieves a list of posts for each word separately. There are over 27K posts with the word “object” in them. There are over 9K posts with the word security. There are only 685 posts that use both words. But in order to determine that, a lot of RAM is dedicated to just that one query.

As an alternative, if you’re looking for object security you should limit your search to the Designer forum (if using XI) or the Supervisor forum (if you’re using a prior version). The word “security” in the context of the Designer forum has a different meaning than the word “security” used in the CMC forum, for example.

When I search the Designer forum for the word “security” I get only 682 topic match results.

Searching is one of the primary functions of this board. There are two main alternatives being considered at this time. First is to create an add-on that will provide an exact phrase search that sits on top of the existing search process. What I mean by this is the standard search will run first (including the stop words logic). After the set of topic (or post) results has been completed a second phase search will run on the set of search results using mysql functions directly, in order to look for the exact phrase. If this were in place the 682 search results from the Designer forum would be reduced to 23 topic matches. Even though the word “object” would not be used in the keyword search the full phrase “object security” would be used in the second phase of the search.

The second alternative being considered is upgrading from phpbb2 to phpbb3. By default phpbb3 uses the same search engine as phpbb2, but there is a project that is using the SPHINX search engine with phpBB.

Neither of these alternatives is on the roadmap with a specific date at this time.

Dave Rathbun (BOB member since 2002-06-06)

Would it be possible to make the full-phrase search take priority over the individual words search?

What I mean is, if you say “object security” then that would be treated as one word - ‘%object security%’ and not ‘%object%’ followed by ‘%security%’ (but ‘%object%’ would be removed because of the stopwords situation).

Doing it the other way about (as suggested in Dave’s post) would mean 6450 results still coming back (assuming a full forum search) but having to look at some indeterminate point near the end to find the most relevant results if the full-phrase search result is done (and placed) after the individual components search.

Angoid (BOB member since 2004-11-30)

I don’t think I explained it very well. The phrase search would be performed on the word search match results, and only those posts that included both words and the exact phrase would be returned. The word search would be used to return a list of candidate posts, and then a second search pass would be done on those posts.

Only those posts with the exact phrase would be shown to you in the end.

Of course that’s just the design; the implementation is another story.

Dave Rathbun (BOB member since 2002-06-06)

The bottom line is that this board is run on open-source software, and is using the search engine that comes with that software. Please note what Dave said:

Bobarella (BOB member since 2004-02-12)

Someone has managed to get sphinx to work on a base phpBB2 board. I am hoping to get details on their implementation so we can test the same.

Bob (BOB member since 2002-06-06)

If you think you have encoutered a bug in the search, please post the process you used. Include the search terms, the forum you searched, and the results that you got. If there is a bug in the search, then it needs to be addressed. Thanks.

Dave Rathbun (BOB member since 2002-06-06)

Again, my only comment is to put the search engine back the way it used to be.

As seen we don’t want or have time to look through 6450 message that the search brings back. This is a major turn-off!

The way it used to be was simple but very effective.

di652 (BOB member since 2005-02-24)

Again, can you be more specific about your issue? The search functionality hasn’t changed. The results may be higher due to the fact there is more content on BOB than there used to be. But it you can provide specifics, we can be of more assistance. What search did you do? What results did you get? What results did you expect to get?

Bobinator (BOB member since 2008-09-09)

Unfortunately that’s not enough to go on. The core search process (indexing and searching by unique words) is the same.

Dave Rathbun (BOB member since 2002-06-06)

I finally found a specific example, which is what I have been asking for all along. I was searching for the trick of using a rank as a filter on a report. I entered the words “rank as filter” fully knowing that the word “as” would be removed as it was too short. However, it was not removed, and in fact I got zero search results. When I searched for the words “rank filter” I got results.

So there is a bug of some sort in the current search.

In the future, a specific example like this (instead of just complaining that “it doesn’t work”) would have gone a long way towards supporting the complaint. I will look at the code and see what’s causing this specific issue and fix it.

Dave Rathbun (BOB member since 2002-06-06)

Another major issue that I have noticed is that the way you enter the search.

For instance if you just enter Reports - You get NO results.

This is wrong because I know for a fact that there are questions with this word!?

I agree, this search used to work great without issue at one time

toscajo (BOB member since 2002-09-04)

The word “report” and “reports” have long been on the stop-words list (a list of words that are not indexed).

Dave Rathbun (BOB member since 2002-06-06)

(I was about to start a new post, but found this one which is of relevance)

When I want to search for something, I go to the search link, typing in words I would like searched on BOB and yet nothing comes up (i read about the stop words, not sure if this is related), so then I copy and paste the same words onto the Google search box at the bottom of the screen, and low and behold, many BOB links are made available about the topic.

I keep the search as generic as possible (search all forums, and for all words), for an example, try putting in “Group By” (without the quotes) into the search box (as in the SQL Group By), and leave all the default values as they are (i.e. search all the words terms, all forums) and you won’t get anything back, try it on the Google search box and links to posts show up.

So… am I doing something wrong ?.

Thanks in advance,
Veronica

Veronica (BOB member since 2002-11-22)

Veronica,

You’re doing it right. But there’s a few things to understand about the BOB search feature.

Words of 3 letters or less are always excluded from the search.
We do not have the capability to search on phrases.
“Group” is a stop word, as it appears in far too many posts to index.

So, looking at your search, “Group by”, as a phrase, cannot be searched. So the individual words must be searched. Since “Group” is a stop word, and “By” is only 2 letters, neither can be searched. That’s why you received no results. Google, not surprisingly, has much more advanced search technology that we do. So, in this case, the Google search is a much better option.

MichaelWelter (BOB member since 2002-08-08)

As has been stated before, reviewing the search process for this board is the top project on the list right now. It’s just a question of getting the time to complete the project.

Dave Rathbun (BOB member since 2002-06-06)

Thank you greatly for the detailed response Michael, at least I now know I’m not misusing the search function !, and thanks for the info Dave.

Veronica (BOB member since 2002-11-22)