1M Posts Search - You must see it !!

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
How long it takes your forum to search 1 Million posts (all with photos to display in page)?

If you think that you need $$$ for big brands, special software and plugins, just give a test here:
http://www.teriakis.com/antibb/index.php

Use any filters you want by enabling/disabling the buttons on the sidebar.

And no, I haven't did any magic. Simple jquery/php code and .....smart construction of search engine.

PS: Just a favor. It's still in testing mode so haven't added lot of security checks in inputs. I'll appreciate a fair use.
 
Last edited:

TrixieTang

Politically Incorrect
Joined
May 4, 2007
Messages
8,509
But how would it fare if it were searching actual user-generated content rather than placeholder content? How would it fare if actual users were searching that amount of threads constantly and concurrently?

It's a nice example of what you're striving to achieve with the search system, but in its current state it says very little about how the final product would actually perform in production use once completed.
 

Paul M

Limeade Addict
Joined
Jun 26, 2006
Messages
3,927
How long it takes your forum to search 1 Million posts (all with photos to display in page)?.
I dont know why a photo on every page would make any difference to a search ?

As to the question, I dont have 1 million, so a search of 2.5 million posts ;
Search took 0.20 seconds. Search: Keyword(s): kitten
(and apparently, I have 161 posts with the word kitten in them :))
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
But how would it fare if it were searching actual user-generated content rather than placeholder content? How would it fare if actual users were searching that amount of threads constantly and concurrently?

It's a nice example of what you're striving to achieve with the search system, but in its current state it says very little about how the final product would actually perform in production use once completed.
Maybe you haven't seen the full search form? Is not only the button filters (which you can let unused). There are also Search by Username and Search by Text (maybe you mean this by saying user-generated content). Also don't underestimate the value of the "Others" section. Most important of them are "Content with zero replies", "Show only content that I follow", "Show only content from Members that I follow".
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
I dont know why a photo on every page would make any difference to a search ?
Do you really believe it? Does not makes a difference? Displaying the search results in the page takes the same time when is just text comparing with the time that system needs to load and display a photo? Then why most forum scripts does not display even the user avatar? Search time is not only the time that the engine needs to filter the content but also the time to display the results on te page.

As to the question, I dont have 1 million, so a search of 2.5 million posts ;
No problem to prepare a database with 25M posts and not just 2.5M You know why? Because for a text (post) that vBulletin (and xenForo as I tested yesterday) are saving in database with 464 words, my smart code saves it with 150 words only having the EXACT SAME content.
Give a look here: http://www.teriakis.com/search.php It's just a screen with echo and print_r()
The first number is the amount of words of a vB saved in search table content. This amount goes down after each "clean" that I'm doing, to endup in 150 words. Now I'm trying to find a way to remove words which exist in both Singular and Plural (eg profile, profiles, user, users etc).

Unless if you think that content like:
Code:
[COLOR=#0000cd][B]
is important to be saved, or duplicate words to be saved etc etc.

So, yes 0.20 seconds is fast. But with my search engine should takes 0.05 seconds, be sure for it.
 
Last edited:

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
TrixieTang Paul M

But both you ignore the most important. That is a "Search as you go" solution. Let me use an example. With the current forum searching style you must set some search criteria, click search and then wait for the results. What happen when you'll see 10.000 results and you want to add more criteria to make the list shorter? You must go back to the form, fill it again, click again the button, wait again the results. Fine now you've 5.000 results only but still need more filters.

To make the long story short. For each filter that you want to add in current search solution you must go back to search form. Again and again. Here the solution is one click away, on the same screen.
 

markoroots

Participant
Joined
Nov 6, 2016
Messages
78
It's wonderful, I like it really more. You are a great!
I will vote you as Europe president. :)
Also eWHMCS is wonderful.
 

TrixieTang

Politically Incorrect
Joined
May 4, 2007
Messages
8,509
Do you really believe it? Does not makes a difference? Displaying the search results in the page takes the same time when is just text comparing with the time that system needs to load and display a photo? Then why most forum scripts does not display even the user avatar? Search time is not only the time that the engine needs to filter the content but also the time to display the results on te page.
Technically, they're two separate things. One is execution time (how long the search actually takes the server), the other is render time (how long the page content takes to display to the user).


Maybe you haven't seen the full search form? Is not only the button filters (which you can let unused). There are also Search by Username and Search by Text (maybe you mean this by saying user-generated content). Also don't underestimate the value of the "Others" section. Most important of them are "Content with zero replies", "Show only content that I follow", "Show only content from Members that I follow".
So, yes 0.20 seconds is fast. But with my search engine should takes 0.05 seconds, be sure for it.
Back when IPB 3 first released it had a feature that let you see a feed of all of a user's recent content in all apps - they had to remove it during the beta testing because the performance wasn't very good once it was tested in an actual production environment.

You don't know for sure what the bottlenecks will be until you actually have software that can be tested in an actual production environment with actual users generating content and searching.
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
Technically, they're two separate things. One is execution time (how long the search actually takes the server), the other is render time (how long the page content takes to display to the user).
This is what I meant. And also the end user does not cares for such details. For what he cares is how fast he/she will see the results on the page with the minimum effort. And back and forh to search form for sure is not what he likes.

Back when IPB 3 first released it had a feature that let you see a feed of all of a user's recent content in all apps - they had to remove it during the beta testing because the performance wasn't very good once it was tested in an actual production environment.

You don't know for sure what the bottlenecks will be until you actually have software that can be tested in an actual production environment with actual users generating content and searching.
First of all. As I'm clearly stated in my scripts thread, (actually is first on the NOT FOR list), I focus to average webmasters, all those who try with a minimal fee to have as more sections as they can. eg To buy a forum script, with additional: Articles, Classifieds, Reviews, Media Gallery etc etc, you need at least $250. For all those I'm coding this forum. The presentation of the search was just to show that it does not lacks of features.
 

mysiteguy

Migration Expert
Joined
Feb 20, 2007
Messages
3,079
I welcome additional competition in the forum world, and new ideas for optimization. We'll see how it does once it's released with real data sets, I'll give it a try on a test setup --- assuming I can import the data from one of my big boards.
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
We'll see how it does once it's released with real data sets
The same search engine with more complicated fields works since March 2016 in a Dating site with 20.000 members and at least 150 online users searching. And as you maybe know, search is A-Z for Dating sites. All members are spending their time on that section.

assuming I can import the data from one of my big boards.
Sorry no. Is not in my plans to code any importer. Never did in my life even for my vB addons. I know that this out of logic, but sorry this is the way that I used to do.
But of course I'm not going to put any border to anyone who may wants to code such importer(s).
 

Attachments

Paul M

Limeade Addict
Joined
Jun 26, 2006
Messages
3,927
Do you really believe it? Does not makes a difference?
Yes.

Displaying the search results in the page takes the same time
Yes, but thats displaying the results, which is not relevant to how long the actual search takes.
Displaying 10 results would take (add) the same time, regardless of how long it took to find the results.

So, yes 0.20 seconds is fast. But with my search engine should takes 0.05 seconds, be sure for it.
Thats certainly a nice improvement on paper.
I hope it works as well for you with real life data and set-ups.

However, a further note of caution, dont get hung up on such small numbers.
In practical terms you are not really going to notice the difference between those two figures when performing a search.
You still have to wait for the results to be sent over the network from the server to you, and then rendered locally by your browser.
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
However, a further note of caution, dont get hung up on such small numbers.
In practical terms you are not really going to notice the difference between those two figures when performing a search.
You still have to wait for the results to be sent over the network from the server to you, and then rendered locally by your browser.
I know it, and I totally agree. As for the others:
  1. I did this post just because I was happy from what I seen. I was expected to be somehow fast, as I did some tricks in search table, but not so much.
  2. The same search engine works in a Dating sites since March 2016 without any problem to support 100+ simultaneus users searching.
  3. But again, as I wrote above, I focus to small sites. As you're also visiting many sites for supporting your mods, most probably you noticed that there are some sites having more categories than threads. I want to give them a choice for something better.
  4. For me mostly counts that I'll be able to release something that I was dreaming years ago, than how many sales I'll have. That's the reason that I'll not release any importer for any forum script (Free or Commercial).
  5. And finally, talking frankly, don't know how long life I've ahead, so no more plans, than being able to release it.
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
I welcome additional competition in the forum world, and new ideas for optimization. We'll see how it does once it's released with real data sets, I'll give it a try on a test setup --- assuming I can import the data from one of my big boards.
Forget real data sets. Wait to see how this will run when people will install this thing on shared providers with their weird configs.
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
Forget real data sets. Wait to see how this will run when people will install this thing on shared providers with their weird configs.
Already runs in a GoDaddy's (worst provider in my opinion) shared account, and yes, still works that "thing". You know why? Because is soooo smart the search engine my dear.

It's amazing how people love to drop the wings of anything coming from "anonymous" people. But those who know me, know how much stubborn I'm and how perfectly I can turn the kicks to boomerang.
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
It's amazing how people love to drop the wings of anything coming from "anonymous" people. But those who know me, know how much stubborn I'm and how perfectly I can turn the kicks to boomerang.
So far this reads like big ego trip on your side with no code or real datasets for us to see how sustainable your claims really are. Hence plenty of us are sceptic to your boasts of being really smart with search engine.

Thanks but graying lads aint my type ;)
 

ChrisTERiS

Neophyte
Joined
Jan 16, 2015
Messages
0
rafalp

After posting my reply before I noticed that you're a Forum developer with your own forum. I wish you the best in your business, but sorry I avoid to start endless convertations with competitors because it's normal that everybody will supports his own "thing".

And yes, I've a big ego. If you've got your knowledge just by reading books as there were no money for studies, if you were speaking English just by trying to compare the subtitles in movies with the sound, again due to lack of money, be sure that even you (any everybody else) should had ego. But this ego is not the bad one. Better use "Proud".
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
After posting my reply before I noticed that you're a Forum developer with your own forum. I wish you the best in your business, but sorry I avoid to start endless convertations with competitors because it's normal that everybody will supports his own "thing".
Don't be one of folk who play "you are competitor!" card, especially to guy who does his project 100% open source and without any commercial intent.

Its very low way to dismiss other.

If you've got your knowledge just by reading books as there were no money for studies, if you were speaking English just by trying to compare the subtitles in movies with the sound
Bringing up life experience is ok, but this experience should also teach you that you aren't the only one in the world who self learned language or craft.
 
Top