Forums

Unfortunately no one can be told what FluxBB is - you have to see it for yourself.

You are not logged in.

#1 2019-05-21 13:57:12

JJones
Banned
Registered: 2019-04-28
Posts: 63

27x Speed improvements ( my edits )

These are a list of changes that i have made that have improved speeds by 27x ( my own bench-marks )

#1 Zend_cache was added / and the fluxbb_write_cache_file function was removed completely I only Incorporated Zend Cache, not the entire Zend_Framework which includes a multi-type cache system ( "memcache", "file cahce", "sql cache", AND "page cache" )... Meme / file / sql are operated as a switch function in which you basically select which option you want ( or completely off ), (file) is default ... keep in mind that "file cache" is basically serialized JSON data in a group/individual/series of groups ( explained reasons later ).

#2 Cats, Forums, Topics & Posts are 99% driven by the cache system to eliminate the need for multiple SQL queries. These "caches" are used in a method that basically "builds itself" in a giant array, however is displayed based on the column structure of the page/forum/topic/post/posts selected.

#3 User Data is stored in its own single cache .... which eliminates the need for 75% "union queries".... AND removes any dependency on "username" as well as any backend "cron tasks" needed to clean the system or additional queries just because someone changes their profile information ( again this basically makes "username" redundant in the core ). This functionality was specifically designed for the purpose of removing any need for additional queries to each topic/post when user information is edited, thereby forcing additional work of having to update all of the other SQL Tables.

#4 Template system was completely removed .... It does STILL check for a tpl file, however defaults to pure CSS... this drastically cut the page generation times due to the overused print( & sprint functions .... This also forces FluxBB to become "mobile friendly" and HTML5 compliant.

#5 File Reduction sizes & quantities ... the edits actually resulted in smaller files due to no longer needed very OLD and SLOW functions. Example: viewforum.php is now a total of 64 lines of code :P Less Function calls = faster page generation.

#6 Unlimited User Groups added ... Much like the rest of the popular forums, Groups AND Group Permissions added to allow each forum to be accessible as one or multiple of the following options:
1 - Unregistered Public Access
2 - Registered Member Access
3 - Group Member Access
4 - Individually Determined Access ( Admin / grants the user privileges )
With each of these Access Rights, a sub right is appended as ( view, post, edit, delete, add-attachments ). Utilizing Bitwise operators to allow the most options from both Permissions & Privileges. With that said, 3 DEFAULT unchangeable groups are "Global Administrator ( the webmaster ), Administrator, Moderator, & GUEST". The reason for these specific groups are to ensure that they can not be deleted, nor can the Global Administrator be removed or modified by other groups ( sort of like a command structure ).

#7 Unlimited User Social ... Allows a single user account to have UNLIMITED Email, Facebook, Twitter, Google, Steam, LinkedIn, Discord Blah blah ( See: https://github.com/hybridauth/hybridaut … c/Provider ) for complete list ... Unlimited Phone numbers will be added sometime in the near future.

#8 VPN & ISP Detection Added ... Most free services like IPHub limit 1000 queries per day. To get around these limits, we are using multiple services in a que format to extend the limit of "free" access. Currently we have a default of 3 Services, which allows a total of 3,000 queries per day. It is also important to note that these validation checks are ONLY done upon the first initial page view, and extend for the life of the $_SESSION. This information will allow for improved user detection, as well as improved security from unwanted social behaviors, AND the ability to allow/disallow proxy services on your website. Upon reaching the limit the function will resort back to default IP detection, but will not include VPN / ISP Detection.

#9 Ckeditor 4 Added ( but will be migrating to version 5 ) ...ALL BBcode was removed! Embedding a pure inline WYSIWYG ( What You See Is What You Get ) editor that is utilized not only in form posting, and editing, but ALSO for creating additional alternate pages in the portal ( for administration only ). The Editor is provided through a CDN while the configuration file is stored locally. The CDN is ONLY loaded if the user has permission to post / edit / that page, otherwise it simply does not load like it never even existed. With the implementation of CKeditor, there is no longer a need for the massive content parser, so it was removed! However, the header still cleans all $_POST data to protect against malicious SQL injections.

[center]Future Edits on my "To-Do-List":[/center]

1# SQL database will be "minified" & re-defined to a more" up to date versions of SQL, since a lot of the fields and columns are no longer needed. Example: ( username ) is now redundant through out the entire system, as well as timestamps will be converted to correct timestamp formats.

2# DB Layers will be re-written to drastically improve SQL queries as well as introduce its own internal cache during run-time functionality ( which seems to be what Franz is complaining is the problem of speed with FluxBB but can't seem to understand that the core must be updated first to get to this step. ( I will only be doing this for Mysqli & PDO ... someone else can do PostgreSQL if they want.

Also adding fetch_rowset( in the DB layer so that the insert ID of any table will automatically be the first available number in a sequence ( if you have 500 topics and you delete topic #55 then the next creation of a topic id will be ID 55.

#3 Topic titles, Subjects, and individual posts ( text ) will be auto converted to fill the "key words" meta tag for improved spiderbot indexing ( some would call this better SEO practices, but i just call it better indexing ).... the same tool used to create the keywords meta is the same function that generates the core files "Search System" thereby removing the need to run "maintenance mode" as it is done automatically, and without any additional SQL queries or cron tasks. ( same concept as current, but done in a way to add additional functionality and remove the current work from the server )

#4 Default mobile functionality embedding .... already through the use of PHP the category list uses a collapse function, however, to make the topic view a lot easier to navigate without having to reload the page with the new pageid, Jquery can be switched on to convert the existing pagination to "swipe screen".

#5 CKEditor Administrative control functions ... Allows the Forum permissions to be set to determine if a user can utilize specific features ( attach files, smiles, templates, blah blah blah ). By Default Administrators have full access to all features.

#6 CKEditor Theme ... Allows theme creators full control over the appearance of the CKeditor, through the same template system.

#7 Template Engine ... Because I am at that point where I need to convert the layout to operate 100% through TPL to allow designers the most access to the data, and allow them to place ANY variable ANYWHERE, I am forced to decide on a Template Engine ( I am currently thinking of a merged system between ( TIG & SMARTY ) attempting to leverage the PROS from both against the CONS of both.

#8 Language System Merged & Converted to definitions... Language variables will be made available through the (global) $_LANG[_KEY] variable instead of multiple $lang_topic, $common_lang variables. ALL language variables throughout the entire Forums will be incorporated into a single php definitions file with the title of that file to use ISO 639-1 standard ( en-us, en-gb en-ca, blah blah ). I will ONLY be providing a few language translations. Someone else can provide additional if they wish. Future Note: The portal system will use 1 language file per module/plugin ( Forums operate as a single module/plugin )

#9 IP Logging The registration_ip will be modified to record the last (5) IPs used to access the account. This is more of an account security system that notifies the user IF their account is logged in through an IP that was not previously used ( Identical to how Google Accounts are secured ).

I do NOT use or support github, so I will not be providing any links to a git repo for link sharing! If you want a copy of the files, and edits, you will have to contact me directly and make a request.

Converting the Forum to operate 1 level down to allow the forum to operate as a portal. This basically means that the "Forums / Bulletin Board" simply operated beside multiple other pages / scripts / & whatever ..

Last edited by JJones (2019-09-12 13:27:30)

Offline

#2 2019-05-22 03:17:00

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Reserved Post for Additional Space

Last edited by JJones (2019-09-12 11:47:35)

Offline

#3 2019-05-28 07:26:33

n3c
New member
Registered: 2019-05-28
Posts: 1

Re: 27x Speed improvements ( my edits )

Hello mr JJones, is it possible to obtain that code from you?

Offline

#4 2019-05-28 09:31:22

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

I apologize, I assumed this forum had a PM systems ..... I have no problem sharing files, how would you propose having them sent to you?

Offline

#5 2019-05-28 18:06:19

quy
Administrator
From: California
Registered: 2008-05-09
Posts: 926

Re: 27x Speed improvements ( my edits )

Maybe you can post as a mod here: https://fluxbb.org/resources/mods/

Offline

#6 2019-05-29 05:28:59

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

its not a mod ... its a complete ( and ongoing ) core re-write

Offline

#7 2019-05-30 05:08:21

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Update: Removing all "Echo" statements from the core, which is being converted to a php variable ( $html ) .... which is only "echoed" from the footer.php file IF the header stipulates to do so!

The purpose of this conversion is in 2 parts:

1) Compression & Buffering is being converted to use what the browser wants ( zip, deflate ( common for chrome ) ) instead of what is defined from the config ...

2) Etag embedding so that the pages no longer need to be served if the content has not changed since last view.

This result allows a user to spam the living crap out of page refresh and all that will happen is a header 304 response code which loads from their browser cache instead of making the server work to server data again...

Offline

#8 2019-09-10 14:29:51

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Original Post Updated .... One day i will get better screenshots added. To be clear, I am converting the system to operate as a "Portal" with multiple pages/scripts, OR as a Stand Alone Forum System.

Last edited by JJones (2019-09-10 14:30:53)

Offline

#9 2019-09-10 20:03:45

LittleSheep
Member
Registered: 2014-04-07
Posts: 30

Re: 27x Speed improvements ( my edits )

Where can i download this release ? tongue

Offline

#10 2019-09-10 20:17:40

GWR
Member
From: Germany
Registered: 2010-08-06
Posts: 214

Re: 27x Speed improvements ( my edits )

Also adding fetch_rowset( in the DB layer so that the insert ID of any table will automatically be the first available number in a sequence ( if you have 500 topics and you delete topic #55 then the next creation of a topic id will be ID 55.

So you plan to reuse IDs? Without proper quotation support this means you will sooner or later have links to the wrong threads or posts ({post} or {topic} tags).
You could avoid that by cleaning up all existing posts from pruned/deleted references.

With the right number type you should be able to give all posts ever done in your forum a unique ID without reaching an overflow.


@ your proposed changes
It is nice to write what you have done - but if there were works in a "modular" way then devs might cherry pick some of your suggestions, improvements, overhauls, ...


bye
Ron

Offline

#11 2019-09-11 04:07:41

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Cherry-picking are GREAT! I thoroughly enjoy debating over functions ... which method creates the fastest return, or creates the greatest amount of dynamic return. This is a standard method of improvement.

Primary Key ID resorting in SQL is easily done with an insert method to combat "gaps"... And yes it was one of the first function appends done to solve any issues of Forced Delete Accounts. ( Albiet we are not in the habit of deleting USER ACCOUNTS, simply DISABLING them - which avoids a lot of future headaches ).

The USER accounts are independent of the posts / topics / ... in fact the forums are basically a giant JSON file, that operates in reverse order where the user info is "merged" onto the topic / record ... There is no way to confuse the "IDs", unless the user account data is missing for some reason ( then it simply displays DELETED ) ... in terms of looping through the topics / posts, it simply does not matter if there are gaps in IDs, check out the foreach( function in PHP ... it has no issues.

No such thing as "overflow" .... no such things as "limits" ...

Define "modular" way?

Offline

#12 2019-09-11 05:43:10

Franz
Lead developer
From: Germany
Registered: 2008-05-13
Posts: 6,675
Website

Re: 27x Speed improvements ( my edits )

I think the problem @GWR describes is that FluxBB supports links to specific posts or topics with the "topic" and "post" tags:

[topic=123]Take me to the blog post[/topic]
[post=456]You said something here[/post]

When you start to re-use IDs (as you seem to be doing?) those links - which should be broken if the topic or post has been deleted - will silently start to link somewhere else.


fluxbb.de | develoPHP

"As code is more often read than written it's really important to write clean code."

Offline

#13 2019-09-11 06:42:36

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Franz wrote:

I think the problem @GWR describes is that FluxBB supports links to specific posts or topics with the "topic" and "post" tags:

[topic=123]Take me to the blog post[/topic]
[post=456]You said something here[/post]

When you start to re-use IDs (as you seem to be doing?) those links - which should be broken if the topic or post has been deleted - will silently start to link somewhere else.

The Json cache array works in reverse order from post -> topic -> sub-forum -> forum ( ignore the subforum for the moment as i have not yet made a decision as to a layout for display during forum display, but did finalize menu display )

each 'post' is an array .... which also contains data from the assigned topic, sub-forum, forum IDs ... the direct link only needs 1 of any ID, however 'p_id' is required to auto focus directly on that specific post ( i am using anchor tags with a JS trigger onpageload to force the browser to focus on that anchor )

As the other assigned ids are stored with the post in the json array, only 1 ID is required .... however the assigned type of id is needed and obligated to exist in the array or it defaults to the top level.

Im not sure how else to explain, other than the SQL query generates ALL forums / sub-forums / topics / posts in a single instance to generate a json cache ( 1 time, and is updated on injection if edited, appended, or modified ).... the individual page display simply uses array_column( to pull exclusively what subforum, topic, post is being requested. In a nut shell, all the forums do is literally display the data associated with the requested ID:type from a very large single 5 dimensional array. Also important to note that the "order" of the array is changed in live run-time IF the user wants the order of each topic different from the default forum configuration which is ( ANNOUNCEMENT > STICKY > LAST_POST_DATE_TIME )

The only POSSIBLE issue is any indexing from spider crawlers, which is repaired on their next visit to the site to obtain updated info. Theoretically that would only be an issue for the length of time it requires the index bot to return ( GoogleBot claims 24 hours, but the prick seems to be on my website every 15 minutes ).

NOW... if you mean a problem with quotations inside of specific posts from one user to another, We use manual moderation to modify replied texts if the text was quoted from a user that was deactivated for some reason and that quote violated a terms of service policy. I did this specifically to ensure that "context" of what is stated is the final basis for a need for moderation.

Was any of that unclear? My edits make "static IDs" redundant with exception to spider bots for indexing ( which is only temporary 24 hours at most ). And we use robots.txt that actually does list ALL Forums/SubForums/Topics/Posts even if those specific forums are hidden from anyone outside of the permission group of access. ( the bot DOES index hidden links, & titles, just not the content )

PPS: This reverse ordering allows a better "keyword" assignment to the header ( sort of like tags ) that are appended to he header keywords, however i have a lot of work to do with multiple translations

( Our forums will be using 1 forum for multiple languages, meaning that English / Vietnamese / German / Russian will all be on the same forum. However, i am still attempting to solve a lot of issues that deal with multi-lingual support ( you would be surprised at how much bullshit is needed to prep a website for content that can be in any language dynamically )

Image Screenshot of the Json array in "pretty print" format to explain the structuring ...
Screenshot-239.png

Last edited by JJones (2019-09-11 14:44:59)

Offline

#14 2019-09-11 07:13:05

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

CKeditor Added to the original post .... eventually resulting in removal of all BBCode Parsing files.

Last edited by JJones (2019-09-11 07:13:45)

Offline

#15 2019-09-11 20:57:09

GWR
Member
From: Germany
Registered: 2010-08-06
Posts: 214

Re: 27x Speed improvements ( my edits )

So to get you right:
- no need for "post.id" as each post contains "data id"
- when building a thread-data-table/container you fetch everything using "data id" (threads, posts ...)
?

Somewhere a connection between "topic" and "post" must be done (and "topic"+"forum" etc.). If you call this "ID", "rowid", ... does not matter. And this "ID" could be reused as there is no need to directly reference it from the "user side"?

This is what I do not understand. Franz already pointed out what I was talking about: the direct referencing of topics/threads and postings via some bbcode or raw web links.
How do you identify threads in urls - via some canonical url?

As soon as you have some static ID somewhere this is what cannot be reused without limiting the way to "link to it easily.
How do you tackle that?


BTW: the manual moderation of quotations of a deleted user can become ... a pretty big job (in other forums I already have 2k posts, in mine over 11k. Others have some thousands too. Now ban one of them and moderate 1000+ posts?). Dunno how this is handled in these forums with "@"support and dynamic quotation information.


PS: How big are your json cache files? Might have an optimizeable memory footprint (only read what is needed).


bye
Ron

Offline

#16 2019-09-12 07:54:39

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

I thought i answered this:

array(ID_TYPE => ID) is how the data is passed ... ID = any number passed! ID_TYPE is passed to determine what the ID is being pointed at ( Forum / Sub-Forum / Post / User / Plugin ( better known as alternative pages ) / and a few other mentions. ID => post_id / or / user_id / or / forum_d / or / sub_forum_id / or / plugin_id

it is NEEDED... its just not "static"

It should be noted that the linking process can take place in ANY the following ways:

$_REQUEST ( passed by browser )
$_POST ( passed by <form> )
$_GET ( passed by browser address )
$_SESSION ( passed by session )
$_COOKIE[temp_cookie] ( passed by cookie / not in the user $_COOKIE[user_cookie] )

a "link" can be passed through any one of those formats! ( $_SESSION, $_GET ) can be used locally, ($_REQUEST & $_POST ) can be used locally, OR remotely, ( $_SESSION & $_COOKIE ) will be mostly used as a last "successful" view of a page for a better history check.

Your difficulty in comprehending this is that you seem to have the impression that my edits are designed to be identical to FluxBB in format, which they are not ... they are designed to be used as a forum with a portal, and based purely on as much native PHP ( No Frameworks allowed, with exception to design / javascript ). Think of it more like the same "motto" of FluxBB but with massively updated functionality ( i constantly bench-mark each function trying to find "faster" solutions ) but based on a PORTAL system as opposed to just a stand alone forum.

The reason for "most" of this is an eventual back-end API for remote posting ( basically a BOT reader that gets content from other websites, and/or services ) AND because i have the full intention of removing the following files ( viewforum.php, viewtopic.php, post.php, edit.php, moderate.php ) I believe i have stated that this modification is specifically for an Expat Community, so i am building to suit their needs/requests. As such, we get a lot of data from embassies, state.gov, ministry of foreign affairs, and some news outlet sources. ( I am nearly sure that people want data feeds from other locations ( but not my job to decide on what is polled, and what is not ). It should also be noted that we take the SMS warning system directly from US, UK, AU, CA, & RU Embassies in Vietnam and post it directly to a forum post / as well as forward SMS to all users within the "expat" group.

userlist.php & profile.php => are being converted to a "module/plugin" ( in the future ) so that it can be used in any other alternative page. ( The reason for this is to compact URLs to either the shortest i can get them, or to non-existent ...meaning index.php will include_once( as stated in the original post, this modification is intended to make fluxBB into a "portal" and/or "stand alone forum".

The json file size depends on the number of ( forums / sub-forums / topics / posts ) but it should also be noted that there is not just 1 json file ... the 1 json file that handles ALL forum data ( does not include the user data with exception to the user_id ) and is bench-marked at 30,000 posts, and 1,000 categories ( topics / forums / sub-forums ) are nothing more than internal flags for separation and are just flat out meaningless due to the loop functionality.

the 1 users json file is bench-marked at 50,000 users and is simply merged to fill in the user_id info on each profile / post or any other page information that needs to display the user details. Its size depends on the amount of data each user account has ( because we allow multiple social information & contact information, this data is not going to be consistent for each user ( 1 person might have 1 email associated, while another might have 5 emails associated, 1 user might have 0 phone numbers, while most of the domestic users will have at least 2 mobile phone numbers (ironic that VN is listed as a poverty nation by over 3/4 of the population has multiple phones) ).

It is also important to note that these "json" caches are literally part of Zend_Cache: https://github.com/zendframework/zend-cache ( know of anything FASTER or MORE OPTIMIZED? )

As for moderation: Banning a User triggers moderation of all associated posts associated with that single person? WHY would you delete/edit/move all posts from a single banned user? Were every post the user ever made a violation of your terms of service? or did you want to delete all association with said person? It seems more like your policy causes more work for yourself & your staff. What about the content that held value in communication prior to whatever caused the ban? Imagine if I violated the terms of service in these forums, and FluxBB admins deleted all of my posts!

Our Expat Community holds a bit over 20k user records ... but i would venture to guess that 14k to 15k are active ( activity in a 7 day period ) visitors... but we do have about 30 staff members. We get a lot of "random traffic" but most can't/won't pass the validity check and are unable to fully participate with the community ( we don't allow "anonymous" users in our community due to the amount of fraud that exists in South East Asia and we DO participate with the OFAC ( US SANCTION REGISTRY ) and the UC FPI ( EU SANCTION REGISTRY ) and i would venture to guess that MOST of our users are domestic nationals attempting to sell bullshit to Expats. In the future, I would like to include support for Korean & Chinese, but i am horribly unfamiliar with the layout needed for those languages .... which is why we are currently building a new forum script in multi-lingual capability ( not just language files for buttons * titles )  for ourselves as what we are using currently  was never intended for our needs, but hacks upon hacks of scripts that have just come to an end of their life-span.

Last edited by JJones (2019-09-12 13:12:48)

Offline

#17 2019-09-12 11:18:01

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

PS: Updated Features & To-Do-List of original Post.

Offline

#18 2019-09-12 20:24:06

GWR
Member
From: Germany
Registered: 2010-08-06
Posts: 214

Re: 27x Speed improvements ( my edits )

Regarding moderation of posts of banned users - maybe I misunderstood it a bit. You told that:

NOW... if you mean a problem with quotations inside of specific posts from one user to another, We use manual moderation to modify replied texts if the text was quoted from a user that was deactivated for some reason and that quote violated a terms of service policy. I did this specifically to ensure that "context" of what is stated is the final basis for a need for moderation.

... you moderate each quotation of such a user. Still this might mean to moderate some hundred of quotations.



Thanks for your elaborative replies. Yet I have some problems understanding how you reference postings when quoting them - if by ID how do you plan reusing IDs without changing stuff?
What happens if you remove a post quoted before?


Regarding JSON caches... are json files kept in memory (just know some dudes having their DB running in memory on their java powered services to allow more concurrent requests/users)? If they are loaded from file then you might add a bottleneck.


bye
Ron

Offline

#19 2019-09-12 21:34:05

JJones
Banned
Registered: 2019-04-28
Posts: 63

Re: 27x Speed improvements ( my edits )

Quotes are stored in the post of the person doing the quoting (the person that replied, and quoted another person ) ... the script does NOT re-read the quoted post during each page view.

We don't delete an entire post in the middle of a conversation, we instead delete the content of the post ( it leaves a flag notice embedded in the offending post displaying which moderator removed the content, on what date, as well as a direct quotation to the rule violation ) )

We don't delete entire posts in a topic for the same reason nobody can understand a chapter in a book when pages are removed. Not to mention we ( as a community ) want to know if our admins are over-reaching their authority of moderation.

json format is the FASTEST method to deliver content to a browser .... which is why it is the standard use format for API serving ... The problem a Forum has is that it has to parse that data into a format that humans can read ( PHP Parses the json data into a template, and the template turns that parsed data into whatever the theme has for a layout ...

Yes, Zend_Cache uses file storage .... in fact it uses 2 files per 1 storage bin ( see Screenshot as an example of the entire websites config ( not including any DB connection information )
Screenshot-242.png

Yes, Zend_Cache will CHECK the validity of that cache and parse it even faster than I can grab the information and parse it through a single sql query ( including PDO )....

Read more about it here: https://docs.zendframework.com/zend-cache/

I STRONGLY suggest their github: https://github.com/zendframework/zend-cache/ and scrolling to the bottom of the page to find a benchmarking tool ... Zend is so damn proud of their speed that they literally brag about it by linking tools to test them.

Keep in mind, that my version of Zend_Cache is a much more stripped down version of the original because i flat out do not need a good portion of their library. However, our edits of FluxBB allow Zend_Cache the option of changing from the json format to Memcache  or flat out rely 100% on SQL queries but there simply is no need for it ( unless I reach a few Millions of Records )

Offline

Board footer

Powered by FluxBB