Github selects Lithium to run its community forums, Discourse dev is none too pleased

Ryan Ashbrook

IPS Developer
Joined
Jan 26, 2004
Messages
3,512
Well, can't wait for XF, IPS, VB, SMF, MyBB and all the other forum software developers to go to their community and have a rant for not picking their software,
I cannot believe they didn't pick IPS!

But really, I don't see what the big deal is. Github is like any other forum owner - they will pick whatever is best suited for them and their situation.
 

Paul M

Limeade Addict
Joined
Jun 26, 2006
Messages
3,876
On behalf of my former employer, I'm surprised, and gutted, they didnt pick vB5 :lildevil:
 

R0binHood

Habitué
Joined
Nov 23, 2011
Messages
1,336
I don't have a ctrl key on mobile?
So use the excellent built in search from the search box or the share sheet search if you need some kind of ‘find on screen’ search?

Risky opinion here - but I quite like the scrubber they have that allows you to easily jump to any point in a thread without having to flick through page after after page and then swipe to scroll further to find a specific post. :eek:

It works very well on mobile.
 

Pete

Flavours of Forums Forever
Joined
Sep 9, 2013
Messages
1,658
So use the excellent built in search from the search box or the share sheet search if you need some kind of ‘find on screen’ search?

Risky opinion here - but I quite like the scrubber they have that allows you to easily jump to any point in a thread without having to flick through page after after page and then swipe to scroll further to find a specific post. :eek:

It works very well on mobile.
If, and only if, you have a good internet connection. If your connection is flaky, it can and will break in many fun ways.
 

Pete

Flavours of Forums Forever
Joined
Sep 9, 2013
Messages
1,658
As will most web apps
Not the way Discourse does. NodeBB handles it better but both flail on inconsistent connections, while phpBB, SMF and XF (I have no experience of trying to use MyBB) on a two hour train commute, where some areas barely hit 3G... they don't break.
 

SAFAD

Developer
Joined
Aug 3, 2011
Messages
61
Not the way Discourse does. NodeBB handles it better but both flail on inconsistent connections, while phpBB, SMF and XF (I have no experience of trying to use MyBB) on a two hour train commute, where some areas barely hit 3G... they don't break.
I have no idea how consistency nor concurrency is handled in both Discourse and NodeBB, but PHP systems *do not* have any control over concurrency and consistency, basically each request "boots up" *the entire system* (well sophisticated development ofcourse will not lead to such thing) that is why you don't face issues with systems built on PHP (consistency and concurrency wise).

On the other hand, NodeJS has this "non blocking I/O" basically the system listens to the host for calls and functions accordingly to the request, and as NodeBB is new to the scene I assume it incorporates the latest technologies and development idioms to achieve maximum usage of the client's capabilities, hence the issue, to put it in more technical words, in NodeBB (without taking a look into the code) the client side has a callbacks system, or async, Qs, or more effeciently subscription system, it basically subscribes to the resource on the server side (let's say for the example's sake /thread?id=10) and updates the client (i.e the interface) content based on the data that is "continuously" received.
More explanation: The server listens for INPUT, it receives a thread request with id 10, it sends OUTPUT to the client (assuming since its all javascript, and ReactJS or any other "Reactive" library is being used) continuously updates the thread content based on the output, it is kind of real-time (ish) thing but it has its major issues especially with the latest versions of NodeJS which introduced many new "features" regarding this technology that basically broke things all around and made some MAJOR developers leave it for Golang (I'm not major but I left it for Go :p)

Speaking of Discourse, I have not used Ruby nor Rails in years, in fact I don't support any of them anymore, but I believe they kept up with the technology and its trends these days, so data concurrency and consistency must be a thing, I assume it would be the same case like NodeJS, thus the issues with network latency people are facing with these two systems.

Conclusion? you really, REALLY, REALLY, have to focus and push lots of resources on data concurrency and consistency to achieve the level of robustness and stability (that the old systems that simply do not incorporate this technology) have, and this is coming from a developer that has switched from "Simple" systems development, to Enterprise-grade real-time systems.
I maybe wrong about this altogether, but if I am correct about them developing the systems the way I said (and for NodeBB there is no way around callbacks and subscriptions, been there done that), they REALLY REALLY REALLY need to look on this subject profoundly, not every resource needs to be a reactive content, Quora for example rather than having the content reactive (and notifying the users in page about the changes that has happened) it simply shows a notification on top saying you need to refresh the page to see the latest content, a similar approach can fix these latency issues in those softwares (again if I am correct about the implementation)

Disclaimer: I am not saying I am better than the guys behind Discourse or NodeBB, I am simply stating that the topic of data is a major thing that should not be looked under just because you have good internet connections so don't worry about network latency and performance.

Fun Fact to add to this long reply: Do you now that we benchmark and measure performance in NANOSECONDS in the Golang community? :D
 

LiquidServe

Aspirant
Joined
Dec 4, 2015
Messages
23
Speaking of Discourse, I have not used Ruby nor Rails in years, in fact I don't support any of them anymore, but I believe they kept up with the technology and its trends these days, so data concurrency and consistency must be a thing, I assume it would be the same case like NodeJS, thus the issues with network latency people are facing with these two systems.
Default implementation of Ruby runs is single core. No concurrency there. There are implementations like Rubinious and some Java one that I seem to forget the name of, which can run on multiple cores but those aren't stable enough.
So there the comparison that Discourse would be better than PHP based in terms of concurrency is not acceptable.

As far as Lithium goes, I think it can work on the scale Github wants it to work on. Something like XenForo would require vertical scaling which means that you can only increase the power of one server, on which PHP based forum is running, to handle the traffic.
Lithium probably has sharding, load balancing, session management on multiple servers built into it right out of the box.
So Lithium can charge $35,000 for the value they provide while a PHP or Ruby based forum can't charge that right now.
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
As far as Lithium goes, I think it can work on the scale Github wants it to work on. Something like XenForo would require vertical scaling which means that you can only increase the power of one server, on which PHP based forum is running, to handle the traffic.
Why can't I spin up additional instances for xenforo and put everything behind loadbalancer?
 

R0binHood

Habitué
Joined
Nov 23, 2011
Messages
1,336
I believe Digitalpoint has some kind of load balancing for his site, with his site split across 8 servers.

https://forums.digitalpoint.com/threads/new-server-hardware-for-the-geeks.2654797/

All 8 servers do the same stuff (for the most part)... each is part of the web server cluster, database cluster (we use ndbcluster storage engine), Elastic Search cluster, etc. Any machine can fail or be taken down for maintenance without the site going down.
  • New gigabit switch (no big deal there... but it's better than the old one)
  • 56Gbit/sec InfiniBand switch
  • 8 new servers (each server has 32 CPU cores, 256GB RAM, really, really, really fast and reliable hard drives [6 of them]).
The blade servers we used over the last many years are now retired (in fact they are sitting in the back of my truck in my garage right now).

Things aren't totally configured how I want them yet, but at the very least, you should see immediate speed increase simply because the CPUs can natively encrypt AES (far less overhead for the SSL traffic) and it's all new hardware, which you know how that goes when you upgrade from 8 year old hardware. :) The servers can communicate amongst themselves at 56Gbit/sec (for example web servers talking to database or memcache servers).

Disk I/O is crazy fast too... can copy filed from one server to another at a sustained rate of ~1.2GB/sec (with Infiniband, the bottleneck for copying files is simply how fast the drives and read/write).

System memory is much higher as well (a total of 2TB of RAM on the new servers).
I'm sure Github HQ could figure out how how to make other forum software scale if they really wanted to, especially given the nature of the business they're in.
 

LiquidServe

Aspirant
Joined
Dec 4, 2015
Messages
23
It’s not OOB. You would have to write customer database adapters just to share the databases.

Setting up session management over a load balancer with multiple server is also a thing to think about.

Everything is possible, but it’s a time Vs money tradeoff and it seems they have a ton of money at their disposal.
 

R0binHood

Habitué
Joined
Nov 23, 2011
Messages
1,336
Yeah, and I'm sure that for 1 months worth of Lithium you could funnel some man hours into pretty heavily customising another solution :D

Lithium doesn't even have code syntax highlighting OOB.
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
It’s not OOB. You would have to write customer database adapters just to share the databases.
Why can't I keep database on dedicated instance and make http instances connect to it? Is that not supported in MySQL?

Setting up session management over a load balancer with multiple server is also a thing to think about.
Why can't I use custom session engine that keeps session data in, say, database or KV store? AFAIK Zend Framework that's under the hood in XF supports and recommends doing that from beginnings of Zend_Session?
 

LiquidServe

Aspirant
Joined
Dec 4, 2015
Messages
23
What you’ll do with SQL is vertical scaling where you’ll increase the power of your single server to a large enough value until adding more hardware becomes so expensive that you realize that the same database could be run off 2 separate servers to distribute the load and at quarter of the cost.
That’s where the problem starts. GitHub is going to hit that on day 1.

For sessions, you could do that. Set up some memchached servers, save the cookies there. Again, I doubt you’d have only 1 of those servers. When you put 2 servers, things change and you’d have to write a custom session engine like you suggested for that as well.

It’s always easier to have pioneers work on giving you a solution than to waste time figuring out on your own. That’s how businesses work. That’s why they acquire other companies, for their assets and expertise.
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
Now go make sure you tell all this stuff to guys running Neogaf, one of most active forums in the web, because apparently what they are doing is impossible using XenForo :]

That’s where the problem starts. GitHub is going to hit that on day 1.
They didn't hit that mark after half year. Its reasonably active forum, but its peanuts completed to, say, bungie.net
 

Karll

Adherent
Joined
Dec 9, 2011
Messages
413
What you’ll do with SQL is vertical scaling where you’ll increase the power of your single server to a large enough value until adding more hardware becomes so expensive that you realize that the same database could be run off 2 separate servers to distribute the load and at quarter of the cost.
That’s where the problem starts.
MySQL/MariaDB replication is a standard feature (has been for years and years), so it's very easy to horizontally scale reads. Write scaling is harder to achieve, though, but I suppose forums are mostly reads.
 

LiquidServe

Aspirant
Joined
Dec 4, 2015
Messages
23
Now go make sure you tell all this stuff to guys running Neogaf, one of most active forums in the web, because apparently what they are doing is impossible using XenForo :]
Never said that it’s impossible. It’s possible to parse your forum content and create a much better search which predicts what you are looking for and gives youbresults even before you hit enter. But that’s not available right out of the box.

I was talking to a developer at a company which has been debating to shift from MySQL to SQL Server by MIcrosoft. Why do you think they would pay for something when they have open source software available to them that they are already using right now?

MySQL/MariaDB replication is a standard feature (has been for years and years), so it's very easy to horizontally scale reads. Write scaling is harder to achieve, though, but I suppose forums are mostly reads.
What would happen to the read latency when a transaction locks a table?
 

rafalp

Desu Ex
Joined
Feb 17, 2008
Messages
1,183
Never said that it’s impossible. It’s possible to parse your forum content and create a much better search which predicts what you are looking for and gives youbresults even before you hit enter. But that’s not available right out of the box.
Humm? Where did the search come from? I haven't brought it up.
 

Karll

Adherent
Joined
Dec 9, 2011
Messages
413
I was talking to a developer at a company which has been debating to shift from MySQL to SQL Server by MIcrosoft. Why do you think they would pay for something when they have open source software available to them that they are already using right now?
SQL Server is a very fine and very popular database system, and for a number of years it has had advanced features not found in MySQL. Though in recent versions, MySQL, and maybe especially MariaDB, are catching up. (Not to mention PostgreSQL, which is another fine Open Source database system.) It's a question of the specific requirements they have as well as the skill set in the developer team.
What would happen to the read latency when a transaction locks a table?
Transactions don't have to lock whole tables (e.g. row-level locking), and you can also use different transaction isolation levels which have an impact on the type of locks used.
 
Top