Reply
Tue 5 Feb, 2008 12:42 am
This is a topic to collect posts from the development team (Nick and I) on the new software. There will be changes big and small and I am going to explain some of them and the reasoning for it. Some of these changes may be canceled and many of them may not be present at launch but since we don't yet know which we'll share the plans that are
Starting with:
CATEGORIES
On the new site, categories are subdomains and can be nested (there can be multiple levels e.g. Literature > Books > Non Fiction). This is a better information architecture that allows us to better scale to more diverse topic coverage. An even more scalable folksonomy solution will be built around this and one of the big fundamental changes we started with are more scalable and relevant organization of information.
DESIGN PRINCIPLES
The new look is starting from a very plain base. In fact there are only two design graphics on the current pre-designer look: one is the logo and the other is a button that's used for all main forms.
The goals we are going for are accessibility, usability and specifically readability. We may not have the greatest typography at launch (because our css files will likely launch a bit of a mess and we'll need to organize it all) but we will strive for it at some point.
The big changes here are in a much more subdued look and feel (lots of white and space) and in the static page width. The subdued design aims to make the content more of a focus point and to be easier on the eyes when reading. As to the narrow static width, research shows that text is more readily digested if it comes in certain column widths (quickly: what's the widest newspaper text column you remember?) and the very wide post widths at some resolutions are not great for readability. Given another of our principles (user-customization and self-determination of one's use and experience) there are also plans for features allowing the user to elect a liquid layout if desired.
The end result of the design changes should be a much faster loading site that is easier to use and subsequently will be used more.
NAVIGATION
The navigation of the site is one of the bigger changes and will be a major work in progress. The big changes are in clarity of the navigation elements and in the scalability of the information architecture. At launch it will be a little awkward since the navigation takes into considerations large sections of the site that will not actually be present right away but there will still be some improvements.
Namely, the site will use very prominent navigation elements including tabs (on the side of each page large tabs will present the most relevant actions within a given context. e.g. "Read Topic" and "Reply" or "My Topics" and "All Topics"), breadcrumbs (e.g. A > B > C) that are consistent and eventually top tabs/nav and search.
It will be messy in the beginning but the basic goal we are going for is to make it more obvious how to use the site by organizing the navigation and presenting it more prominently when appropriate.
WYSIWYG
That stands for "what you see is what you get" and is a technical description of editing platforms that allow you to see the result of the formatting you apply to text on the very text that you apply it to. An example of this would be Microsoft Word, where you don't need to know and often never even see the underlying markup. Contrast that to our current editor where you have to learn a pseudo-markup (BBCode) to apply formatting that you often only see the result of after the post is live.
The new site will use BBCode, but also will provide an optional Javascript WYSIWYG editor. You will be able to format text, links and images without needing to know anything about BBCode.
THREADING
The new site will begin to treat all replies as replies to a post, not a thread. This will allow us to enable threaded or hybrid threaded/flat views that can help users follow conversations within large threads. The launched application will likely not feature any of the tools meant to help organize discussions but we will launch with the data structure in place to begin to do so.
TOPIC TYPES
The new site may or may not launch with topic types but they are a fundamental part of the technology we will build. To summarize, the site is used in various different ways. From long political discussions to short questions and an answer. We are planning to help support the different types of uses by having different topic types that behave slightly differently. Examples of topic types we have in mind are Questions, Discussions, News, Videos....
For example, the questions topic type will be differentiated from discussions in that it can be marked as "answered". Not just "replied to" which is our current very limited view of our Q&A topic filtering capabilities. News topics may be displayed with a more time-sensitive context than discussions.
TOPIC CATEGORIZATION
While the site, at launch, will have its topics moved around similarly to how it is currently done (by moderators) one of its biggest updates are in its tagging feature which will provide social indexing (folksonomy) to the topics and allow us to eventually do better content matching on the site (e.g. related topics, user-defined interest in certain topics only...).
This also extends to the way the threads are prioritized. Right now it's a "featured threads" process that few people directly participate in and consequently many find fault with. There will eventually be tools to put this in the hands of the users at large (and this is going to be a recurring theme here) which won't stop many people from disagreeing with what's featured but at least the blame can go around and is not taxing the volunteer human resources. It has the additional benefit of being able to scale beyond what a handful of individuals are able to organize as well.
There may still be user-differentiation (real simply a moderator has different access and rights on the software than a user) in future processes but it will be blended in with much more hands-on access for the end users.
CACHING
One of the big problems we wanted to solve is the ability for the site to scale. The site was crippled for years at levels of use that exceeded the software (phpBB) or hardware capacity we had.
We are designing this new software to handle much more load. One of the fundamental changes are in the use of object caching. We'd previously cached at various levels but this is true scalable object caching that will impact the site dramatically.
The point of failure for this site, and almost any of its type, is the database. The overhead from processing the application language (in our case PHP) and the overhead of serving the files (html, images, css, etc) is very manageable. The database is a different question and very low levels of concurrent use can bring the site to its knees.
We are addressing this with object caching built into the application and stored in the servers memory. This can potentially allow us to serve hundreds of times the number of users on the same hardware. An example of this would be a topic page. Let's say it gets linked to on a popular site and a lot of users hit the page at once. Right now, each visit would result in an array of queries to the database, bringing the site down with moderate spikes in traffic. With object caching the query would be saved in RAM memory for a certain duration and each subsequent request would be served out of memory without touching the database until it expires.
This will make a fast site both in that the requests served out of memory will usually be generated in about half the time but also because more concurrent users can be supported per-server and the sluggishness of load will be felt less often.
Now all of that's background to the end user and you may wonder why I bother to explain it and that's mainly because of the lag it will sometimes introduce.
Let's say we use a 30-second cache (we'll use all sorts of custom caching depending on the object we are caching but that's an easy example). That will mean, in some cases, that changes on the site aren't reflected for 30 seconds. So you may post a thread but not see it on the topics list for 30 seconds.
Striking the perfect balance of freshness and cache hit ratio (ideally over 80% of all requests would be served out of cache, for an idea of how extensive and robust the caching plan is) is going to be tricky. And in a related change in architecture we are going to be queuing trivial writes.
A read of the database is when data is requested, e.g. a post selected from the database. A write is when data is inserted or modified on the database. With caching and load balancing reads can be scaled linearly fairly easily. But writes are always a bottleneck until the very last level of scale (sharding, but that's far off) and with trivial writes like views that are widely used you get very significant bottlenecks. In the case of views, every visit to a topic requests a write to the database but even worse is that every visit to a particular topic requests a write to a particular row in the database and each write needs to finish before the other can begin for data intgegrity reasons. Now hundreds if not thousands of such requests can be made per minute on most server configurations but that's still a low bottleneck for a site with this kind of traffic when a spike comes along.
So we'll also delay some of the writes to consolidate them and hit the database fewer times. So we may store visit records in memory, or on flatfiles individually then have background code that collects, counts and inserts them in groups.
The end result is that some data will lag even more. But we'll try our best to prioritize what we allow to get stale (I'm considering anything over 30 seconds stale) but you'll see some staleness in things like topic views. They may be stale to the tune of being several minutes late.
These are small things but any staleness adds to a usability problem for the site, and the reason these will be introduced (the current site doesn't have these problems) is because ability to scale is being introduced (the current site falls apart if sneezed on). We'll do our best to make this as transparent to the user as possible but when we fail to do so there are very important performance considerations that are motivating us to do what we do with the caching.
Phoenix32890 wrote:I don't know what the hell that you are talking about, but it sounds good!
Just nod sagely and say "Uh huh, I see". and "yes that sounds like a plan"
That fools most people into thinking you know exactly what's going on.
Uh huh, I see , mmm scaled caching, excellent idea craven. Are you sure you can pull it off?
well I have every confidence. I know you'll do your best.
Dont forget to drill down and become client focused, we want this to be a win -win.
dadpad wrote:
Uh huh, I see , mmm scaled caching, excellent idea craven. Are you sure you can pull it off?
We already did and are using an open source software for this called memcached that works very well and is used for Wikipedia, Live Journal, Word Press etc. It's a very good wheel so we aren't going to reinvent it.
Now we just need to build the application to use it intelligently.
How will advertising be handled?
Hard to say, since just last week we decided not to use existing ad servers and make our own (and have yet to think about how).
Initially it will be very simple and not much different than what we currently do.
George-
If you have a problem with the ads on the side and top of the page, I really suggest using Mozilla.
Their ad blocker stops all of that. Even those noisy , irritating little smilie ones.
shewolfnm wrote:George-
If you have a problem with the ads on the side and top of the page, I really suggest using Mozilla.
Their ad blocker stops all of that. Even those noisy , irritating little smilie ones.
Nope, not a problem.
Just wondering how they'd be presented in the new format.
(I use Mozilla Firefox at home.)
Phoenix32890 wrote:I don't know what the hell that you are talking about, but it sounds good!
I'm glad
you said that, Phoenix. Exactly what I was thinking but would have been too embarrassed to say!
It probably seems more complicated to use than it will end up being.
Will adding art, photos and , say , graphs be accomplished as now?
Apropos adding files - it would be nice if we had capabilities to upload
files (pictures, maps etc.) directly from the post, instead of looking for
a host and then use the URL from there.
Like here from a German site that incorporates the upload in its post
Craven-
First off I think the site is great as it is now.
Second- will the present record of discussions still be available for revisiting?
Thanks.
BBB
I don't understand any of this but Butrflynet can explain it all to me.
BBB
Re: BBB
BumbleBeeBoogie wrote:I don't understand any of this but Butrflynet can explain it all to me.
BBB
Clueless in da Big Apple. When she explains it to you, BBB, please pass it along.
margo wrote:msolga wrote:Phoenix32890 wrote:I don't know what the hell that you are talking about, but it sounds good!
I'm glad
you said that, Phoenix. Exactly what I was thinking but would have been too embarrassed to say!
Olga
Almost no-one does!
Oh Phew! I thought I was having (yet another!
) Luddite moment!
It is so wonderful not to be utterly alone & in the dark for a change!
spendius wrote:Craven-
First off I think the site is great as it is now.
Second- will the present record of discussions still be available for revisiting?
Thanks.
Speny has a definite aversion to change. Have you got used to women voting yet spend?