Will You Comment On This Post?
Post categories: social
| 19:24 UK time, Tuesday, 14 October 2008
If you do, you'll use a ³ÉÈËÂÛ̳ system called DNA. It powers most of the comments, message boards, forums, blog comments, and other social media applications across bbc.co.uk.
Just think of all the comments on , Robert Peston's blog, h2g2, the Archers message board, C³ÉÈËÂÛ̳ boards and many, many more...
On the morning of Thursday October 9th, there was a major code release - three months' worth of updates and fixes were scheduled to go live after extensive testing on the staging server. Usually, releases happen monthly, but due to the Olympics, we didn't release over summer.
The DNA team had planned on rolling out between 7 and 10am, but because of the popularity of Peston's Picks, the blog team felt we couldn't take any chances.
So the developers came in early and took down DNA between 4 and 6am. By 7am, everything was done, the software was back up and everything was working fine. By 10am, all was still well, until the central communities team picked up a minor little problem. Some threads on a single message board were being automatically closed. It wasn't a major problem; rather a minor problem that needed an urgent fix.
Paul and Jay from our communities team ran between their desks and the developer team to try and identify and to isolate the problem. Working closely with Mark and Mark on Martin's DNA developer team, they quickly identified the offending line of code.
Because DNA is used in so many different ways on so many different parts of the ³ÉÈËÂÛ̳ website, it's very difficult to set up tests that exactly mimic all the possible different permutations.ÌýVery occasionally, a problem will slip through the net.
In less than an hour, the problem was identified and the one line of code was fixed. There were less than 500 threads affected, but they still needed to be reopened. Once again, Paul and the DNA team jumped in and put together a script implementing some of our moderation rules, sort of in reverse, to fix the problem. Shortly after 11am, this was run, and we were back to normal.
Or so we thought.
contacted us saying that a lot of their threads were still closed - the fix had apparently not worked for them. The 606 service is by far the biggest single platform using DNA. Luckily, this turned out to be a caching issue, as we have to cache their boards because they're so big. The cache was cleared and 606 was back to its wild ways.
Not everyone was happy, though, as this unhappy 606 poster posted:
Why am I getting this message, I've been a member since 2006 not yesterday, how is that new 606 has been a complete mess today, sort it out
At the end of business on Thursday, it was also clear that we had another problem: the so-called "new user hole". Despite being around for a while, some users were identified as brand new users. This also meant their comments didn't show up immediately, as they were pre-modded in accordance to our house rules.
However, this turned out to affect only 24 users out of all the tens of thousands, and was fixed by close of business on Friday.
By noon on Thursday, members of the DNA team had already put in more than eight hours at the office. The central communities team had helped them to fix all the problems less than two hours after the first issues cropped up.
There was no data loss at any stage, everything was back to normal and the central communities team was just in time for a boring weekly team meeting.
Join in - leave a comment and become a DNA user yourself.
Tom Van Aardt is Communities Editor, bbc.co.uk.
Comment number 1.
At 14th Oct 2008, SheffTim wrote:Some sites shut down (without warning) completely during maintenance and upgrades, so it is pretty good to keep the most of the site operational whilst this was being done.
Is possible to fix the most irritating issue, that of apostrophes, hyphens etc turning into question marks? (And pound symbols into diamonds.) I`ve just looked at the comments in Justin Webb`s blog and it still seems to happen e.g. # 110
Complain about this comment (Comment number 1)
Comment number 2.
At 14th Oct 2008, Michael Walsh wrote:Will You Comment On This Post?
Tom Van Aardt 15 Oct 08, 6:08 PM
Yes!
Michael Walsh 14 Oct 08 7:00 PM
Time travel not restricted to Doctor Who then?
Complain about this comment (Comment number 2)
Comment number 3.
At 14th Oct 2008, Nick Reynolds wrote:Something very wierd happened there.
The post was saying it was not published in our system even though it had been.
Anyway, since it had been published I brought the date forwards.
Apologies for the time travel.
Nick Reynolds (editor, ³ÉÈËÂÛ̳ Internet blog)
Complain about this comment (Comment number 3)
Comment number 4.
At 14th Oct 2008, Michael Walsh wrote:No need to apologise Nick - it just confirms that the ³ÉÈËÂÛ̳ is just a bunch o' left-wing, time travellin', tree huggin' meeja luvvies!
Complain about this comment (Comment number 4)
Comment number 5.
At 14th Oct 2008, tomVersus wrote:@SheffTim (#1) I'll look into the special characters issue and get back to you.
@Michael-Walsh (#2) Under the "One ³ÉÈËÂÛ̳" policy there's a lot of sharing and openness between all the different divisions, so we get some pretty cool stuff from Dr Who, yes :-)
Complain about this comment (Comment number 5)
Comment number 6.
At 14th Oct 2008, alaninbelfast wrote:@Nick - Any reason why the Blogs homepage is has been stick in a time warp showing September stories for the last month?
/blogs/
All thse blogs featured in the top half have had posts long since 8 September ... yet they seem stuck around that date on the blog homepage.
Complain about this comment (Comment number 6)
Comment number 7.
At 15th Oct 2008, johndrinkwater wrote:Have you fixed the UTF–8 handling yet? Probably not…
Complain about this comment (Comment number 7)
Comment number 8.
At 15th Oct 2008, _marko wrote:Preview function?
Stop repetition
(accidental) of identical comments from the same user, often added when there are more than 500 comments and the comment doesn't appear to have been published on the first page.
Complain about this comment (Comment number 8)
Comment number 9.
At 15th Oct 2008, Dave Parker wrote:Will You Comment On This Post?
No
Damn, I appear to have created a paradox.
Complain about this comment (Comment number 9)
Comment number 10.
At 15th Oct 2008, Briantist wrote:Strange that you've updated DNA (after Douglas N Adams, of course), but have not switched the blogs to Barlesque layout (named after the fictional Nathan Barley).
Probably worth pointing out the former is on and the latter's creator is doing a Big Brother with Zombies on E4!
Complain about this comment (Comment number 10)
Comment number 11.
At 15th Oct 2008, BettyHur wrote:Can you comment on this post?
"Yes"
Which one? I or _ or / or \ ?
I - It's still standing up.
I seem to have created a para Docs.
Complain about this comment (Comment number 11)
Comment number 12.
At 16th Oct 2008, DamnyoureyesF wrote:Maybe H2G2 could get a revamp and a bit of money and love spent on it, as the ³ÉÈËÂÛ̳'s social media / community site. It always gets over looked, it was doing blogs long before anyone else. It'll be 10 years old next year!!
Complain about this comment (Comment number 12)
Comment number 13.
At 9th Sep 2009, felicioo wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 13)
Comment number 14.
At 20th Sep 2009, mrdonovanin wrote:This comment was removed because the moderators found it broke the house rules. Explain.
Complain about this comment (Comment number 14)
Comment number 15.
At 20th Sep 2009, paul_geaton wrote:Tom, will DNA ever get the facility to edit posts. Many is the time when I've pressed Post Comment, only to spot a few seconds later that I've written "their" when I meant "there" or "whose" when I meant "who's". Not earth shattering I know, but it niggles me that I can't correct it.
Complain about this comment (Comment number 15)
Comment number 16.
At 21st Sep 2009, Nick Reynolds wrote:There is a preview button Paul.
Complain about this comment (Comment number 16)
Comment number 17.
At 21st Sep 2009, paul_geaton wrote:You've told me that before Nick, it hasn't helped. It's only when I get to the stage where I can't change them that I notice the errors, or else I've simply forgotten to press Preview. So an edit button would be nice, as incidentally would a spell-checker in Preview mode.
Complain about this comment (Comment number 17)
Comment number 18.
At 21st Sep 2009, TV Licence fee payer against ³ÉÈËÂÛ̳ censorship wrote:#17
"So an edit button would be nice"
Might be workable on a reactively or post moderated blog but on any pre-moderated blog the edited comment would have to be resubmitted for moderation, thus causing even more work to the mods and slowing the busy blogs up even more...
Nost poeple can accept typose on this sort of fast moving, interactive, socail media.
Complain about this comment (Comment number 18)
Comment number 19.
At 22nd Sep 2009, paul_geaton wrote:Pardon?
Complain about this comment (Comment number 19)
Comment number 20.
At 22nd Sep 2009, Nick Reynolds wrote:Boilerplated's reasons for why we don't allow people to edit comments after they've been published are good ones. Once it's published, it's published I'm afraid.
Complain about this comment (Comment number 20)
Comment number 21.
At 22nd Sep 2009, paul_geaton wrote:Nick, I was joking - I get the point. Cheers.
Complain about this comment (Comment number 21)