Sorry for the info vacuum of the past few days; i have been trying to get things together to work out some clear options. Here’s where we are so far:
- The hosts recommend that “basically you should junk your server; it’s old and inefficient and the disks are shot”. They’re offering us a virtual server for the same price which has less CPU but the same amount of memory and disk. It’s not a bad deal — and ILX is unbelievably light on CPU anyway — but I also think we should get other quotes, so...
- Have asked a number of colo and virtual server places to quote us for enough power to run ILX. The big issue is memory — the DB is heading towards 4gb now, most of which the code tries to keep in memory. But virtual hosts are usually quite sparing with the memory. Given that ...
- The memory requirement we’re at now is already pretty expensive, and the ILX DB has doubled in size since we switched to Keith-code. If it keeps on growing like that we could eventually hit some relatively expensive memory requirements. Changing this setup would be a big chunk of work, and it might not be necessary, but would also dictate a bit what kind of hosting we’re looking for. Surprisingly, among the more reasonable big-memory online hosts are Amazon, so:
- Also looking at running on Amazon’s server, using their EC2 service. That’s what this sandbox is running on right now (on a smaller, free, server — the full-size one would cost a fair bit more.) There’s some real advantages here, as they’ve also got great tools and monitoring and can do real-time backups and I can spin up a backup “server” in seconds and it’s generally great. But:
- it’ll take some code changes to run properly here, as there are some things we do which are incompatible with the Amazon setup, and it could also be expensive. That said:
- Virtually all the options are going to cost some cash. When I get the quotes back (day or two) we should know how much those options will cost, but won’t really know how much the Amazon setup will be until we try it out for real. At an guess, I think an upper limit for basic running on any of these options would be $100 a month, which is what we pay now for hosting.
So: Right now I think Amazon’s my preferred option, although our hosts have given us really personal service so don’t want to rule their virtual server offer out either. I think either of them is better than buying or renting another server and just shifting all our eggs from one basket to another.
I’d like to keep investigating Amazon for at least this week. A couple of people have offered to help with the work I need to get done, which is v. appreciated and I’ve sent them details of what’s still required to get proper ILX running on this setup.
If we can get that done, hopefully in a week or two at most, we’ll know pretty quickly what the costs are likely to be. Then, depending which option we go with, it may be cap-in-hand time again. Will keep you posted on what’s going on.
<3
thx stet!
surely we can raise whatever money we need fairly quickly
thanks for the update!
is sb extra
Brilliant. I’m away during the cap-in-hand period but am more than willing to stump up before or after.
open ended fundraising pls
I will definitely donate to the cause when necessary. I has a jobz now!
Is the DB plate inevitably going to get too big to keep spinning on a stick?
Apols for asking that in a dumb way
great work, thx much stet!
presumably this kinda space will be cheaper in the future?
WmC, I think that was the perfect way to ask.
whatever price he asks, we shall pay
apprec your work, stet.
Wmc: We’re being outrun by falling prices for now, but there are other complications — the bigger it gets the more unwieldy it gets too. Running backups was more of a pain than it needed to be on the old server, for instance.
(In database terms, it’s not that huge — there are databases of hundreds of GBs and more happily chuntering along. Those tend to have more dedicated stewards and need care and watering, tho.)
Thanks, stet. Sorry for being crochety on the other thread. Really been missing/needing ILM of late.
WmC, I think that was the perfect way to ask.
― Making like Melusine (Pyth), Monday, December 5, 2011 3:27 PM (9 minutes ago)
heh, thanks. I keep thinking “would it be possible to reduce the load on the DB by making the first several years of it read-only and sending it off to live quietly with relatives or something?” but really, there’s too much value in revisiting and reviving old threads. And having it accessible to read doesn’t reduce the load anywhere, I guess. It’s like having to lug around a copy of The Riverside Shakespeare all the time instead of just the one play you’re reading at the time.
Amazon seems like a pretty sound option IMO
I’m not gonna read all that cause it’s long and I’m trying to leave work but I’m happy to donate and thank you stet for your hard work. Also, the sandbox is pretty good for the time being imo.
Thanks stet for handling this, much appreciated.
xp Amazon option seconded.
+1 Amazon. I can’t help in any meaningful way other than by injecting funds. William (C), you should buy a Kindle.
“would it be possible to reduce the load on the DB by making the first several years of it read-only and sending it off to live quietly with relatives or something?” but really, there’s too much value in revisiting and reviving old threads.
I have been thinking similar things myself. The quick and easy thing to do is make everything that hasn’t been updated in say a year just a read-only HTML thread. That’d be super-easy to serve, and we could do it from a tiny server like this one.
But it’s also a bit crap — wouldn’t be properly integrated with search, and you couldn’t just bump an old thread. I’m trying to see if there isn’t something cleverer we could do that would fix those problems — perhaps the threads live in cold storage as static HTML and then if someone tries to bump one we “defrost” it and insert all its posts into the DB again? Can foresee problems doing that, too.
Either way, by far the simplest option for the immediate future given funding (eg at least another 2/3 years the way things have been going) is just to get a bit more memory.
I can live with the sandbox for a while. As an example of Amazon EC2 performance it’s pretty impressive.
^^ yeah, definitely
would our existing host folks be willing to allow us to spin up another sandbox using their virtual solution so we can compare performance?
ah, got it — those were the same questions and reservations I had from a layman’s perspective — just wasn’t sure if I was asking the right questions or the programming equivalent of “can we turn this block of solid granite inside out?”
xp to stet
is sb extra
― Sad Banter (p much resigned to deems), Monday, 5 December 2011 21:19 (34 minutes ago) Bookmark Permalink
this, this should be like carbon tax, we should get a credit & have to buy more
good work stet!, let us know when you need $$$. when your head explodes from the continuous effort dedicated to server maintenance &c we will chip in for replacement parts for that, also
That’s probably not a bad idea to ask about. I would guess they’d want to put the existing server out of its misery before they do that though, and I want to keep it zombieing on until I have a live running copy of proper ILX on another server.
(I have a copy running on my Mac, and the DB all checks out, but it’s not quite the same)
xp thanks everybody btw <3 u all
I am in favor of a sb tax—we would never need a funding drive
frogbs would keep ilx running for 100 years
stet, there is one way I might be able to help depending on (a) how you go about doing what you intend to go about doing and (b) how busy I am when you get to that point. Bung me an email if you so desire.
Yes, adding to the thank-yous to stet for service above and beyond!
I want to volunteer to help but this is about to become silly crazy busy season for me and I don’t want to be the flake who backs out
can’t believe you’re already flaking dan. sorry everyone—it seems like the orphans won’t get into their new home in time for christmas after all.
so unthoughtful, jeez
awesome, adding my thanks here, and hopefully some $ at some point.
hey stet, thanks for all yr work!
fwiw i support using amazon too, best to use the futuristic hosting styles since we r a cutting edge community after all
can’t believe dan is being such a huge jerk about this
hey stet, thanks for all yr work!
― your voice of treason, Monday, December 5, 2011 5:23 PM (15 minutes ago)
echoing frogbs here, thanks stet
dan needs to take some time off and think abt his priorities
xpost
\(^o^)/
\(^o^)/
Thank you, stet! I have no skills to contribute but would love the opportunity to throw money at the problem.
btw in case I sounded too much like a huge wanker I was offering to develop the sort of supporting documents that would be megaboring for an ilx thread.
ditch the db, keep the sb
GOOGLE ADS HAVE WE TREID GOGGLE ADS
Amazon affiliate setup where every word on ILX is a link to a product.
loll yes
i know this comes up a lot but ads for unregistered posters and posters i dont like would get help us pay for a more expensive server if we needed it
xp
Given that the annual fund drive is always oversubscribed, do we really need to look into novel solutions?
Stet, you are awesome.
catching up now but thx stet you are a prince
Charge people $5 every time they change their display name.
Readmission fee after a ban
£51 fine for self-excelsioring
50¢ per otm charge
graduated lol tax
Tithes to be paid by all posters to The Church
that medieval church puggle ain’t looking too happy
he can’t afford to go on ye olde ILM
lol tax revolts of 2012
Am I the only person who thinks “sb” should be left behind?
^no
thanks 1,000,000 stet for all yr hard work
I’d like to poll that question, I’ll admit. But I have ‘mixed feeligns’.
In theory, if someone’s just posting offensive crap or spam, 50 good and faithful ILXors should be able to point them out and the perp gets xored out.
At the moment, it’s more a ‘custard pie for not liking Animal Collective in the right way’.
Maybe a ‘ban’ should last all of an hour, in the spirit of “we love you but maybe you should go and chill out for a bit”
And so on. Don’t mind me.
If someone’s just posting offensive crap or spam, they’ll get banned by the mods before even reaching 51.
I know, and there’s the ‘borderline’ stuff, and there’s the ‘offensive to relatively few people’, etc.
Do most SBs happen clumped together after someone has said something very not cool, or are they usually spread out because most people are low-level horrible rather than balls-out nasty? Because these two scenarios maybe require something different? Maybe?
it seems ‘low-level’.
I’ve never sb’d anyone, but then again nobody’s really been b-o-n to me.
I can think of a couple of scenarios where I could have been sb’d to heck (OK, one scenario), but because it didn’t exist I was able to explain my side, and people were able to either agree/forgive/cope/whatever.
Is that so wrong?
I’ve never sb’d anyone
Same here. I hate the whole idea of it.
I think I’ve only ever done it to people threatening other people with an SB in a kind of twisted Karma way.
God i wish we had SB on sandbox
but maybe we shd debate it AGAIN given the huge number of inoffensive posters 51’d every week for being inoffensive
OK, parc the sarc, let’s talk about it.
I’m guessing you’ve had your fair share of being sb’d to 51, right?
How many of those would you say you ‘deserved’, or ‘helped’ you?
hehehe i’m not who you think i am ;-)
thing is Mark, we have this debate every fortnight or so, a bunch of people don’t like SB and will never participate in it, good for them. the system doesn’t exist independently of moderation, it functions as a guide to mods. its biggest flaw is that on the odd occasion when a user has a bit of a meltdown an unseemly pile-on can take them to 51 quite quickly—as i understand it, mods will now try to step in and prevent this happening. the only other way to be 51’d is to piss off 51 separate posters enough to click the button within a limited time frame—under this system it’s proven all but impossible to get 51’d, and peeps who have been have almost always been at least aware that part of their online persona is “controversial” to other posters. some people’s ILX experience is spoiled by meanness and swears, some people’s ILX experience is spoiled by dickwittery and not-quite-spamming. SB is a reasonable leveller of these differences of opinion.
I think that’s a fair summary of the current thinking on SB. I love a clusterfuck as much as the next misanthrope, so by all means everybody who hates SB can keep telling us all about it. Don’t think that’s gonna make it go away, at the mo.
God i wish we had SB on sandbox
oh, and it’s got nothing to do with hosting ILX so personally i’d prefer it had its own thread so we can address relevant issues relevantly.
i agree w/ you MarkG sb is tryrannical and cruel & should be outlawed
for one thing idk how we are all surviving without it on this here sandbox, julie you must be having an extremely hard time my apologies where o where is the sb button miss u boo
yeah, y’see i find it useful for when you’ve made a reasoned argument and then people just carry on as if you hadn’t
lol @ ppl thinking you’re actually LJ also lol @ this discussion AGAIN
we can never rest until the brutal shadow of the SB button has been lifted and people feel free to post their innermost thoughts again
Well, it’s neither one thing nor the other.
Anyway, thanks for the reasoned defence of it. I still would like it gone, but then that’s my opinion.
I dunno if it needs a full debate, just .
or we can go on as we seem to be managing to do right now in perfect harmony xp
You guys just think there’s no SB on the sandbox. I found the secret button and am SBing you all repeatedly right now.
That’s the wrong button, you’ve got us all signed up for the Sugababes audition!
u r so mean! funny how such a bully would be in favor of a system designed to let ppl gang up on some1 & ‘vote him off the island’
Mmm, interesting! Intense levels of sarcasm, but. (ref: that website)
Anyway, to get back on topic—I’d love to help out but donation drives always seem to come at a bad time for me: either I’m off-board and they finish before I get there or they turn up when I’m skint. It would be good if we could open donations year-round, but I seem to remember there were tax/legal issues with this, right?
The drives were always open very briefly, and closed when the target was reached. In my time here, i’ve only managed to contrib to one!
hooray for stet! hooray for more memory! otm’ing everyone who wants to be able to donate whenever.
also I hope that the ability to bump ancient threads doesn’t get phased out in the interest of db performance, since that is like the most charming thing about ilx
^^^
yeah clearly u have to be able to bump, thats a given
In theory, if someone’s just posting offensive crap or spam, 50 good and faithful ILXors should be able to point them out and the perp gets xored out.
otoh we had about 14 active users and no mods the morning Newgod arrived. sb failed us that morning. and all this “oh i’ve never banned anyone, i’m so special” bullshit is so fucking lame. spammers like newgod are in theory what it is there for and if i was the only fucking one pressing the button for 2 hours then ilx deserved to break.
anyway, post passive aggression, thx stet! i will try to contribute, even if it is a time of the year i have absolutely no £££
um, I wasn’t saying I was “so special”, I did caveat “nobody’s upset me personally here”
only fucking one pressing the button for 2 hours then ilx deserved to break
While I personally take a lot of satisfaction in repeatedly hitting SB on certain people just for the hell of it, you do know that it only counts once when you do it even if you sit there and hit it 200 times, right S?
I missed the downtime! I just logged in and saw that nobody had updated for 4 hours. thought it seemed a little suspicious.
the sandbox was out celebrating step’s birthday probably
Thank you for the opportunity. Xp
darragh sounding like an old hollywood agent there
I have turned on some other site, to see how this level of server takes the load. Don’t spread the word too far yet.
Wow, that went badly.
ha
;)
www.geocities.ilx.com!!
ilx.xxx
dear god what have we done
lol
hey why does this keep crashing?
What do you mean? Last night I was doing a lot of work on it so it was up and down, but not around the time you posted.
hi stet i know you’re new around here so i think it’s be good to show you round and introduce you to a few of the ‘regulars’ just so you know what’s what
any new news? any old news?
me. you.
Old news: hit some unexpected hiccups with the last switch on. New news: trying again tonight.
thank you, kind fellow.
would love to get a play by play when youre done, i only understand this stuff v much not really, but still would be interesting
Um, I know I volunteered to dig into ILX code in the past and didn’t really do anything due to no specific direction and a lack of free time, but... I should be home a fair bit over the holidays and my day job is Java development so I might be able to help look at some of the code.
Refresh my memory, but doesn’t the code currently only load threads to memory from the DB if they’re actually accessed? Getting the memory load back down might be as easy as expiring out the cache more aggressively.
aren’t you knackered though?
god i so don’t want to relive that. but I need to document this stuff, so I will.
anyway, let’s try this again folks http://ilxor.com/ILX/
how long is this for? (thanks stet, you’re the best!)
i’m going to leave it running. if it’s still going tomorrow, I’ll redirect the front page back to it and that’ll be us in business.
RIP CK Dexter
sandbox is slowing down; think a lot of people are hitting the search quite hard to find old threads
my gods, man. can it handle to ILXes at once?! what portals to the netherworld will this open??
answer: no.
actually it just ran out of memory with the volume of searching going on. I have raised the limits and now I’m going to leave it closed but running while it gathers its wits (and the poll bumps etc all happen)
IT LIVES
*jumps for joy*
thank u Mr Stet
well, not in just yet...
actually it just ran out of memory with the volume of searching going on.
sorry but i had a mark wahlberg pic that i NEEDED to post
ty for yr efforts. feel like planning to do this pre-holiday should set you up PERFECTLY to spend crucial celebratory time occupied w/this.
actually it just ran out of memory with the volume of searching going on.
I think I broke it thirty minutes ago looking for a Tears For Fears thread :(
So it’s currently down?
It’s running, but it’s closed. So it’s half back.
you shoulda been there it was really something
i saw a kjb post and i tried to start the metal thread poll nominations thread but i dont think it made it
OK, it made it through the night. Turning it back on to see what happens now.
Right, shifting to Plan B. Will split the sandbox and ILX on to different servers. Have cranked memory to the max on this instance for ILX now.
crank dat
better do somethin i’m not gonna put up with this for much longer tbh
srsly, is some stupid shit, this. apols.
:D
better do somethin i’m not gonna put up with this for much longer tbh
― twice banned gabbage is death (p much resigned to deems), Friday, 23 December 2011 18:03 (7 minutes ago) Bookmark Permalink
lol
thx stet, like a family member slipping away this is just reminding me how much i love & rely on ILX so thanks for striving to provide it
Yeah, am just watching the memory tank steadily when ILX is up. There’s something not right here. Going to need to investigate.
Have closed ILX for now
what setup did you decide to go with
not amazon looks like, no
did i break it stet?
My sense is that, when ILX is reopened with the intention of long term stability, that about 3 or 4 days’ worth of the pre-crash threads ought to be displayed to anyone who arrives there. This should cut down considerably on the new arrivals bashing the search function.
would it work if it was open but if jimmy riddle was banned? if you open it real quick the first 51 in can help out
maybe we should just not do ilm could be a solution
haha
i didnt search anything. I just started 2 threads
and were they really worth it
Yeah this is still on Amazon. The setup is largely the same as before — mysql, tomcat and apache — but it’s somehow consuming massive amounts of memory. It’s not the actual caches, they’re staying about 500mb (which is what I’d expect), but I am seeing free memory steadily dwindle to nowt.
There’s two big differences from the old box — one is that we’re running on the OpenJRE instead of the Oracle one (we used to be on the Sun JRE), and the other is that the recreated Lucene search index is now a massive 4gb file (it was previously chunks built up over time).
I am first of all going to try disabling the search and see how it goes with that, and failing that will shift to the Oracle JRE.
It’s hugely annoying that the next size of server up is 8gb of memory and $150/month but if that’s what it takes so be it.
as a hilarious behavioural experiement can we try an ‘essential ilx’ where people only try to post things that are worthwhile and important
^ lol
^rite
well one was nominations thread for metal poll
lol
as a hilarious behavioural experiement can we try an ‘essential ilx’ where people only try to post things that are worthwhile and important
― Never translate German (schlump), Friday, 23 December 2011 18:29 (Yesterday) Bookmark Permalink
ilf reduced to the gareth bale v. theo walcott thread
ilm reduced to rolling gun sounds and rolling r&b thread and maybe something metal and steely dan
ile just that thread about trucker hats on couches and any thread surmounter ever started
just let me know if we won’t be back by the Oscars
^^ itchy thread-trigger finger
i seriously miss ilxor
I didn’t even know there was a Miss Ilxor contest!
We’re gonna bring a case of wine
Hey, let’s go mess and fool around
You know, like we used to
thanks for everything stet
my new year resolution is: i want my ilx
+1
Thanks for everything.
Thanks for getting ILX up again before the Oz Open tennis, and for all your hard work generally.
Will all the old threads from the old site ever be accessible? Sorry if this ? has already come up....
They are OK, they are in cryogenic freeze next to Ted Williams’s head
Have tweaked this and fiddled that and turned on ILX again. It looks a lot healthier this time, so here’s hoping. If it conks again, will need to get the big guns in.
Sweeet, so far so good. Thanks, stet!
Dumb question, but how do I set zing back to ilx (not sandbox)?
Just change the instance setting to “ILX”
ilxor.com is still redirecting to the sandbox btw. direct link http://www.ilxor.com/ILX/index.jsp
Yes — want to keep it that way. I want to add load back slowly so I can be sure it’s working. Right now it seems OK, and has been up longer than any previous attempt, but it’s a quiet time generally.
ok cool.
got a tomcat error (which i didn’t save sorry) when i logged in, but i manually visited /ILX after that and i was logged in ok.
I never reset zing to sandbox for that very reason
Also had trouble over there posting from zing for some unknown reason
It Is A Mystery
OK, problem was I had logged myself off fooling around with the settings.
THANK YOU!
are we back now?
are we back now?
YES
so old skool ilx is kinda slow right now?
seems fine to me
Slow for me too — 3–4 seconds for SNA page load. But it’s better than it was a couple of hours ago.
As soon as I say that, now it’s loading instantly.
yah its been intermittently slow for me
slow for me,as has sandbox
I no longer even have a working login for “real” ILX—can someone who’s managed to log in pls be a dear and tell me if it’s still got SB before I bother going to the trouble of reregistering? I’ve enjoyed it here but I’m not going back to that stress.
Yeah, the new ILX does seem to be quite slow. After I clicked “login”, it took half a minute before it actually logged in.
FT, regular ILX hasn’t changed at all, and so, yes, does still have SB.
OK, thanks for letting me know.
yeah, prohibitively slow in both spaces now.
fundraiser
fundraiser.