[ADMIN] Future hosting of ILX (Part 8080)

Sorry for the info vacuum of the past few days; i have been trying to get things together to work out some clear options. Here's where we are so far:

- The hosts recommend that "basically you should junk your server; it's old and inefficient and the disks are shot". They're offering us a virtual server for the same price which has less CPU but the same amount of memory and disk. It's not a bad deal -- and ILX is unbelievably light on CPU anyway -- but I also think we should get other quotes, so...

- Have asked a number of colo and virtual server places to quote us for enough power to run ILX. The big issue is memory -- the DB is heading towards 4gb now, most of which the code tries to keep in memory. But virtual hosts are usually quite sparing with the memory. Given that ...

- The memory requirement we're at now is already pretty expensive, and the ILX DB has doubled in size since we switched to Keith-code. If it keeps on growing like that we could eventually hit some relatively expensive memory requirements. Changing this setup would be a big chunk of work, and it might not be necessary, but would also dictate a bit what kind of hosting we're looking for. Surprisingly, among the more reasonable big-memory online hosts are Amazon, so:

- Also looking at running on Amazon's server, using their EC2 service. That's what this sandbox is running on right now (on a smaller, free, server -- the full-size one would cost a fair bit more.) There's some real advantages here, as they've also got great tools and monitoring and can do real-time backups and I can spin up a backup "server" in seconds and it's generally great. But:

- it'll take some code changes to run properly here, as there are some things we do which are incompatible with the Amazon setup, and it could also be expensive. That said:

- Virtually all the options are going to cost some cash. When I get the quotes back (day or two) we should know how much those options will cost, but won't really know how much the Amazon setup will be until we try it out for real. At an guess, I think an upper limit for basic running on any of these options would be $100 a month, which is what we pay now for hosting.

So: Right now I think Amazon's my preferred option, although our hosts have given us really personal service so don't want to rule their virtual server offer out either. I think either of them is better than buying or renting another server and just shifting all our eggs from one basket to another.

I'd like to keep investigating Amazon for at least this week. A couple of people have offered to help with the work I need to get done, which is v. appreciated and I've sent them details of what's still required to get proper ILX running on this setup.

If we can get that done, hopefully in a week or two at most, we'll know pretty quickly what the costs are likely to be. Then, depending which option we go with, it may be cap-in-hand time again. Will keep you posted on what's going on.

<3

thx stet!

surely we can raise whatever money we need fairly quickly

thanks for the update!

is sb extra

Brilliant. I'm away during the cap-in-hand period but am more than willing to stump up before or after.

open ended fundraising pls

I will definitely donate to the cause when necessary. I has a jobz now!

Is the DB plate inevitably going to get too big to keep spinning on a stick?

Apols for asking that in a dumb way

great work, thx much stet!

presumably this kinda space will be cheaper in the future?

WmC, I think that was the perfect way to ask.

whatever price he asks, we shall pay

apprec your work, stet.

Wmc: We're being outrun by falling prices for now, but there are other complications -- the bigger it gets the more unwieldy it gets too. Running backups was more of a pain than it needed to be on the old server, for instance.

(In database terms, it's not that huge -- there are databases of hundreds of GBs and more happily chuntering along. Those tend to have more dedicated stewards and need care and watering, tho.)

Thanks, stet. Sorry for being crochety on the other thread. Really been missing/needing ILM of late.

WmC, I think that was the perfect way to ask.

― Making like Melusine (Pyth), Monday, December 5, 2011 3:27 PM (9 minutes ago)

heh, thanks. I keep thinking "would it be possible to reduce the load on the DB by making the first several years of it read-only and sending it off to live quietly with relatives or something?" but really, there's too much value in revisiting and reviving old threads. And having it accessible to read doesn't reduce the load anywhere, I guess. It's like having to lug around a copy of The Riverside Shakespeare all the time instead of just the one play you're reading at the time.

Amazon seems like a pretty sound option IMO

I'm not gonna read all that cause it's long and I'm trying to leave work but I'm happy to donate and thank you stet for your hard work. Also, the sandbox is pretty good for the time being imo.

Thanks stet for handling this, much appreciated.

xp Amazon option seconded.

+1 Amazon. I can't help in any meaningful way other than by injecting funds. William (C), you should buy a Kindle.

"would it be possible to reduce the load on the DB by making the first several years of it read-only and sending it off to live quietly with relatives or something?" but really, there's too much value in revisiting and reviving old threads.

I have been thinking similar things myself. The quick and easy thing to do is make everything that hasn't been updated in say a year just a read-only HTML thread. That'd be super-easy to serve, and we could do it from a tiny server like this one.

But it's also a bit crap -- wouldn't be properly integrated with search, and you couldn't just bump an old thread. I'm trying to see if there isn't something cleverer we could do that would fix those problems -- perhaps the threads live in cold storage as static HTML and then if someone tries to bump one we "defrost" it and insert all its posts into the DB again? Can foresee problems doing that, too.

Either way, by far the simplest option for the immediate future given funding (eg at least another 2/3 years the way things have been going) is just to get a bit more memory.

I can live with the sandbox for a while. As an example of Amazon EC2 performance it's pretty impressive.

^^ yeah, definitely

would our existing host folks be willing to allow us to spin up another sandbox using their virtual solution so we can compare performance?

ah, got it -- those were the same questions and reservations I had from a layman's perspective -- just wasn't sure if I was asking the right questions or the programming equivalent of "can we turn this block of solid granite inside out?"

xp to stet

is sb extra

― Sad Banter (p much resigned to deems), Monday, 5 December 2011 21:19 (34 minutes ago) Bookmark Permalink

this, this should be like carbon tax, we should get a credit & have to buy more

good work stet!, let us know when you need $$$. when your head explodes from the continuous effort dedicated to server maintenance &c we will chip in for replacement parts for that, also

That's probably not a bad idea to ask about. I would guess they'd want to put the existing server out of its misery before they do that though, and I want to keep it zombieing on until I have a live running copy of proper ILX on another server.

(I have a copy running on my Mac, and the DB all checks out, but it's not quite the same)

xp thanks everybody btw <3 u all

I am in favor of a sb tax - we would never need a funding drive

frogbs would keep ilx running for 100 years

stet, there is one way I might be able to help depending on (a) how you go about doing what you intend to go about doing and (b) how busy I am when you get to that point. Bung me an email if you so desire.

Yes, adding to the thank-yous to stet for service above and beyond!

I want to volunteer to help but this is about to become silly crazy busy season for me and I don't want to be the flake who backs out

can't believe you're already flaking dan. sorry everyone - it seems like the orphans won't get into their new home in time for christmas after all.

so unthoughtful, jeez

awesome, adding my thanks here, and hopefully some $ at some point.

hey stet, thanks for all yr work!

fwiw i support using amazon too, best to use the futuristic hosting styles since we r a cutting edge community after all

can't believe dan is being such a huge jerk about this

hey stet, thanks for all yr work!

― your voice of treason, Monday, December 5, 2011 5:23 PM (15 minutes ago)

echoing frogbs here, thanks stet

dan needs to take some time off and think abt his priorities

xpost

http://i.imgur.com/xsUWy.gif

\(^o^)/

\(^o^)/

Thank you, stet! I have no skills to contribute but would love the opportunity to throw money at the problem.

btw in case I sounded too much like a huge wanker I was offering to develop the sort of supporting documents that would be megaboring for an ilx thread.

ditch the db, keep the sb

GOOGLE ADS HAVE WE TREID GOGGLE ADS

Amazon affiliate setup where every word on ILX is a link to a product.

loll yes

My sense is that, when ILX is reopened with the intention of long term stability, that about 3 or 4 days' worth of the pre-crash threads ought to be displayed to anyone who arrives there. This should cut down considerably on the new arrivals bashing the search function.

would it work if it was open but if jimmy riddle was banned? if you open it real quick the first 51 in can help out

maybe we should just not do ilm could be a solution

haha
i didnt search anything. I just started 2 threads

and were they really worth it

Yeah this is still on Amazon. The setup is largely the same as before -- mysql, tomcat and apache -- but it's somehow consuming massive amounts of memory. It's not the actual caches, they're staying about 500mb (which is what I'd expect), but I am seeing free memory steadily dwindle to nowt.

There's two big differences from the old box -- one is that we're running on the OpenJRE instead of the Oracle one (we used to be on the Sun JRE), and the other is that the recreated Lucene search index is now a massive 4gb file (it was previously chunks built up over time).

I am first of all going to try disabling the search and see how it goes with that, and failing that will shift to the Oracle JRE.

It's hugely annoying that the next size of server up is 8gb of memory and $150/month but if that's what it takes so be it.

as a hilarious behavioural experiement can we try an 'essential ilx' where people only try to post things that are worthwhile and important

^ lol

^rite

well one was nominations thread for metal poll

lol

https://www.youtube.com/watch?v=s6pLn_rF4_c

as a hilarious behavioural experiement can we try an 'essential ilx' where people only try to post things that are worthwhile and important

― Never translate German (schlump), Friday, 23 December 2011 18:29 (Yesterday) Bookmark Permalink

ilf reduced to the gareth bale v. theo walcott thread
ilm reduced to rolling gun sounds and rolling r&b thread and maybe something metal and steely dan
ile just that thread about trucker hats on couches and any thread surmounter ever started

just let me know if we won't be back by the Oscars

^^ itchy thread-trigger finger

i seriously miss ilxor

I didn't even know there was a Miss Ilxor contest!

We're gonna bring a case of wine
Hey, let's go mess and fool around
You know, like we used to

thanks for everything stet

my new year resolution is: i want my ilx

+1

Thanks for everything.

Thanks for getting ILX up again before the Oz Open tennis, and for all your hard work generally.

Will all the old threads from the old site ever be accessible? Sorry if this ? has already come up....

They are OK, they are in cryogenic freeze next to Ted Williams's head

Have tweaked this and fiddled that and turned on ILX again. It looks a lot healthier this time, so here's hoping. If it conks again, will need to get the big guns in.

Sweeet, so far so good. Thanks, stet!

Dumb question, but how do I set zing back to ilx (not sandbox)?

Just change the instance setting to "ILX"

ilxor.com is still redirecting to the sandbox btw. direct link http://www.ilxor.com/ILX/index.jsp

Yes -- want to keep it that way. I want to add load back slowly so I can be sure it's working. Right now it seems OK, and has been up longer than any previous attempt, but it's a quiet time generally.

ok cool.

got a tomcat error (which i didn't save sorry) when i logged in, but i manually visited /ILX after that and i was logged in ok.

I never reset zing to sandbox for that very reason

Also had trouble over there posting from zing for some unknown reason

It Is A Mystery

OK, problem was I had logged myself off fooling around with the settings.

THANK YOU!

are we back now?

http://www.ilxor.com/ILX/

are we back now?

YES

so old skool ilx is kinda slow right now?

seems fine to me

Slow for me too -- 3-4 seconds for SNA page load. But it's better than it was a couple of hours ago.

As soon as I say that, now it's loading instantly.

yah its been intermittently slow for me

slow for me,as has sandbox

I no longer even have a working login for "real" ILX - can someone who's managed to log in pls be a dear and tell me if it's still got SB before I bother going to the trouble of reregistering? I've enjoyed it here but I'm not going back to that stress.

Yeah, the new ILX does seem to be quite slow. After I clicked "login", it took half a minute before it actually logged in.

FT, regular ILX hasn't changed at all, and so, yes, does still have SB.

OK, thanks for letting me know.

yeah, prohibitively slow in both spaces now.

fundraiser

fundraiser.