Is there a way to block ai scrapers from accessing the site? That’s kind of gross, worse than ad scrapers even.
Hey N, well no way to block them in a sense, strap yourself in... They make themselves look like normal web requests, because if there were some clear marker it were an AI bot, people would just block them. What a lot of work here has been to do is to at least cope with that situation as best I can.
Traffic is placed in different buckets:
Anonymous traffic
Logged in users
The front page
People trying to log in
Also anonymous API requests (computer to computer messages)—these are just blocked
And within that there are two controls
How many concurrent requests you are allowed (i.e. requests at pretty well exactly the same time), and
How many requests in a thirty second period you are allowed
The numbers are unimportant and I don’t know what they’ll actually be yet; they’ll require tweaking, but:
Anonymous users (AI etc.) get a certain amount of computer, but it’s capped at a certain point (based on the two aforementioned criteria)
People trying to log in get certain amount of computer—quite small, as it doesn’t require much, but it’s ring-fenced from the anonymous user stuff so it’s not overwhelmed—it’s so real users are able to log in during an ‘attack’
The front page also gets it’s own allowance to make sure people can find the site, even if it’s ‘under attack’
Logged in users get unlimited access (i.e. the rest of the available compute)
So you can’t prevent this stuff happening, but if this works the way I intended, then it should mean that even under attack, logged in (and trying to log in / front page) users can still operate as if nothing is happening.
I have tested it at 5–10,000 transactions per second on my laptop, depending on the type of transaction. By way of comparison, the existing site managed 400 transactions per second when I last tested it, although that was years ago.
Thanks,
Lemmy.
Oh and if an AI thing tried to register as a user and do it that way, I haven’t done this yet, but it’s quite easy to deal with, because real users don’t throw thousands of requests per second at the machine, so I would just cap logged in users at something way above real users generate, but that would stop AI bots easily—basically at that point if they’re logged in, then you can identify them.
The live notifications of new posts makes me fantasize about a Keith is typing... sort of notification so you know who’s imminently about to post also.
Sorry, I draw the line about fantasizing about Keith.
Wide view seems to work! Thank you Keith
i tried my best to be open to the new settings, and i bet a lot of people will enjoy them, but yes, for me the
wide view + metadata below post
is very good, immediately makes things easier (for me), appreciate the option!
I can see that the wide view has been tweaked again, thanks for that. Wide + metadata to the left was what I was after. Thanks.
No bother.
this is all very exciting. i think in general my feeling is that nothing is particularly wrong with the current site so any changes should be incremental rather than revolutionary
maybe i’ve missed it but where’s “site new answers”? that view and my bookmarks view (keyed to the POST i bookmarked rather than just the thread in toto) are easily, far and away, the two most common ways i access threads, and i suspect for others as well
i find it complicated to look to the left of a post to see the poster. maybe i’d get used to it, idk. but i wonder if vertically centering the name and time of post means that your eye doesn’t have a reliable place to scan—the spatial relationship of metadata to post data is constantly varying, rather than just “above” or “below” or “at the top”
anyway i can appreciate all the thought and time that you’ve put in keith. have you ever seen “Discourse”? it’s a system that has most of what you’re going for here. i wonder if we wanted to have such a huge change to the site if it might be easier and more sustainable longterm to just go with a discourse deployment?
that said i’d be in favour of NOT doing this, i like our roll-your-own attitude. but for me i’d like to keep the simplicity of the current site. i like how there are still strong UX links with the original greenspun site. there’s a DNA there. just with maybe an upgrade to polls, or the ability to notify. sorry, i know the default position of like 99% of habitual users of internet services is “i hate change” and i guess i’m falling into that camp. but i really have very few issues or problems with ilx as it is
Hi Tracer! I hope you are well. Thanks for your comments…
First perhaps a bit of context… When you say there’s nothing wrong with it, you’re meaning in a functional sense. There’s a lot wrong in a non-functional sense. It’s been being propped up for some time now (in fact, it seemed to fall over for a bit on Friday nigh) and there are fundamental issues that need to be resolved. I appreciate people don’t see this stuff, so won’t be aware of it, but it was analogous to twenty years ago, where no-one had an issue with the existing side, except that it simply stopped working for weeks on end, and was being repeatedly hacked by people, due to lax security. External environmental conditions change over time, and stuff that was suitable X years ago becomes unsuitable over time, so something needs to happen. That said, I assume you do actually understand this, given your mention of Discourse, so I’ll come to that
Site New Answers, I called ‘Recent Activity’ for two reasons: I think it’s clearer, because most comments are not answers, and it’s a bit shorter, and practically speaking it’s a tough job fitting everything on that menu. Most people seem to have received that well, but I’m also not wedded to it if people agree on something else
There are a bunch of options that cover I think pretty much what you’re asking for. I think if you go into the settings and play around with (a) The ‘modes’ (I know they’re terribly explained, but if you want the name at the top, try mode 4) and/or (b) the experimental settings, which let you shift it around in multiple other ways. In fact, if you don’t like change, you can broadly put it back the way it is now
Re: ‘Discourse’, no you mentioning it is the first I’ve heard of it. Stet and I have said many times over the years that if anyone, or any group of people, want to take on this and build their own one, or indeed, migrate to an off-the-shelf package, and if that’s what the community wants, then we are 100% fine with that… The only real bar to clear is that someone needs to volunteer to do it (well, and Discourse needs to give you some route to migrating stuff, I guess)
Re: your points about links to the original site, and I will try not to remember twenty years ago when lots of people told me pretty well the opposite, I have tried to maintain those links. I would appreciate it if you were to mess around with (a) the aforementioned options and (b) the functionality, because pretty well all of it is there. There’s even a blog view that I didn’t make public because I found it difficult to squeeze on a menu! (https://sandbox.ilxor.com/blogview), so you’ll be the first to see that. Never particularly understood it myself, but some people seem to like it. It’s a good example really, because it has a board selector, where the old one did not—I think that’s an improvement; same is true of random threads and unanswered threads. What’s been added largely comes from conversations over the years—obviously you know there are a lot of us who know each other well, and the ideas thread. Some stuff I added because I thought it was a good idea (like I did for Site New Answers and Bookmarks twenty years ago)
Re: your point about roughly ‘the existing site, but with an upgrade to polls’, could you take a look at my response to Sleeve (https://sandbox.ilxor.com/message/15/47/198); in particular, the last paragraph. It’s doesn’t really affect your point, which is your view, but just to understand the challenges in general. Cheers!
If you want some idea of the background around addressing one part of the ‘non-functional’ issues, I replied upthread about some of the stuff that’s been done to at least control the AI scrapers that continually hit the site:
Rodice Andelia Olsun: Is there a way to block ai scrapers from accessing the site? That’s kind of gross, worse than ad scrapers even.
Hey N, well no way to block them in a sense, strap yourself in... They make themselves look like normal web requests, because if there were some clear marker it were an AI bot, people would just block them. What a lot of work here has been to do is to at least cope with that situation as best I can.
Traffic is placed in different buckets:
- Anonymous traffic
- Logged in users
- The front page
- People trying to log in
- Also anonymous API requests (computer to computer messages)—these are just blocked
And within that there are two controls
- How many concurrent requests you are allowed (i.e. requests at pretty well exactly the same time), and
- How many requests in a thirty second period you are allowed
The numbers are unimportant and I don’t know what they’ll actually be yet; they’ll require tweaking, but:
- Anonymous users (AI etc.) get a certain amount of computer, but it’s… View message
Although TBH, I’m sure you’re not interested!
Oh and I should say that despite building all the functions, there are ways in which I built some of them that block unintended consequences of how I did it twenty years ago, e.g. you can’t bookmark an older post at the moment. That’s simply because I didn’t intend for it to be used that way back then, but people have. This is totally cool, and I’m doing some work right now, in John Lewis café to try and reinstate it.
SNA is now “Recent Activity,” but does not have bookmarks. I agree that it ought to—that’s been such a smart one-stop shop for all purposes. Catching up on threads you’re already following, or seeing if there’s some new threads of interest, you can see it all without even scrolling (depending how many bookmarks you’ve got going, of course). The 3.0 “red notifier number” unfortunately separates those things, and also requires two clicks instead of one to get to the thread. Keith, I’d say this is one of the many things you got right twenty years ago, in terms of setting up a simple system of a few clear features that people then were able to incorporate into a wide range of personal habits and practice.
“Recent Activity” is definitely more transparent than “Site New Answers.” I’m guessing the thinking here is probably “what things would most mystify a new arrival,” which I think wouldn’t be a bad thing for us to be talking through as a community. But since “SNA” is woven into the board culture, I wonder if something like “Sitewide New Activity” could remove one small speed-bump there. And if it must have a graphical icon, maybe the icon library has like a “newspaper front page” or something like that? I would just say it’s “Home,” but maybe the list of all boards kinda needs that spot, as the “top” of the site’s hierarchy. Just thinking out loud here.
TBH I’ve hated the front/boards page for a long time, as it just looks like a list of in-jokes and incomprehensible stuff, unless you’ve been here for ever. ILM gets broadly equal billing to HTML playground. In the past, people arrived at ILX mostly because they Googled something and it linked to a thread... From there they could orientate themselves.
Google is going to stop this pretty soon, I understand, in favour of AI summaries of stuff, in part I expect scraped off of sites like these, so that’s not how people are going to find it in future. My personal view is that the front page wants to say much more clearly what this place is. Of course, that requires people to agree on actually what it is.
SNA is now “Recent Activity,” but does not have bookmarks. I agree that it ought to—that’s been such a smart one-stop shop for all purposes.
Hey Dr C, not quite true that it doesn’t have bookmarks, they just don’t work quite the same as they used to in your use case... You don’t need to lobby for it any more, I am literally trying to do it now!
One more point of detail: it was definitely more luck than smart from my point of view!
Keith, sorry, I don’t think I’d seen your most recent run of posts, so I was really just responding to Tracer there!
Hey, no bother!
I have just put something in that I hope will help you. On the post menu, there is now a bookmark icon. It’s just an icon, because I can’t squeeze any more words in without it looking ugly! Maybe I can refine that in time. It lets you bookmark any post on a thread. It’s quite possible stuff might break, so please bear with me, it was a bit more complicated than I initially thought.
Also, I hear you re: the red blob vs a list of bookmarks on the screen. The main logic there was just space, but if people would like an option to put them on the screen, then I can do that as an option. I do need some time to think it through. Another reason is that I’d like it to live update, but it likely isn’t practical, some people have 10K bookmarks and huge numbers of them unread, and I can’t have the server using that kind of bandwidth on a ‘transmission’—it’s that kind of stuff I need to think through.
Just tested mid-thread bookmarking and it seems to be a-ok! Nice, thank you!
No bother, have fun!
And that makes sense about the potential load of bookmarks on the page. By live updating, do you mean like, if you had the page open, things would be switching over from “7 new posts” to “8 new posts” and swapping in new threads that got new posts?
Thank you for wide view, Keith—I love it! I’m trying it out with poster info on the left and that seems to work fine for me, and I think I might prefer keeping the info there rather than going back to old-ilx under the post.
Yeah, I mean I can also see that there are quite a number of issues with it notwithstanding the server load thing—it could be awful. That’s what I mean about thinking it through for a while.
No bother, Emily!
Yeah, I’d spare yourself the headache! I don’t think anyone really needs it to do that, and it might actually be harder to use that way (information overload, things moving around and popping into view).
I’ll bear it in mind. Someone else mentioned it, so let’s just see how things play out. Others might be keen.
took me a few to figure out how the shortcuts to load bookmarked threads 1–9 work, but seems to work flawlessly here—thanks for preserving that functionality
PBKR, I put in Divider 17, which is a longer horizontal line. I tried to make it look nice, but let me know if you want it tweaked. I also fixed a few bugs with wide mode etc. so that stuff is properly horizontally centered.
sadbonx, I put in an option to always show post menu. See if that helps.
I think I’m with you on that!
I also fixed a few bugs with wide mode etc. so that stuff is properly horizontally centered. (Keith)
I noticed this—wasn’t going to complain but it’s much more aesthetically satisfying now, thank you!
Thank you, Keith. This looks good to me. I will sit with it a while and let you know if I have any further thoughts.
wide mode is gopping imo. if there’s going to be an implementation which aims to mimic ILX 2, it can’t be an approximation. it has to be 1:1. my tier list is S+ tier ILX 2 old.css, C tier ILX 3 default, D tier ILX 3 wide mode
this is correct ^
What is ‘gopping’?
TBH I’ve hated the front/boards page for a long time, as it just looks like a list of in-jokes and incomprehensible stuff, unless you’ve been here for ever. ILM gets broadly equal billing to HTML playground. In the past, people arrived at ILX mostly because they Googled something and it linked to a thread... From there they could orientate themselves.
Google is going to stop this pretty soon, I…
This assumes that one of the goals of the site is to attract new users. Do you think most regular posters would agree with that view?
Agree, and it might not be, I don’t know. Intuitively, in my opinion, it would seem a pity that if someone shows up and they don’t engage simply because they don’t know what it is, but if people actively don’t want new users, then sure, that’s a totally different thing, and fine if that’s the view.
What I do know is that its usage for the last 20 years has declined on average 8% each year, so it doesn’t stand still—roughly an 80% drop between 2005 and 2025.
Just out of interest, are you actually against attracting new users to the board?
No, I’m not against it on principle. But I guess I chafe at decisions being made under a presumption of a goal that I don’t think has been adequately proven to be the consensus of the community. I guess I also worry that changes aimed at attracting new users could have an unwanted side effect of alienating current users and wonder if perhaps retaining users should be considered more of a priority for this particular redesign.
Anyway, going to back to regular ILX now.
I think that’s a good point, re: retention. TBH the new stuff wasn’t based on attracting new users, retaining them or otherwise, it was primarily in making all the same pieces of functionality that already exist, plus a bunch that’s been requested over the years—I just asked that question upthread.