Should have called it “Black ICE”.
Definitely falls under the category of a Trap ICE card.
I have no idea why the makers of LLM crawlers think it’s a good idea to ignore bot rules. The rules are there for a reason and the reasons are often more complex than “well, we just don’t want you to do that”. They’re usually more like “why would you even do that?”
Ultimately you have to trust what the site owners say. The reason why, say, your favourite search engine returns the relevant Wikipedia pages and not bazillion random old page revisions from ages ago is that Wikipedia said “please crawl the most recent versions using canonical page names, and do not follow the links to the technical pages (including history)”. Again: Why would anyone index those?
Because you are coming from the perspective of a reasonable person
These people are billionaires who expect to get everything for free. Rules are for the plebs, just take it already
People complain about AI possibly being unreliable, then actively root for things that are designed to make them unreliable.
I find this amusing, had a conversation with an older relative who asked about AI because I am “the computer guy” he knows. Explained basically how I understand LLMs to operate, that they are pattern matching to guess what the next token should be based on a statistical probability. Explained that they sometimes hallucinate, or go of on wild tangents due to this and that they can be really good at aping and regurgitating things but there is no understanding simply respinning fragments to try to generate a response that pleases the asker.
He observed, “oh we are creating computer religions, just without the practical aspects of having to operate in the mundane world that have to exist before a real religion can get started. That’s good, religions that have become untethered from day to day practical life have never caused problems for anyone.”
Which I found scarily insightful.
Oh good.
now I can add digital jihad by hallucinating AI to the list of my existential terrors.
Thank your relative for me.
Here’s the key distinction:
This only makes AI models unreliable if they ignore “don’t scrape my site” requests. If they respect the requests of the sites they’re profiting from using the data from, then there’s no issue.
People want AI models to not be unreliable, but they also want them to operate with integrity in the first place, and not profit from people’s work who explicitly opt-out their work from training.
I’m a person.
I dont want AI, period.
We cant even handle humans going psycho. Last thing I want is an AI losing its shit due from being overworked producing goblin tentacle porn and going full skynet judgement day.
Got enough on my plate dealing with a semi-sentient olestra stain trying to recreate the third reich, as is.
We cant even handle humans going psycho. Last thing I want is an AI losing its shit due from being overworked producing goblin tentacle porn and going full skynet judgement day.
That is simply not how “AI” models today are structured, and that is entirely a fabrication based on science fiction related media.
The series of matrix multiplication problems that an LLM is, and runs the tokens from a query through does not have the capability to be overworked, to know if it’s been used before (outside of its context window, which itself is just previous stored tokens added to the math problem), to change itself, or to arbitrarily access any system resources.
This will only make models of bad actors who don’t follow the rules worse quality. You want to sell a good quality AI model trained on real content instead of other misleading AI output? Just follow the rules ;)
Doesn’t sound too bad to me.
Maybe it will learn discretion and what sarcasm are instead of being a front loaded google search of 90% ads and 10% forums. It has no way of knowing if what it’s copy pasting is full of shit.
i mean this is just designed to thwart ai bots that refuse to follow robots.txt rules of people who specifically blocked them.
I’m imagining a sci-fi spin on this where AI generators are used to keep AI crawlers in a loop, and they accidentally end up creating some unique AI culture or relationship in the process.
I guess this is what the first iteration of the Blackwall looks like.
Gotta say “AI Labyrinth” sounds almost as cool.
“I used the AI to destroy the AI”
And consumed the power output of a medium country to do it.
Yeah, great job! 👍
We truly are getting dumber as a species. We’re facing climate change but running some of the most power hungry processers in the world to spit out cooking recipes and homework answers for millions of people. All to better collect their data to sell products to them that will distract them from the climate disaster our corporations have caused. It’s really fun to watch if it wasn’t so sad.
We had to kill the internet, to save the internet.
We have to kill the Internet, to save humanity.
Surprised at the level of negativity here. Having had my sites repeatedly DDOSed offline by Claudebot and others scraping the same damned thing over and over again, thousands of times a second, I welcome any measures to help.
I think the negativity is around the unfortunate fact that solutions like this shouldn’t be necessary.
Burning 29 acres of rainforest a day to do nothing
Bitcoin?
Imagine how much power is wasted on this unfortunate necessity.
Now imagine how much power will be wasted circumventing it.
Fucking clown world we live in
On on hand, yes. On the other…imagine frustration of management of companies making and selling AI services. This is such a sweet thing to imagine.
My dude, they’ll literally sell services to both sides of the market.
I…uh…frick.
I just want to keep using uncensored AI that answers my questions. Why is this a good thing?
Because it only harms bots that ignore the “no crawl” directive, so your AI remains uncensored.
Good I ignore that too. I want to towards a world where information is shared. I can get behind the
Get behind the what?
Perhaps an AI crawler crashed Melvin’s machine halfway through the reply, denying that information to everyone else!
That’s not what the
no followcommand meansdon’t worry, information is still shared. but with people. not with capitalist pigs
Because it’s not AI, it’s LLMs, and all LLMs do is guess what word most likely comes next in a sentence. That’s why they are terrible at answering questions and do things like suggest adding glue to the cheese on your pizza because somewhere in the training data some idiot said that.
The training data for LLMs come from the internet, and the internet is full of idiots.
LLM is a subset of AI
That’s what I do too with less accuracy and knowledge. I don’t get why I have to hate this. Feels like a bunch of cavemen telling me to hate fire because it might burn the food
Considering how many false positives Cloudflare serves I see nothing but misery coming from this.
Lol I work in healthcare and Cloudflare regularly blocks incoming electronic orders because the clinical notes “resemble” SQL injection. Nurses type all sorts of random stuff in their notes so there’s no managing that. Drives me insane!
In terms of Lemmy instances, if your instance is behind cloudflare and you turn on AI protection, federation breaks. So their tools are not very helpful for fighting the AI scraping.
Can’t you configure exceptions for behaviours?
I’m not sure what can be done at the free tier. There is a switch to turn on AI not blocking, and it breaks federation.
You can’t whitelist domains because federation could come from and domain. Maybe you could somehow whitelist /inbox for the ActivityPub communication, but I’m not sure how to do that in Cloudflare.
And soon, the already AI-flooded net will be filled with so much nonsense that it becomes impossible for anyone to get some real work done. Sigh.
Some of us are only here to crank hog.
AROOO!
You have Thirteen hours in which to solve the labyrinth before your baby AI becomes one of us, forever.
While AI David Bowie sings you rock lullabies.
I’m glad we’re burning the forests even faster in the name of identity politics.
Do you identify as retarded?
What has this anything to do with identity politics?
Don’t feed the troll
So the web is a corporate war zone now and you can choose feudal protection or being attacked from all sides. What a time to be alive.
There is also the corpo verified id route. In order to avoid the onslaught of AI bots and all that comes with them you’ll need to sacrifice freedom, anonymity, and privacy like a good little peasant to prove you aren’t a bot… and so will everyone else. You’ll likely be forced to deal with whatever AI bots are forced upon you while within the walls but better an enemy you know I guess?
Not exactly how I expected the AI wars to go, but I guess since we’re in a cyberpunk world, we take what we get
Next step is an AI that detects AI labyrinth.
It gets trained on labyrinths generated by another AI.
So you have an AI generating labyrinths to train an AI to detect labyrinths which are generated by another AI so that your original AI crawler doesn’t get lost.
It’s gonna be AI all the way down.
All the while each AI costs more power than a million human beings to run, and the world burns down around us.
This is the great filter.
Why isn’t there detectable life out there? They all do the same thing we’re doing. Undone by greed.
The same way they justify cutting benefits for the disabled to balance budgets instead of putting taxes on the rich or just not giving them bailouts, they will justify cutting power to you before a data centre that’s 10 corporate AIs all fighting each other, unless we as a people stand up and actually demand change.
Vote Blue No Matter Who
Any Democrat is Better than Any Republican
Plenty of Democrats are voting to put trump nominees in office, plenty are voting on partisan spending bills. The CR vote should tip you off that any democrat is not better than any republican… half of them are complicit too. 10 Senate Dems just financed this authoritarian takeover.
Not a single Democrat voted to confirm Hegseth and 3 Republicans also didnt but he still got confirmed.
Every single Democrat was present and voted no for the Budget which passed the House and it still passed.
Even if 10 dems voted not to shutdown government and enter congressional recess, the CR only exists because Republicans wrote it and won’t compromise.
Any Democrat is Better than Any Republican.
Scheumer rubberstamped autocracy by not filibustering the CR. I think anyone who protects the constitution and their constituents is better than someone who doesn’t. Not that any repuclicans fit the bill, but its not like we can just trust any old democrat. Look at Gavin Newsome sliding to the right to maintain power. That the kinda dems we want?
No
Then you will get Republicans
Blue no matter who is precisely how we got Trump.
In my country blue is conservatives… But I agree with the sentiment! It worked for California, it can work for your whole country, let the Dems stop fearing they’ll lose elections, give them comfortable margins and then massively support progressives who can bring in the good stuff, they won’t have a chance if the party core thinks the very future of elections is on the line, but if they think they’ll likely win anyway, you might just be able to push through a progressive candidate and end the Neoliberal decay.
To be fair, California is kind of dysfunctional and constantly trips over its own regulations when trying to get anything built. For instance, needing excessive environmental impact review for things like trains that will obviously help the environment, or limiting ferry boats crossing the bay to protect the environment even though it likely results in more people driving instead.
It’s so rare to get positive reinforcement like this, these days. Thank you.
LLMs tend to be really bad at detecting AI generated content. I can’t imagine specialized models are much better. For the crawler, it’s also exponentially more expensive and more human work, and must be replicated for every crawler since they’re so freaking secretive.
I think the hosts win here.
Yeah that’s how we achieve singularity.
