/sci/ - Soyence and Technology » Thread #24271

[20 / 5 / 1]

Chud Sun 07 Jan 2024 20:04:54 No.24271 View View Original Report

>try to use go's http library to access https://soyjak.party/soy/threads.json
>error 403, cloudflare has decided the request is le bad and a captcha must be solved first
>try to access it through curl, wget and python's requests library
>all run into the same issue
>have the idea to include my browser's user-agent in the request header
>this fixes it for curl, wget and python but go is still running into the 403 error
how do i get around this? i've tried adding other headers like accept-language and referer to no avail
it doesn't make sense that it's rejecting all GET requests sent through go's http client no matter what yet everything else just werks once i add a user-agent header

View Same Similar Google ImgOps iqdb SauceNAO ClipboardImage.png, 6.6KiB, 299x63

Chud Sun 07 Jan 2024 20:25:55 No.24272 Report

Quoted By: >>24273

ive had this problem in the past. it didnt involve the sharty but the problem was, essentially, the same, cloudflare breaking my balls when i wanted to scrape stuff.
what i ended up doing was creating a Python microservice I used to route all my HTTP requests through that used this package
https://pypi.org/project/cloudscraper/
I have no idea of what this thing does internally, but it does work.
in pic related you can see how i've used this thing in the microservice

Chud Sun 07 Jan 2024 20:37:58 No.24273 Report

Quoted By: >>24312

>>24272
thanks, i'll give this a try

Chud Sun 07 Jan 2024 20:41:05 No.24274 Report

Quoted By: >>24314

>http
just stop being a contrarian and use python

Chud Mon 08 Jan 2024 14:20:09 No.24312 Report

Quoted By: >>24314

>>24273
did it work?

View Same Similar Google ImgOps iqdb SauceNAO ClipboardImage.png, 129KiB, 573x602

Chud Mon 08 Jan 2024 14:57:46 No.24314 Report

Quoted By: >>24317 >>24322 >>27547

>>24312
the library he linked works great though i realised the microservice approach was overkill for what i'm doing since after getting the json i'm just throwing the data i needed from it straight into a sql database
so i've instead opted to take the advice of >>24274 and just write the scraper part in python since i can access the database it writes to with go code anyway
pic related, /soy/ post ids and urls selected from the database and sorted by date

Chud Mon 08 Jan 2024 15:45:38 No.24317 Report

Quoted By: >>24318

>>24314
why are you scraping html files of threads when vichan already makes .json versions of them?

Chud Mon 08 Jan 2024 15:51:32 No.24318 Report

Quoted By: >>24322

>>24317
that's not what i'm doing at all, i'm downloading threads.json, then downloading the json of every thread in that list
then i use that data to generate a link to every post on the board, even replies (you can't lookup replies through the json api, only threads)

Chud Mon 08 Jan 2024 17:03:49 No.24322 Report

Quoted By: >>24323

>>24314
nice, whatever works for you i guess
i went the microservice way because i dislike python enough that i didnt want to have to deal with it at all lel, but yea it was kinda overkill for my usecase too
are you the same guy who was building a 4stats for the sharty?
>>24318
you actually can look up the replies too. replace ".html" with ".json" and you'll see.

Chud Mon 08 Jan 2024 17:27:46 No.24323 Report

Quoted By: >>24326

>>24322
>are you the same guy who was building a 4stats for the sharty?
yep

>you actually can look up the replies too. replace ".html" with ".json" and you'll see.
that's looking up a thread, which i'm already making heavy use of
the issue isn't getting a thread's replies, it's finding out which thread a reply was posted in
if you try to do that with a post number that's not the OP of a thread you'll get a 404 error because it's a reply and not a thread

threads.json contains the post number, page number and last modified timestamp of every thread on the board
[thread_number].json contains every post in a thread, including both the op and replies

the problem with this that i'm working around by scraping the entire board and building a local index is that there's no API for finding posts by their post numbers that works for replies
the reason i want to be able to look up post numbers and find their threads is because i also want have a page like get watcher's most recent view where you can see the most recent posts on the board including sages

Chud Mon 08 Jan 2024 17:42:38 No.24326 Report

Quoted By: >>24328 >>27225

>>24323
>there's no API for finding posts by their post numbers that works for replies
you can do https://soyjak.party/search.php?search=id%3A24323&board=sci

Chud Mon 08 Jan 2024 18:13:31 No.24328 Report

>>24326
oh cool, didn't realise you could use search.php to search by id
going to stick to my current method though because it's faster and i needed to build a local index to get PPH stats anyway

Chud Sun 14 Apr 2024 10:28:12 No.27223 Report

bump

Chud ## Developer Sun 14 Apr 2024 11:27:26 No.27225 Report

>>24326
I know this is a 3 month old post but scraping via search.php is an excellent way to get it removed.

View Same Similar Google ImgOps iqdb SauceNAO 1711680566534x.gif, 12MiB, 350x640

Chud Sun 14 Apr 2024 19:13:53 No.27232 Report

>>>24326
>I know this is a 3 month old post but scraping via search.php is an excellent way to get it removed.

View Same Similar Google ImgOps iqdb SauceNAO ClipboardImage(7).png, 76KiB, 636x822

Chud Sun 14 Apr 2024 23:18:42 No.27234 Report

>>>24326
>I know this is a 3 month old post but scraping via search.php is an excellent way to get it removed.

Chud Wed 17 Apr 2024 00:52:43 No.27257 Report

Quoted By: >>27280 >>27465

Ever wonder why
these blatant troll
threads that violate
the only rule in the
sticky are never
deleted while the
mod is literally in
these threads all day
deleting posts?
>>100032795 (Dead)

Because the
apple nigger mód
is the OP posting
from his iphone.
He spends 20 hours
a day of his worthless
NEET life posting and
babysitting these falseflag
anti-linux anti-windows
flamewar troll threads,
deleting posts
exposing him as
the OP, and saving
his troll threads
from page 10 after
every 1-4 hours
of no bites.

The only way to purge these shit threads off /g/ is to put a bullet in his head.
Jayy
Louis
lrwin
DoB: 2/2/1983
Age: 41
3004 Nor.folk Dr.
Austin TX 78745
(864) 421-3980
2020 Toyota Camry SE Nightshade VIN# 4T1G11AK8LU913695
jay.irwin@draftfcb.com
jlirwin@gmail.com
thascourge@gmail.com
thascourge@yahoo.com
https://web.archive.org/web/20220323094011/https://twitter.com/invisibro

Relatives:
Larry Richard Irwin
8 Manly Drive, Greenville, SC 29609
(864) 232-2849
DoB: 3/20/1958

Alicia Hilley Irwin
DoB: 8/5/1956
Died at 58 on
April 3, 2015 when
she realized her
son was a faggot.
He then immediately
blew his inheritance
on fag virtue
signaling toys:
https://desuarchive.org/g/thread/50602486

Lawrence Richard Irwin
DoB: 5/19/1985
Cocain addict
shoplifter brother

Leila Alexandra Scogin
DoB: 1/27/1991
Thot halfsister
between his mom and
Richard Gordon Scogin.
TL;DR mom was a whore,
dad was a cuck.
And in one trip to DC,
Rachel Bieder
simultaneously
dumped Jay and
engaged David Woolston
(must have been
going on for a
long time for
it to all happen
in one trip),
making Jay
a cuck as well.
This is why cuck
in all caps is
word banned.

Capcode	All Only Moderator Posts Only Manager Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Embed Type	All Only With YouTube Only With Vocaroos Only With Sounds Only With Flashes
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Post Type	All Only Sticky Threads Only Locked Threads Only Looped Threads Only Opening Posts Only Reply Posts
Data Source	All From archived.st From Log Warehouse From Internet Archive
Results	All Grouped By Threads
Favorites	All Only My Favorites
Order	Latest Posts First Oldest Posts First AI Most Relevant First

On these boards

Your latest searches