/b/ - Random » Thread #22891

[33 / 5 / 1]

191KiB, 265x400, bouncing wholesome soyjak heart.gif

Chud Tue 26 Oct 2021 23:01:12 No.22891 View View Original Report

Quoted By: >>22892

I finished programming a scraper for the 'ru, if someone needs to get a lot of jakkies with minimal effort this could be a good way.
https://github.com/somegenericdev/Shimmie2Scraper/releases/tag/v1.0.0

View Same Similar Google ImgOps iqdb SauceNAO 1630256894467.png, 88KiB, 600x800

Chud Tue 26 Oct 2021 23:02:49 No.22892 Report

Quoted By: >>22893 >>22895

>>22891
holy based, great work soybro

Chud Tue 26 Oct 2021 23:04:50 No.22893 Report

>>22892
That smile is creepy

Chud Tue 26 Oct 2021 23:06:39 No.22894 Report

soot pin this to the 'ridge

Chud Tue 26 Oct 2021 23:08:57 No.22895 Report

>>22892
i dont like this jak

Chud Wed 27 Oct 2021 00:59:29 No.22905 Report

Quoted By: >>22907 >>22909

>(((C#)))
Fuck off kike pajeet

Chud Wed 27 Oct 2021 01:05:59 No.22907 Report

>>22905
incel

Chud Wed 27 Oct 2021 01:11:00 No.22909 Report

Quoted By: >>22931

>>22905
it's comfy af though. standard library is huge, syntax is consistent, every major update brings a lot of cool new stuff and nugets are extremely comfy too. are you one of those guys that refuse to use anything other than C?

Chud Wed 27 Oct 2021 02:34:24 No.22910 Report

Quoted By: >>22944

But the tags though

Chud Wed 27 Oct 2021 03:28:48 No.22931 Report

>>22909
Fuck off pajeet

Chud Wed 27 Oct 2021 03:29:36 No.22932 Report

esrel phull sapport sirs

Chud Wed 27 Oct 2021 06:53:24 No.22944 Report

im not a pajeet retards
>>22910
they get saved in the filenames

Chud Wed 27 Oct 2021 10:33:31 No.22948 Report

Quoted By: >>22949

rewrite it in rust

Chud Wed 27 Oct 2021 10:49:44 No.22949 Report

>>22948
this

View Same Similar Google ImgOps iqdb SauceNAO 15948372849212.png, 9.5KiB, 1000x800

Chud Wed 27 Oct 2021 11:28:08 No.22950 Report

Quoted By: >>22951 >>22953 >>22955 >>22997

Just wrote one in 'ash

[code]
#!/bin/bash
BOORU_URL=${1}
if [ -z "${BOORU_URL}" ]; then echo "Usage: ./scraper [BOORU URL]"; exit 1; fi

i=1
while true; do
response=$(curl -f ${BOORU_URL}/post/view/$i 2>/dev/null)
if [ $? -ne 0 ]; then echo "Error, or end of 'jaks"; exit 0; fi

img_url=$(echo "${response}" | grep " main image

echo "${i}: ${BOORU_URL}${img_url}"
wget -q "${BOORU_URL}${img_url}"

sleep 0.25s
i=$((i+1))
done
[/code]

View Same Similar Google ImgOps iqdb SauceNAO ClipboardImage.png, 591KiB, 768x719

Chud Wed 27 Oct 2021 11:53:26 No.22951 Report

Quoted By: >>22952

>rewrite it in rust
>>22950
baste
do we have a page on the wiki for programming resources related to 'jaking already?

Chud Wed 27 Oct 2021 11:54:41 No.22952 Report

Quoted By: >>22954

>>22951
There are these pages
https://wiki.soyjak.party/index.php?title=Artificial_Soyduel
https://wiki.soyjak.party/index.php?title=User_JS

Chud Wed 27 Oct 2021 12:18:57 No.22953 Report

>>22950
boutta 'ash in your skull with these HAYNDS

Chud Wed 27 Oct 2021 13:05:37 No.22954 Report

>>22952
why is the ‘iki down

Chud Wed 27 Oct 2021 13:21:08 No.22955 Report

Quoted By: >>22992 >>22997

>>22950

- u dont need ${...}. it's enough to write "$BOORU_URL"
- I prefer: [ ...] || { echo ...; exit 1; }
- grep "

" relies on on line "img" tag, but if it will be changed in the future then your code fails
- u already use curl, why wget? It would be better to use either curl or wget
- better to rewrite it to Python/Ruby and similar

View Same Similar Google ImgOps iqdb SauceNAO 1632675876071.jpg, 91KiB, 656x843

Chud Wed 27 Oct 2021 13:24:37 No.22956 Report

Quoted By: >>22991

>>22955
>py*hon

Chud Wed 27 Oct 2021 14:38:24 No.22989 Report

Quoted By: >>22995

>Run it with sudo (important!)
doesn't seem to be necessary

Chud Wed 27 Oct 2021 14:42:21 No.22990 Report

Quoted By: >>22995

how does it deal with duplicate filenames?

Chud Wed 27 Oct 2021 14:42:26 No.22991 Report

>>22956
nas

Chud Wed 27 Oct 2021 14:59:48 No.22992 Report

Quoted By: >>22997 >>23002

>>22955
>u dont need ${...}. it's enough to write "$BOORU_URL"
I know, but I find it easier and cleaner to just the the ${} syntax everywhere
>I prefer: [ ...] || { echo ...; exit 1; }
Cool trick, thanks
>grep "<img alt=...>" relies on on line "img" tag, but if it will be changed in the future then your code fails
Yeah it's a hacky script. But it took me 25% of the time it would have taken if I was to do it properly in python or something. So I can just use that saved time to spend 30 seconds fixing the script if it ever breaks
>u already use curl, why wget? It would be better to use either curl or wget
Fair point
>better to rewrite it to Python/Ruby and similar
Not really, would just make it take more effort to install and run

Chud Wed 27 Oct 2021 15:16:45 No.22994 Report

Quoted By: >>22997

Actually replacing wget with curl doesn't work because curl doesn't decode html escapes in filenames.
All the names end up like this
>2%20-%20glasses%20smile%20soyjak%20stubble%20template%20variant%3Awholesome_soyjak%20wholesome.png
and there's no way to make it decode them

Chud Wed 27 Oct 2021 15:18:03 No.22995 Report

Quoted By: >>22997

>>22989
i tested it real fast on Ubuntu WSL yesterday and it was giving me some weird error when I launched it without sudo, but if you managed to get it working without it then good for you ig
>>22990
it appends a number, so you'll have tag.jpg, tag (1).jpg, tag (2).jpg and so on.

Chud Wed 27 Oct 2021 15:19:16 No.22996 Report

Quoted By: >>22997

I cleaned up the script a bit though

#!/bin/bash
BOORU_URL=${1}
[ -z "${BOORU_URL}" ] && { echo "Usage: ./scraper [BOORU URL]"; exit 1; }

i=1
while true; do
response=$(curl -f ${BOORU_URL}/post/view/$i 2>/dev/null)
[ $? -ne 0 ] && { echo "Error, or end of 'jaks"; exit 0; }

img_url=$(echo "${response}" | grep " main image

url=${BOORU_URL}${img_url}
echo "${i}: ${url}"
wget -q "${url}"

sleep 0.25s
i=$((i+1))
done

Chud Wed 27 Oct 2021 15:32:02 No.22997 Report

Quoted By: >>23001 >>23015

>>22996
>>22995
>>22994
>>22992
>>22955
>>22950
'eak 'lish

Chud Wed 27 Oct 2021 15:33:05 No.22998 Report

Quoted By: >>23015

it broke for me after reaching a file that surpasses default filename limit on windows (250 characters)

Chud Wed 27 Oct 2021 16:00:03 No.23001 Report

>>22997
'earn 'o 'ode

Chud Wed 27 Oct 2021 16:43:34 No.23002 Report

>>22992
Ok, then
Ship it

Chud Wed 27 Oct 2021 20:42:04 No.23015 Report

>>22998
it was probably because the program wasn't checking the filename's length a second time after appending the index to duplicates. It should be fixed now, try again and let me know if you have other issues.
https://github.com/somegenericdev/Shimmie2Scraper/releases/tag/v1.0.1
>>22997
whats 'lish though

Capcode	All Only Moderator Posts Only Manager Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Embed Type	All Only With YouTube Only With Vocaroos Only With Sounds Only With Flashes
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Post Type	All Only Sticky Threads Only Locked Threads Only Looped Threads Only Opening Posts Only Reply Posts
Data Source	All From archived.st From Log Warehouse From Internet Archive
Results	All Grouped By Threads
Favorites	All Only My Favorites
Order	Latest Posts First Oldest Posts First AI Most Relevant First

On these boards

Your latest searches