edshot99/eBooru
1
0
Fork 0
forked from e621ng/e621ng
Code Issues 11 Releases 1 Activity
11,473 Commits 3 Branches 60 Tags 67 MiB
master2
Commit Graph

141 Commits

Author SHA1 Message Date
Earlopain
c60ee96133
[Posts] Fix an error during uploads for 404/etc. 
This line wasn't properly updated for faraday
 2024-05-07 16:47:57 +02:00
Earlopain
d01c8c131d
[Misc] Replace httparty with faraday 
We already pull in faraday from opensearch-ruby.
httparty hasn't had a release in a while and it printing warnings on ruby 3.3
 2024-04-27 23:01:10 +02:00
Earlopain
75ea36b700
[Cleanup] Remove unused size method from file class 2024-04-26 15:58:18 +02:00
Earlopain
fc7d84affd
[RuboCop] Enable Style/FrozenStringLiteralComment 
This reduces allocations on the posts page by about 5%, from basic testing
 2024-02-25 18:15:55 +01:00
Earlopain
7dcaf81979
[Uploads] Don't follow redirects to non-whitelisted urls 
Also improves the existing tests a bit by being checking the error message
 2023-10-20 21:00:32 +02:00
Earlopain
776866b873
[Cleanup] Move blocked ip checking out of config 
These three used methods do basically the same, except more correct.
fc00::/7 are considered private while here only fd00::/8 was checked
 2023-10-20 19:58:10 +02:00
Earlopain
cc6e18797a
[Uploads] Correctly follow redirects 
Some sites serve urls that redirect by default. This would append the
"You are being redirected" html to the file being downloaded after that
and subsequently report that text/html is not a valid format
 2023-10-15 13:52:38 +02:00
Earlopain
7adac7b250
[Gems] Bump rubocop/regenerate todo 
987 files inspected, 6509 offenses detected, 6224 offenses autocorrectable
 2023-08-01 19:32:24 +02:00
Earlopain
9ae9fa51c0
[Posts] Fix uploading of urls with square brackets 
Mostly encountered on furaffinity.
Since this is now also a problem for iqdb I though why not try a fix
 2023-04-24 21:24:48 +02:00
Earlopain
04b18d0852
[Tests] Disallow and stub all requests being made 
Make CloudflareService a module for easier stubbing
 2022-11-25 17:27:38 +01:00
Earlopain
a0b2f4639f
[Cleanup] Remove unused header parameter 
This was set with httparty_options, while the passed parameter got ignored
 2022-03-16 18:39:40 +01:00
Earlopain
2a4f97610e
[Cleanup] Remove more upload attributes 2022-03-16 18:34:23 +01:00
Earlopain
92a0a36589
[Cleanup] Remove upload referer_url 2022-03-16 17:39:58 +01:00
Earlopain
3aa731c408
[Cleanup] Don't return strategy on download 2022-03-16 17:15:45 +01:00
Kira
e685fb87c2 [Tests] Update tests and fix deprecations 2021-02-25 12:08:09 -08:00
Kira
08e08b18b4 Add whitelist to upload services and file downloaders 2019-02-07 10:05:50 -08:00
evazion
7524d52276 Fix #3985: Uploads: 405 Method Not Allowed. 2018-11-11 20:18:21 -06:00
r888888888
192ac128cd add additional meta tags for seo 2018-10-20 17:15:29 -07:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498). 
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
 2018-09-19 20:11:53 -05:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498). 
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
 2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498). 
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
 2018-09-18 12:16:27 -05:00
evazion
488b1fd994 Downloads::File: memoize strategy. 
Also remove unused options param from constructor
 2018-09-18 10:22:15 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload. 
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
 2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic. 
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
 2018-09-18 09:44:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
e620abad08 add checks for width=640 and width=500h in tumblr rewriter (#3732) 2018-07-24 16:42:30 -07:00
Albert Yi
f6440ca70d fix upload preprocessing for pixiv posts 
fixes #3782
 2018-07-23 16:24:31 -07:00
Albert Yi
f9b63702bf support artist-path urls in deviant art rewrite strategy (#3771) 2018-07-09 16:47:59 -07:00
Albert Yi
01250faaac temporary fix for nil urls 2018-05-25 09:39:59 -07:00
Albert Yi
23bef55632
Merge pull request #3700 from evazion/fix-3659 
Fix uploads getting stuck in 'processing' state (fix #3659)
 2018-05-07 17:36:03 -07:00
evazion
181a906766 Fix #3695: Bookmarklet breaks on pixiv fanbox direct image links. 2018-05-05 12:21:11 -05:00
evazion
c584ca5b19 Fix uploads getting stuck in 'processing' state (fix #3659). 
Bug: if an upload timed out while downloading the file, Upload#process!
would catch the error and attempt to retry, but since the upload was
already in the 'processing' state, on the second try `process!` would
bail out immediately and leave the upload stuck in the 'processing' state.

Fix: remove the retry logic from Upload#process!. Let Downloads::File#download!
(which had its own retry logic) handle it instead.
 2018-05-05 11:42:40 -05:00
evazion
c76463f34d uploads: use storage manager to distribute files. 
Refactors the upload process to pass around temp files, rather than
passing around file paths and directly writing output to the local
filesystem. This way we can pass the storage manager the preview /
sample / original temp files, so it can deal with storage itself.

* Change Download::File#download! to return a temp file.

* Change DanbooruImageResizer and PixivUgoiraConverter to accept/return
  temp files instead of file paths.

* Change Upload#generate_resizes to return temp files for previews and samples.

* Change Upload#generate_resizes to generate ugoira .webm samples
  synchronously instead of asynchronously.
 2018-03-20 19:49:06 -05:00
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed. 
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
 2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
e48b75c261 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-24 11:23:27 -06:00
Albert Yi
d8340f83db
Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
1eff41a6f9 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-17 11:11:50 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images. 
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
 2017-11-21 14:40:39 -06:00
evazion
fa22e419af Fix #3385: Bookmarklet sets wrong source for Tumblr posts 
Due to missing || operators, for Pawoo/Tumblr/Artstation uploads the
source wasn't changed from the direct image URL to the html page URL
after upload.
 2017-11-18 15:44:16 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
evazion
96fddc5bec Fix #3234: Incorrect filesize on upload page. 2017-07-22 00:34:30 -05:00
evazion
8125d8e796 Set the default User-Agent sent for outgoing http requests. 
* Add a `Danbooru.config.http_headers` config option.

* Refactor various places to use this option instead of setting the user
  agent manually.
 2017-07-19 20:25:27 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
5066f75d8c Fix Net::HTTPOK#read_body called twice error during upload (#3135) 
Fixes this exception:

    An error occurred: error: IOError - Net::HTTPOK#read_body called twice

    /home/admin/.rbenv/versions/2.3.4/lib/ruby/2.3.0/net/http/response.rb:196:in `read_body'
    /home/admin/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/httparty-0.15.5/lib/httparty/response.rb:91:in `method_missing'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:43:in `block (2 levels) in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `open'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `block in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb💯in `http_get_streaming'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:39:in `download!'
    /home/admin/src/danbooru/app/models/upload.rb:408:in `download_from_source'
    /home/admin/src/danbooru/app/models/upload.rb:113:in `block in process_upload'
 2017-07-13 15:41:36 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00