Commit Graph

88 Commits

Author SHA1 Message Date
Earlopain
c60ee96133
[Posts] Fix an error during uploads for 404/etc.
This line wasn't properly updated for faraday
2024-05-07 16:47:57 +02:00
Earlopain
d01c8c131d
[Misc] Replace httparty with faraday
We already pull in faraday from opensearch-ruby.
httparty hasn't had a release in a while and it printing warnings on ruby 3.3
2024-04-27 23:01:10 +02:00
Earlopain
75ea36b700
[Cleanup] Remove unused size method from file class 2024-04-26 15:58:18 +02:00
Earlopain
fc7d84affd
[RuboCop] Enable Style/FrozenStringLiteralComment
This reduces allocations on the posts page by about 5%, from basic testing
2024-02-25 18:15:55 +01:00
Earlopain
7dcaf81979
[Uploads] Don't follow redirects to non-whitelisted urls
Also improves the existing tests a bit by being checking the error message
2023-10-20 21:00:32 +02:00
Earlopain
cc6e18797a
[Uploads] Correctly follow redirects
Some sites serve urls that redirect by default. This would append the
"You are being redirected" html to the file being downloaded after that
and subsequently report that text/html is not a valid format
2023-10-15 13:52:38 +02:00
Earlopain
9ae9fa51c0
[Posts] Fix uploading of urls with square brackets
Mostly encountered on furaffinity.
Since this is now also a problem for iqdb I though why not try a fix
2023-04-24 21:24:48 +02:00
Earlopain
00c1dc27e1
[Tests] Use file_fixture/fixture_file_upload 2022-11-25 18:44:36 +01:00
Earlopain
04b18d0852
[Tests] Disallow and stub all requests being made
Make CloudflareService a module for easier stubbing
2022-11-25 17:27:38 +01:00
Earlopain
3740599920
[Tests] Fix a bunch of tests 2022-04-05 18:50:34 +02:00
Earlopain
3aa731c408
[Cleanup] Don't return strategy on download 2022-03-16 17:15:45 +01:00
Earlopain
ac5e8673ca
[Cleanup] Remove most source strategy code 2022-02-19 15:55:26 +01:00
evazion
7524d52276 Fix #3985: Uploads: 405 Method Not Allowed. 2018-11-11 20:18:21 -06:00
evazion
5525bbe1ca nijie: normalize all thumbnail urls (#3919). 2018-09-23 20:08:14 -05:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498).
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
2018-09-19 20:11:53 -05:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498).
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498).
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
2018-09-18 12:16:27 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload.
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic.
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
2018-09-18 09:44:15 -05:00
evazion
96e89cecfb tests: move twitter canonical url test. 2018-09-17 23:27:53 -05:00
evazion
cae78fa8ee moebooru: move tests from unit/downloads to unit/sources. 2018-09-16 21:00:11 -05:00
Albert Yi
266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
evazion
0d7b7e6e85 twitter tests: fix dead tweet. 2018-09-09 14:10:43 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
77854349e5 testing 2018-07-26 18:11:19 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
6839249e9d add logic for persisting pixiv sessions in tests 2018-05-10 13:59:57 -07:00
Albert Yi
5bca31bad1 add retry monkey patch for mechanize 2018-05-09 16:45:03 -07:00
Albert Yi
e23814be92 fix pixiv agent shutdown 2018-05-09 16:13:47 -07:00
Albert Yi
d4a2521eec reset connection for pixiv tests 2018-05-09 14:59:18 -07:00
Albert Yi
99012ff342 fix tests 2018-05-09 11:59:51 -07:00
evazion
181a906766 Fix #3695: Bookmarklet breaks on pixiv fanbox direct image links. 2018-05-05 12:21:11 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
d089be9f8a tests: fix upload tests. 2018-03-20 19:49:58 -05:00
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed.
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
b859a1f714 downloads: add tests for untested sites. 2018-02-24 13:42:00 -06:00
Albert Yi
d8340f83db
Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
a923a64220 tests: fix pixiv profile image test. 2018-01-20 14:51:05 -06:00
evazion
53f7207716 tests: remove bad_id pixiv test. 2018-01-20 14:50:44 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images.
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
2017-11-21 14:40:39 -06:00
evazion
5ba63733d3 download/pixiv_test.rb: fix broken pixiv download tests.
This post was deleted:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46304614

Switch to this instead:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46324488
2017-11-21 13:48:07 -06:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
22980c845a tests: fix pixiv ugoira urls. 2017-07-25 00:36:18 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
030052bf14 tumblr: rewrite html pages to image url. 2017-06-25 15:34:15 -05:00