0 Replies
・
Started by KimtaroAdmin
at 2021-04-18 03:30:17 UTC
Rate limiting on API and search
This week Jisho went down a couple of times because someone was aggressively crawling the site.
This forced me to add rate limiting to the site. If you make more than 10 requests per 10 seconds to the api or web search, you will be rate limited and will receive HTTP 429 response until you back off.
In general you should not crawl Jisho. All the data on the site comes from open data sources, and if you think you need to crawl the site for data you are much better off using the same source files that Jisho uses. You can read about them on the About page: https://jisho.org/about
If you think you have a legitimate reason to crawl the site or if you are getting incorrectly rate limited because of normal use of the site, please email me at jisho.org@gmail.com.
Rate limiting on API and search
This week Jisho went down a couple of times because someone was aggressively crawling the site.
This forced me to add rate limiting to the site. If you make more than 10 requests per 10 seconds to the api or web search, you will be rate limited and will receive HTTP 429 response until you back off.
In general you should not crawl Jisho. All the data on the site comes from open data sources, and if you think you need to crawl the site for data you are much better off using the same source files that Jisho uses. You can read about them on the About page: https://jisho.org/about
If you think you have a legitimate reason to crawl the site or if you are getting incorrectly rate limited because of normal use of the site, please email me at jisho.org@gmail.com.