diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 49 |
1 files changed, 49 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..54d636d --- /dev/null +++ b/README.md @@ -0,0 +1,49 @@ +# Google Group Scrapper + + + +A small script as a replacement of [the old PHP +script](http://saturnboy.com/2010/03/scraping-google-groups/) for +downloading messages stored in the black hole of the Google Groups. + +## How to use it? + +This script requires `formail(1)` from `procmail` package. Any version +is OK, so please install it from your distribution's repositories. Then +run: + + pip install beautifulsoup4 PyYAML + python gg_scraper.py 'https://groups.google.com/forum/#!forum/<group_name>' + +## Background + +I would never start without an inspiration from the +[comment](https://luther.ceplovi.cz/blog/2013/09/19/we-should-stop-even-pretending-google-is-trying-to-do-the-right-thingtm/#comment-133-by-sean-hogan) +by Sean Hogan on my previous post on the theme of locked down nature of +Google Groups: + +> At least google-groups appears to follow google\'s own advice on +> making AJAX sites accessible to the google web-crawler. See +> <https://developers.google.com/webmasters/ajax-crawling/docs/getting-started> +> +> So for me, +> <http://groups.google.com/d/forum/jbrout:FBv1oMXRZkxB6YShaFuNHc3-Moc&cuid=3654582> +> redirects (eventually) to +> <https://groups.google.com/forum/#!forum/jbrout> which can be viewed +> in raw HTML as +> <https://groups.google.com/forum/?_escaped_fragment_=forum/jbrout> +> +> Google Groups seems a perfect use-case for +> extreme-progressive-enhancement, but what would I know. +> +> regards, Sean + +------------------------------------------------------------------------ + +All issues, questions, complaints, or (even +better!) patches should be send via email to +[~mcepl/<devel@lists.sr.ht>](mailto:~mcepl/devel@lists.sr.ht) +email list (for patches use [git +send-email](https://git-send-email.io/)). For the issue tracking +I use [git-bug](https://github.com/MichaelMure/git-bug) in this +repo. |