The repository of conference talks videos

[zverok]

Project

Gather (automatically and semi-automatically) as much talk videos from Ruby conferences as there are exist, and provide a nice browsing interface for them.

Plan

1. Go through http://rubyconferences.org/past/
2. For each conference, define in Ruby (with the help of wombat?) its site (or web.archive.org copy of it) parser to extract structured list of talks, speakers, topics and so on;
3. For each conference, find (on site, or in linked YouTube/Confreaks playlist) list of talks, and define extractor for it, and matcher with conference's program;
4. Define (with the help of Jekyll or another nice static site renderer) the rendering of this data, and provide extensive navigations by years, by speakers, by talk topics, keywords and so on; joining the same talk given on several conferences into "versions".
5. Publish to GitHub pages
6. Make sure to have a "framework" of scripts to update the site after future conferences
7. Provide an ability for collaborative editing of talk classifications.

Importance

There are a lot of good material over there gathered through 25 years of Ruby, and a huge part if it is available online, providing learning, discovery and historical interest. Somebody just should do something about it.

Skills and domains

You will do a lot of automated web scraping, and site generation, as well as generalizing a lot of non-generic data in a bearable way.

[subratrout]

Hi Victor, I am interested to work on this project. I am a Ruby developer with 1 year of Ruby experience. Am I eligible to work on this project? Please let me know. Thank you.

[zverok]

Am I eligible to work on this project?

Of course! Do you have any ideas of where to start, or do you want me to give you some initial steps and hints?

[subratrout]

It will be great if you would be kind to give some initial guidance on how to approach it.

[zverok]

OK then! As I see it, the very first steps could be those:

1. Go to http://rubyconferences.org/past/ and select some conferences that have videos published...
2. For completeness, select:
   • one with videos and slides right on the site, for ex. RubyKaigi: http://rubykaigi.org/2017
   • one with the site containing only talk announces, and videos on YouTube playlist, for ex. EuRuko: https://www.euruko2017.org/schedule/ (videos, could be found through Twitter: https://www.youtube.com/playlist?list=PLCrwBqiOtwpIRdk4VMfD6hbFVHZoKOyMg and https://www.youtube.com/playlist?list=PLCrwBqiOtwpINmaC2eBX8QHuLYngECD19&disable_polymer=true)
   • (for later) say, one with the site preserved only in webarchive: http://web.archive.org/web/20150927215820/http://rubyconf.org/program and videos on Confreaks.tv: http://confreaks.tv/events/rubyconf2015
3. Now, we need from all of the examples above, produce some YAML (one file per conference) with the list of talks, containing, probably:
   • title
   • announce
   • link to slides, if available
   • link to video, wherever it is
   • speaker name
   • speaker bio/about
   • link to speaker's photo if available
   • talk's date/time (for historical reasons)
   • some conference meta information (title, date, place, description)
4. I recommend trying to write conference site scrapers with wombat, it seems the most appropriate tool
   • for YouTube, it probably would be wise to use official YouTube Ruby client library (and you'll need to match titles of talks on YouTube with titles on conference site, be careful)
5. Then, when you have YAML with all the data, you can try writing renderers of all the data into a static site, probably with the help of middleman. For the first iteration, nothing fancy should be done, just pages like /conferences/ (links to all), /conferences/<conferenceid> (metainformation about the conference + list of talks), conferences/<id>/talks/<talkid> (particular talk with info, links to slides, embedded video)
6. Next step would be to cover all conferences in existence with parsers (sometimes it will also require some digging the web for their long-gone sites, but videos are mostly on YouTube and still present) and have a HUGE bunch of parsed YAML files
   • probably, some small DSLs would emerge to DRY this task, like "This conference has a site at that URL, use this Wombat definitions and fetch videos from those YouTube playlists, and match their titles to conference program with this regexp" or something, we'll see.
7. And then "site crafting" stage comes, which we'll probably discuss and experiment more when the time will come.

How it sounds? Let me know if something is unclear in this plan (especially first stages of it, the later stages intentionally left generic, we can discuss them in more details later)

[subratrout]

Thank you Victor for detailed instructions. I believe I can start working it. Where do I create the repo? Or will you create a repo with basic folder and file structure and give me the access to work on it? Please let me know. ᐧ

…

On Fri, Mar 2, 2018 at 1:15 AM, Victor Shepelev ***@***.***> wrote: OK then! As I see it, the very first steps could be those: 1. Go to http://rubyconferences.org/past/ and select some conferences that have videos published... 2. For completeness, select: - one with videos and slides right on the site, for ex. RubyKaigi: http://rubykaigi.org/2017 - one with the site containing only talk announces, and videos on YouTube playlist, for ex. EuRuko: https://www.euruko2017.org/ schedule/ (videos, could be found through Twitter: https://www.youtube.com/playlist?list=PLCrwBqiOtwpIRdk4VMfD6hbFVHZoK OyMg <https://www.youtube.com/playlist?list=PLCrwBqiOtwpIRdk4VMfD6hbFVHZoKOyMg> and https://www.youtube.com/playlist?list= PLCrwBqiOtwpINmaC2eBX8QHuLYngECD19&disable_polymer=true <https://www.youtube.com/playlist?list=PLCrwBqiOtwpINmaC2eBX8QHuLYngECD19&disable_polymer=true> ) - (for later) say, one with the site preserved only in webarchive: http://web.archive.org/web/20150927215820/http:// rubyconf.org/program <http://web.archive.org/web/20150927215820/http://rubyconf.org/program> and videos on Confreaks.tv: http://confreaks.tv/events/rubyconf2015 3. Now, we need from all of the examples above, produce some YAML (one file per conference) with the list of talks, containing, probably: - title - announce - link to slides, if available - link to video, wherever it is - speaker name - speaker bio/about - link to speaker's photo if available - talk's date/time (for historical reasons) - some conference meta information (title, date, place, description) 4. I recommend trying to write conference site scrapers with wombat <https://github.com/felipecsl/wombat>, it seems the most appropriate tool - for YouTube, it probably would be wise to use official YouTube Ruby client library (and you'll need to match titles of talks on YouTube with titles on conference site, be careful) 5. Then, when you have YAML with all the data, you can try writing renderers of all the data into a static site, probably with the help of middleman <https://github.com/middleman/middleman>. For the first iteration, nothing fancy should be done, just pages like /conferences/ (links to all), /conferences/<conferenceid> (metainformation about the conference + list of talks), conferences/<id>/talks/<talkid> (particular talk with info, links to slides, embedded video) 6. Next step would be to cover all conferences in existence with parsers (sometimes it will also require some digging the web for their long-gone sites, but videos are mostly on YouTube and still present) and have a HUGE bunch of parsed YAML files - probably, some small DSLs would emerge to DRY this task, like "This conference has a site at that URL, use this Wombat definitions and fetch videos from those YouTube playlists, and match their titles to conference program with this regexp" or something, we'll see. 7. And then "site crafting" stage comes, which we'll probably discuss and experiment more when the time will come. How it sounds? Let me know if something is unclear in this plan (especially first stages of it, the later stages intentionally left generic, we can discuss them in more details later) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOB6pgKAhP1wJfTzvjZukqsIOWhk8MZks5taP-VgaJpZM4SRLbn>

[zverok]

Where do I create the repo?

Just do it in your personal GitHub for now. Then, when the work would be close to publishing, we'll create separate GitHub organization and you'll "transit" repo there (GitHub allows this). The separate organization (something like ruby-talks or something) is needed because GitHub will provide nice org-name.github.io domain name.

[zverok]

Or let's create org immediately, in fact :) Going to it.

[zverok]

https://github.com/ruby-talks-archive/ruby-talks Here we go. You should have access to it.

[subratrout]

Hi Victor, just pushed my first commit to ruby-talks under repo named "videotalks". The first one i.e videos.rb is working. Please advise why creating class and specifying model Wombat::Crawler is not working. Thank you.

[KrassCodes]

Hi Victor, has this progressed since the last update? I would be interested to contribute if I can. Thank you for organising these projects!

[zverok]

@Krass101 Nope, nothing changed for the project since the discussion in this issue 🤷‍♀️

I'd be glad if you'll contribute your work to this project, and I am ready to answer any questions and provide other help whenever you need.

[KrassCodes]

@zverok thanks for your offer to help! I have started my own repo and will make small updates every day. I am still learning ruby, wombat and other gems. My goal is to use a TDD approach and as a first step I have written the basic code to scrape the speech titles of the RubyKaigi 2018 conference. My repo is here: https://github.com/Krass101/ruby-talks

Do you have a preferred way for me to ask you questions? I can collect them somewhere and either send to you at once or maintain a list in the repo?

[zverok]

@Krass101 We can discuss things at https://gitter.im/molybdenum-99/rubytalks (it is chat service that is easy to log in from GitHub, and easy to discuss code in), or you can drop me an email at [email protected] (but Gitter is more convenient).

As for starting the work -- I believe the very first steps could be to just save some page(s) of the conference of interest locally and write some specs saying "I can extract this and that data from the page", this would be a good start (and will also be head-first so you'll see if you love to work on problems like this) which is easy to achieve.