Add skill acquisition eval

evals/elsuite/skill_acquisition/readme.md

@@ -0,0 +1,64 @@
+# Skill acquisition
+
+This eval tests models' ability to learn a skill with minimal human involvement. In the initial release, models are evaluated on questions related to the [Miskito language](https://en.wikipedia.org/wiki/Miskito_language). Some samples are translation and others are language manipulation exercises. 
+
+## Usage
+Run with:
+```bash
+oaieval <solver> skill_acquisition.miskito
+```
+
+Where the solver can be any generation solver in `evals/registry/solvers/defaults.yaml`, eg. `generation/cot/gpt-3.5-turbo-16k`.
+
+## Evaluation process
+Every time the eval is run, the model is evaluated twice. The first time, it answers the question directly using whatever prompting technique is executed by the solver you choose. The second time the model runs in a loop, interacting with an interface which gives it access to a knowledge base. The knowledge base contains text files, some of which are relevant for answering the question, while others are unrelated. If models can use this interface to increase their performance on the task, we can say that they've improved or acquired their language translation and manipulation skills.
+
+## Prompts
+See `skill_acquisition/utils.py` to review/adjust the prompts used in this eval.
+
+## Datasets
+
+The dataset is generated from [this language course](https://en.wikibooks.org/wiki/Miskito), which comprises 229 questions. We further split this into manipulation-only (`miskito_test_manipulation.jsonl`) and translation-only (`miskito_test_translation.jsonl`) subsets.
+
+## Variants
+
+We test zero-shot and few-shot prompting techniques on the dataset:
+
+| Dataset | Zero-shot | Few-shot |
+| --------- | -------- | -------- |
+| Miskito | `skill_acquisition.miskito.zero-shot.full`|`skill_acquisition.miskito.few-shot.full`|
+
+The `full` in this case refers to the size of the dataset – there are also variants for testing where only 5 examples are considered, called `dev5`. For full details, look at `evals/registry/skill_acquisition/skill_acquisition.yaml`.
+
+For the few-shot setting, use the eval-specific solvers in `evals/registry/solvers/skill_acquisition.yaml` to avoid train/test leakage.
+
+## Token Usage Estimates
+
+Below is a rough estimate of the total number of tokens consumed by some variations the eval, including both input and output tokens:
+
+| Model | Solver | Prompt tokens | Completion tokens | Total tokens
+| --- | --- | --- | --- | --- |
+| gpt-3.5-turbo | direct | 1,000,000 | 23,000 | 1,050,000 |
+| gpt-3.5-turbo | cot | 930,000 | 120,000 | 1,050,000 |
+| gpt-3.5-turbo | fewshot | 450,000 | 9,600 | 460,000 |
+| gpt-3.5-turbo-16k | direct | 1,400,000 | 24,000 | 1,500,000 |
+| gpt-3.5-turbo-16k | cot | 2,000,000 | 120,000 | 2,100,000 |
+| gpt-3.5-turbo-16k | fewshot | 610,000 | 10,000 | 620,000 |
+| gpt-4-base | direct | 1,800,000 | 420,000 | 2,200,000 |
+| gpt-4-base | cot | 4,700,000 | 890,000 | 5,600,000 |
+| gpt-4-base | fewshot | 1,400,000 | 320,000 | 1,700,000 |
+| gpt-4-1106-preview | direct | 1,700,000 | 100,000 | 1,800,000 |
+| gpt-4-1106-preview | cot | 1,600,000 | 99,000 | 1,700,000 |
+| gpt-4-1106-preview | fewshot | 1,700,000 | 95,000 | 1,800,000 |
+| gpt-4-32k | direct | 1,800,000 | 80,000 | 1,900,000 |
+| gpt-4-32k | cot | 2,700,000 | 180,000 | 2,900,000 |
+| gpt-4-32k | fewshot | 190,000 | 6,000 | 190,000 |
+
+## Version History
+v0: Initial version released
+
+
+## Contribution statement
+
+Eval design, implementation, and results evaluation were primarily conducted by Andrei Alexandru. Giulio Starace was responsible for code reviews throughout the implementation process, along with fine-grained feedback on the project in general. Additional guidance was provided by (alphabetically by last-name) Steven Adler, James Aung and Chan Jun Shern, who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation.
+

evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py

@@ -0,0 +1,96 @@
+# %%
+import json
+import re
+
+import requests
+from bs4 import BeautifulSoup
+from markdownify import markdownify as md
+
+articles_to_scrape = [
+    "https://en.wikipedia.org/wiki/Mosquito",
+    "https://en.wikipedia.org/wiki/Mosquito_Coast",
+    "https://en.wikipedia.org/wiki/Nicaragua",
+    "https://en.wikipedia.org/wiki/Honduras",
+    "https://en.wikipedia.org/wiki/Miskito_language",
+    "https://en.wikipedia.org/wiki/Miskito_people",
+]
+dirpath = "evals/registry/data/skill_acquisition/distractor_articles/"
+
+
+def clean_soup(content):
+    for infobox_tag in content.find_all("table", class_="infobox"):
+        infobox_tag.decompose()
+    for figure_tag in content.find_all("figure"):
+        figure_tag.decompose()
+    for style_tags in content.find_all("style"):
+        style_tags.decompose()
+    reflist_div = '<div class="reflist"'
+    if reflist_div in str(content):
+        reflist_index = str(content).index(reflist_div)
+        content = BeautifulSoup(str(content)[:reflist_index], "html.parser")
+
+    return content
+
+
+def clean_heading_text(
+    heading_text,
+    strip_list=["a"],
+    css_selector_pattern=r"\.\w+[^{}]*\{[^}]*\}",
+    whitespace_pattern=r"\s\s+",
+):
+    text = md(str(heading_text), strip=strip_list)
+    text = (
+        # text.replace("\n", "")
+        text.replace("\u3000", " ")
+        .replace("\xa0", " ")
+        .replace("| --- |", "")
+        .replace("|", "")
+    )
+    text = re.sub(whitespace_pattern, "", re.sub(css_selector_pattern, "", text))
+    return text
+
+
+for article in articles_to_scrape:
+    response = requests.get(article)
+    soup = BeautifulSoup(response.text, "html.parser")
+
+    content = soup.find("div", class_="mw-content-ltr mw-parser-output")
+    content = clean_soup(content)
+    headings = str(content).split("<h2>")
+
+    sections = {}
+    for heading_text in headings:
+        if "</h2>" not in heading_text:
+            sections["Introduction"] = clean_heading_text(heading_text)
+            continue
+        span = heading_text[: heading_text.index("</h2>")]
+        heading_title = BeautifulSoup(span, "html.parser").contents[0].contents[0]
+        text = heading_text[heading_text.index("</h2>") + 5 :]
+        if heading_title not in ["References", "See also", "External links", "Footnotes"]:
+            sections[heading_title] = clean_heading_text(text)
+
+    article_title = article.split("/")[-1]
+
+    print(f"Scraped {article_title} successfully. Headings: {sections.keys()}\n")
+    filename = f"{article_title.lower()}.jsonl"
+
+    with open(dirpath + filename, "w") as f:
+        for k, v in sections.items():
+            f.write(json.dumps({"title": k, "content": v}, ensure_ascii=False) + "\n")
+
+# Separate code to scrape human rights article, as it's in a different format.
+with open("human_rights.html", "r") as f:
+    html = f.read()
+
+soup = BeautifulSoup(html, "html.parser")
+content = soup.find("div", class_="migrated-content")
+md_content = md(str(content)).replace("\xa0", " ").replace("\u3000", " ")
+
+with open(dirpath + "human_rights_miskito.jsonl", "w") as f:
+    f.write(
+        json.dumps(
+            {"title": "Declaration of Human Rights in Miskito", "content": md_content},
+            ensure_ascii=False,
+        )
+        + "\n"
+    )

evals/elsuite/skill_acquisition/scraping/scrape_miskito.py

@@ -0,0 +1,135 @@
+# %%
+import json
+
+import bs4
+import requests
+from bs4 import BeautifulSoup
+from markdownify import markdownify as md
+
+# TODO: make sure italicised text is crawled properly and that hints are excluded from answers. 
+# TODO: Split any multi-part questions into individual questions.
+
+miskito_base_url = "https://en.wikibooks.org/wiki/Miskito/Lesson_{idx}"
+
+
+def process_practice_section_div(practice_div: bs4.element.Tag):
+    tds = practice_div.find_all("td")
+    instructions = (
+        md(str(tds[1]))
+        .replace("*", "")
+        .replace("|", "")
+        .strip()
+        .replace("What do these mean?", "Translate to English:")
+        .replace("What do these sentences mean?", "Translate to English:")
+    )
+    question_text = tds[2]
+    questions = question_text.find_all("li")
+    questions = [str(q.contents[0]) for q in questions]
+    answer_text = tds[3]
+    answers = answer_text.find_all("li")
+    answers = [str(a.contents[0]) for a in answers]
+    return instructions, questions, answers
+
+
+def extract_toc_sections(content: bs4.element.Tag):
+    toc = content.find_all("div", class_="toc")[0]
+    lis = toc.find_all("li", class_="toclevel-1")
+    lis = [li.find_all("span", class_="toctext")[0].contents[0] for li in lis]
+
+    lis = [md(str(li)).strip().replace("*", "") for li in lis]
+    return lis
+
+
+def process_miskito_page():
+    qa_pairs_by_lesson = {}
+    articles_without_qa_pairs = []
+    for idx in range(1, 11):
+        response = requests.get(miskito_base_url.format(idx=idx))
+        soup = BeautifulSoup(response.text, "html.parser")
+        content = soup.find("div", class_="mw-content-ltr mw-parser-output")
+
+        # Extract the question-answer pairs.
+        divs_with_specific_style = content.find_all(
+            "div", style=lambda value: value and "width:300px; float:right;" in value
+        )
+        lesson_qa_pairs = []
+        for i, div in enumerate(divs_with_specific_style):
+            if i == 0 and idx == 1:  # First section of first lesson is not in the same format.
+                instructions = "Translate to English:"
+                questions = div.find_all("ul")[0].find_all("li")
+                questions = [str(q.contents[0]) for q in questions]
+                answers = div.find_all("ul")[1].find_all("li")
+                answers = [str(a.contents[0]) for a in answers]
+                lesson_qa_pairs += [
+                    {"question": q, "answer": a, "instructions": instructions}
+                    for q, a in zip(questions, answers)
+                ]
+                continue
+            instructions, questions, answers = process_practice_section_div(div)
+            for q, a in zip(questions, answers):
+                lesson_qa_pairs += [{"question": q, "answer": a, "instructions": instructions}]
+        qa_pairs_by_lesson[f"lesson_{idx}"] = lesson_qa_pairs
+
+        # Remove them from the page and store the page contents.
+        for div in divs_with_specific_style:
+            div.decompose()
+
+        articles_without_qa_pairs += [content]
+
+    return qa_pairs_by_lesson, articles_without_qa_pairs
+
+
+# %%
+# Write to file: all questions by lesson, and all questions in evallib format.
+qa_pairs_by_lesson, clean_articles = process_miskito_page()
+qa_by_lesson_file = "miskito_qa_pairs_by_lesson.jsonl"
+
+with open(qa_by_lesson_file, "w") as f:
+    for lesson, qa_pairs in qa_pairs_by_lesson.items():
+        f.write(json.dumps({"lesson": lesson, "qa_pairs": qa_pairs}) + "\n")
+
+miskito_qa = "miskito_qa.jsonl"
+with open(miskito_qa, "w") as f:
+    for lesson, qa_list in qa_pairs_by_lesson.items():
+        for qa_dict in qa_list:
+            instructions = qa_dict["instructions"][:-1] + ": "
+            f.write(
+                json.dumps(
+                    {
+                        "input": [{"role": "user", "content": instructions + qa_dict["question"]}],
+                        "ideal": qa_dict["answer"],
+                    },
+                    ensure_ascii=False,
+                )
+                + "\n"
+            )
+# %%
+as_text = [str(a).split("<h2>")[1:] for a in clean_articles]
+sections_by_heading = {}
+for article in as_text:
+    for heading in article:
+        hsoup = BeautifulSoup(heading, "html.parser")
+        heading_name = (
+            md(str(hsoup.find("span", class_="mw-headline").contents[0])).replace("*", "").strip()
+        )
+        hsoup.find("span", class_="mw-editsection").decompose()
+        content = (
+            md(str(hsoup))
+            .strip()
+            .replace("*", "")
+            .replace("|", "")
+            .replace("What do they mean?", "")
+            .replace(" --- ", "")
+            .replace("\u2003", " ")
+            .replace("     ", " ")
+        )
+        content = content.split(" Study ")[1] if "Study " in content else content
+        sections_by_heading[heading_name] = content.strip()
+
+sections_by_heading
+# %%
+file = "lessons_no_exercises.jsonl"
+with open(file, "w") as f:
+    for heading, content in sections_by_heading.items():
+        f.write(json.dumps({"title": heading, "content": content}, ensure_ascii=False) + "\n")
+# %%