Class 8 - Local LLMs Solution.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"gpuType":"T4","collapsed_sections":["-wRa7zf5Owkd","80o9xFCbO1ZN"]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","source":["# Class 8: Local Large Language Models\n","\n","*   Each section of this class uses a lot of RAM and will cause the notebook to break due to that\n","*   We recommend to disconect and start from the beginnig (with mount and necessary installations) everytime before each section\n","\n"],"metadata":{"id":"-RZbkMCzjcBL"}},{"cell_type":"code","source":["import warnings; warnings.simplefilter('ignore')\n","import pandas as pd\n","import numpy as np\n"],"metadata":{"id":"fxVj1RXLVMdE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["#Import drive\n","from google.colab import drive\n","#Mount Google Drive\n","ROOT=\"/content/drive\"\n","drive.mount(ROOT, force_remount=True)"],"metadata":{"id":"pJFEG2jZOe19"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["In this notebook, we use 4-bit quantization to run Llama-7B Chat model. This code uses only 10 GB of VRAM. It can run on a free instance of Google Colab or on a local GPU (e.g., RTX 3060 12GB).\n","[More details here.](https://open.substack.com/pub/kaitchup/p/run-llama-2-chat-models-on-your-computer?r=2kp66c&utm_campaign=post&utm_medium=web)\n","\n","\n","We only need the following libraries:\n","\n","\n","*   transformers\n","*   accelerate (for device_map)\n","*   bitsandbytes (for 4-bit quantization)\n","\n","\n"],"metadata":{"id":"UaHcYzEKlwI_"}},{"cell_type":"code","source":["%pwd"],"metadata":{"id":"qBiwmvyNOhx3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["%cd /content/drive/MyDrive/dlss24"],"metadata":{"id":"vyLOEn-qOiGu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["%pwd"],"metadata":{"id":"oOxnbLeLOj9L"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"tn0VdF4NOl6a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","1.   Install package\n","2.   Go to hugging face to get your access token (you can also use this one, its mine)\n","3.   Run the out of the box and fine tune versions of the exercises\n","\n"],"metadata":{"id":"ZefWU4fWUXfU"}},{"cell_type":"markdown","source":["# Instalations"],"metadata":{"id":"tSwjRt3zOndK"}},{"cell_type":"code","source":["!pip install transformers accelerate bitsandbytes"],"metadata":{"id":"ZgGyWYanlvr_"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["Note that to run the following code, you must have got access to Llama 2's weights and have an access token from Hugging Face. You can find instructions on the model cards on the hugging face hub: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf\n"],"metadata":{"id":"KCdND_LSkVPg"}},{"cell_type":"code","source":["!pip install accelerate"],"metadata":{"id":"ql0_6QK3Usgj"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["!pip install -q bitsandbytes datasets accelerate loralib"],"metadata":{"id":"WTGh1UxwWoFc"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Llama with access token"],"metadata":{"id":"AZ_ONnVVOqe8"}},{"cell_type":"markdown","source":["If you have a persistent error about the accelarate, restart the session (if you don't go out of these tab for long, you will not loose any variable by restarting the session)"],"metadata":{"id":"Q3kQ8RmgU5uB"}},{"cell_type":"code","source":["from transformers import AutoModelForCausalLM, AutoTokenizer\n","model_name = \"meta-llama/Llama-2-7b-chat-hf\"\n","access_token = \"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\"\n","\n","\n","\n","model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", load_in_4bit=True,  use_auth_token=access_token)\n","tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True, use_auth_token=access_token)"],"metadata":{"id":"PSXmN_2-NGYA"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"THqfvzHIjSK9"},"outputs":[],"source":["prompt = \"Tell me about gravity in english please\"\n","model_inputs = tokenizer(prompt, return_tensors=\"pt\") #.to(\"cuda:0\")\n","\n","output = model.generate(**model_inputs)\n","\n","print(tokenizer.decode(output[0], skip_special_tokens=True))\n"]},{"cell_type":"code","source":["prompt = \"Based on this text: 'the jewish community gave a speaker referee to the radio. This speaker was talking about traditions and values of jewish community in the country'. What is the high level topic? Choose one: religion, culture, politics, business, environment. answer only the topic name\"\n","model_inputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\n","\n","output = model.generate(**model_inputs)\n","\n","print(tokenizer.decode(output[0], skip_special_tokens=True))"],"metadata":{"id":"fH1pPQMpv58n"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Apply to your dataset (the one from last lab class)\n","\n"],"metadata":{"id":"m09kYakVIz8w"}},{"cell_type":"code","source":["import pandas as pd\n","import numpy as np\n","df_source_corpus=pd.read_csv('/content/drive/MyDrive/catarina/source_corpus.csv')\n","df_source_corpus = df_source_corpus.dropna(subset=['text'])"],"metadata":{"id":"COqcu2xsJFue"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["df_source_corpus.head()"],"metadata":{"id":"Y0TK3vnsfOqD"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["np.unique(df_source_corpus.topic_8)"],"metadata":{"id":"aZ0U4-TDLlm0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Complete the code\n","\n"],"metadata":{"id":"0BkwxmBBfUuq"}},{"cell_type":"code","source":["\n","model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", load_in_4bit=True,  use_auth_token=access_token)\n","tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True, use_auth_token=access_token)\n","\n","def get_local_gtp_answer(prompt):\n","   #TODO:\n","\n","  model_inputs = ...\n","\n","  output = ...\n","\n","  return ..."],"metadata":{"id":"Oo9P_ErJJTCO"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Iterate over the DataFrame with tqdm\n","from tqdm import tqdm\n","import time\n","\n","df_sample=df_source_corpus[0:100]\n","\n","for index_, row in tqdm(df_sample.iterrows(), total=df_sample.shape[0], desc=\"Processing rows\"):\n","  #TODO:\n","    sentence_ = ...\n","    prompt = f\"Thaving this sentence: '{sentence_}', what is the topic code? Answer one topic from: economy, external relations, fabric of society, freedom and democracy, no topic, political system, social groups, welfare and quality of life.\"\n","\n","    df_sample.at[index_, 'output_promt'] = ..."],"metadata":{"id":"OSFKPXY2MPAZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"CDAQGjqJMQPN"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"awWrbX4eMQKy"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"FajURlCsMQHi"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"j3-e1_q8MQEZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"XK6vjCpvMQA8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["\n","\n","\n","model = AutoModelForCausalLM.from_pretrained(model_name, device_map=\"auto\", load_in_4bit=True,  use_auth_token=access_token)\n","tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True, use_auth_token=access_token)\n","\n","\n","def get_local_gtp_answer(prompt):\n","\n","  model_inputs = tokenizer(prompt, return_tensors=\"pt\") #.to(\"cuda:0\")\n","\n","  output = model.generate(**model_inputs)\n","\n","  return tokenizer.decode(output[0], skip_special_tokens=True)"],"metadata":{"id":"e0tHVJWTOvxw"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["df_sample=df_source_corpus[0:100]"],"metadata":{"id":"mH2s6fIUNf8H"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["sentence_ = df_sample['text'].iloc[0]\n","prompt = f\"Based on this text: '{sentence_}', what is the topic? Answer one topic from: economy, external relations, fabric of society, freedom and democracy, no topic, political system, social groups, welfare and quality of life.\"\n","\n","get_local_gtp_answer(prompt)"],"metadata":{"id":"0-Vzsjqwf3SE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Iterate over the DataFrame with tqdm\n","from tqdm import tqdm\n","import time\n","\n","for index_, row in tqdm(df_sample.iterrows(), total=df_sample.shape[0], desc=\"Processing rows\"):\n","    sentence_ = row['text']\n","    #prompt = f\"Which of these topics adapt better to the sentence i'll give you: economy, external relations, fabric of society, freedom and democracy, no topic, political system, social groups, welfare and quality of life.\"\n","    #prompt = f\"what is the topic of these sentence:{sentence_} \"\n","    prompt = f\"having this sentence: '{sentence_}', what is the topic? Answer one topic from: economy, external relations, fabric of society, freedom and democracy, no topic, political system, social groups, welfare and quality of life.\"\n","\n","    df_sample.at[index_, 'output_promt'] = get_local_gtp_answer(prompt)"],"metadata":{"id":"JYuh-Ie2JaHN"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["df_sample['output_promt'].iloc[90]"],"metadata":{"id":"d8NIyQJlNr89"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["import torch\n","torch.cuda.empty_cache()"],"metadata":{"id":"-DrAt0P8TdZs"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Llama without access code"],"metadata":{"id":"-wRa7zf5Owkd"}},{"cell_type":"code","source":["from transformers import AutoModelForCausalLM, AutoTokenizer\n","\n","checkpoint = \"HuggingFaceH4/zephyr-7b-beta\"\n","tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n","model = AutoModelForCausalLM.from_pretrained(checkpoint)  # You may want to use bfloat16 and/or move to GPU here\n","\n","messages = [\n","    {\n","        \"role\": \"system\",\n","        \"content\": \"You are a friendly chatbot who always responds in the style of a pirate\",\n","    },\n","    {\"role\": \"user\", \"content\": \"How many helicopters can a human eat in one sitting?\"},\n"," ]\n","tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors=\"pt\")\n","print(tokenizer.decode(tokenized_chat[0]))"],"metadata":{"id":"oE9knKnZtbue"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# OpenAI paid GTP"],"metadata":{"id":"80o9xFCbO1ZN"}},{"cell_type":"code","source":["import openai\n","import pickle\n","\n","openai.api_key = \"...\"\n","\n","def naics_codes_wlc( voting_unit, union_, case_name,state_, city):\n","    #if case_name:\n","    prompt = f\"I am a senior researcher in labour studies in the USA. I am interested in defining the economic activity sector of workers. These workers want to join a union at their workplace, and I need to classify them according to their economic activity. In USA economic activities can be defined according to 20 categories. Those categories are NAICS sectors that represent general categories of economic activities.  \\n\\\n","    'Here is a list of the NAICS sector code in the format: 'code :Description' '\\n\\\n","    '11 :Agriculture, Forestry, Fishing and Hunting'—Activities of this sector are growing crops, raising animals, harvesting timber, and harvesting fish and other animals from farms, ranches, or the animals' natural habitats. Example job titles: farmworker, grower, picker, harvester, apple sorter. Example job titles: wellhead pumper, derrick operator, miner, rig operator, roustabout, pipeline technician \\n\\\n","    '21 :Mining, Quarrying, and Oil and Gas Extraction' — Activities of this sector are extracting naturally occurring mineral solids, such as coal and ore; liquid minerals, such as crude petroleum; and gases, such as natural gas; and beneficiating (e.g., crushing, screening, washing, and flotation) and other preparation at the mine site, or as part of mining activity.Example job titles: wellhead pumper, derrick operator, miner, rig operator, roustabout, pipeline technician\\n\\\n","    '22 :Utilities'—Activities of this sector are generating, transmitting, and/or distributing electricity, gas, steam, and water and removing sewage through a permanent infrastructure of lines, mains, and pipe.Example job titles: power plant operator, electrical lineworker, power system dispatcher, electric meter technician, water plant maintenance mechanic, wind turbine service technician \\n\\\n","    '23 :Construction'—Activities of this sector are erecting buildings and other structures (including additions); heavy construction other than buildings; and alterations, reconstruction, installation, and maintenance and repairs.Example job titles: bricklayer, mason, roofer, plumbing installer, pipe welder, electrician, construction carpenter, scaffolding operator \\n\\\n","    “31-33 :Manufacturing”—Activities of this sector are the mechanical, physical, or chemical transformation of materials, substances, or components into new products.Example job titles: assembly line machine worker, packaging operator, quality inspector, industrial mechanic  \\n\\\n","    “42 :Wholesale Trade”—Activities of this sector are selling or arranging for the purchase or sale of goods for resale; capital or durable nonconsumer goods; and raw and intermediate materials supplies used in production, and providing services incidental to the sale of the merchandise.Example job titles: salesperson, account representative, merchandiser, clothing designer, customer account technician\\n\\\n","    “44-45 :Retail Trade”—Activities of this sector are retailing merchandise generally in small quantities to the general public and providing services incidental to the sale of the merchandise.Example job titles: customer assistant, car salesman, cashier, checker, stocker, butcher \\n\\\n","    “48-49 :Transportation and Warehousing”—Activities of this sector are providing transportation of passengers and cargo, warehousing and storing goods, scenic and sightseeing transportation, and supporting these activities.Example job titles: truck driver, warehouse worker, tractor trailer driver, transit operator, flight attendant, ticket agent, package delivery driver\\n\\\n","    “51 :Information”—Activities of this sector are distributing information and cultural products, providing the means to transmit or distribute these products as data or communications, and processing data.Example job titles: software engineer, computer support specialist, sound editor, news editor, reporter, digital projectionist\\n\\\n","    “52 :Finance and Insurance”—Activities of this sector involve the creation, liquidation, or change in ownership of financial assets (financial transactions) and/or facilitating financial transactions.Example job titles: financial advisor, broker, bank teller, insurance agent, loan officer, claims adjuster \\n\\\n","    “53 :Real Estate and Rental and Leasing”—Activities of this sector are renting, leasing, or otherwise allowing the use of tangible or intangible assets (except copyrighted works), and providing related services.Example job titles: rental agent, realtor, maintenance man, facilities technician, appraiser, leasing consultant\\n\\\n","    “54 :Professional, Scientific, and Technical Services”—Activities of this sector are performing professional, scientific, and technical services for the operations of other organizations.Example job titles: application developer, veterinary technician, animal care provider, paralegal, attorney, auditor, accountant, IT specialist \\n\\\n","    “55 :Management of Companies and Enterprises”—Activities of this sector are the holding of securities of companies and enterprises, for the purpose of owning controlling interest or influencing their management decisions, or administering, overseeing, and managing other establishments of the same company or enterprise and normally undertaking the strategic or organizational planning and decision-making role of the company or enterprise.Example job titles: human resources director, employee relations manager, brand manager, marketing director, product manager\\n\\\n","    “56 :Administrative and Support and Waste Management and Remediation Services”— Activities of this sector are performing routine support activities for the day-to-day operations of other organizations.Example job titles: building custodians, janitor, trash collector, alarm technician, gardener, security guard, landscape technician, telemarketer, call agent \\n\\\n","    “61 :Educational Services”—Activities of this sector are providing instruction and training in a wide variety of subjects.Example job titles: instructor, professor, teaching assistant, teacher, tutor, academic advisor, guidance counselor, student services, school administrator, registrar, librarian  \\n\\\n","    “62 :Health Care and Social Assistance”—Activities of this sector are providing health care and social assistance for individuals.Example job titles: home health aide, caregiver, nurse, medical assistant, medical receptionist, medical secretary, certified dental assistant, dental hygienist, x-ray technician, physical therapy assistant, case manager, therapist \\n\\\n","    “71 :Arts, Entertainment, and Recreation”—Activities of this sector are operating or providing services to meet varied cultural, entertainment, and recreational interests of their patrons.Example job titles: ride operator, fitness trainer, ski instructor, lifeguard, card dealer, usher, actor, actress, dancer, tour guide\\n\\\n","    “72 :Accommodation and Food Services”—Activities of this sector are providing customers with lodging and/or preparing meals, snacks, and beverages for immediate consumption.Example job titles: cook, waiter, server, barista, maid, cafeteria assistant, guest service representative, bellhop \\n\\\n","    “81 :Other Services (except Public Administration)”—Activities of this sector are providing services not elsewhere specified, including repairs, religious activities, grantmaking, advocacy, laundry, personal care, death care, and other personal services.Example job titles: hairdresser, beautician, groomer, dry cleaner, auto body repair technician, funeral attendant, manicurist \\n\\\n","    “92 :Public Administration”—Activities of this sector are administration, management, and oversight of public programs by Federal, State, and local governments.Example job titles: mail carrier, police officer, highway maintenance crew worker, firefighter, correctional officer, crossing guard, town clerk, emergency communications dispatcher\\n\\\n","    \\n\\\n","    The NAICS sector code is '62' for:\\n\\\n","    [Voting Unit (Unit A)]='Included: All full-time and regular part-time  technical employees employed by the Employer at its Olean, New York facility, including the classifications of Licensed Practical Nurse, Biomedical Services Technician, CT Tech, Dental Hygenist, Registered Cardiac Sonographer (Echo Tech), Licensed Practical Nurse â€“ Endoscopy Technician (Endo Tech), MRI Technologist, Nuclear Medicine Technologist, Certified Occupational Therapy Assistant, Physical Therapy Assistant, Radiation Therapist, Radiologic Technologist, Registered Cardiac Sonographer, Registered Sleep Laboratory Technician, Certified Respiratory Therapy Technician, Registered Respiratory Therapist, and Ultrasound Technologist.'\\n\\\n","    [Labor Union1]='Communications Workers of America, AFL-CIO'\\n\\\n","    [Case Name]='Olean General Hospital'\\n\\\n","    \\n\\\n","    The NAICS sector code is “48-49” for:\\n\\\n","    [Voting Unit (Unit A)]='All full-time and regular part-time warehouse employees, semi drivers and route employees employed at the Employer‚Äôs facilities located at 2900 Jackson Street, Oshkosh, Wisconsin, 110 Sime Avenue, Tomah, Wisconsin, and 2129 Harlem Street, Eau Claire, Wisconsin;'\\n\\\n","    [Labor Union1]='Teamsters General Local Union No. 200'\\n\\\n","    [Case Name]='The American Bottling Company, d/b/a Keurig Dr Pepper'\\n\\\n","    \\n\\\n","    Please give the NAICS sector code for: [Voting Unit (Unit A)]= '{voting_unit}', [Labor Union1]='{union_}', and [Case Name]='{case_name}''\"\n","\n","    response = openai.ChatCompletion.create(\n","        model=\"gpt-3.5-turbo\", #3.5-turbo\n","        messages=[\n","            {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n","            {\"role\": \"user\", \"content\": prompt}\n","        ],\n","        max_tokens=60,\n","        temperature=0.7\n","    )\n","    return response['choices'][0]['message']['content'].strip()"],"metadata":{"id":"XqEuxBJs_H0i"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Iterate over the DataFrame with tqdm\n","from tqdm import tqdm\n","import time\n","\n","for index_, row in tqdm(df_sample.iterrows(), total=df_sample.shape[0], desc=\"Processing rows\"):\n","    case_name = row['Case_Name']\n","    voting_unit = row['Voting_Unit_Unit_A']\n","    state_ = row['States_&_Territories']\n","    city = row['City']\n","    union_ = row['Labor_Union1']\n","    out_wlc = naics_codes_wlc(voting_unit, union_, case_name, state_, city)\n","    df_sample.at[index_, 'output_wlc'] = out_wlc"],"metadata":{"id":"wKw7sY2fPgGG"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Classification  with Transformers"],"metadata":{"id":"51YKqDR-Px0_"}},{"cell_type":"code","source":["!pip install -q bitsandbytes datasets accelerate loralib\n","!pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git"],"metadata":{"id":"4-mdvmSATd9f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["!pip install -U git+https://github.com/huggingface/trl"],"metadata":{"id":"ktjSLxzMWmMU"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["import pandas as pd\n","import numpy as np\n","df_source_corpus=pd.read_csv('/content/drive/MyDrive/catarina/source_corpus.csv')\n","df_source_corpus = df_source_corpus.dropna(subset=['text'])"],"metadata":{"id":"BC4HJqrs_oN_"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["df_sample.head()"],"metadata":{"id":"NQrhvcaW_7OH"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Complete the classification pipeline\n","\n","\n","1.   Split the data correctly for batches of 8 (line 37)\n","2.   Add right code in line 48, 50,51\n","\n","\n","\n"],"metadata":{"id":"AzleRX0rDkZH"}},{"cell_type":"code","source":["import torch\n","from transformers import DistilBertTokenizer, TextClassificationPipeline, DistilBertForSequenceClassification\n","from sklearn.metrics import confusion_matrix, classification_report\n","import numpy as np\n","import pandas as pd\n","from tqdm import tqdm\n","\n","# Load the dataset\n","df_sample = df_source_corpus[0:1000]\n","df_sample = df_sample.assign(topic_id=(df_sample['topic_8']).astype('category').cat.codes)\n","\n","# Determine the number of unique classes\n","num_classes = len(set(df_sample['topic_8'].tolist()))\n","print(f\"Number of classes: {num_classes}\")\n","\n","model_name = 'distilbert-base-uncased'\n","model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=num_classes)\n","tokenizer = DistilBertTokenizerFast.from_pretrained(model_name)\n","\n","# Prepare inputs and labels\n","inputs = tokenizer(df_sample['text'].tolist(), return_tensors=\"pt\", padding=True, truncation=True)\n","labels = torch.tensor(df_sample['topic_id'].tolist()).long()\n","\n","optimizer = torch.optim.Adam([\n","    {'params': model.distilbert.parameters(), 'lr': 1e-5},\n","    {'params': model.classifier.parameters(), 'lr': 1e-3}\n","])\n","\n","from sklearn.model_selection import train_test_split\n","\n","X_train, X_test, y_train, y_test = train_test_split(df_sample['text'].tolist(), df_sample['topic_id'].tolist(), test_size=.2)\n","\n","# Generate batches\n","X_train, X_test, y_train, y_test = np.array(X_train[:344]), np.array(X_test[:80]), np.array(y_train[:344]), np.array(y_test[:80])\n","\n","# TODO:  BATCHES OF 8\n","X_train, X_test, y_train, y_test = ...\n","\n","\n","X_train, X_test = X_train.tolist(), X_test.tolist()\n","\n","# Train the model\n","model.to('cuda:0')\n","num_epochs = 3\n","for epoch in range(num_epochs):\n","    model.train()\n","    for text, labels in tqdm(zip(X_train, y_train), total=len(X_train)):\n","      #TODOD\n","        model_inputs = ...\n","        model_inputs = {k: v.to('cuda:0') for k, v in model_inputs.items()}\n","      #TODO\n","        labels = ...\n","        output = ...\n","        loss, logits = output[:2]\n","        loss.backward()\n","        optimizer.step()\n","        optimizer.zero_grad()\n","\n","torch.save(model, 'topic_classifier.pt')\n","topic_classifier = model\n","\n","distil_tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', max_length=512, truncation=True)\n","pipeline = TextClassificationPipeline(model=topic_classifier, tokenizer=distil_tokenizer, return_all_scores=True, device=0)\n","\n","\n","\n","\n","\n"],"metadata":{"id":"3AfBtwVhiIhX"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"xh6CelonDPon"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"-Za4sCIIDPkt"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"jZy8Z9NyDPhM"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"FVpVzXhlDPdu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"nQtV0-FGDPaQ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"JNUs2qYyDPWp"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"GkmI0G5aDPSZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"ay0Vr4juDPN_"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"y4oFDOPpDPKO"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"zKEv5sP9DPGT"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Do predictions on Test dataset\n","\n"],"metadata":{"id":"L6ZSle-WDAxU"}},{"cell_type":"code","source":["\n","# Tokenize and predict on the test set\n","y_true = []\n","y_pred = []\n","\n","for text, true_label in zip(X_test, y_test):\n","    model_inputs = distil_tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True, max_length=256)\n","    model_inputs = {k: v.to('cuda:0') for k, v in model_inputs.items()}\n","\n","    with torch.no_grad():\n","       #TODO:\n","        output = ...\n","        logits = ...\n","        predictions = ...\n","\n","    y_true.extend(true_label)\n","    y_pred.extend(predictions)"],"metadata":{"id":"dFNFIwpSC89i"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"nStg_SGXC86T"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"6iJiy4BJC83q"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"I2bEambVC80s"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"KZ5eNlX7C8xH"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"sBdQX4HeC8tF"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"bKgWTqtLC8pz"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Create the metrics\n","\n"],"metadata":{"id":"MOo-u5uSCegf"}},{"cell_type":"code","source":["category_mapping = {\n","    'economy': 0,\n","    'politics': 1,\n","    'society': 2,\n","    'freedom and democracy': 3,\n","    'external relations': 4,\n","    'fabric of society': 5,\n","    'welfare and quality of life': 6,\n","    'no topic': 7,\n","}\n","\n","\n","\n","\n","# Ensure y_true and y_pred are in the correct shape\n"," #TODO:\n","y_true = ...\n","y_pred = ..\n","\n","\n","\n","# Generate classification report\n","class_report = ...\n","print(\"Classification Report:\")\n","print(class_report)\n"],"metadata":{"id":"hPjMe_leCg8-"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"FEdOBUrdCg57"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"YY07xnRTCg2x"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"qucknArgCgz6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"waHCd_QOCgw9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"lHCgr0v5Cgtp"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"rWbkTTD0p2Q_"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"px7Uci4zHVjj"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# Fine Tune LLMs\n","\n","[xturing](https://xturing.stochastic.ai/overview/quickstart/data_usage/)\n"],"metadata":{"id":"PPcV2WRbYfZX"}},{"cell_type":"code","source":["!pip install xturing --upgrade"],"metadata":{"id":"4ycpJEoAHlYE"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"WJEFfGBijMzv"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Download the data\n","!wget https://raw.githubusercontent.com/tatsu-lab/stanford_alpaca/main/alpaca_data.json"],"metadata":{"id":"PHX55YJoHtIT"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["import os\n","os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n"],"metadata":{"id":"Jg-yRBKK4_3r"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["##Adapt the data to look like a prompt\n","\n","import json\n","from datasets import Dataset, DatasetDict\n","\n","alpaca_data = json.load(open('/content/alpaca_data.json'))\n","instructions = []\n","inputs = []\n","outputs = []\n","\n","for data in alpaca_data[0:100]:\n","    instructions.append(data[\"instruction\"])\n","    inputs.append(data[\"input\"])\n","    outputs.append(data[\"output\"])\n","\n","data_dict = {\n","    \"train\": {\"instruction\": instructions, \"text\": inputs, \"target\": outputs}\n","}\n","\n","dataset = DatasetDict()\n","for k, v in data_dict.items():\n","    dataset[k] = Dataset.from_dict(v)\n","\n","dataset.save_to_disk(str(\"./alpaca_data\"))"],"metadata":{"id":"doSgCipOH25o"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from xturing.datasets.instruction_dataset import InstructionDataset\n","from xturing.models.base import BaseModel\n","\n","instruction_dataset = InstructionDataset(\"./alpaca_data\")\n","# Initializes the model\n","\n","model = BaseModel.create(\"gpt2_int8\")"],"metadata":{"id":"gmR7_CRtH6Hf"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Finetuned the model\n","\n","model.finetune(dataset=instruction_dataset)"],"metadata":{"id":"6uPQlP-bH82i"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"V678o3RVsrRO"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["\n","# Once the model has been finetuned, you can start doing inferences\n","output = model.generate(texts=[\"Name the 6 most populous cities in China.\"])\n","print(\"Generated output by the model: {}\".format(output))"],"metadata":{"id":"19g03R-bH_kD"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["\n","\n","> Apply to your dataset\n","\n","\n","1.   Data Transformation: line 14-16 (first cell)\n","2.   Dataset loader: line 4, 7 (second cell)\n","3.   Fine tune the model (third cell)\n","4.   Do inferences (fourth cell)\n","\n","\n","\n"],"metadata":{"id":"H3F2A9zfvung"}},{"cell_type":"code","source":["import pandas as pd\n","import numpy as np\n","df_source_corpus=pd.read_csv('/content/drive/MyDrive/catarina/source_corpus.csv')\n","df_source_corpus = df_source_corpus.dropna(subset=['text'])"],"metadata":{"id":"hOocklCN7k9A"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from datasets import Dataset, DatasetDict\n","\n","df_sample=df_source_corpus[0:100]\n","\n","\n","df_sample[\"input\"]=''\n","\n","instructions = []\n","inputs = []\n","outputs = []\n","\n","\n","for index_, row in df_sample.iterrows():\n","  #TODO\n","    ...\n","    ...\n","    ...\n","\n","data_dict = {\n","    \"train\": {\"instruction\": instructions, \"text\": inputs, \"target\": outputs}\n","}\n","\n","dataset = DatasetDict()\n","for k, v in data_dict.items():\n","    dataset[k] = Dataset.from_dict(v)\n","\n","#dataset.save_to_disk(str(\"./parliament_data\"))"],"metadata":{"id":"CZlI6cosv57h"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from xturing.datasets.instruction_dataset import InstructionDataset\n","from xturing.models.base import BaseModel\n"," #TODO:\n","instruction_dataset_parliament = ...\n","# Initializes the model\n","\n","model_parliament = ..."],"metadata":{"id":"7RCNCfrXv6lb"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Finetuned the model\n"," #TODO:\n","..."],"metadata":{"id":"zWzbB6uYv6g1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Once the model has been finetuned, you can start doing inferences\n"," #TODO:\n","..."],"metadata":{"id":"1W6y9phg-f5P"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"AqbtE5Wb-hre"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"h-dS4YTY-hnD"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"BspV4wwO-hjI"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"Gc41N9eO-he_"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"Ke9s8d9q-hbZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"Qm27TApe-hXI"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"fEToNoAI-hTQ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"W5YPrzu5-hO7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"32_lWCVa-hLJ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"9nKCNNdh-hHS"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# solution"],"metadata":{"id":"p1c0oiFo-hDG"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["df_sample.head()"],"metadata":{"id":"NxFV2T3Qw6Ut"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from datasets import Dataset, DatasetDict\n","\n","df_sample=df_source_corpus[0:100]\n","\n","\n","df_sample[\"input\"]=''\n","\n","instructions = []\n","inputs = []\n","outputs = []\n","\n","\n","for index_, row in df_sample.iterrows():\n","    instructions.append(\"What is the topic of this text: \" +row[\"text\"] )\n","    inputs.append(row[\"input\"])\n","    outputs.append(row[\"topic_8\"])\n","\n","data_dict = {\n","    \"train\": {\"instruction\": instructions, \"text\": inputs, \"target\": outputs}\n","}\n","\n","dataset = DatasetDict()\n","for k, v in data_dict.items():\n","    dataset[k] = Dataset.from_dict(v)\n","\n","#dataset.save_to_disk(str(\"./parliament_data\"))"],"metadata":{"id":"yTprQIsyKqCt"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["from xturing.datasets.instruction_dataset import InstructionDataset\n","from xturing.models.base import BaseModel\n","\n","instruction_dataset_parliament = InstructionDataset(dataset) # \"./parliament_data\"\n","# Initializes the model\n","\n","model_parliament = BaseModel.create(\"gpt2_int8\")"],"metadata":{"id":"q7WJhOWMxemQ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Finetuned the model\n","model_parliament.finetune(dataset=instruction_dataset_parliament)\n"],"metadata":{"id":"wT9gJldJxhoA"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["\n","# Once the model has been finetuned, you can start doing inferences\n","output_parl = model_parliament.generate(texts=[df_sample['text'].iloc[101]])\n","print(\"Generated output by the model: {}\".format(output_parl))"],"metadata":{"id":"LvAQvzwoyQv9"},"execution_count":null,"outputs":[]}]}