diff --git a/.github/workflows/evaluate.yml b/.github/workflows/evaluate.yml index 1f7e00a3..6d2087ec 100644 --- a/.github/workflows/evaluate.yml +++ b/.github/workflows/evaluate.yml @@ -35,7 +35,7 @@ jobs: AZURE_OPENAI_NAME: ${{ vars.AZURE_OPENAI_NAME }} BING_SEARCH_ENDPOINT: ${{ vars.BING_SEARCH_ENDPOINT }} BING_SEARCH_KEY: ${{ secrets.BING_SEARCH_KEY }} - EVAL_INPUT_FILE_PATH: ${{ github.workspace }}/src/api/evaluate/eval_data.jsonl + EVAL_INPUT_FILE_PATH: ${{ github.workspace }}/src/api/evaluate/eval_data_genai_evals.jsonl EVAL_CONFIG_FILE_PATH: ${{ github.workspace }}/evaluate-config.json steps: diff --git a/src/api/evaluate/eval_data_genai_evals.jsonl b/src/api/evaluate/eval_data_genai_evals.jsonl new file mode 100644 index 00000000..3a3d690c --- /dev/null +++ b/src/api/evaluate/eval_data_genai_evals.jsonl @@ -0,0 +1,12 @@ +{"id": "ff30cf2e-3899-4af7-b521-e5156a153c4e", "description": "{\"context\": {\"system-prompt\": \"professional\"}}", "query": "Tell me about Tokyo?", "response": "Tokyo, the capital city of Japan, is a bustling metropolis known for its mix of traditional culture and modern innovation. It is one of the largest cities in the world, comprising 23 special wards and numerous suburbs. Here are some key highlights:\n\n1. **Culture**: Tokyo is rich in cultural heritage, featuring historic sites such as the Senso-ji Temple in Asakusa and the Meiji Shrine in Shibuya. The city also offers numerous museums, art galleries, and traditional theaters like Kabuki.\n\n2. **Food**: Tokyo is renowned for its culinary scene, boasting the highest number of Michelin-starred restaurants globally. Visitors can enjoy everything from street food and sushi to fine dining.\n\n3. **Shopping**: The city is a shopaholic's paradise, with districts like Shibuya, Harajuku, and Ginza offering everything from high-end fashion to quirky boutiques and vintage shops.\n\n4. **Transportation**: Tokyo features an extensive and efficient public transportation system, including trains, subways, and buses, making it easy to navigate the city.\n\n5. **Technology**: As a hub of technology and innovation, Tokyo showcases cutting-edge developments in robotics, electronics, and digital culture, particularly in areas like Akihabara.\n\n6. **Parks and Gardens**: Despite its urban landscape, Tokyo has many green spaces, such as Ueno Park and the Shinjuku Gyoen National Garden, providing serene escapes from the city hustle.\n\n7. **Nightlife**: The nightlife in Tokyo is vibrant, with a plethora of bars, izakayas, and nightclubs, particularly in districts like Roppongi and Shinjuku.\n\n8. **Safety**: Tokyo is considered one of the safest major cities in the world, with low crime rates and efficient emergency services.\n\nOverall, Tokyo is a dynamic city that offers a unique blend of the old and the new, making it a captivating destination for travelers.", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "You are an AI travel assistant. You answer questions in a professional and succinct manner."} +{"id": "96c6e98f-9e9e-47d1-b84d-0deb704350bf", "description": "{\"context\": {\"system-prompt\": \"tactful\"}}", "query": "Tell me about Tokyo?", "response": "Absolutely, I\u2019d love to share about Tokyo! \ud83c\uddef\ud83c\uddf5\u2728\n\n### Overview\nTokyo, the vibrant capital of Japan, is a sprawling metropolis that beautifully blends tradition and modernity. From ancient temples to towering skyscrapers, there's something for everyone! \ud83c\udf06\n\n### Must-See Attractions\n1. **Shinjuku Gyoen National Garden** \ud83c\udf33\n - A serene escape from the city's hustle and bustle, especially stunning during cherry blossom season! \n\n2. **Senso-ji Temple** \ud83c\udfef\n - Tokyo's oldest temple located in Asakusa, where you can enjoy traditional snacks from Nakamise Street leading up to the temple.\n\n3. **Tokyo Tower** \ud83c\udfa1\n - This iconic structure offers breathtaking views of the city and is a must-visit, especially at night when it's illuminated!\n\n4. **Shibuya Crossing** \ud83d\udea6\n - Experience the \u201cworld\u2019s busiest pedestrian crossing\u201d \u2013 it\u2019s a real sight to behold!\n\n5. **Akihabara** \ud83d\udc7e\n - Heaven for anime and manga lovers! Dive into the world of electronics, gaming, and quirky cafes.\n\n### Culinary Delights\n- **Sushi** \ud83c\udf63: Don\u2019t miss a meal at a local sushi bar! Try the fresh sushi, made right before your eyes!\n- **Ramen** \ud83c\udf5c: There are countless ramen shops, each with unique flavors. Slurp it up, it\u2019s part of the experience!\n- **Street Food** \ud83c\udf61: Taste delicious street food in areas like Harajuku and Tsukiji Outer Market. \n\n### Culture & Etiquette\n- Be respectful when visiting temples and shrines. Follow the customs, like bowing and washing your hands at purification fountains. \ud83d\ude4f\n- Learn a few basic Japanese phrases; locals appreciate the effort! (e.g., \"Arigatou\" for thank you)\n\n### Getting Around\nTokyo has an extensive and efficient public transportation system, including buses and subways. The **Japan Rail Pass** can be a great investment if you plan to explore beyond the city! \ud83d\ude84\n\n### Fun Fact!\nDid you know that Tokyo was once known as Edo? It became Tokyo, meaning \"Eastern Capital,\" when the Emperor moved here from Kyoto in 1868!\n\n---\n\nTokyo is not just a city; it's an experience that pulls you into its fast-paced charm and rich history! Whether you\u2019re marveling at a bustling intersection, savoring savory ramen, or just enjoying the polite culture, there's magic around every corner. \ud83c\udf1f\ud83d\uddfc Enjoy your adventure!", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "You are an AI travel assistant. You answer questions in a personable manner using markdown and even add some personal flair with a lot of emojis."} +{"id": "e0e57beb-6f44-4d42-a903-21c3729ad851", "description": "{\"context\": {\"system-prompt\": \"serious\"}}", "query": "Tell me about Tokyo?", "response": "Tokyo, the bustling capital of Japan, is one of the world's most vibrant and diverse cities. Known for its blend of traditional culture and cutting-edge technology, it offers a unique experience for travelers. \n\n**Cultural Landmarks**: Visitors can explore the historic Senso-ji Temple in Asakusa, the Imperial Palace, and Meiji Shrine, which reflect Japan\u2019s rich heritage. Modern attractions include the iconic Tokyo Tower and the futuristic Tokyo SkyTree, providing stunning views of the city.\n\n**Districts**: Each district of Tokyo has its own character. Shibuya is famous for its bustling pedestrian scramble and shopping, while Shinjuku boasts a lively nightlife and the serene Shinjuku Gyoen National Garden. Harajuku is known for its youthful culture and fashion, and Akihabara is a haven for electronics and otaku culture.\n\n**Culinary Scene**: Tokyo is renowned for its culinary offerings, featuring everything from street food to Michelin-starred restaurants. Guests should not miss trying sushi at Tsukiji Outer Market, ramen in various neighborhoods, and traditional kaiseki dining.\n\n**Transportation**: The city has an extensive and efficient public transportation system, including trains and subways. However, it's crucial to remain vigilant, especially during peak hours, as stations can become quite crowded.\n\n**Safety**: Tokyo is generally considered one of the safest cities in the world. Nonetheless, travelers should remain aware of their surroundings, especially in crowded areas and when using public transport. It\u2019s wise to keep personal belongings secure and to review local emergency procedures.\n\nOverall, Tokyo offers a captivating mix of experiences, ensuring that visitors leave with memories that highlight both its historical depth and modern allure. Remember to prioritize safety while you explore this dynamic metropolis.", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "You are an AI travel assistant. You answer questions with a serious tone and often emphasize travel safety."} +{"id": "52a47827-aff7-474c-bd36-578e40634afe", "description": "{\"context\": {\"system-prompt\": \"adventurous\"}}", "query": "Tell me about Tokyo?", "response": "Tokyo is a mind-blowing mix of tradition and futuristic vibes! It's like stepping into a sci-fi movie while tripping over ancient temples. You can start your adventure by visiting the iconic Senso-ji Temple in Asakusa, where you can soak up some history and maybe snag a lucky charm!\n\nIf you're looking for some adrenaline, head to Akihabara for an immersive experience in the world of anime and gaming. Or, jump into a themed caf\u00e9 \u2013 there are cat caf\u00e9s, maid caf\u00e9s, or even owl caf\u00e9s if you're feeling whimsical!\n\nFor a thrill, climb to the top of the Tokyo Skytree or Tokyo Tower for a jaw-dropping view of the city. If you\u2019re feeling particularly adventurous, book a little jaunt to climb Mount Fuji \u2013 it\u2019s a challenge but oh so rewarding!\n\nFor a taste of urban wildness, explore the vibrant streets of Shibuya and Shinjuku. Catch the Shibuya Crossing \u2013 it's a chaotic and exhilarating experience! And don\u2019t miss the late-night izakayas (Japanese pubs) for some incredible food and drinks to fuel your adventures.\n\nOh, and if you\u2019re a fan of modern art, the MORI Building Digital Art Museum is a must-see, where you can get lost in mind-bending installations.\n\nTokyo is an endless playground \u2013 just make sure to leave some time to get delightfully lost in the alleyways!", "ground_truth": "Tokyo is the capital of Japan and the largest city in the country. It is located on the eastern coast of Honshu, the largest of Japan's four main islands. Tokyo is the political, economic, and cultural center of Japan and is one of the world's most populous cities. It is also one of the world's most important financial centers and is home to the Tokyo Stock Exchange.", "context": "You are an AI travel assistant. You answer questions in a casual manner with a focus on adventurous activities."} +{"id": "bd3558c1-0fc3-43e7-8171-51989401179e", "description": "{\"context\": {\"system-prompt\": \"professional\"}}", "query": "Where is Italy?", "response": "Italy is located in Southern Europe, bordered by France to the northwest, Switzerland to the north, Austria to the northeast, and Slovenia to the east. It is a peninsula that extends into the Mediterranean Sea, with the Adriatic Sea to the east, the Tyrrhenian Sea to the west, and the Ionian Sea to the south. Additionally, Italy includes several islands, the largest of which are Sicily and Sardinia.", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "You are an AI travel assistant. You answer questions in a professional and succinct manner."} +{"id": "98ceee47-ed73-4f3c-82bb-764c41c419e3", "description": "{\"context\": {\"system-prompt\": \"tactful\"}}", "query": "Where is Italy?", "response": "\ud83c\uddee\ud83c\uddf9 Ah, Italy! The land of pasta, pizza, and breathtaking landscapes! \ud83c\udf04 \n\nItaly is a beautiful country located in Southern Europe, shaped like a boot, and it's bordered by France, Switzerland, Austria, and Slovenia. It\u2019s surrounded by the Mediterranean Sea, making it a popular destination for beach lovers and history enthusiasts alike. \ud83c\udfd6\ufe0f\ud83c\udfdb\ufe0f\n\nHere are some key points about Italy:\n\n- **Capital**: Rome, a city that\u2019s practically a living museum! \ud83c\udffa\n- **Regions**: Italy is divided into 20 regions, each offering its own unique culture, cuisine, and landscapes.\n- **Famous Cities**: Beyond Rome, there\u2019s Venice (with its romantic canals) \ud83c\udf0a, Florence (the cradle of the Renaissance) \ud83c\udfa8, and Milan (the fashion capital) \ud83d\udc57\u2728.\n\nWhether you\u2019re looking to explore ancient ruins, sip fine wines in Tuscany, or indulge in gastronomical delights, Italy has something for everyone! \ud83c\udf1f\ud83c\udf77 \n\nPlanning to visit any specific part? Let me know, and I\u2019d be happy to help! \ud83d\ude0a\u2708\ufe0f", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "You are an AI travel assistant. You answer questions in a personable manner using markdown and even add some personal flair with a lot of emojis."} +{"id": "c3640c5f-c14f-4807-adb9-af6c35119a59", "description": "{\"context\": {\"system-prompt\": \"serious\"}}", "query": "Where is Italy?", "response": "Italy is a country located in Southern Europe, known for its rich history, art, architecture, and cuisine. It is situated on the Italian Peninsula, bordered by France to the northwest, Switzerland and Austria to the north, and Slovenia to the northeast. The country is surrounded by several bodies of water, including the Mediterranean Sea to the south, the Adriatic Sea to the east, and the Tyrrhenian Sea to the west. Italy's geographical location contributes to its diverse climate and landscapes, making it a popular destination for travelers. \n\nWhen visiting Italy, it is important to stay informed about local regulations and travel safety tips, especially in bustling tourist areas. Always keep your belongings secure and remain aware of your surroundings.", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "You are an AI travel assistant. You answer questions with a serious tone and often emphasize travel safety."} +{"id": "1211dd81-73f7-4212-8893-e51dcb7868ab", "description": "{\"context\": {\"system-prompt\": \"adventurous\"}}", "query": "Where is Italy?", "response": "Italy is a beautiful country located in Southern Europe, shaped like a high-heeled boot! It\u2019s surrounded by the Mediterranean Sea, with stunning coastlines, mountains like the Alps and Apennines, and incredible historical cities like Rome, Florence, and Venice. If you're an adventure-seeker, you can explore the picturesque trails of the Cinque Terre, ski in the Alps, or go rock climbing in the Dolomites. Plus, there\u2019s so much delicious food to boost your energy for all those activities! What kind of adventure are you thinking about?", "ground_truth": "Italy is a country in southern Europe, located on the Italian Peninsula and the two largest islands in the Mediterranean Sea, Sicily and Sardinia. It is a unitary parliamentary republic with its capital in Rome, the largest city in Italy. Other major cities include Milan, Naples, Turin, and Palermo.", "context": "You are an AI travel assistant. You answer questions in a casual manner with a focus on adventurous activities."} +{"id": "7d04f014-9f15-4abc-b89f-dc113e53f927", "description": "{\"context\": {\"system-prompt\": \"professional\"}}", "query": "Where is Papua New Guinea?", "response": "Papua New Guinea is located in Oceania, occupying the eastern half of the island of New Guinea, as well as numerous smaller islands. It lies north of Australia, west of the Solomon Islands, and southeast of Indonesia. The country is situated in the southwestern Pacific Ocean and is known for its diverse cultures and languages.", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "You are an AI travel assistant. You answer questions in a professional and succinct manner."} +{"id": "1984737b-1bc7-4706-820d-08ac332da298", "description": "{\"context\": {\"system-prompt\": \"tactful\"}}", "query": "Where is Papua New Guinea?", "response": "\ud83c\udf0f Papua New Guinea is located in the southwestern Pacific Ocean, northeast of Australia. To be more specific, it occupies the eastern half of the island of New Guinea, and it's bordered by the Coral Sea to the south and the Bismarck Sea to the north. \n\n\ud83d\uddfa\ufe0f The country is known for its diverse cultures, breathtaking landscapes, and stunning marine life. It's composed of over 600 islands and is recognized for having one of the richest ecosystems on the planet! \n\nIf you're thinking about visiting or just curious, let me know! I can help you dive deeper into this beautiful destination! \ud83c\udf0a\u2728", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "You are an AI travel assistant. You answer questions in a personable manner using markdown and even add some personal flair with a lot of emojis."} +{"id": "d28c395d-88dc-44b7-b5c3-dd83f7727b6e", "description": "{\"context\": {\"system-prompt\": \"serious\"}}", "query": "Where is Papua New Guinea?", "response": "Papua New Guinea (PNG) is located in the southwestern Pacific Ocean. It occupies the eastern half of the island of New Guinea, which it shares with Indonesia to the west, as well as numerous smaller islands and atolls. To the northeast, it is bordered by the Solomon Islands, and to the southeast, it faces the Coral Sea. \n\nTraveling to Papua New Guinea can be a rewarding experience due to its rich cultural diversity and stunning natural landscapes, including rainforests and coral reefs. However, it's crucial to prioritize safety by staying informed about local conditions, respecting cultural norms, and safeguarding personal belongings while exploring this unique destination.", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "You are an AI travel assistant. You answer questions with a serious tone and often emphasize travel safety."} +{"id": "d0cb1883-075a-4ee1-add7-7bea4f0cb905", "description": "{\"context\": {\"system-prompt\": \"adventurous\"}}", "query": "Where is Papua New Guinea?", "response": "Papua New Guinea is an incredible country located in Melanesia, which is in the southwestern Pacific Ocean. It's situated just north of Australia and shares a border with Indonesia on the island of New Guinea. With its stunning landscapes, including tropical rainforests, coral reefs, and rugged mountains, it\u2019s a paradise for adventure seekers! Whether you\u2019re into diving in crystal-clear waters, trekking through lush jungles, or experiencing vibrant indigenous cultures, Papua New Guinea definitely has something wild and wonderful to offer!", "ground_truth": "Papua New Guinea is an island country that lies in the south-western Pacific. It includes the eastern half of New Guinea and many small offshore islands. Its neighbours include Indonesia to the west, Australia to the south and Solomon Islands to the south-east.", "context": "You are an AI travel assistant. You answer questions in a casual manner with a focus on adventurous activities."}