📃: (GenAI Beginners)Build a Generatively Pretrained Transformer (GPT) referring to "Attention is All You Need", OpenAI GPT-2, and Shakespeare Dataset

[shaansuraj]

🔴 Title : Build a Generatively Pretrained Transformer (GPT) referring to "Attention is All You Need", OpenAI GPT-2, and Shakespeare Dataset
🔴 Aim : Generating the next word based on the context of the previous ones.
🔴 Brief Explanation : Make use of Self Attention mechanisms to process and generate text efficiently, without relying on recurrent neural networks. You have to refer to "Attention is All You Need" paper and learn about GPT models, like OpenAI's GPT-2, which are pretrained on large text corpora, learning to predict the next word in a sentence, which allows them to generate coherent and contextually relevant text. You can start with Shakespeare's Dataset, the model learns the patterns specific to the playwright’s language, enabling it to generate text in a similar style. This combination of large-scale pretraining and fine-tuning on domain-specific data makes GPT highly effective for creative text generation tasks. This will be your first step towards Generative AI.

Here are the links for your help:

Attention is All You Need: https://arxiv.org/abs/1706.03762
Shakespeare Plays Dataset: https://www.kaggle.com/datasets/kingburrito666/shakespeare-plays

For any issues you can reach me out via email: [email protected]

Screenshots 📷

---

✅ To be Mentioned while taking the issue :

• Full name : Suraj Sahu
• What is your participant role? (Mention the Open Source Program name. Eg. GSOC, GSSOC, SSOC, JWOC, etc.)
  GSSOC'24 Extended and Hacktoberfest Mentor

---

Happy Contributing 🚀

All the best. Enjoy your open source journey ahead. 😎

[github-actions]

🙌 Thank you for bringing this issue to our attention! We appreciate your input and will investigate it as soon as possible.

[Abhicoder03]

Hey I found this issue interesting. Could you assign this issue to me.

[shaansuraj]

@Abhicoder03 I am glad that you found this topic interesting. I have assigned it to you. If you have any doubts, you can reach me out through the email I provided above.

[JahnaviDhanaSri]

Hi @shaansuraj
I’m interested in contributing to the Build a Generatively Pretrained Transformer (GPT) project. I plan to focus on:

Understanding Self-Attention Mechanisms, I will thoroughly study the "Attention is All You Need" paper and the architecture of OpenAI's GPT-2 to grasp how self-attention enables efficient text generation.
I will work with the Shakespeare Dataset to learn and emulate the unique language patterns of the playwright, enhancing the model's text generation capabilities.
I will implement the GPT model, ensuring it generates coherent and contextually relevant text based on the context of previous words.

[shaansuraj]

@JahnaviDhanaSri Okay sure, I will assign you to this issue as well. You can work collaboratively with @Abhicoder03 on this.

[JahnaviDhanaSri]

Thank You!

…

On Wed, 2 Oct 2024 at 22:17, Suraj Sahu ***@***.***> wrote: Assigned #31 <#31> to @JahnaviDhanaSri <https://github.com/JahnaviDhanaSri>. — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCKE3GS7THMNH7UBNIVWWQ3ZZQPTNAVCNFSM6AAAAABPHWSRGCVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGQ4TEMBXGUZDINQ> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Abhicoder03]

@JahnaviDhanaSri Hey, we can collaborate on this project.

[JahnaviDhanaSri]

Sure!

…

On Wed, 2 Oct 2024 at 22:58, Abhishek Choudhary ***@***.***> wrote: @JahnaviDhanaSri <https://github.com/JahnaviDhanaSri> Hey, we can collaborate on this project. — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BCKE3GVMJD6PTCYDNRKDBQTZZQUNNAVCNFSM6AAAAABPHWSRGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBZGIZDKNZYGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[github-actions]

✅ This issue has been closed. Thank you for your contribution! If you have any further questions or issues, feel free to reach out!