Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Offline DeepSeek Model #119

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

varshith257
Copy link

@varshith257 varshith257 commented Apr 5, 2025

This PR implements an offline DeepSeek model loader and inference wrapper fulfilling all requirements in issue. It provides a lightweight, memory-efficient, dependency-minimal way to load and run DeepSeek models from HuggingFace which supports all official DeepSeek R1 & Distill variants using HuggingFace + safetensors. Dynamic model config parsing and deeper memory optimizations (e.g., Triton/offloading) can be addressed in a follow-up issue if needed.

NOTES TO REVIEWERS

Tested on DeepSeek-R1-Distill-Qwen-7B that works offline and follows low-level-only policy and here's o/p in local testing :


> python -m unittest intelli.test.integration.test_deepseek_wrapper

----------------------------------------------------------------------
Ran 1 test in 129.257s

OK
Downloading model.safetensors.index.json from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B...
Index downloaded to /home/runner/.cache/deepseek/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/snapshots/91[6](https://github.com/varshith257/Intelli/pull/1/checks#step:6:7)b56a44061fd5cd7d6a8fb63255[7](https://github.com/varshith257/Intelli/pull/1/checks#step:6:8)ed4f724f60/model.safetensors.index.json
Downloading config.json from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B...
Config downloaded to /home/runner/.cache/deepseek/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/snapshots/916b56a44061fd5cd7d6a[8](https://github.com/varshith257/Intelli/pull/1/checks#step:6:9)fb632557ed4f724f60/config.json
Downloading model.safetensors.index.json from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B...
Index downloaded to /home/runner/.cache/deepseek/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/snapshots/[9](https://github.com/varshith257/Intelli/pull/1/checks#step:6:10)16b56a44061fd5cd7d6a8fb632557ed4f724f60/model.safetensors.index.json
Downloading model-00002-of-000002.safetensors from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B...
Model downloaded to /home/runner/.cache/deepseek/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/snapshots/916b56a44061fd5cd7d6a8fb632557ed4f724f60/model-00002-of-000002.safetensors
Downloading model-00001-of-000002.safetensors from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B...
Model downloaded to /home/runner/.cache/deepseek/models--deepseek-ai--DeepSeek-R1-Distill-Qwen-7B/snapshots/916b56a44061fd5cd7d6a8fb632557ed4f724f60/model-00001-of-000002.safetensors
Model weights loaded successfully from split safetensors.
Inference successful, output shape: torch.Size([1, 16, 152064])

DETAILED SETUP AND TESTING IS DOCUMENTED IN README

@varshith257 varshith257 changed the title Add Offline DeepSeek Model Loader feat: Add Offline DeepSeek Model Loader Apr 5, 2025
@varshith257
Copy link
Author

varshith257 commented Apr 5, 2025

@intelligentnode @Barqawiz Let me know if I am missing anything and it's ready for review

@varshith257 varshith257 changed the title feat: Add Offline DeepSeek Model Loader feat: Add Offline DeepSeek Model Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant