Iphone 12 pro, ipad and iphone 15 or some other device this llama.rn pakage has issue. Even the device has latest os and enough gpu to run the model. #102
Labels
help wanted
Extra attention is needed
Comments
PratikBhatti83
changed the title
PratikBhatti83
changed the title
|
@jhen0409 @a-ghorbani is there any solution for this? could you please help me to solve this problem |
|
I can't figure the cause based on the error message. Could you provide more information, like Xcode logs or what models are you using? |
|
@jhen0409 i used Llama-3.2-1B-Instruct-Q4_K_M.gguf this model! Xcode log is like below error only Context initialization failed: Error: Failed to load model. Meanwhile on other device like Iphone 13, Iphone 14 Pro I can able to run this model. but some of the device it's failed to load model while intiazation. i think you should update lllama.cpp version or sync the latest code because I checked llama.cpp doesn't have this type of initialization issue. |
|
@jhen0409 is there any update for this? please help me to solve this issue! |
|
I can update llama.cpp tomorrow, but not sure if it will help you. You should be able to get Xcode logs like this: Logs: Load Llama-3.2-1B-Instruct-Q4_K_M.gguf (success)If it fails, there should be some logs that can help us find the reason. |
|
@jhen0409 let me share the logs LOG Model info (took 64ms): {"alignment": 32, "data_offset": 7831552, "general.architecture": "llama", "general.basename": "Llama-3.2", "general.file_type": "27", "general.finetune": "Instruct", "general.languages": "["en", "de", "fr", "it", "pt", "hi", "es", "th"]", "general.license": "llama3.2", "general.name": "Llama 3.2 1B Instruct", "general.quantization_version": "2", "general.size_label": "1B", "general.tags": "["facebook", "meta", "pytorch", "llama", "llama-3", "text-generation"]", "general.type": "model", "llama.attention.head_count": "32", "llama.attention.head_count_kv": "8", "llama.attention.key_length": "64", "llama.attention.layer_norm_rms_epsilon": "0.000010", "llama.attention.value_length": "64", "llama.block_count": "16", "llama.context_length": "131072", "llama.embedding_length": "2048", "llama.feed_forward_length": "8192", "llama.rope.dimension_count": "64", "llama.rope.freq_base": "500000.000000", "llama.vocab_size": "128256", "quantize.imatrix.chunks_count": "125", "quantize.imatrix.dataset": "/training_dir/calibration_datav3.txt", "quantize.imatrix.entries_count": "112", "quantize.imatrix.file": "/models_out/Llama-3.2-1B-Instruct-GGUF/Llama-3.2-1B-Instruct.imatrix", "tokenizer.chat_template": "{{- bos_token }} {#- This block extracts the system message, so we can slot it into the right place. #} {#- System message #} {#- Custom tools are passed in a user message with some extra guidance #} {%- for message in messages %} |
|
@jhen0409 is there any update? please check my logs and let me know the solution. |
|
@jhen0409 is there any update? please help me to solve this issue! |
|
Can you try v0.4.8?
This looks like logs from JS, not Xcode logs. |
|
bro still having same issue |
|
Same issue here. iPhone 12 Pro Max. Works well on simulator but not on device. Here are the Xcode logs |
Looks like some kernels has been required to use Metal 3.2 (ref) I haven't noticed, so you may need to upgrade your iOS version to 18. |
|
But I set |
Due to llama.cpp changes, the metal kernels still load even set n_gpu_layers: 0, which it used to offload some other compute ops if available. |
Is it possible to somehow disable the ability to connect metal? On iPad 7 (2019) I get this error when starting: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When using the llama.rn package, the initLlama function fails to initialize, throwing a "Failed to load model" error.
The text was updated successfully, but these errors were encountered: