[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for cross-gpu device mapping #462

Merged
merged 10 commits into from
Jun 23, 2024
Merged

Add support for cross-gpu device mapping #462

merged 10 commits into from
Jun 23, 2024

Conversation

EricLBuehler
Copy link
Owner
@EricLBuehler EricLBuehler commented Jun 22, 2024 •
edited
Loading

Refs #395.

@EricLBuehler EricLBuehler added new feature New feature or request backend Backend work labels Jun 22, 2024
@github-actions GitHub Actions
Copy link
Code Metrics Report 
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 32         1256         1075           37          144
 TOML                   16          444          403            1           40
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               17         1297            0          965          332
 |- BASH                 5          100           97            0            3
 |- Python               6          122          110            0           12
 |- Rust                 3          151          135            6           10
 (Total)                           1670          342          971          357
-------------------------------------------------------------------------------
 Rust                  119        36082        32630          629         2823
 |- Markdown            59          659           13          609           37
 (Total)                          36741        32643         1238         2860
===============================================================================
 Total                 197        39576        34523         1632         3421
===============================================================================

@EricLBuehler EricLBuehler merged commit 196c893 into master Jun 23, 2024
10 of 11 checks passed
@EricLBuehler EricLBuehler deleted the cross_gpu_map branch June 23, 2024 03:17
@EricLBuehler EricLBuehler mentioned this pull request Jun 23, 2024
Closed
@chenwanqq
Copy link
Contributor

mistral.rs/mistralrs-core/src/models/llama.rs

Lines 302 to 303 in e04f840

let vb = vb.set_dtype(mapper.get_min_dtype()?);

I think maybe this line copy the entire weights..
It does cost serious memory usage problem, can not even start a 7b llama model in 4090 GPU...

@chenwanqq chenwanqq mentioned this pull request Jun 27, 2024
Merged
11 tasks
@EricLBuehler
Copy link
Owner Author

@chenwanqq can you please raise an issue about this?

@chenwanqq
Copy link
Contributor

@chenwanqq can you please raise an issue about this?

The problem is that I can not track this problem again.. after using memory usage to check:

println!("vb.device: {:?}", vb.device());
        println!("memory usage: {:?}", memory_track.get_memory_available(vb.device()));
        let vb = vb.set_dtype(mapper.get_min_dtype()?);
        println!("memory usage: {:?}", memory_track.get_memory_available(vb.device()));

It is same before and after. And I did not encounter oom problem today...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backend Backend work new feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@EricLBuehler @chenwanqq