Make the application launch faster. #990

Steve235lab · 2024-02-02T15:13:13Z

Is your feature request related to a problem? Please describe.

The application launch a bit too slow especially for cases that users just want some basic functions like -h, --conversations and so on.

Describe the solution you'd like

I find that the entry point for current version is:

def main():
    interpreter = OpenInterpreter()
    try:
        start_terminal_interface(interpreter)
    except KeyboardInterrupt:
        pass

which means the OpenInterpreter instance was initialized first before processing the arguments.

I think we can have two enumerates for those arguments need the OpenInterpreter instance to work and those don't, so we can avoid initializing the instance and have a faster application.

Describe alternatives you've considered

No response

Additional context

I will open a pull request to do so in the near future.

The text was updated successfully, but these errors were encountered:

Steve235lab · 2024-02-03T17:17:42Z

I have tried to init some heavy objects with multithreads, here's the code: my fork, just check the latest commits However, it doesn't work well. I run a simple benchmark to measure the launch time:

Write a benchmark script time_this.sh:

#!/bin/bash

time_this() {
  START=$(python -c "import time; print(int(time.time() * 1000))")
  $@
  END=$(python -c "import time; print(int(time.time() * 1000))")
  DIFF=$(( $END - $START ))
  echo "Execution time: $DIFF milliseconds"
}

Source the bash script:

chmod +x time_this.sh && source time_this.sh

Then run benchmarks:
```
time_this interpreter -h
```

With this simple benchmark we can get the time cost to launch the app, and compare the difference after modifications.

So I think that I'd better locate lines that take the most time to run first. Then I generated a performance report performance.txt with cProfile:

python -m cProfile -o performance.pstats interpreter -h

Here's the file: performance.txt
Please be noticed that this is not a real txt file, but a .pstats file which is not a supported format on GitHub comments. To read this file, you need pstats which is a module from python standard library.

Are there anyone can help me with this problem? Or I had to finish this alone?

Steve235lab · 2024-02-03T17:20:17Z

@yuhanj2 HELP!!!

tyfiero · 2024-02-13T18:40:27Z

I did some digging, it looks like the computer API imports are what are slowing it down substantially. With all of these dynamic imports, it takes 5.189 seconds to start up. If I comment them out, it takes 1.655 seconds. There must be a better way to import these packages. I'll figure this out today and make a PR

Steve235lab · 2024-02-13T23:04:44Z

I did some digging, it looks like the computer API imports are what are slowing it down substantially. With all of these dynamic imports, it takes 5.189 seconds to start up. If I comment them out, it takes 1.655 seconds. There must be a better way to import these packages. I'll figure this out today and make a PR

You are right, some imports take the most time instead of initializing instances.

Actually this is not a big problem for most cases if you are not using OI in China or any other places need a proxy to access the OpenAI services. The launch time depends a lot on the user's network quality. I am on vacation in Canada these days and it almost takes no time to list my history conversations with interpreter --conversations. However, there's definitely something to improve like purging all unnecessary network accessing actions. I'm always here to help if you need.

tyfiero · 2024-02-16T23:16:10Z

I made a PR to lazy load modules to speed up the start-up time. There also was an arbitrary 5 second time delay added to --conversations for some reason, I removed that so it should start up faster. Although I did not do anything to speed it up on slow network speeds, I will look into that later on.

Notnaton · 2024-02-19T12:56:24Z

When I run llama.cpp I can get 100Tok/sec on 7B model
Seems interpreter have a max speed of 20tok/sec as there is little difference between 7B and a bigger model. Too many for loops? 😅

tyfiero · 2024-02-19T17:19:23Z

Huh... that's a weird one. That might even deserve its own issue, with tech like groq coming out, we want Open Interpreter to be able to support ultrafast inference times. I wonder why fast models are still slow, maybe it is too many loops? Worth looking into!

rustom · 2024-04-04T07:21:52Z

I'm still hitting almost 10-second startup times while working with the HEAD of main which should include your fix @tyfiero. Shouldn't have anything to do with network speed, I'm well above general benchmarks for "good" wifi. Any ideas?

csaroff · 2024-06-08T03:38:53Z

21 seconds just to print the version number on a fresh pip install...

→ time interpreter --version
Open Interpreter 0.2.5 New Computer Update
interpreter --version  1.25s user 2.71s system 18% cpu 21.268 total

csaroff · 2024-06-08T03:50:02Z

I ran it a handful of times and did ctrl + c at 3 seconds, 10 seconds, 17 seconds, and I always got back the same stack trace:

→ time interpreter --version
^C^CTraceback (most recent call last):
  File "/Users/csaroff/.local/bin/interpreter", line 7, in <module>
    from interpreter.terminal_interface.start_terminal_interface import main
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/interpreter/__init__.py", line 2, in <module>
    from .core.core import OpenInterpreter
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/interpreter/core/core.py", line 17, in <module>
    from .llm.llm import Llm
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/interpreter/core/llm/llm.py", line 1, in <module>
    import litellm
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/litellm/__init__.py", line 288, in <module>
    model_cost = get_model_cost_map(url=model_cost_map_url)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/litellm/__init__.py", line 271, in get_model_cost_map
    with requests.get(
         ^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/csaroff/Library/Application Support/pipx/venvs/open-interpreter/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
KeyboardInterrupt

interpreter --version  0.84s user 2.69s system 20% cpu 17.443 total

Seems like the slowness is probably coming from litellm's get_model_cost_map.

Potentially related to BerriAI/litellm#2677

Running with LITELLM_LOCAL_MODEL_COST_MAP=True sped things up significantly:

→ export LITELLM_LOCAL_MODEL_COST_MAP="True"
→ time interpreter --version
Open Interpreter 0.2.5 New Computer Update
interpreter --version  1.16s user 2.79s system 219% cpu 1.803 total

Steve235lab added the Enhancement New feature or request label Feb 2, 2024

Steve235lab mentioned this issue Feb 3, 2024

[WIP] Fix broken unit tests. #995

Merged

6 tasks

tyfiero mentioned this issue Feb 16, 2024

Boost performance by lazy loading modules #1020

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the application launch faster. #990

Make the application launch faster. #990

Make the application launch faster. #990

Make the application launch faster. #990

Comments

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context