If documents added as context to GPT4all take too long to embed, try changing the embedding device in the settings

#fosai #gpt4all

Been testing those past few days, and something I couldn’t help but to notice, since it’s the reason I’m testing it, is how long it would hang at any given percentage. Even an extremely small TXT file would suffer from that.

The issue was in the CPU.

Under Settings > LocalDocs > Embedding Devices, it automatically comes set as “Application default”. Having two CPUs I can toggle between, I saw the dedicated CPU was listed in the dropdown options, and upon selecting it and restarting GPT4all, it suddenly became much faster.

Adding files too big still take quite a while, though much faster too. Trying to add a generated CSV from a super old SQ Lite database I still feed totaling ~300 MB, it still took a few hours to index 10%, though in the same time frame previously, it would barely get to 1%.

As a bonus, the speed of the models to reply is also affected by the chosen device, though in another part of the settings. Respectively, Settings > Application > Device.

Further optimizations may be possible for both embedding and reply generation from what I skimmed in the settings, though I have yet to confirm.

Hope it helps someone. =)