One of the key parts of any knowledge base is to be able to find similar ideas so that the creator can expand them, refute them or build on top of them.

The problem is that often times the idea is fuzzy and the complex to slim it down to a query search. That's why keywords exists where the creator puts some important words on top of the article to hint readers about the content. However this has flaws since it requires effort from the original creator and even with that effort, it is impossible to summarize complex concepts in just words

That is why a search has to be semantic, where we take into account the meaning of words. In order to do that we should translate the words to a series of vectors that later we can relate to and measure the relative distance from our now modified query search.

First we have to vectorize the texts. We can use llama.cpp to generate the embeddings. This comes handy as we can load a llama server and we can call it locally in a way that is compatible with openai api. Then once we have the vectors we can store them in the database. Its worth mentioning that this vectorizing process is multilingual so we can support EN/ES and major languajes, We just have to use the right model like the BGE-M3