How can AI enrich Projektron BCS? An overview of the basics and development approaches

This article gives you an insight into the technology behind the currently leading AI language models. In particular, we will discuss the transformer architecture, which enables machines to accurately understand language and generate text. These models can be customized by adapting and training them to specific requirements – for example, as an assistant in Projektron BCS. In a follow-up article, we will show how we developed the Projektron AI Help and other applications and how they enrich our software.

In the first step, the user input is “tokenized”. This involves breaking the input text into smaller units, known as tokens. A token can be a word, part of a word, or even a single character, depending on the tokenization method chosen.

The models work with a fixed character set, which in the case of OpenAI products is 100,256 tokens. A token can be a single character, a part of a word or an entire word. The maximum text size that can be processed at one time is also measured in tokens, as are the costs. The goal is to divide the text into processable units that the model can efficiently analyze.

On the page https://platform.openai.com/tokenizer, you can convert a text into tokens on a test basis. If you want to take a look at the complete token set of OpenAI's GPT 4, you can do so at https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193. 

The token set is generated by analyzing large amounts of text using byte-pair encoding. An explanation can be found here:  https://huggingface.co/learn/nlp-course/chapter6/5

Since English is by far the most prevalent language in the training texts, English texts with the same meaning can be tokenized more cheaply, i.e. they require fewer tokens, the response comes faster, and the costs are lower than for other languages.

The attention mechanism is essentially based on vector and matrix calculations. After the text has been split into tokens, each of these tokens is translated into a vector (“vector embeddings”). This gives the model a numerical representation of the text, which serves as a basis for processing and calculating the relationships.

Which vector a token maps to is determined by analyzing large amounts of text. The vector of a word is determined by analyzing the neighboring words. For example, one can imagine that synonyms such as “cat” and “house cat” yield very similar vectors because they occur in largely the same word environments. “Stubentiger” will appear more often in domestic contexts, so the vector will be somewhat more ‘domestic’. An explanation of how vectorization works in detail can be found here: https://medium.com/@zafaralibagh6/a-simple-word2vec-tutorial-61e64e38a6a1

The vectors used by GPT 4 have 512 entries (“dimensions”).

Since the transformer architecture processes all user input in a single step, no information about the order of the tokens is available per se. In order for the model to understand which tokens are related, positional information is added to each token. This positional encoding ensures that the model can capture not only the meaning of the individual tokens, but also their order and relationships to each other. This is important because the meaning of a sentence depends heavily on the word order.

The output of the model is finally created by repeatedly selecting the most probable tokens until the end-of-sequence token (<EOS>) is reached. This iterative process results in a text sequence that reads as if it were written by a human.

The output is no longer deterministic due to the use of sampling.

AI alignment is concerned with the question of how to align AI with desirable behavior. The aim is to minimize risks such as bias or misinformation. Language models like the current AI systems are trained with enormous amounts of text, including data from the internet. However, this training data often contains distortions and incorrect information. Language models are able to abstract from the examples at hand, recognize patterns and apply them to new situations. At the same time, the prejudices or false assumptions contained in the training data flow unnoticed into the results.

Two examples illustrate this problem:

1. The issue gained prominence when cases such as Amazon's automatic application scoring system came to light. Training was done with past decisions – and Amazon had mainly hired men in the past. The Amazon AI then systematically discriminated against women, even though the application texts did not contain any explicit information about gender. It was enough for the AI to be able to infer gender indirectly from hobbies, sports or the type of social engagement, for example. Alignment did not succeed here, and use was largely discontinued. 
    
2. On the other hand, the Google developers had a somewhat excessive success in their efforts to align the weighting of members of minorities who are considered disadvantaged in the output. We know that the Google AI generated images of black popes, and some early US presidents were also depicted as dark-skinned.

The pre-trained language models such as GPT4, Gemma or Llama have language comprehension and general training knowledge. To use these models for your own application, you need to be able to handle company-specific specialized knowledge. The following diagram shows possible ways to adapt them. Completely new training from scratch is usually out of the question due to the costs. Prompt engineering, fine-tuning and RAG are practical for medium-sized companies.

Our attempts at fine-tuning have been less successful. Fine-tuning is done with small, specific data sets from a particular domain. The goal is to convert a general-purpose model into a specialized model. The model is “retrained”, i.e. the internal weights in the neural network are changed. This is the same procedure as for the initial training. A new model is created.

In our previous attempts, fine-tuning impaired the general abilities of the models without sufficiently building up specific abilities. The models had understood that they had learned something. However, they gave useless, generic answers, mainly based on their general prior knowledge from the training. General control questions were answered rather worse after fine-tuning.

We see the cumbersome adaptation to new data as a major disadvantage of fine-tuning: for a help assistant, the model would have to be retrained with each new version. An agile workflow, as established in the Projektron documentation, with a help release per week, is thus practically not reproducible. For an application based on RAG, on the other hand, this is not a problem. Therefore, we did not invest any further effort in fine-tuning at first.

Further sources for methods and techniques are, for example: https://www.promptingguide.ai or https://platform.openai.com/docs/guides/prompt-engineering

In our experience with prompts, it works well to have an AI create the first prompt: briefly describe what the prompt should do as input, and then let a tool like ChatGPT create the prompt.

It is important to be as specific and clear as possible about what you expect the result to be. If the outcome is different, a closer look at the AI's answer often reveals that it is a good and direct implementation of the prompt – it's just that our requirements didn't accurately reflect the prompt. You get what you specify, not what you want. That's why testing and iterative improvement are important: prompting is an iterative process.

Prompts in English are better understood. I've already mentioned the advantage of English in the context of tokenization. “Don't” instructions in the system prompt are less well understood than positive formulations of what is desired.

The first step is to split the documents. There are various methods for doing this, such as fixed-length splits, splits at certain formatting characters like double line breaks, or splits based on semantics.

The text splitter recommended in the literature for general text is “recursive splitting by character”. The splitter is parameterized by a list of characters, by default [ “\n\n”, “\n”, “ ',”, “]” . The splitter divides the text at the delimiters in the list, i.e. first at double line breaks. If this results in too long splits, the splitter splits at single line breaks. This process continues until the pieces are small enough. This way, you want to keep together paragraphs, then sentences, then words, since these are generally the semantically coherent pieces of text.

In the third step, the vector database (e.g. FAISS) takes the vectors and uses them to build a vector index. The vector database is specialized to find a predefined number of nearest neighbor vectors for a query vector from a very large number of vectors. This completes the first phase. 

The result

GPT4 was used as the language model in this experiment.

Another advantage is that the output can also be provided in foreign languages without the need for translation. The database is easy to expand with new knowledge.

Our positive experiences with the retrieval-augmented generation (RAG) have convinced us to use this method in our first main application, the AI software help. As soon as the technology works successfully in one application, it can easily be transferred to other areas, such as contract negotiations. The important thing is to create the appropriate data set and develop a suitable system prompt.

Further details on the development of the software help and the underlying AI frameworks will follow in our next blog article. We will also discuss the RAG technology in more detail in a future post, as it plays a central role in our applications.