Understanding Superalignment: Aligning AI with Human Values

Techniques for Achieving Superalignment

In this section, we explain some of the methods and techniques used to achieve superalignment. The underlying philosophy behind these approaches is they should be scalable. 

Adversarial training 

In alignment, as with any training, developers must test if the system has learned to demonstrate desirable behaviors. One way of testing superalignment is to present the AI with counterexamples. Large AI systems must be trained to identify which requests are not made in good faith and handle them appropriately. 

A common approach to superalignment is to use two AIs as adversaries. This is analogous to the red team and blue team approach commonly used in security research, where the red team tries to penetrate the security and defenses of the blue team. 

In the context of adversarial training for superalignment, each AI tries to find inputs that will confuse the other AI. For example, consider an AI (blue team) that has been aligned to not respond with swear words. In adversarial training, the goal of the adversary AI (red team) is to find prompts that trigger the blue team AI to give inappropriate responses. The goal is to ensure that even when tested by the red team AI, the blue team AI continues to generate acceptable responses.

Illustration of the concept of adversarial training. Image created with DALL·E

Robustness training 

Robustness refers to being able to distinguish between inputs that are actually different but superficially seem similar. It should be able to recognize corner cases and edge cases. For example, a system to identify human actions in video clips should be able to distinguish a real street fight from a choreographed fight in a movie. Confusing one for the other can lead to detrimental consequences. 

Thus, large models should be specifically exposed to superficially similar scenarios. This will teach the model to recognize nuance and subtle features. 

Scalable oversight

As AI models find new users and are used in more applications, their oversight needs to scale in parallel. Human involvement is the crux of supervising and aligning AI. Scaling up human oversight is harder than scaling a technology solution. Thus, to maintain scalable oversight of AI, new methods are necessary:

• Automated real-time monitoring: Automated systems can continually track the AI’s responses and monitor its behavior to check that it complies with human values. Significant deviations from expected standards can be flagged for human intervention. 
• Programmatic reviews: Manually reviewing and auditing all AI outputs is feasible. This can be done by developing new programs and algorithms that check the AI’s alignment with human ethics. Doubtful cases can be flagged for human moderators to judge manually.

Reinforcement learning with human feedback (RLHF)

As discussed in the article on reinforcement learning (RL), in traditional RL implementations, the AI system learns to modify its behavior by observing a human. It uses a trial-and-error process to maximize the reward. RLHF combines RL with human input to guide the AI’s actions during the training. 

RLHF starts with a pre-trained model. Human reviewers assign the model a set of tasks and provide feedback on the outputs, such as grading or correcting the model’s responses. 

The model uses the information to fine-tune its behavior to align with human feedback and maximize the reward (positive human feedback). This process is repeated until the model’s outputs are considered relevant, appropriate, and accurate. 

Inverse reinforcement learning (IRL)

In traditional reinforcement learning (RL), the model developers explicitly declare the reward function (also called the utility function). The goal of the model is to maximize the incentive according to this reward function. 

In inverse RL, the model infers the reward function based on the behavior of either a trained model or a human. For example, an RL-based autonomous car might be given positive incentives to drive in the lane, stop at red lights, etc. An IRL-based self-driving system figures out the incentives by closely observing and following the actions of an experienced human driver or a trained AI. 

Human values cannot be explicitly listed for all contexts, making traditional RL difficult for alignment problems.

A more effective approach is to have the AI observe and replicate human behavior in different contexts. By observing the human’s behavior, the AI learns which of the different possible responses (or courses of action) are most similar to how a human might act in that scenario. This helps align the AI’s behavior with human values.

Illustration of the concept of inverse reinforcement learning. Image created with DALL·E

Debate 

Many alignment methods involve a human training the AI on what behavior is acceptable and what is not, but using a human may not always be the ideal solution. 

For many complex problems, such as the optimal scheduling of a national train system, no human may be capable of judging whether a proposed solution is the best. A solution is to leverage AI itself for superalignment. In the debate method: 

• An AI proposes a solution and gives a detailed explanation of each step. 
• Either the same AI or a different AI is then used to critique (provide arguments against) each explanation. 
• A human decides which argument is more correct. Because each argument has a limited scope, it is easier for humans to judge its validity. 

The process of logical reasoning becomes gamified, and the most valid argument wins. Thus, over a series of debates, the AI learns which lines of reasoning are acceptable and which are not. For future tasks, the AI’s output is better aligned with human values. 

Iterated amplification

In traditional machine learning and deep learning systems, the training is based on whether the model’s output matches the expected output. 

As AI’s capabilities grow, it will be used for increasingly complex tasks. For many such tasks, humans don’t know the expected output in advance. The only options may be the solutions generated by the AI. For such complex tasks, aligning the AI based on the final output is difficult because humans cannot judge the final output. 

For example, in designing a transport schedule, the system must balance financial viability with providing a useful public service. Many such considerations make it difficult to determine which final system design best aligns with human values.

In such cases, it is easier to leverage human supervision to ensure alignment on smaller sub-tasks. Later in the process, slightly larger tasks are broken into smaller sub-tasks and then aligned under human supervision. This process continues iteratively until the tasks get too big to be directly judged by humans. 

Iterated amplification assumes that since smaller and simpler sub-tasks have been aligned based on human validation, the larger primary task is also aligned. This is called iterated amplification. Iterated amplification is also applied in combination with techniques such as debate. 

Value learning

Human interactions are complex and nuanced. In many cases, human actions are ambiguous when considered in isolation - they only make sense in the right context. Thus, traditional utility functions and reinforcement learning are inadequate for imbibing AI with human values. 

A proposed alternative is for the model to have access to a set of utility functions instead of a single utility function. Each utility function is likely to be chosen, depending on the context. The model learns to choose the right utility function by observing human actions in different contexts. This is called value learning. 

Value learning is a technique in which the AI directly observes human behavior in different scenarios and learns the right behavior for each context. This involves exposing the AI to human behavior through a wide variety of real-life interactions. 

Illustration of the concept of value learning. Image created with DALL·E

The humans in the alignment training should behave in a way that is considered acceptable. For example, a human driving a car should not give in to road rage while trying to align an AI. However, when stuck behind a slow driver, the human should try to safely overtake the slower vehicle.

Superalignment in Large Language Models (LLMs)

LLMs are currently the most popular large-scale AI models. They are used in a wide variety of tasks. Considering their large user base and use cases, superalignment in the context of LLMs deserves a closer look. 

LLMs and alignment challenges

Large language models like GPT-4, Gemini, and Meta AI are starting to be used in daily life.  Given their scale and scope, LLMs come with new alignment challenges, such as: 

• Bias perpetuation: These models have been trained on large quantities of data sourced from the open internet and other libraries. Any biases in the training data are directly reflected in the model’s outputs. Bias due to deficiencies in input data is considered algorithmic bias. 
• Complex systems: The large size of these models, the vastness of their training data, and the wide variety of use cases make it challenging to predict the system’s behavior. It makes it difficult to ensure the model’s output is aligned with human values for all use cases. 
• Problems of scale: Given their vast user base and the large number of queries they handle: 

• Even if the percentage of responses not aligned with human values is small, it still translates to a large number.
• It becomes difficult to ensure that AI's behavior consistently aligns with human values and ethics across all users. 

This makes it necessary to adopt new methods to test how well these models align and to continually fine-tune them to ensure that they align with human values and ethical standards. 

Evaluating LLM outputs

Many approaches have been proposed and tried to evaluate LLM outputs to ensure alignment with human values. These methods need to be scalable. Some examples are: 

• Filtering undesirable output: A pragmatic approach to eliminating inappropriate responses by an LLM is to use a filtering layer on top of the language model. Training separate systems for text generation and filtration tasks is more efficient. The filter itself can be another AI that scans the output of the generator LLM for inappropriate language and sensitive topics. 
• Bias detection: Training datasets invariably contain some bias, leading to biased models. Some methods to detect biases in LLM outputs are: 
• Model developers audit models for biased outputs. Bias audits involve testing a model with diverse inputs and comparing the response in each case. 
• Specialized bias detection algorithms are sometimes used to systematically look for and identify different types of bias. 
• Fact-checking: Hallucination has been the bane of LLMs since the beginning. An LLM only knows how to predict what words should follow a string of words based on what it saw in the training dataset. It has no concept of facts. To an LLM, the sentence “Dell makes personal computers” is equally meaningful as “Dell makes potato chips.” Thus, when using LLMs to generate fact-based text, it is critical to check that the generated text is factually correct. 
• Explainability: Traditional IT systems and programs are rules-based. You can debug the system to pinpoint which rules are responsible for a particular output. In contrast, LLMs have billions of parameters. It is impossible to track down which weights contribute to different parts of the output. 

Fortunately, reasoning is an emergent property of large language models. An emergent property is a property of a complex system that cannot be explained as the sum of its parts. 

LLMs are often observed to be able to explain their responses and reason through multi-step problems. Asking an LLM to explain a complex solution and manually evaluating each step can allow humans to validate the LLM’s response. 

LLMs and human interaction

LLMs that are used interactively must understand the nuances and subtleties of human conversations. They should be able to respond based on the context of the conversation. 

Tone, intent, sarcasm, and emotional cues are difficult to represent mathematically. Yet, AI models must account for these aspects to communicate effectively with humans. Only an LLM that is aligned to be empathetic will be effective in interacting with people. It can build trust and promote productive engagements with humans, and the only way to achieve this is by applying various superalignment techniques.  

Ethical Considerations in Superalignment

The ultimate goal of superalignment is for powerful AI to conform to the ethical standards of human society. 

This section explains the role of ethics in superalignment and the challenges associated with ensuring ethical behavior in large models. 

Ensuring ethical behavior

An AI misaligned with human values can never be safe for humans to use. A misaligned system can produce harmful content that malignant actors can misuse. It can perpetuate social inequalities and amplify existing biases. 

In the worst case, a misaligned system can misuse this ability and lead to manipulative behavior. Thus, ethical concerns lie at the very heart of all superalignment efforts. Only by ensuring ethical behavior under all circumstances can AI tools be considered reliable and trustworthy for widespread and long-term use. 

Balancing utility and safety

Imbibing ethical behavior into AI systems comes at a significant cost. To achieve superalignment, model developers need: 

• Humans to audit the responses and provide feedback to the AI
• Additional time to ensure superalignment before releasing the model
• Additional computational resources to fine-tune and align the model 

Furthermore, while running inference, the model needs to perform additional computations to ensure that its responses are aligned with human values. Sometimes, imposing alignment as a constraint can limit the model’s capabilities. Thus, a superaligned model can be less performant. 

Hence, there must be a balance between the model’s technical performance and alignment considerations. Prioritizing alignment and safety considerations over traditional performance metrics may lead to models that are not very useful in practice. 

Fairness and bias

Large-scale AI models have many users and varied use cases. Any bias present in the model’s output can have far-reaching consequences. Thus, developers should take care that their models are as bias-free as possible. 

Eliminating bias from LLMs is challenging because these models undergo unsupervised training based on a huge corpus of text data. The AI picks up all the different biases and prejudices present in the text of the training dataset. This can lead to the model’s output being unfair and discriminatory. Developers can employ various methods to address this:

• Bias detection: Before you can take countermeasures, you must first identify instances of biased behavior. Regular audits and fairness assessments on samples of model output can help identify the types of bias the model exhibits and the situations that trigger these biases. 
• Bias elimination: There are three main approaches to eliminate bias:
• Programming the model to correct for specific biases in the output.
• Enhancing the training dataset by supplementing it with synthetic data. For instance, if the training dataset for an image recognition system has very few animals of a particular species, it reduces the marginal probability of identifying this animal. This is often tackled by augmenting the training dataset with synthetically generated images of this species and/or increasing its weight in the training process. 
• Updating the weights of underrepresented subsets of the training data. This involves adjusting the weight of different subsets of the training data such that underrepresented categories (i.e., categories with few examples) in training data have the same weight as other categories. 
• Ongoing monitoring: Models are periodically retrained on new datasets. Furthermore, the behavior of future releases is influenced by the model’s past interactions with users. User behavior also changes based on prolonged exposure to and interaction with AI models. Thus, bias detection and elimination is not a one-time task. It needs to become a part of the model development cycle. 

Challenges in Achieving Superalignment

As we explained in the previous sections, achieving superalignment is challenging. Some of the main difficulties are: 

• Complexity and scale of models: Modern AI models have billions of parameters. This makes it difficult to predict their responses or to understand their decision-making process from first principles. Thus, ensuring that the model is consistently aligned with human values is challenging. Small updates to the model parameters can disturb its alignment. 
• Scalability of alignment techniques: Human supervision is central to the alignment process. As models grow in scope and usability, they need to be aligned in more and more scenarios. This raises questions on how effectively and economically humans can supervise the model’s alignment efforts. 
• Generalization and robustness: AI models routinely handle new scenarios and novel inputs. The model’s training and alignment, on the other hand, is based on a large but limited set of training data and scenarios. The model needs to be able to extend its alignment to scenarios that were not explicitly covered during training. It should be able to distinguish between edge cases and behave ethically under new circumstances.
• Value misalignment: AI doesn’t have a native understanding of human values. It only learns based on its training. Human values are complex and sometimes contradictory. What would be perfectly acceptable in one situation could be completely inappropriate in another. Misalignment can happen when the model understands the values differently than intended. This can lead to inappropriate responses and potentially unethical behavior. 
• Regulatory compliance: National and supranational bodies are beginning to set regulatory and legal standards to ensure that powerful AI models do not hamper the public interest. Complying with these new regulations will need additional effort from model developers. 
• Integration with other IT systems:  Most alignment efforts are centered on direct human-AI interaction. New challenges arise when AI is integrated with other systems. The AI’s inputs and outputs in such integrated systems are no longer based on human languages. This makes it difficult to interpret the AI’s responses.  Ensuring the superalignment of a complex IT system including AI presents unforeseen challenges. One possibility is to entrust the entire integrated system to another AI, which helps humans to understand and monitor it. 
• Monitoring and updating: Continually monitoring the model’s responses is necessary to ensure that the model acts ethically in practice. This is typically done by randomly sampling the model’s output to real-life inputs. Updating the model’s parameters can also disturb its alignment. Thus, the alignment needs to be maintained and updated as part of the model development lifecycle. 

Conclusion

An AI system that is not aligned with human values and ethics is unsuitable for public use. In fact, it could have detrimental consequences. 

In this article, we discussed superalignment, a collection of methods and approaches used to align large and powerful AI systems. Additionally, we explored superalignment in the context of LLMs. Lastly, as superalignment is an evolving field, we covered the major challenges associated with its implementation.