Modern pretrained language models are critical components for natural language processing. Yet, they suffer from spurious correlations, poor out-of-domain generalization, and biases. This is a framework to learn invariant representations that should generalize across training environments.