In an earlier note, I discussed how monitoring and incentive contracts can alleviate the asymmetric information problem in the principal-agent relationship. Perfect monitoring, apart from being impossible in many cases, is also too expensive. As a result, most principals will monitor to the extent that the expense is justified by the reduced incentive mismatch. In most industries, this approach is good enough. The menu of choices available to an agent is usually narrow and the principal only needs to monitor for the most egregious instances of abuse.

In fact, this was the case in banking as well until the advent of derivatives. Goodhart's Law by itself does not guarantee arbitrage by the agent - the agent also needs a sufficiently wide menu of choices that the principal cannot completely monitor or contract for.

As discussed in an earlier note, agents in banking have a strong incentive to enter into bets with negatively skewed payoffs. The limiting factor was always the supply of such financial instruments. For example, supply of AAA corporate bonds has always been limited. Securitisation and tranching technology increased this limit substantially by using a diverse pool of credits with a lower rating to produce a substantial senior AAA tranche. But the supply was still limited by the number of mortgages or bonds that were available.

The innovation that effectively removed any limit on the agent's ability to arbitrage was the growth of the CDS market and the development of the synthetic CDO. As the UBS shareholder report notes:

“Key to the growth of the CDO structuring business was the  development of the credit default swap (”CDS”) on ABS in June 2005 (when ISDA published  its CDS on ABS credit definitions). This permitted simple referencing of ABS through a CDS. Prior to this, cash ABS had to be sourced for inclusion in the CDO Warehouse.”