Brainstorm AI London 2024: Striking The Right Balance In AI Regulation
Balbir Bakhshi, Chief Risk Officer, London Stock Exchange Group Ian Hogarth, Chair, Phasecraft, AI Safety Institute; Partner, Plural Connor Leahy, Founder and CEO, Conjecture; Co-founder, EleutherAI Moderator: May Habib, Co-founder and CEO, Writer; Co-chair, Fortune Brainstorm AI London
Category
🤖
TechTranscript
00:00 - Really excited to get into the conversation.
00:03 National approaches may differ is an understatement.
00:06 Thank you, Jeremy.
00:07 I'm excited to get into it with everybody.
00:10 Ian, I'd like to start with you.
00:12 You have been writing about not just the opportunity
00:16 for AI to transform society,
00:18 but the risks as well for two decades.
00:20 And you now chair the AI Safety Institute in the UK,
00:25 a hundred million dollar investment, pound investment
00:28 by the government to make sure that the ecosystem for AI
00:33 in the UK is the right balance of grabbing the opportunity
00:37 as well as mitigating the risks.
00:39 So walk us through what you think is kind of state
00:43 of the art.
00:43 You guys have published the AI progress report,
00:46 safety progress report not too long ago.
00:48 What is the UK's pro innovation stance
00:53 on managing AI safety?
00:56 And how do you compare it to what the EU is doing
00:59 and maybe what's happening in the US as well?
01:02 - I think in the UK, I mean, the prime minister is,
01:05 I think he would sort of describe himself
01:06 as a technopomist.
01:08 Very excited about potential of technology
01:11 and startups to transform the economy.
01:13 And so I think the UK sort of tried to chart a course
01:18 which really is sort of fundamentally built
01:20 around empiricism.
01:22 So he created this AI Safety Institute
01:26 with the intention of basically creating research capacity
01:31 inside government.
01:32 So we've recruited some great researchers from industry,
01:36 from academia to come into government.
01:37 So people like Yaren Gal, who's a professor at Oxford,
01:42 Jeffrey Irving, who was previously in charge of alignment
01:45 and safety at Google DeepMind,
01:47 Jade Leung, who was previously at OpenAI.
01:51 And these people have come into government
01:53 and we've built a research team.
01:54 And the research team is essentially looking at a variety
01:57 of different risks at the frontier
01:59 and trying to assess kind of how meaningful they are
02:02 and then feeding that up to people making decisions
02:05 about regulation, people making decisions about policy.
02:07 So the idea is to sort of put risk on an empirical basis
02:11 so that the government can make appropriate decisions
02:13 around how and when it regulates.
02:15 And so the approach really is building state capacity,
02:20 building the capacity to understand what's going on.
02:23 I think in Europe and in the US,
02:26 there's been maybe a bit more of a kind of earlier move
02:30 towards regulation with the EU AI Act
02:33 and the Biden executive order,
02:34 but the UK hasn't yet regulated.
02:36 We've just built this kind of empirical capacity.
02:39 And in terms of the innovation risk trade-off,
02:44 I mean, I think that maybe a good place to frame it
02:47 would be around the ability of AI systems to write software.
02:51 So I think it's something that potentially
02:53 is very, very exciting because there's so many things
02:56 that we could build if we had more capable
03:00 kind of software tooling.
03:01 So I remember when I first kind of encountered
03:04 more powerful programming languages,
03:05 it was like it was kind of like having a superpower, right?
03:08 The ability to do, write code faster
03:10 and ship features faster.
03:13 And so I think that, for example,
03:14 AI that makes coding more productive
03:16 is potentially really, really positive
03:19 in that we can generate loads of new applications
03:21 and value for society.
03:22 Obviously, there's always a risk with software,
03:25 which is cyber crime, cyber attacks.
03:28 And so, one of the work streams that we have
03:31 at the Eye Safety Institute is just understanding
03:34 how much these kind of frontier models
03:36 would give a sort of malicious actor uplift
03:39 in trying to perpetrate some kind of cyber attack.
03:41 And we work in tandem with organizations like GCHQ
03:45 and NCSC, National Cyber Security Center on that.
03:49 So that's a good example of an area
03:51 where there's lots and lots of opportunity
03:53 and lots of opportunities for value creations for society,
03:56 but then there's also this risk to mitigate.
03:58 - Yeah, thank you, Ian.
04:00 Connor, you have really, I think,
04:04 kicked off the initial open source work with GPTJ
04:10 and we build models at Rider.
04:12 And I remember looking at that project
04:15 with a keen interest to see how it develops.
04:18 You've also expressed grave concerns.
04:21 I believe the exact wording was,
04:22 "God-like AI could be super-effed."
04:25 So there's definitely a large plurality
04:30 of AI researchers who agree with you.
04:33 You've since started Conjecture,
04:35 which is a effort to build models with boundedness.
04:40 Can you walk everybody through what you think that means?
04:43 Do you still believe, you guys are two years in
04:45 and this is a for-profit company.
04:49 Tell us about what you have learned on the frontier side,
04:52 to Ian's point, around the potential
04:55 to build in benevolence to humanity
04:57 inside of these non-deterministic models.
05:01 - So fundamentally, the way I got into AI originally
05:04 when I was a teenager was I was thinking about
05:06 how can I do the most good for the world?
05:07 How can I solve the most problems?
05:09 Eventually I figured out, you know,
05:11 I'm just one guy with one brain.
05:12 So if I figure out intelligence,
05:14 well, then I can make a lot of brains
05:15 and they could work on all the problems.
05:17 I could solve all the diseases and all the problems.
05:19 Great, so I'll go do that.
05:20 How hard can it be?
05:21 And so, turns out, relatively hard,
05:25 but maybe not as hard as you might think.
05:28 So when I, eventually though, I figured out
05:31 that I was thinking about that the heart problem of AI,
05:34 of AI, is not how to build it, it's how to control it.
05:38 It's much easier to build something that is intelligent,
05:41 that does something, than something that does what you want.
05:45 Because think of yourself, like, what is what we want?
05:48 That's a really hard question.
05:50 Even someone we love very dearly,
05:51 we can sometimes hurt them,
05:52 or we can misunderstand what they want,
05:53 even if we're trying our very, very best.
05:55 Now imagine a system built by for-profit companies
05:58 to maximize profit, presumably,
06:00 what do you think that thing's gonna do?
06:02 Who knows?
06:03 If we build systems that are smarter than humans,
06:05 which is the stated goal of all the major players,
06:09 that's better at business, science, technology,
06:14 politics, and everything else,
06:17 and we don't know how to control it,
06:18 which we don't currently know how to do,
06:20 please think for a moment, what do you think happens?
06:23 Seriously, what happens?
06:25 And so this is the question I've always been working on.
06:28 Well, then what?
06:29 So, well, even having the luxury of a then what is a luxury.
06:34 So at Conjecture, we work on this problem,
06:36 on what I call the technical problem,
06:38 or even hypothetically,
06:39 how can you make an artificial intelligence system
06:42 do what you want, what does that even mean?
06:44 So we focus on a sub-problem of this,
06:46 which we call boundedness,
06:47 which is the question of how can you know
06:49 what a system won't do?
06:51 So this isn't even asking what it can do,
06:54 or guaranteeing that it will do the right thing,
06:56 it's just can you guarantee
06:57 that it will never do specific things?
07:00 This is currently extremely difficult,
07:02 as I'm sure anyone here who has tried to use
07:04 a GPT model in production can tell you.
07:07 So we work on a specific paradigm
07:11 that we personally call cognitive emulation,
07:13 where we try to work on very, very precisely
07:15 emulating the exact cognitive processes that humans use,
07:19 using LLMs trained very, very precisely
07:21 on very, very precise tasks,
07:23 so we know what these systems can and cannot do.
07:26 This is not a full solution to the problem.
07:28 It is an important step forward
07:30 to how can we get systems that in your business,
07:33 in your applications, whatever, bring you real value,
07:36 do what you want them to do,
07:38 do it in ways that is legible and comprehensible to humans,
07:41 and don't go off the rails.
07:43 But this is not enough for a good future
07:45 in totality, of course,
07:46 since as we are on a panel here about regulation,
07:49 the problem of AI in general, of full job automation,
07:52 which is something that we will see in the near future,
07:54 is a question that is not just technical,
07:57 it is also a social and a political problem,
07:59 it is a humanist problem.
08:00 Techno-optimism is nice, I love technology,
08:04 I have worked with technology my entire life,
08:06 I've worked with open source,
08:08 I've been deeply involved in movements, I love technology,
08:10 but techno-optimism is not a replacement for humanism.
08:14 If we just build better and better technology,
08:16 it does not mean we will build a better future,
08:18 that we make a better place for people.
08:20 We will make a better place for capital, let me tell you.
08:22 Those stock prices are gonna go up, I guarantee you that.
08:24 That's a prediction I'm happy to make.
08:26 But will it be a better world for us to live in?
08:28 I think that is a very difficult question.
08:31 - Yeah, I appreciate that, that perspective.
08:33 The gap between the haves and have-nots in AI enablement,
08:38 when we think about people at work
08:43 who can be as productive as the next 20 or 30 colleagues,
08:47 and folks who haven't learned those skills,
08:50 it's definitely not a recipe for social cohesion.
08:53 And you think about the government's role
08:56 in regulating the social ills and harms of social media,
09:00 and it is not a pretty picture of their role
09:04 to be able to really bring the best
09:06 of this next platform shift.
09:09 In that hairy world, Bill Beer,
09:13 as chair of the responsible AI group
09:16 at London Stock Exchange,
09:19 you are really in the middle of trying to figure out
09:22 how to incorporate AI internally in your business,
09:27 while keeping one eye out for what the government is doing
09:31 and is likely to regulate, right, in the UK.
09:35 Nothing is law yet around where and where not to invest.
09:39 And then also where the models and the technology is going.
09:43 So take us through a day in the life of Bill Beer,
09:47 trying to really help, hopefully,
09:49 push the envelope internally on the innovation side
09:52 while balancing all of the risks and potential harms.
09:56 - Yeah, thank you.
09:57 I mean, LSEG is an exchange, a clearinghouse,
10:00 but most of our business is data and analytics.
10:03 So when we started looking at the control side of things
10:08 at the beginning of last year,
10:10 we established a set of principles aligned to our values,
10:12 responsible AI principles,
10:15 and then we set up a steering committee
10:19 to really govern what was going on,
10:21 because there was a huge amount of hype in Q1 last year,
10:24 and we wanted to get a handle on,
10:26 okay, what's going on in a 25,000-person organization?
10:29 How can we control it?
10:30 So we started, so I established that as an oversight group,
10:34 not because I'm the risk officer,
10:36 because this is a truly multidisciplinary activity,
10:40 and we see it fundamental to business transformation.
10:43 So we had an overall steering committee
10:46 that really collected use cases, evaluated them,
10:49 looked at feasibility commercially as well,
10:51 but also looked at the risk.
10:53 And then a product-level working group,
10:56 an efficiency working group,
10:58 and a responsible AI working group.
11:01 And that's really the construct that we've used ever since.
11:06 And we keep the principles updated
11:09 as new regulations come out,
11:11 so they're in line with the EU AI Act.
11:13 As NIST brought out standards, we've aligned them to that.
11:16 So all of these are very helpful,
11:18 but we largely have taken a self-regulation route.
11:22 And in terms of managing the risk,
11:24 I think our approach there has very much been
11:27 upgrading, constant upgrading of our capabilities,
11:30 so particularly data management, security,
11:33 but also model risk management,
11:35 and thinking about how AI systems
11:38 can regulate other AI systems.
11:42 And we've recently tested a couple of our LLM models.
11:46 So we're working very closely with Microsoft
11:48 in developing AI tools that really expose
11:52 our data and analytics and make clients' lives easier.
11:55 And then we're also working on tools
11:57 to make internal efficiency improvements,
11:59 like a QA tool to improve customer inquiries.
12:04 And as we looked at our model validation approach
12:07 and designed that in conjunction with some professors,
12:11 we used LLM as a judge as part of that,
12:14 as well as human oversight.
12:16 And that showed interesting results,
12:18 particularly for out-of-domain questions
12:22 that we might throw at a model.
12:23 So that actually didn't surpass our expectations,
12:26 but actually was very helpful in feeding back
12:29 into the model development lifecycle.
12:32 So very much evolving the risk management approach
12:35 at the same time, but the self-regulation approach,
12:38 I think, is really helpful,
12:40 keeping it updated as new regulations
12:43 and principles arise.
12:45 - Yeah, I love that.
12:46 One framework I've really found helpful
12:49 in talking to execs about AI safety
12:52 is the concept of guardrails.
12:54 And I think what I'm hearing here is
12:57 there is a level of guardrails that the tech vendors
13:02 are going to be competing to build into their systems.
13:06 There is a level of guardrails
13:08 that responsible executives like Belbir
13:11 are going to be building into their own
13:14 responsible AI and governance frameworks
13:16 at a company level.
13:17 And then there are AI guardrails
13:19 that the government should be insisting on.
13:23 And so with that framing in mind,
13:26 could we do rapid fire, maybe starting with Connor?
13:29 What are the top one or two guardrails?
13:33 If you do the vendor level, Connor,
13:36 we'll go then to Belbir government level,
13:38 and then, or company level,
13:40 and then Ian with government level.
13:43 Top one or two guardrails
13:44 that folks should be building into products.
13:47 - Oh God.
13:48 (laughing)
13:51 I mean, for your own business,
13:52 I highly recommend that you do not trust
13:55 any outputs of LLMs in general.
13:57 I don't think there's any clear technical solutions there,
14:01 other than filters and the like,
14:04 but even those don't really cut it.
14:05 If you're asking about the general societal level questions,
14:08 these are the kinds of questions
14:09 that I don't think companies should even have to answer.
14:12 It's not, this might be controversial,
14:14 but it's not, shouldn't be the responsibility
14:16 of oil companies to solve climate change.
14:18 It should be the responsibility of governments
14:20 to stop oil companies from causing climate change,
14:23 or at least make them pay to clean it up
14:26 after they cause the mess.
14:27 So really, guardrails should be something
14:30 that is coming down from the governmental level,
14:32 of in terms of societal level risks.
14:34 As for risks that apply to like a given business application
14:37 or a given thing, I can just say like,
14:40 careful what your consultants tell you.
14:41 AI is super unreliable at the moment.
14:43 It is not at human level reliability at the moment
14:46 for basically any application that isn't fun fiction writing.
14:49 So just be very careful there.
14:51 - All right, well,
14:54 Connor's thrown the responsibility over to you guys, Ian.
14:56 What are the top one or two guardrails
14:59 that you think the UK, let's start with the UK,
15:03 or keep it to the UK, should insist on?
15:06 - I think that I would say in most of our efforts right now
15:11 are really trying to understand whether large models
15:14 enhance any existing national security threats.
15:17 And so that would seem to be a really clear area
15:20 where we already are doing work to mitigate against,
15:25 for example, cyber attacks.
15:27 And so understanding how a model can and can't contribute
15:31 to some sort of malicious code-based attack
15:35 would be the kind of thing.
15:36 But again, I think we're still quite early
15:39 in building the empirical base for talking about this stuff.
15:42 But the great thing about what we're doing
15:46 at the Safety Institute is we are working very closely
15:49 with the experts in the UK national security community,
15:53 people who are experts in biosecurity,
15:54 experts in cybersecurity and nuclear security,
15:57 who can help us kind of understand
15:59 the level of uplift these systems truly provide
16:02 'cause they think about these risks day in, day out.
16:04 I think that's the sort of, I guess, the easy answer.
16:07 I think the sort of thing I think we would wanna be
16:10 really on the lookout for with these models
16:12 is something like deceptiveness.
16:14 You wouldn't want a model that is deceptive.
16:16 And that's the sort of thing that's a bit broader
16:20 than national security that I think it'd be very important
16:22 to sort of put guardrails in around.
16:26 - Yeah, thank you.
16:27 Valvere.
16:28 - Yeah, I think, so we would worry a lot
16:31 at a corporate level around being responsible
16:34 for any misinformation, disinformation.
16:37 So in terms of ensuring that you're putting things out there
16:42 for people to make decisions on that are robust,
16:45 I think having watermarks content and auditability.
16:49 So that's what we're trying to build into our AI tools
16:52 so that clients can see what was used to produce the output.
16:56 And I think just in terms of copyright type issues,
17:01 I think having really strong data management
17:06 and tagging your data as to what's the licensing agreement
17:08 you have and where can you use it and who can have,
17:12 what are the permissions associated with that data
17:14 will give you just a bit more robust assurance
17:17 that what you're producing at the end
17:19 doesn't infringe anyone else's copyrights.
17:21 But both of these areas,
17:23 you're already seeing some litigation
17:25 that is concerning for organizations.
17:27 So I think data management is probably quite fundamental.
17:32 It's the fuel for AI.
17:33 So getting that right, I think,
17:35 puts you in a much better place.
17:37 - In your usage inside of London Stock Exchange,
17:42 have you created a guardrail around using models
17:46 that are not trained on copyrighted data?
17:48 We're seeing that now in some RFPs.
17:50 - That are not trained.
17:52 - Yeah.
17:53 - Yeah, so I think there's--
17:55 - Have you gone that far is what I'm asking.
17:57 - The main thing that we are focused on
18:02 is actually training models on our own data.
18:07 And a lot of our data, we actually contract,
18:10 some we create ourself
18:11 and some we actually have license agreements
18:14 with third parties.
18:15 So in doing so, we're being really careful.
18:17 It's very easy for our own data,
18:19 but where we actually contract data from somebody else,
18:22 we're very careful about how and when we would use that
18:25 within the training of an LLM
18:28 that we would use with a client.
18:30 - So trained and not fine-tuned.
18:31 - Yes.
18:32 - Yeah, very interesting.
18:33 Well, I can't believe it, but that is all we have time for.
18:37 Would have loved to do a Q&A session with everybody,
18:39 but we'll have to save that for the break.
18:41 So thanks everybody.
18:42 (upbeat music)
18:45 [BLANK_AUDIO]