WEBVTT

00:19.930 --> 00:22.152
This week , we will discuss topological

00:22.159 --> 00:24.103
features and large language models

00:24.103 --> 00:26.215
which have received attention because

00:26.215 --> 00:29.200
they can syn synthesize humanlike text .

00:29.469 --> 00:31.580
Most of the theoretical literature on

00:31.580 --> 00:33.779
deep neural networks assume that the

00:33.790 --> 00:36.380
space of possible vectors is a manifold .

00:36.389 --> 00:38.389
But it seems that no one has really

00:38.389 --> 00:41.169
checked to see if this is true today .

00:41.180 --> 00:43.347
A discussion on this topic will be led

00:43.347 --> 00:45.513
by Professor Michael Robinson from the

00:45.513 --> 00:47.624
Mathematics and Statistics Department

00:47.624 --> 00:49.847
at American University . There's been a

00:49.847 --> 00:51.736
lot of interest this week in this

00:51.736 --> 00:53.680
discussion including Patrick Grace

00:53.680 --> 00:56.150
George . Uh So without delay over to

00:56.159 --> 00:58.437
you Michael to kick us off . All right .

00:58.439 --> 01:00.661
Well , thank you for the invitation and

01:00.661 --> 01:02.772
actually a thanks to uh Katrina who I

01:02.772 --> 01:05.029
saw on the line join in here for uh for

01:05.040 --> 01:07.373
letting me know about this group at all .

01:07.373 --> 01:09.484
This has been uh it's been a fun part

01:09.484 --> 01:11.429
of my Friday . So uh these , these

01:11.429 --> 01:13.540
discussions are great . So uh this is

01:13.540 --> 01:15.818
uh a little bit of that . I don't know .

01:15.818 --> 01:17.949
I , I don't . Yeah , Kevin Kevin said

01:17.959 --> 01:20.070
it said it right . Uh No one seems to

01:20.070 --> 01:22.126
have checked that certain things are

01:22.126 --> 01:24.292
manifold in the machine learning world

01:24.292 --> 01:26.519
people are starting to uh and there's

01:26.529 --> 01:29.160
some surprise now how this there is

01:29.169 --> 01:32.080
upon consciousness proper . Uh I'm not

01:32.089 --> 01:34.311
a consciousness researcher . So maybe ,

01:34.311 --> 01:36.589
maybe there's , there's something here .

01:36.589 --> 01:40.379
All right . Excellent . So uh first of

01:40.389 --> 01:42.556
all , J work with a bunch of people uh

01:42.556 --> 01:44.556
funding from DARPA Shana Suit at uh

01:44.556 --> 01:48.089
DARPA I 20 formerly at GA uh is a , is

01:48.099 --> 01:50.321
a big collaborator on this . Uh Most of

01:50.321 --> 01:52.543
the most of the initial data processing

01:52.543 --> 01:54.766
by the good folks at Galois and then my

01:54.766 --> 01:55.988
students next slide .

01:58.800 --> 02:01.199
So I want you to take away visually .

02:01.410 --> 02:04.389
Uh I if we had a space that was a

02:04.400 --> 02:07.050
manifold , um you can ask great

02:07.059 --> 02:09.115
questions about it if you know about

02:09.115 --> 02:11.259
its geometry . Uh Most of the time ,

02:11.270 --> 02:13.270
the machine learning people imagine

02:13.270 --> 02:15.437
that their data are vector , like they

02:15.437 --> 02:17.548
live on vector spaces , vector spaces

02:17.548 --> 02:19.714
typically have some kind of measure of

02:19.714 --> 02:21.770
distance on them . There are lots of

02:21.770 --> 02:23.929
these and when you ask ma machine

02:23.940 --> 02:26.051
learning practitioners , hey , what ,

02:26.051 --> 02:28.162
what , what , what's the right metric

02:28.162 --> 02:30.218
for your data ? They usually look at

02:30.218 --> 02:32.440
you a little funny . Uh And so we tried

02:32.440 --> 02:34.496
a bunch and we found that they , the

02:34.496 --> 02:36.607
the spaces if they were to be medical

02:36.607 --> 02:38.607
tend to be negatively curved . What

02:38.607 --> 02:40.829
does that mean ? From an interpretation

02:40.829 --> 02:42.884
standpoint ? I don't know but from a

02:42.884 --> 02:44.773
math standpoint , seeing as I'm a

02:44.773 --> 02:46.940
mathematician , they look kind of like

02:46.940 --> 02:48.940
this . So this uh this neat knitted

02:48.940 --> 02:51.051
surface is a two dimensional manifold

02:51.051 --> 02:53.273
uh that has negative curvature . And as

02:53.273 --> 02:55.496
it turns out for whatever reason , kind

02:55.496 --> 02:57.551
of the internal state space of large

02:57.551 --> 02:59.773
language models seems like at least the

02:59.773 --> 03:01.996
parts of it that are manifold and there

03:01.996 --> 03:04.162
are parts of it which aren't , but the

03:04.162 --> 03:06.273
parts of it that are manifold seem to

03:06.273 --> 03:08.496
be negatively curved or hyperbolic . So

03:08.496 --> 03:10.496
this is kind of a visual but now it

03:10.496 --> 03:12.607
turns out this is just a manifold and

03:12.607 --> 03:14.662
it looks like there are things which

03:14.662 --> 03:16.773
are not manifold . So imagine a bunch

03:16.773 --> 03:18.884
of these uh of different dimensions .

03:18.884 --> 03:20.884
So this is two dimensional , if you

03:20.884 --> 03:22.940
were a bug sitting on the surface of

03:22.940 --> 03:25.051
this , this knitted thing , you could

03:25.051 --> 03:27.329
move sort of forward , backward , left ,

03:27.329 --> 03:29.329
right , but not off the surface two

03:29.329 --> 03:31.440
dimensions . Uh There are also higher

03:31.440 --> 03:33.440
dimensional pieces that cut kind of

03:33.440 --> 03:35.162
glued together like a union of

03:35.162 --> 03:37.218
manifolds of different sizes . So if

03:37.218 --> 03:39.496
you can kind of fit that in your brain ,

03:39.496 --> 03:41.551
that's the visual of what it appears

03:41.551 --> 03:43.551
that the kind of hidden state space

03:43.551 --> 03:45.662
large language model looks like . And

03:45.662 --> 03:47.718
we're gonna kind of walk you through

03:47.718 --> 03:49.829
why I think this is the case . Um And

03:49.829 --> 03:52.190
so yes , I'm in a math department . I'm

03:52.199 --> 03:54.199
also in a stat department . We're a

03:54.199 --> 03:56.589
joint math stat department . So really ,

03:56.600 --> 03:58.489
I wanna compute stuff that's like

03:58.489 --> 04:00.711
differential geometry fully in the math

04:00.711 --> 04:02.933
world . But then I wanna run hypothesis

04:02.933 --> 04:04.878
tests against it fully in the stat

04:04.878 --> 04:07.100
world . And I wanna do that for machine

04:07.100 --> 04:09.267
learning systems in a particular large

04:09.267 --> 04:11.649
language model . Next slide . So what

04:11.660 --> 04:13.771
do I mean by a large language model ?

04:13.771 --> 04:15.993
Anyway , there's certainly things which

04:15.993 --> 04:18.049
uh I , I would say it's dubious that

04:18.049 --> 04:19.993
they , that they're modeling human

04:19.993 --> 04:22.049
brain in any way . They're more like

04:22.049 --> 04:24.049
AAA statistical regression and they

04:24.049 --> 04:26.160
work on text , you put in some text ,

04:26.160 --> 04:28.216
they do not meddle in the affairs of

04:28.216 --> 04:30.839
wizards and the machine should somehow

04:30.850 --> 04:32.906
know that the right response to that

04:32.906 --> 04:35.779
famous uh quotation is for their subtle

04:35.790 --> 04:37.734
and quick to anger . So somehow it

04:37.734 --> 04:40.089
needs to fill in with this arrow . Next

04:40.100 --> 04:44.040
slide , the high uh at a

04:44.049 --> 04:46.271
high level where a large language model

04:46.271 --> 04:49.390
works is it takes this text words ,

04:49.399 --> 04:51.890
characters incidentally , they tend to

04:51.899 --> 04:54.121
be Multilingual . So they're in unicode

04:54.130 --> 04:56.859
somehow uh and turns them into vectors

04:56.869 --> 05:00.170
RN for N big like

05:00.529 --> 05:03.589
billions is uh And then there is this

05:03.600 --> 05:06.019
function called the transformer that

05:06.029 --> 05:08.170
takes that vector billions ish and

05:08.179 --> 05:10.290
gives you a new vector billions ish .

05:10.679 --> 05:12.901
In fact , I've drawn the ends being the

05:12.901 --> 05:15.123
same , but they don't really have to be

05:15.123 --> 05:17.380
in practice . Um And they're the , the

05:17.390 --> 05:20.700
our ends are broken up in various ways

05:20.709 --> 05:23.519
uh to make them easier to handle . Uh

05:23.880 --> 05:26.119
So that there's some notion of locality

05:26.130 --> 05:28.359
words come in a particular order ,

05:28.670 --> 05:30.910
there's a context window that gets that

05:30.920 --> 05:33.790
that gets dealt with . But if you take

05:33.799 --> 05:36.730
the view from earth's orbit on what a

05:36.739 --> 05:38.739
transformer is really , really high

05:38.739 --> 05:41.459
level view . It's a piecewise smooth

05:41.470 --> 05:43.581
function that's globally continuous .

05:43.581 --> 05:46.600
So here , often people talk about relu

05:46.609 --> 05:48.553
and deep learning with deep neural

05:48.553 --> 05:50.442
networks . Uh they have different

05:50.442 --> 05:52.387
activation functions . They're not

05:52.387 --> 05:54.442
smooth . I can't take derivatives at

05:54.442 --> 05:56.442
everywhere . I can take derivatives

05:56.442 --> 05:58.760
most everywhere . So piecewise smooth .

05:59.190 --> 06:00.968
But overall , it's a continuous

06:00.968 --> 06:02.746
function and that puts a lot of

06:02.746 --> 06:04.968
constraints . Now , how do you get this

06:04.968 --> 06:07.023
function ? Well , at great expense ,

06:07.023 --> 06:08.690
they're found by some kind of

06:08.690 --> 06:11.500
statistical regression . Um And to my

06:11.510 --> 06:13.566
eyes , again , earth view from earth

06:13.566 --> 06:16.600
orbit , a function that's continuous

06:16.609 --> 06:19.829
from r end to RM most of the time ,

06:19.839 --> 06:23.200
probably if it's a reasonable function

06:23.209 --> 06:25.540
as these tend to be , uh it feels like

06:25.549 --> 06:27.716
a dynamical system of some sort , it's

06:27.716 --> 06:29.827
got some kind of hidden state space .

06:29.827 --> 06:31.938
And then you can kind of ask , well ,

06:31.938 --> 06:33.660
where is the attractor of that

06:33.660 --> 06:35.549
dynamical system ? Where does the

06:35.549 --> 06:37.493
system spend most of its time that

06:37.500 --> 06:39.850
should be on the vectors that

06:39.859 --> 06:42.399
correspond to comprehensible ,

06:42.410 --> 06:44.570
meaningful ? Maybe maybe that's too

06:44.579 --> 06:46.079
pretentious , but at least

06:46.079 --> 06:49.140
syntactically correct sentences that

06:50.179 --> 06:52.179
now you could ask , well , how do I

06:52.179 --> 06:54.401
know this ? For sure , there's actually

06:54.401 --> 06:54.399
some good proofs in the literature that

06:54.410 --> 06:56.809
tell me that I have a dynamical system ,

06:56.820 --> 06:59.042
I could start to do stochastic dynamics

06:59.042 --> 07:02.290
on this . But I'm I'm again trying to

07:02.299 --> 07:04.355
be very naive and just ask , what is

07:04.355 --> 07:06.630
the shape of the space of where the

07:06.640 --> 07:09.980
words that make sense , live and , and

07:09.989 --> 07:12.100
even that question already gets us in

07:12.100 --> 07:14.100
the kind of deep water next slide .

07:16.239 --> 07:18.519
So I , I really wanna just focus right

07:18.529 --> 07:20.696
now right today on , on how do we turn

07:20.696 --> 07:24.350
the text inter vectors uh in the in the

07:24.359 --> 07:26.192
literature , people call this an

07:26.192 --> 07:29.630
embedding . Now as a mathematician , I

07:29.640 --> 07:32.440
think of myself as , as a pathologist ,

07:32.450 --> 07:34.589
the word embedding has a definite

07:34.600 --> 07:37.450
meaning . To me , it means that I have

07:37.459 --> 07:40.109
two topological spaces . I have notions

07:40.119 --> 07:42.609
of measuring neighborhood on both the

07:42.619 --> 07:45.670
domain , the text and the co domain ,

07:45.920 --> 07:48.730
the real numbers , our and vectors and

07:48.739 --> 07:50.795
of course , our and vectors . I know

07:50.795 --> 07:52.350
how to manage distances and

07:52.350 --> 07:54.517
neighborhoods there . That's perfectly

07:54.517 --> 07:56.517
fine text . Can you tell me how far

07:56.517 --> 07:58.572
apart two pieces of text are maybe a

07:58.572 --> 08:00.683
lot of contention how we do this . So

08:00.683 --> 08:02.739
it cannot strictly be an embedding .

08:03.339 --> 08:05.980
It's a function , it should be mostly 1

08:05.989 --> 08:08.269
to 1 because different words should end

08:08.279 --> 08:10.390
up as different vectors . But besides

08:10.390 --> 08:12.750
of that , I don't know . So really what

08:12.779 --> 08:14.890
we're going to do is we're just gonna

08:14.890 --> 08:16.668
say , hey , suppose it were and

08:16.668 --> 08:18.668
abetting and suppose that the words

08:18.668 --> 08:20.612
were sampling from some continuous

08:20.612 --> 08:23.459
space . This is probably not entirely

08:23.470 --> 08:26.640
true , but it's sort of true because we

08:26.649 --> 08:28.871
don't imagine that these large language

08:28.871 --> 08:31.093
models that they've been trained on all

08:31.093 --> 08:32.982
the human text and whatnot , they

08:32.982 --> 08:35.205
haven't seen all human language and all

08:35.205 --> 08:37.316
human language that is to come people

08:37.316 --> 08:39.316
create new words and they get drawn

08:39.316 --> 08:41.650
from neighboring words somehow . So

08:41.659 --> 08:43.659
thinking of words as , as something

08:43.659 --> 08:45.826
like a sampling from some hidden state

08:45.826 --> 08:47.992
states is kind of vaguely reasonable .

08:47.992 --> 08:50.326
And and there's some notion of locality ,

08:50.326 --> 08:52.492
a sliding window on text . So really ,

08:52.492 --> 08:54.548
we don't end up embedding the entire

08:54.548 --> 08:56.603
text wholesale , we break it up into

08:56.603 --> 08:58.715
tokens and embed each of those tokens

08:58.715 --> 09:01.320
and then string together to get a AAA

09:01.330 --> 09:04.250
vector of vectors . Next slide .

09:06.239 --> 09:08.350
So just to give you an idea , this is

09:08.350 --> 09:10.919
this is the first few dimensions of the

09:10.929 --> 09:14.190
token embedding for GP T two GP T two

09:14.200 --> 09:17.250
has 50,000 tokens . So if you look on

09:17.260 --> 09:19.316
the left , there's a set of tokens .

09:19.316 --> 09:21.649
The first token uh appears to be a , an ,

09:21.649 --> 09:25.590
a uh exclamation point that a backslash ,

09:25.799 --> 09:27.688
right ? And , and , and Patrick ,

09:27.688 --> 09:29.855
you're absolutely right . People start

09:29.855 --> 09:32.077
to think about , hey , can I do algebra

09:32.077 --> 09:31.859
on this once I got a vector , add

09:31.869 --> 09:34.000
subtract and hopefully words that are

09:34.010 --> 09:36.330
nearby do something reasonable when I

09:36.340 --> 09:39.750
do algebra on them , let me say that

09:39.760 --> 09:41.927
the , the mathematician is all excited

09:41.927 --> 09:44.093
by this . The statistician is a little

09:44.093 --> 09:48.030
word because last I checked words

09:48.039 --> 09:50.380
are a classic example of a categorical

09:50.390 --> 09:52.590
variable . And vectors are a classical

09:52.599 --> 09:54.710
example of a numerical variable . And

09:54.710 --> 09:57.530
we're mixing data types that tends to

09:57.539 --> 10:01.330
lead to the problem . Anyway , let's

10:01.340 --> 10:03.599
get it point of the matter . Here is

10:04.109 --> 10:07.400
each one of these 50,000 tokens sort of

10:07.409 --> 10:09.465
imagine that this being a table with

10:09.465 --> 10:11.631
50,000 rows , uh Each one of them gets

10:11.631 --> 10:13.742
turned into a vector . In the case of

10:13.742 --> 10:17.159
GP T two , it is 768 dimensions . So

10:17.169 --> 10:19.336
768 columns , I've shown you the first

10:19.336 --> 10:21.558
eight here and in various numbers , you

10:21.558 --> 10:23.613
can see they mostly lie between zero

10:23.613 --> 10:27.190
and one . Uh If anyone's a history buff ,

10:27.940 --> 10:30.051
I know a terrible calling . History .

10:30.051 --> 10:32.162
Uh Why is the number 768 ? Leave that

10:32.162 --> 10:34.051
as a as a question to you guys to

10:34.051 --> 10:36.162
figure out there's a real good reason

10:36.162 --> 10:38.384
why . That's the case . So anyway , the

10:38.384 --> 10:40.329
point of the matter is that uh the

10:40.329 --> 10:42.551
topology and geometry lives all in this

10:42.551 --> 10:44.662
latent space , this vector space land

10:44.662 --> 10:46.829
and the topology does not exist on the

10:46.829 --> 10:48.996
token , but we can induce one and then

10:48.996 --> 10:51.051
ask , well , if I have a token , how

10:51.051 --> 10:52.996
big is its neighborhood ? How many

10:52.996 --> 10:55.218
neighbors does it have ? And if we were

10:55.218 --> 10:57.384
to imagine this is a sample drawn from

10:57.530 --> 10:59.697
a continuous thing , you'd say , hey ,

10:59.697 --> 11:01.752
what's the dimension ? How many free

11:01.752 --> 11:03.919
parameters does the space of neighbors

11:03.919 --> 11:06.080
of a give and token have ? Maybe even

11:06.090 --> 11:08.090
since we have geometry on the , the

11:08.090 --> 11:10.257
late space , you can ask , mm , what's

11:10.257 --> 11:14.239
its curvature ? Excellent . So we

11:14.250 --> 11:16.739
even care about doing this . And uh

11:16.750 --> 11:18.979
this is a , a sort of a blobby picture

11:18.989 --> 11:21.211
here . It kind of looks like a bacteria

11:21.211 --> 11:23.545
with some cilia off of that . This is a ,

11:23.545 --> 11:25.433
a drawing of a Whitney stratified

11:25.433 --> 11:27.489
manifold . It's a stratified space ,

11:27.489 --> 11:29.656
not a manifold , but it's a union of a

11:29.656 --> 11:31.711
bunch of one dimensional manifolds ,

11:31.711 --> 11:33.489
little , little hairs and a two

11:33.489 --> 11:35.433
dimensional blob in the middle . I

11:35.433 --> 11:37.656
think this is kind of the sort of thing

11:37.656 --> 11:39.878
that's going on . So it's not even just

11:39.878 --> 11:41.933
that the spaces are , are singular ,

11:41.933 --> 11:44.156
they've got crossings in them that they

11:44.156 --> 11:43.729
have vastly different dimensions . And

11:43.739 --> 11:45.850
I'll show you why this is why I think

11:45.850 --> 11:48.017
this in a moment , but just to kind of

11:48.017 --> 11:50.350
keep your , your , your head right here ,

11:50.350 --> 11:52.572
the way the large language model has to

11:52.572 --> 11:54.795
work is it's gonna take a token at some

11:54.795 --> 11:56.683
point in this space , not all the

11:56.683 --> 11:58.739
vectors correspond to valid tokens .

11:58.739 --> 12:00.683
And it's gonna tell you that's the

12:00.683 --> 12:02.461
query . It's gonna tell you the

12:02.461 --> 12:04.683
response is somewhere else . Now , if I

12:04.683 --> 12:06.795
have a continuous map , it's entirely

12:06.795 --> 12:08.906
possible that , that there'll be some

12:08.906 --> 12:11.072
interpolation going on and I might end

12:11.072 --> 12:13.183
up between , not on the state of to .

12:13.183 --> 12:15.295
And in fact , this happened with some

12:15.295 --> 12:17.461
nano frequency . This is re related to

12:17.461 --> 12:19.295
the idea of perplexity and large

12:19.295 --> 12:21.350
language model . It's not , I've got

12:21.350 --> 12:23.572
something now , a response that isn't a

12:23.572 --> 12:25.795
token validly . Uh How do I get it back

12:25.795 --> 12:29.119
to a token so that I can emit text ? I

12:29.130 --> 12:31.241
don't know , right . Find the nearest

12:31.241 --> 12:33.408
one A and the problem is that if I had

12:33.408 --> 12:35.686
a stratified manifold , not a manifold .

12:35.686 --> 12:37.463
Yeah , that problem is somewhat

12:37.463 --> 12:39.352
ambiguous . There can be multiple

12:39.352 --> 12:41.297
closest tokens that I might try to

12:41.297 --> 12:43.352
project onto . And that means that I

12:43.352 --> 12:45.463
may get really different responses ie

12:45.463 --> 12:47.859
there some built in instability in how

12:47.869 --> 12:50.280
the model works . So this is if , if

12:50.289 --> 12:52.456
we're not dealing with manifold , this

12:52.456 --> 12:54.789
problem is endemic and it's unavoidable .

12:54.789 --> 12:57.011
So let's see if we're in that problem .

12:57.011 --> 12:59.390
Thanks a lot . OK . So how do we have

12:59.400 --> 13:01.511
to make the mention ? There's a bunch

13:01.511 --> 13:03.678
of these in the literature . Um And uh

13:03.678 --> 13:05.900
maybe it's because I'm not a very smart

13:05.900 --> 13:08.122
programmer . I tried some of them on GP

13:08.122 --> 13:10.122
T two and Lema and a bunch of other

13:10.122 --> 13:12.289
large language models . And as soon as

13:12.289 --> 13:14.011
I ran on my , not very new but

13:14.011 --> 13:15.900
certainly performance server , it

13:15.900 --> 13:18.178
immediately crashed , ran out of memos ,

13:18.178 --> 13:20.233
but I , I'm , I'm obviously doing so

13:20.233 --> 13:22.456
wrong . Try it again . Mess around with

13:22.456 --> 13:24.622
a little bit different . Still crash .

13:24.622 --> 13:26.844
What am I doing wrong ? Let me just try

13:26.844 --> 13:26.780
something a little back to first

13:26.789 --> 13:29.159
principles . How do IE the s what's the

13:29.169 --> 13:32.440
volume of that ? Fear ? Anyone remember ?

13:32.450 --> 13:34.561
Volume is a spar . What's the formula

13:34.561 --> 13:36.672
for that ? All right . Yeah . What is

13:36.672 --> 13:38.839
that ? Uh All right , fine . I , I'm a

13:38.839 --> 13:40.839
math teacher . It's flippers . Py ,

13:41.130 --> 13:43.297
what's the area of the circle ? I note

13:43.297 --> 13:45.352
that that's an easy one , right ? IR

13:45.352 --> 13:47.690
squared , what's the volume ? I would

13:47.700 --> 13:50.090
say a 10 dimensional sphere ? Uh

13:50.400 --> 13:52.549
Something in pi is involved sort of

13:52.559 --> 13:54.503
that . But it's probably something

13:54.503 --> 13:57.640
proportional volume proportional to R

13:57.650 --> 14:00.349
to the 10 that Xon that tells you the

14:00.359 --> 14:02.669
dimension . So roughly speaking , the

14:02.679 --> 14:06.270
volume of a ball in dimension N is like

14:06.510 --> 14:09.140
proportional to R to the end . And

14:09.150 --> 14:11.094
actually , there's some correction

14:11.094 --> 14:13.150
terms that , that , that , that come

14:13.150 --> 14:15.094
out into this formula according to

14:15.094 --> 14:17.317
curvature . And you know what I thought

14:17.317 --> 14:19.483
again with my stat hat on , I said , I

14:19.483 --> 14:21.650
want that and I want to find it . If I

14:21.650 --> 14:23.872
take the log of both sides , I now have

14:23.872 --> 14:26.094
a linear regression . The dimension the

14:26.094 --> 14:28.094
end there is the slope of my linear

14:28.094 --> 14:30.150
aggression . So if I plot the log of

14:30.150 --> 14:32.261
volume against the log of radius , uh

14:32.261 --> 14:34.428
Lager , yeah Lager volume versus lager

14:34.428 --> 14:36.761
radius , I I should get a straight line .

14:37.119 --> 14:39.419
If I got a medical , if I don't have a

14:39.429 --> 14:41.596
manifold , I don't get a straight line

14:41.596 --> 14:43.707
that's like basically all there is to

14:43.707 --> 14:45.762
it . Um There'll be some corrections

14:45.762 --> 14:47.929
there , there's the R squared terms of

14:47.929 --> 14:47.849
what I might have to bend and that has

14:47.859 --> 14:49.748
to do with curvature next slide .

14:51.659 --> 14:53.609
So let's try , let me estimate the

14:53.619 --> 14:55.841
volume by Monte Carlo estimate . Here's

14:55.841 --> 14:57.710
a , here's a sphere , uh two

14:57.719 --> 14:59.941
dimensional sphere . So if I , if I sit

14:59.941 --> 15:02.108
myself down on the point that I marked

15:02.108 --> 15:04.108
with the crosshairs on the left and

15:04.108 --> 15:06.330
plot log of radius versus log of volume

15:06.330 --> 15:08.441
on , on the left there on the right .

15:08.441 --> 15:10.497
Rather uh I get the points that I've

15:10.497 --> 15:12.497
marked and you notice it's mostly a

15:12.497 --> 15:14.608
straight bond , but it kind of curves

15:14.608 --> 15:16.775
down a little bit and that curves down

15:16.775 --> 15:18.886
a little bit is an expression of , of

15:18.886 --> 15:20.830
positive ricci curvature . Fear is

15:20.830 --> 15:22.941
positively curved . It's not like the

15:22.941 --> 15:24.997
uh the knitted surface that I showed

15:24.997 --> 15:27.163
you on the second slide . It's got the

15:27.163 --> 15:29.219
opposite opposite side . OK . So the

15:29.219 --> 15:31.219
point of the matter is you can just

15:31.219 --> 15:34.669
like polynomial regression your way to

15:34.679 --> 15:36.901
finding the dimension and the curvature

15:36.901 --> 15:40.270
of any embedded space that's in samples

15:41.210 --> 15:43.432
cool . Let's try it on something better

15:43.432 --> 15:45.669
than a sphere . Right . Next slide

15:47.539 --> 15:49.650
I said we might be dealing with not

15:49.710 --> 15:51.766
manifolds but stratified manifolds .

15:51.766 --> 15:53.988
Here's a stratified manifold . It looks

15:53.988 --> 15:56.099
a bit like one of those exercise ball

15:56.099 --> 15:58.210
things that's attached to a ring . So

15:58.210 --> 16:00.432
there's a one dimensional circle that's

16:00.432 --> 16:02.543
a one dimensional sphere glued onto a

16:02.543 --> 16:04.599
two dimensional disk and glued on to

16:04.599 --> 16:06.821
that is a three dimensional ball . OK .

16:06.821 --> 16:08.821
If you set yourself down on the red

16:08.821 --> 16:10.932
point , so squarely inside the circle

16:11.159 --> 16:13.270
and you look at the log volume versus

16:13.270 --> 16:15.437
log radius on the , the right , that's

16:15.437 --> 16:17.492
the red curve . And you can see it's

16:17.492 --> 16:19.603
got a nice slope one because we're in

16:19.603 --> 16:21.881
dimension one at least for small radii .

16:21.881 --> 16:23.770
But what's interesting is as that

16:23.770 --> 16:25.992
radius to the sphere gets bigger radius

16:25.992 --> 16:28.048
of the ball gets bigger . Eventually

16:28.048 --> 16:30.103
you start picking up points from the

16:30.103 --> 16:32.159
other pieces , two dimensional piece

16:32.159 --> 16:34.270
and the three dimensional piece . And

16:34.270 --> 16:36.103
what what happens with the slope

16:36.103 --> 16:38.559
changes abruptly . So the point is that

16:38.570 --> 16:40.237
slope changing abruptly is an

16:40.237 --> 16:42.181
indication that things have gotten

16:42.181 --> 16:44.292
glued together . Approach it . If you

16:44.292 --> 16:46.580
look plot log volume versus log radius

16:46.590 --> 16:48.812
and you have a straight line , but then

16:48.812 --> 16:50.923
it has a , a knee and a sharp bend in

16:50.923 --> 16:53.419
it , you don't have a manifold . And ,

16:53.429 --> 16:55.596
and in fact , actually based on a cool

16:55.596 --> 16:58.239
paper by Charlie Fefferman from 2016 ,

16:58.409 --> 17:00.353
uh he actually tells you about the

17:00.353 --> 17:02.576
distribution of these estimates and you

17:02.576 --> 17:04.687
could do a full on hypothesis test to

17:04.687 --> 17:06.465
get a P value for . Do I have a

17:06.465 --> 17:08.520
manifold or not ? Freaking awesome ?

17:08.520 --> 17:11.400
But here it's visible , you can see the

17:11.410 --> 17:13.577
corner and you know , I've got a bunch

17:13.577 --> 17:15.632
of manifolds glued together . And of

17:15.632 --> 17:17.688
course , if you sit yourself down to

17:17.688 --> 17:19.966
the green point , uh you see slope too ,

17:19.966 --> 17:19.839
you're on the disk until you hit that

17:20.439 --> 17:22.606
ball of radiant of , of , of dimension

17:22.606 --> 17:24.606
three . And there's a little corner

17:24.606 --> 17:26.495
marked with the green arrow . And

17:26.495 --> 17:28.717
actually , if you're got the ball start

17:28.717 --> 17:30.939
in , in the blue point , eventually you

17:30.939 --> 17:33.106
fall out of the ball and start picking

17:33.106 --> 17:35.272
up mostly this . So the slope goes the

17:35.272 --> 17:36.772
other way at that point is

17:36.772 --> 17:38.995
stratification are visible next slide .

17:40.949 --> 17:43.369
So here we go . Well , I did this on GP

17:43.380 --> 17:45.269
T two . It's an open source large

17:45.269 --> 17:47.324
language model where we have all the

17:47.324 --> 17:49.491
data . And if you sit yourself down on

17:49.491 --> 17:51.602
the , on the token . That's the pound

17:51.602 --> 17:53.713
sign for as kids these days . Call it

17:53.713 --> 17:57.550
the hashtag let's look 20 oh like ,

17:57.680 --> 17:59.459
ok , that part of the space is

17:59.469 --> 18:01.580
basically a manifold . But if you sit

18:01.580 --> 18:04.180
yourself down on the fence sign , it

18:04.189 --> 18:05.911
starts out with a really steep

18:05.911 --> 18:08.630
dimension 500 then abruptly changes

18:09.000 --> 18:11.167
down to like , I don't know , a little

18:11.167 --> 18:13.278
less than that . Maybe in the , maybe

18:13.278 --> 18:15.556
in tens ish and there's another corner ,

18:15.556 --> 18:17.556
it's back up to the 40 right away .

18:17.556 --> 18:21.089
Visually . Full stop GP D two is

18:21.099 --> 18:23.719
using a state space that is not a

18:23.729 --> 18:27.369
manifold from this slide . OK . Now you

18:27.380 --> 18:29.102
could say , all right , that's

18:29.102 --> 18:31.158
interesting . Next slide , what else

18:31.158 --> 18:33.436
can I tell what else goes on with this ?

18:33.436 --> 18:35.713
Can I try some other things ? And yeah ,

18:35.713 --> 18:37.658
so I went around and checked a few

18:37.658 --> 18:39.824
other things . I checked them . So for

18:39.824 --> 18:39.140
instance , if I look at all the

18:39.150 --> 18:41.449
different currency symbols , they're in

18:41.459 --> 18:43.626
black , they all have tokens , they've

18:43.626 --> 18:45.792
got stratification . Notice the dollar

18:45.792 --> 18:47.848
sign also a currency symbol . Last I

18:47.848 --> 18:51.369
checked has a different pattern . Um

18:51.380 --> 18:55.119
Why ? Oh Right .

18:55.130 --> 18:57.719
Because actually GP T two like was

18:57.729 --> 18:59.785
trained on all sorts of stuff on the

18:59.785 --> 19:01.896
internet . A lot of which is code and

19:01.896 --> 19:04.118
the dollar sign plays an important role

19:04.118 --> 19:06.118
in code in the way that these other

19:06.118 --> 19:09.150
guys don't . This suggests that the

19:09.160 --> 19:10.938
different pieces that are glued

19:10.938 --> 19:12.827
together , the different manifold

19:12.827 --> 19:15.010
strata that are getting assembled have

19:15.020 --> 19:17.030
some kind of syntactic or semantic

19:17.040 --> 19:18.810
meaning to them that's kind of

19:18.819 --> 19:20.986
reflected and has gotten regressed out

19:20.986 --> 19:23.097
in the , in the ways of building this

19:23.097 --> 19:24.986
large language model that are now

19:24.986 --> 19:28.760
visible next slide . So of

19:28.770 --> 19:30.826
course , I said , well , great . Now

19:30.826 --> 19:32.881
let me look at them all . How do you

19:32.881 --> 19:35.214
look at at , at a 768 dimensional space ?

19:35.214 --> 19:37.437
I don't know , but I do know if I could

19:37.437 --> 19:39.548
project it down to two dimensions . I

19:39.548 --> 19:38.859
have a chance of understanding what's

19:38.869 --> 19:40.869
going on . So what we're looking at

19:40.869 --> 19:43.739
here on the left is a projection of the

19:43.750 --> 19:46.670
768 point C . The dimensional point

19:46.680 --> 19:50.630
cloud that's been unwrapped using PSNE ,

19:50.640 --> 19:52.751
which is a , a standard technique for

19:52.751 --> 19:54.696
dimension reduction . It's got its

19:54.696 --> 19:56.918
flaws , but it's also rather nice . And

19:56.918 --> 19:59.510
I've colored each of the 50,000 tokens .

19:59.750 --> 20:01.806
Each of those are the points colored

20:01.806 --> 20:03.917
them based on dimension . You can see

20:03.917 --> 20:05.917
that there are some places that are

20:05.917 --> 20:08.083
kind of uniformly one dimension , sort

20:08.083 --> 20:10.250
of the cyan ish color . But then there

20:10.250 --> 20:10.219
are some islands that are really low

20:10.229 --> 20:12.699
dimensional , in particular , they do

20:12.709 --> 20:15.380
seem to have some semantic meaning . So

20:15.390 --> 20:17.334
for instance , there's a bit of an

20:17.334 --> 20:19.168
archipelago , uh low dimensional

20:19.168 --> 20:21.112
stratum , that's where most of the

20:21.112 --> 20:23.334
numbers are . In fact , actually all of

20:23.334 --> 20:25.168
the numbers are there . But then

20:25.168 --> 20:27.334
there's , there's some other things in

20:27.334 --> 20:27.030
there that are , that are date related

20:27.040 --> 20:29.890
things like months and , and year names

20:29.900 --> 20:32.250
and such GP TS token set is very

20:32.260 --> 20:34.989
strange . Um I've marked with the , the

20:35.000 --> 20:37.800
dollar sign and , and then hashtag and

20:37.810 --> 20:40.032
whatnot that we were looking at are and

20:40.032 --> 20:41.921
then there's this other blog that

20:41.921 --> 20:43.977
consists of tokens that have leading

20:43.977 --> 20:46.088
spaces effectively , their beginnings

20:46.088 --> 20:48.530
of words show up as a separate piece of

20:48.540 --> 20:50.651
low dimension . Well , that's kind of

20:50.651 --> 20:52.873
interesting . The other thing too is to

20:52.873 --> 20:54.651
note the visual similarities of

20:54.651 --> 20:56.596
hyperbolic plane there kinda kinda

20:56.596 --> 20:59.829
looks a bit like that . Next one , uh

20:59.839 --> 21:01.930
we can take a slice through and see

21:01.939 --> 21:03.995
that , that if I , if I look at that

21:03.995 --> 21:06.217
tokens with leading space , you can see

21:06.217 --> 21:08.217
that the distribution of dimensions

21:08.217 --> 21:10.439
which I'm showing on the left are , are

21:10.439 --> 21:12.495
little slices of distribution as you

21:12.495 --> 21:14.828
move through , it becomes bimodal there .

21:14.828 --> 21:14.719
That's an indication that actually

21:14.729 --> 21:16.729
there's two layers that have gotten

21:16.729 --> 21:18.507
glued together . This is really

21:18.507 --> 21:21.390
definitely not a manifold . No way . No ,

21:21.400 --> 21:23.859
how I did not compute the P value from

21:24.349 --> 21:28.060
Pepper's uh uh uh Pepper's estimates in

21:28.069 --> 21:30.630
part because uh as I tell my freshman

21:30.640 --> 21:32.696
stat students , what you wanna do is

21:32.696 --> 21:34.900
you wanna find a hypothesis test and

21:34.910 --> 21:37.021
you want , you really want to be in a

21:37.021 --> 21:39.077
situation where it's just so obvious

21:39.077 --> 21:41.299
you don't have to compute the P value .

21:41.299 --> 21:41.180
That's what we got here . This is

21:41.189 --> 21:44.750
definitely not a medical . OK . Next

21:44.760 --> 21:47.849
slide . So I can go ahead and do the

21:47.859 --> 21:50.026
same thing for Lema , which is another

21:50.026 --> 21:52.081
large language model . And oh by the

21:52.081 --> 21:54.137
way , I computed the ricci curvature

21:54.137 --> 21:56.248
for all of these guys and , and , and

21:56.248 --> 21:58.415
for every single model I've laid hands

21:58.415 --> 22:00.581
on they're strongly negative . So they

22:00.581 --> 22:02.581
have this hyperbolic feel to them ,

22:02.581 --> 22:04.803
there's a lot of extra points , they're

22:04.803 --> 22:07.137
very roughly if you will . But out here ,

22:07.137 --> 22:08.915
yeah , plotting dimension , the

22:08.915 --> 22:11.137
dimensions are really a lot smaller and

22:11.137 --> 22:13.359
actually these are kind of in line with

22:13.359 --> 22:15.581
the dimensions computed by other people

22:15.581 --> 22:17.803
uh for , for natural language text . Um

22:17.803 --> 22:19.748
So that's kind of interesting GP T

22:19.748 --> 22:21.970
seems to be uh much higher for whatever

22:21.970 --> 22:24.248
reason . But you can see right in here ,

22:24.248 --> 22:26.248
a stratification boundary , a place

22:26.248 --> 22:28.359
where the dimension kind of goes very

22:28.359 --> 22:30.581
low dimensional and then abruptly along

22:30.581 --> 22:32.637
this edge becomes high dimensional .

22:32.637 --> 22:34.692
What does that mean ? Uh beat ? They

22:34.692 --> 22:37.270
have absolutely no idea . Next slide .

22:43.920 --> 22:46.329
Yes , there is one . Maybe that's the

22:46.339 --> 22:48.506
end of the line , it's not advancing .

22:48.506 --> 22:50.506
Oh OK . Maybe that's the end of the

22:50.506 --> 22:52.728
line actually . Hold on . Hold on . Let

22:52.728 --> 22:52.310
me , let me , let me make another

22:52.319 --> 22:54.486
observation here . I think this is the

22:54.486 --> 22:56.708
last spot . Um If you , if you notice a

22:56.708 --> 22:58.708
little bit to uh to the , to the uh

22:58.708 --> 23:00.763
left of where the , the point of the

23:00.763 --> 23:02.875
arrow is where it says stratification

23:02.875 --> 23:02.839
boundary , you'll notice that there's

23:02.849 --> 23:05.329
like a little dark spot , that little

23:05.339 --> 23:07.219
dark spot is very , very low

23:07.229 --> 23:09.285
dimensional . In fact , it mostly is

23:09.285 --> 23:11.229
isolated points but also very high

23:11.229 --> 23:13.062
curvature . If you look into the

23:13.062 --> 23:15.285
literature that suggests that there's a

23:15.285 --> 23:17.396
lot of overfitting going on there . I

23:17.396 --> 23:19.562
was curious , I looked at them . Uh It

23:19.562 --> 23:19.420
turns out they're all things that I

23:19.430 --> 23:22.160
can't print for you . The reason why I

23:22.170 --> 23:24.226
can't print them is because they are

23:24.226 --> 23:26.630
non printing characters . Why does

23:26.640 --> 23:29.290
lemma process non printed characters ?

23:30.010 --> 23:32.288
I don't know , but that's pretty weird .

23:32.288 --> 23:34.343
Uh So at this point , uh basically ,

23:34.343 --> 23:36.399
what I wanted to do is use this as a

23:36.399 --> 23:38.121
jumping off point to start the

23:38.121 --> 23:40.729
conversation . I don't know why we have

23:40.739 --> 23:43.119
stratified manifolds here . Exactly . I

23:43.130 --> 23:46.500
also don't know if there's , this is an

23:46.510 --> 23:48.677
artifact of these being large language

23:48.677 --> 23:50.621
models or if they're actually like

23:50.621 --> 23:52.732
properly features of human language ,

23:52.732 --> 23:55.719
they could be , I don't also know

23:55.729 --> 23:58.420
if uh I if these are the sorts of

23:58.430 --> 24:02.020
things that happen in other spaces uh

24:02.030 --> 24:04.319
that , that are associated to human

24:04.329 --> 24:06.819
understanding of things , it seems like

24:06.829 --> 24:09.500
I can kind of intuit based on say the ,

24:09.510 --> 24:12.560
the , the GP T two case uh where the

24:12.569 --> 24:14.770
islands had some different meaning to

24:14.780 --> 24:18.449
them . That that might also be useful

24:18.560 --> 24:20.699
when thinking about how say neural

24:20.729 --> 24:23.630
activation patterns work . My guess is

24:23.640 --> 24:25.751
and here , here , here I'm here , I'm

24:25.751 --> 24:27.640
gonna be a little bit provocative

24:27.640 --> 24:29.751
hopefully is to say that we have been

24:29.751 --> 24:31.918
thinking about manifold learning for a

24:31.918 --> 24:33.584
long time out of mathematical

24:33.584 --> 24:35.307
convenience and there are good

24:35.307 --> 24:37.473
theoretical reasons why certain things

24:37.473 --> 24:39.640
might be manifolds . But the moment we

24:39.640 --> 24:41.473
step away from the world of hard

24:41.473 --> 24:44.829
physics , those reasons are don't

24:44.839 --> 24:47.310
really hold water anymore . And so we

24:47.319 --> 24:49.430
need to be aware of the fact that our

24:49.430 --> 24:52.880
spaces state spaces are not manifold

24:52.890 --> 24:55.112
should not be manifold and we shouldn't

24:55.112 --> 24:57.223
expect them . And so we should really

24:57.223 --> 24:59.446
be testing others with that . I want to

24:59.446 --> 25:02.540
open up the Florida discussion . Um I

25:02.550 --> 25:04.772
was wondering if you could quickly just

25:04.772 --> 25:06.828
go over the method again , the , the

25:06.828 --> 25:09.569
slide with the sphere . Um And that

25:09.579 --> 25:12.349
would be super awesome . Um This , this

25:12.359 --> 25:14.526
one was , was great uh back one more ,

25:14.526 --> 25:18.410
maybe uh maybe 11 more , but

25:18.760 --> 25:20.816
this one , the form , this one right

25:20.816 --> 25:22.927
here . OK . So I've got an equation ,

25:22.927 --> 25:25.060
volume is equal to A K , some random

25:25.069 --> 25:27.750
coefficient involving P and junk times

25:27.760 --> 25:29.927
radius to the end . That's your normal

25:29.927 --> 25:33.849
volume formula that and then there's

25:33.859 --> 25:36.329
this parenthesis plus one plus order of

25:36.339 --> 25:38.750
R squared . So actually the , there ,

25:38.760 --> 25:40.871
there's what I'm doing is I'm doing a

25:40.871 --> 25:42.982
tailor expansion of the volume of the

25:42.982 --> 25:45.038
functional radius . And it turns out

25:45.038 --> 25:47.204
that uh there's a nice paper from 1974

25:47.204 --> 25:50.349
where some gentleman Alfred Gray uh sat

25:50.359 --> 25:52.303
down and figured out where all the

25:52.303 --> 25:54.359
terms of the tailor expansion were .

25:54.359 --> 25:56.581
The next term in that expansion happens

25:56.581 --> 25:58.470
to involve something called ricci

25:58.470 --> 26:00.637
scalar curvature effectively . When it

26:00.637 --> 26:02.526
tells you , it tells you how much

26:02.526 --> 26:04.692
volume excess or depth that you get as

26:04.692 --> 26:07.026
you expand the ball of a certain radius .

26:07.026 --> 26:09.026
Uh So now what I did is I looked at

26:09.026 --> 26:11.248
this formula and I said , all right , I

26:11.248 --> 26:13.303
can estimate volume by drawing a , a

26:13.303 --> 26:15.689
ball of radius R measure distance and

26:15.699 --> 26:17.859
find and count up how many points are

26:17.869 --> 26:19.925
within that radius . How many tokens

26:19.925 --> 26:22.250
are within that distance ? If I

26:22.260 --> 26:24.316
estimate that volume , by way , it's

26:24.316 --> 26:26.930
effectively a Monte Carlo . So as such

26:26.939 --> 26:29.010
from a stat standpoint , it has a

26:29.020 --> 26:31.020
certain uncertainty down that comes

26:31.020 --> 26:33.187
with it . And again , the supper , the

26:33.187 --> 26:35.409
paper does a fantastic job of , of , of

26:35.409 --> 26:38.170
elucidating this . OK , great . I wanna

26:38.180 --> 26:40.124
estimate N all I'm gonna do is I'm

26:40.124 --> 26:42.449
gonna take the log of both sides . I

26:42.459 --> 26:44.626
think the log on both sides that the K

26:44.626 --> 26:46.670
times our turn into the log K plus

26:46.680 --> 26:49.520
stuff . Uh And that plus stuff , the

26:49.530 --> 26:52.030
end comes down . Uh And now all I'm

26:52.040 --> 26:54.207
going to do is I'm gonna say I'm gonna

26:54.207 --> 26:55.984
plot logarithm of radius versus

26:55.984 --> 26:58.040
logarithm of my Monte Carlo estimate

26:58.040 --> 27:01.689
how many points . Uh And then just say ,

27:01.699 --> 27:04.770
do a linear aggression on that data .

27:05.020 --> 27:07.131
And I'm gonna look at the slope , the

27:07.131 --> 27:11.130
slope is in this up an offset , you

27:11.140 --> 27:13.362
know , it's open intercept . That's the

27:13.362 --> 27:15.473
one K , who cares what K is ? I don't

27:15.473 --> 27:17.529
know what it is all pie somehow . Uh

27:17.529 --> 27:19.584
And then there's a residual and that

27:19.584 --> 27:21.640
residual is the O of R squared . And

27:21.640 --> 27:23.751
from that , most of the residuals due

27:23.751 --> 27:25.973
to the curvature . So you can just back

27:25.973 --> 27:25.630
the curvature out that way . That's the

27:25.640 --> 27:29.640
method really , really , really , very ,

27:29.650 --> 27:33.589
very nice and has the benefit that

27:33.750 --> 27:35.917
it's easy to kind of visually query to

27:35.917 --> 27:37.972
say . Tell me why this spot here has

27:37.972 --> 27:39.917
that flip . I can look at the data

27:39.917 --> 27:42.028
additionally , you know , paralyze is

27:42.028 --> 27:44.028
easy and I can read it , you know ,

27:44.028 --> 27:46.083
without too much effort in Python or

27:46.083 --> 27:48.139
what what . And so then you just put

27:48.139 --> 27:50.589
tokens in and then you count how , how

27:50.599 --> 27:53.050
many of each token as a functional

27:53.060 --> 27:55.227
radius ? Just count how many tokens in

27:55.227 --> 27:57.227
the functional radius log the , the

27:57.227 --> 27:59.839
count log , the radius linear aggress

27:59.849 --> 28:00.310
done .

28:06.380 --> 28:08.324
And then , but what is the radiant

28:08.324 --> 28:12.030
operationally ? Then what is the radius

28:12.040 --> 28:14.040
when you're running the experiments

28:14.040 --> 28:16.262
then ? Right ? OK . I I run , you run a

28:16.262 --> 28:18.859
range of ra you sweep radius , you say

28:18.939 --> 28:20.995
what , start out with a small radius

28:20.995 --> 28:22.939
and , and sweep it out . There's a

28:22.939 --> 28:22.760
bunch of different ways you can do this .

28:22.770 --> 28:24.992
You could say , let me just look at the

28:24.992 --> 28:27.119
radii of the nearest hils , then the

28:27.130 --> 28:29.989
counts go 12345678 and so on . And the

28:30.000 --> 28:33.020
radii are what they are . So it's

28:33.030 --> 28:35.141
really very simple . You just say all

28:35.141 --> 28:37.363
right sort the tokens based on distance

28:37.363 --> 28:39.363
to the token you're interested in .

28:42.420 --> 28:45.040
It's like very , very naive . Wow .

28:46.400 --> 28:48.456
All the other methods for estimating

28:48.456 --> 28:50.178
dimension are much , much more

28:50.178 --> 28:52.233
elaborate . And as I said , they all

28:52.233 --> 28:53.456
crashed my computer ,

28:58.229 --> 29:00.340
I can't see the chat . Let me pull up

29:00.340 --> 29:02.562
the chat . I couldn't either . Well , I

29:02.562 --> 29:04.618
took a look but so super basic basic

29:04.618 --> 29:06.729
question , right ? What , what , what

29:06.729 --> 29:08.840
would , what would there be advantage

29:08.840 --> 29:08.800
if they were manifold ? The advantages

29:08.810 --> 29:10.532
are that then , then there are

29:10.532 --> 29:12.254
approvable guarantees that are

29:12.254 --> 29:14.254
available . They're not necessarily

29:14.254 --> 29:16.588
strong . But I could start to say , hey ,

29:16.588 --> 29:18.810
if I know this is a manifold and I know

29:18.810 --> 29:18.630
it's got certain curvature . There's

29:18.640 --> 29:20.584
some really wonderful theorem from

29:20.584 --> 29:22.640
differential geometry that say great

29:22.640 --> 29:24.862
since I know of what the curvature is .

29:24.862 --> 29:26.862
And I know I've got a manifold of a

29:26.862 --> 29:29.084
certain dimension , then I can tell you

29:29.084 --> 29:31.140
something about its topology or , or

29:31.140 --> 29:33.251
other way around . I could say I know

29:33.251 --> 29:32.880
something about its topology . I'll

29:32.890 --> 29:35.168
tell you something about its curvature .

29:35.168 --> 29:37.739
These then impinge on how the , the

29:37.750 --> 29:40.290
ultimately the dynamical system ie what

29:40.300 --> 29:42.300
the larger language model does with

29:42.300 --> 29:45.109
your query uh behave because we know

29:45.119 --> 29:47.230
the transformer models are continuous

29:47.500 --> 29:49.389
and they are continuous dynamical

29:49.389 --> 29:51.333
systems on manifolds are very well

29:51.333 --> 29:53.556
understood things . There's a , there's

29:53.556 --> 29:55.667
been decades of literature that study

29:55.667 --> 29:57.667
this , the benefit is if there were

29:57.667 --> 29:59.444
manifolds , we can actually say

29:59.444 --> 30:01.444
something about it . And so part of

30:01.444 --> 30:03.167
what I'm what I'm saying , the

30:03.167 --> 30:05.389
explainability problem with these large

30:05.389 --> 30:04.989
language models and especially

30:05.000 --> 30:07.056
preventing them from running off the

30:07.056 --> 30:08.944
rails and saying things that they

30:08.944 --> 30:11.056
shouldn't uh all of those theoretical

30:11.056 --> 30:13.278
guarantees rely on manifolds . The fact

30:13.278 --> 30:15.444
that we don't have manifolds means the

30:15.444 --> 30:17.778
theoretical guarantees are inaccessible ,

30:17.778 --> 30:21.359
at least presently . Uh

30:21.369 --> 30:23.425
Now let me , let me pick up the next

30:23.425 --> 30:25.647
question here . How non manifolding can

30:25.647 --> 30:27.813
we be ? Well , I mean , you put on the

30:27.813 --> 30:29.925
mask hat to say , uh I can't move the

30:29.925 --> 30:33.290
serum anymore with the staff hat on .

30:33.569 --> 30:35.829
Uh The nice thing that semans paper

30:35.839 --> 30:38.006
gives you is it gives you a bound on ,

30:38.006 --> 30:40.619
on the uh the probability of ending up

30:40.630 --> 30:42.852
in one of these weird places ie falling

30:42.852 --> 30:45.630
off of one piece and on to another . Um

30:45.849 --> 30:47.905
So then it becomes a tolerance . How

30:47.905 --> 30:50.127
tolerant are you ? All right . And then

30:50.127 --> 30:51.460
I see a hand from Ted .

30:53.900 --> 30:55.900
Yeah , I'm just , I'm just thinking

30:55.900 --> 30:58.890
it's hard for me to imagine them being

30:58.910 --> 31:01.719
um anything but just uh stratified in ,

31:02.339 --> 31:04.395
in that we think so differently from

31:04.395 --> 31:06.450
each other . And I just wonder if we

31:06.450 --> 31:09.400
went to the uh you know , to the

31:09.410 --> 31:13.079
Library of Congress . Uh um uh And

31:13.089 --> 31:16.369
we made , made these LL MS , you know ,

31:16.380 --> 31:19.150
for each , each uh stack in the , in

31:19.160 --> 31:21.280
the , in , we might find ourselves

31:21.290 --> 31:23.400
making manifolds by , by doing the

31:23.410 --> 31:25.530
stratification in our choice of

31:25.540 --> 31:27.596
languages . That's , that's entirely

31:27.596 --> 31:29.540
possible . So let , let me make an

31:29.540 --> 31:31.707
observation here , the pictures that I

31:31.707 --> 31:33.959
showed you for GP T two and Lema ,

31:34.219 --> 31:37.150
they're both mostly trained on fluent

31:37.160 --> 31:39.049
English text . There's some other

31:39.049 --> 31:41.216
Multilingual stuff in it , but they're

31:41.216 --> 31:43.160
mostly fluent in English text . If

31:43.160 --> 31:44.993
these models are really learning

31:44.993 --> 31:47.216
language in a sense , you might imagine

31:47.216 --> 31:49.382
that those faces ought to be about the

31:49.382 --> 31:51.382
same and , and they're simply not ,

31:51.382 --> 31:53.604
they're simply vastly different . Uh So

31:53.604 --> 31:55.549
there's certainly some instability

31:55.549 --> 31:58.609
that's present . Um Now flip side

31:59.449 --> 32:01.930
stratification could mean . And I don't

32:01.939 --> 32:04.050
know if this is true . This is now an

32:04.050 --> 32:06.106
open question , in fact . So this is

32:06.106 --> 32:08.050
funded by DARPA , this is now what

32:08.050 --> 32:10.161
we're doing subsequent to this work .

32:10.161 --> 32:12.328
So thinking about these questions , uh

32:12.328 --> 32:14.495
certainly it seems like there's some ,

32:14.495 --> 32:16.495
some semantics going on at the most

32:16.495 --> 32:20.359
basic level , clustering of topics

32:20.369 --> 32:22.425
is the thing that you would expect ,

32:22.425 --> 32:24.258
right ? And you would expect the

32:24.258 --> 32:26.591
Library of Congress for a great example .

32:26.591 --> 32:28.647
Uh It , it's hierarchical and if you

32:28.647 --> 32:30.813
hierarchically cluster things and kind

32:30.813 --> 32:32.869
of fit them together , you do end up

32:32.869 --> 32:34.702
with something that feels like a

32:34.702 --> 32:36.591
stratification . Is that now that

32:36.591 --> 32:38.758
that's a long , that that's a long way

32:38.758 --> 32:40.680
from being a , a proven statement

32:41.089 --> 32:43.256
either statistically or mathematically

32:43.256 --> 32:45.200
or otherwise , but at least it's a

32:45.200 --> 32:48.349
hypothesis we can try . Yeah .

32:49.199 --> 32:51.430
And in fact , I , I think that you know

32:51.439 --> 32:53.272
what it's gonna take to get to a

32:53.272 --> 32:55.272
manifold is probably more than that

32:55.349 --> 32:58.400
because because we do , I mean , we ,

32:58.410 --> 33:01.979
we don't write or think or , or act

33:02.239 --> 33:05.199
uh you know , in continuous ways .

33:05.349 --> 33:07.880
Absolutely . Yeah , I , I mean ,

33:07.890 --> 33:09.890
actually just , just even trying to

33:09.890 --> 33:12.223
write down how a sentence works , right ?

33:12.223 --> 33:14.334
And go back to diagramming a sentence

33:14.334 --> 33:16.390
if you think each of these guys here

33:16.390 --> 33:18.557
has got turned into vectors . The fact

33:18.557 --> 33:20.612
that , that that sentence could be a

33:20.612 --> 33:22.501
variable length that has a lot of

33:22.501 --> 33:22.469
structures , dependent clauses and

33:22.479 --> 33:25.099
modifiers and such that already feels

33:25.109 --> 33:27.729
very stratified system . Yeah . So , so

33:27.739 --> 33:31.599
111 thought or question is what

33:31.900 --> 33:35.560
uh can we make something that

33:36.369 --> 33:39.030
like a manifold has some sense of

33:39.040 --> 33:42.270
continuity that represents the way that

33:42.890 --> 33:45.223
people make continuity out of the world .

33:46.060 --> 33:48.260
Uh It's entirely possible . Yeah , I I

33:48.589 --> 33:52.410
be me all right .

33:55.030 --> 33:57.709
Um Following up on my question in the

33:57.719 --> 34:00.599
chat is so um you know , if it's not a

34:00.609 --> 34:02.498
manifold , we can't prove certain

34:02.498 --> 34:05.219
theorems . But could we um fit a

34:05.229 --> 34:08.620
manifold to the non manifold ? So that

34:08.629 --> 34:10.518
we can say OK , fine , you know ,

34:10.518 --> 34:12.969
locally this manifold is within epsilon

34:12.979 --> 34:15.199
of whatever the real thing is . Could

34:15.209 --> 34:17.600
we then apply those theorems to that

34:17.610 --> 34:20.860
manifold ? And then in turn say OK ,

34:20.870 --> 34:23.540
fine . So the real , the manifold is ,

34:23.810 --> 34:25.866
you know , within a this is true for

34:25.866 --> 34:28.949
the manifold . Therefore , this is true

34:28.959 --> 34:31.409
plus or minus some function of epsilon

34:31.419 --> 34:33.197
for the true surface . Is there

34:33.197 --> 34:35.252
anything OK . So actually I can give

34:35.252 --> 34:37.570
you a definite answer to that question

34:37.580 --> 34:39.580
because you can't even do the first

34:39.580 --> 34:41.469
approximation . The reason why is

34:41.469 --> 34:43.636
because these are so that you can have

34:43.668 --> 34:46.009
stratified spaces ratified manifolds

34:46.019 --> 34:48.130
for instance , uh that are , that are

34:48.130 --> 34:49.908
effectively are called immersed

34:49.908 --> 34:52.019
manifolds , immersed sub manifolds in

34:52.019 --> 34:54.241
which each of the pieces basically have

34:54.241 --> 34:56.463
the same dimension . Uh Then you can do

34:56.463 --> 34:58.519
what you're asking , it's not unique

34:58.519 --> 35:00.741
how to approximate with an animal , but

35:00.741 --> 35:02.850
you can do it . These are Whitney

35:02.860 --> 35:04.749
stratified manifolds in which the

35:04.749 --> 35:06.916
dimensions are very different . And so

35:06.916 --> 35:09.138
the problem is that , that imagine that

35:09.138 --> 35:11.360
sort of the uh the , the amoeba picture

35:11.360 --> 35:13.249
is the good one to keep in mind ,

35:13.249 --> 35:15.416
you've got a blob this two dimensional

35:15.416 --> 35:17.416
and it's glued onto in a really non

35:17.416 --> 35:19.527
changeable way , some one dimensional

35:19.527 --> 35:21.638
hairs and it may even have like a two

35:21.638 --> 35:23.860
or 15 dimensional blob attached to that

35:23.860 --> 35:25.804
somehow . Um There's no way you're

35:25.804 --> 35:28.419
gonna get a single manifold to fit that

35:28.429 --> 35:30.485
unless you pick a manifold that's of

35:30.485 --> 35:32.429
the maximum dimension of the whole

35:32.429 --> 35:34.429
works . So for instance , if you're

35:34.429 --> 35:36.540
dealing with this Amoeba , you've got

35:36.540 --> 35:38.762
the two dimensional blob . But now what

35:38.762 --> 35:38.429
you'll be forced to do if you're trying

35:38.439 --> 35:40.489
to fit the manifold is one of two

35:40.500 --> 35:43.260
things either spread the thing out so

35:43.270 --> 35:45.679
that it covers all of the , the , the ,

35:45.689 --> 35:48.629
the little cilia and now lots of points

35:48.639 --> 35:51.820
in this two dimensional manifold model

35:51.870 --> 35:54.639
don't correspond to anything or you

35:54.649 --> 35:56.760
have to kind of try to squish it down

35:56.760 --> 35:59.270
to make it fit . And now you have a lot

35:59.280 --> 36:02.260
of non uniqueness as in lots of points

36:02.659 --> 36:04.437
correspond to the actual in the

36:04.437 --> 36:06.492
manifold model correspond to lots of

36:06.500 --> 36:09.389
that the same point in the stratified

36:09.399 --> 36:11.830
space . Both of these are bad . Both of

36:11.840 --> 36:15.239
these uh reacted badly to trying to do

36:15.250 --> 36:17.659
any of the theorems that we want to do .

36:17.969 --> 36:20.080
And in fact , they break a lot of the

36:20.080 --> 36:22.247
theorems right out of the right out of

36:22.247 --> 36:24.469
the gate . That's in the case where you

36:24.469 --> 36:26.191
can do this . And actually the

36:26.191 --> 36:28.580
squishing down case actually completely

36:28.590 --> 36:30.812
fails in higher dimensions . Uh There's

36:30.812 --> 36:32.923
a whole theory which amusingly enough

36:32.923 --> 36:35.030
is called surgery on manifolds that

36:35.040 --> 36:37.939
tries to do it tries to do this , that

36:37.949 --> 36:40.171
completely fails in higher dimensions .

36:40.171 --> 36:42.393
The only one available to you is trying

36:42.393 --> 36:44.393
to do the covering thing . In which

36:44.393 --> 36:44.179
case , you're forced to have lots of

36:44.189 --> 36:46.199
points in your manifold model that

36:46.209 --> 36:49.290
don't correspond to reality , which of

36:49.300 --> 36:51.300
course is what's already being done

36:51.300 --> 36:53.522
with these large language models . Most

36:53.522 --> 36:57.330
of the vectors by vast majority 768

36:57.340 --> 37:01.330
dimensions in in GP T all , but a

37:01.350 --> 37:04.419
handful of them correspond to nothing .

37:04.790 --> 37:06.846
Most of the data is on a much , much

37:06.846 --> 37:09.012
lower dimensional subspace . Uh In the

37:09.012 --> 37:11.068
case of lemma , it's even worse than

37:11.068 --> 37:13.290
4000 dimensional ambient space and only

37:13.290 --> 37:15.540
20 or so dimensions actually are useful

37:15.979 --> 37:18.035
that that that's sort of problematic

37:19.689 --> 37:23.040
Patrick . OK . Great . Uh This is great .

37:23.050 --> 37:25.879
So um I'm thinking about this , uh some

37:25.889 --> 37:27.722
colleagues and I have given this

37:27.722 --> 37:29.722
thought , this topic thought from a

37:29.722 --> 37:31.889
vision perspective , a computer vision

37:31.889 --> 37:34.280
perspective . Um And maybe we can , I

37:34.290 --> 37:36.457
think you had mentioned in the email ,

37:36.457 --> 37:38.679
something about a radar example which I

37:38.679 --> 37:40.830
might be keen to , to hear about . Um

37:40.840 --> 37:43.350
So the what kind of keyed us onto it

37:43.360 --> 37:45.689
was this idea of adversarial examples .

37:45.860 --> 37:47.749
So in computer vision , there's a

37:47.749 --> 37:50.110
thought that , you know , if you train

37:50.120 --> 37:52.830
on enough data , you know , then slight

37:52.840 --> 37:55.659
variations in illumination or in

37:55.669 --> 37:59.510
position or angle will be uh will ,

37:59.520 --> 38:01.631
will be handled right . In fact , the

38:01.631 --> 38:03.798
the vision models are , are especially

38:03.798 --> 38:05.909
the convolutional neural networks are

38:05.909 --> 38:08.610
kind of architected in a way to try to

38:08.620 --> 38:11.489
make that , you know , they're

38:11.500 --> 38:13.699
basically architected in a way with

38:13.709 --> 38:16.659
that in mind , right ? Um However , it

38:16.669 --> 38:18.891
turns out right , that by just slightly

38:18.891 --> 38:20.947
perturbing the examples , uh the the

38:20.947 --> 38:23.739
inputs in a , in a imperceptible human

38:23.750 --> 38:27.750
wise way , uh you fall off of whatever

38:27.760 --> 38:29.979
area you are on , right ? And , and it

38:29.989 --> 38:32.211
can completely misc categorize things .

38:32.219 --> 38:34.429
So this , this makes it look very un

38:34.439 --> 38:37.209
manifold in , in , in our view and like

38:37.560 --> 38:39.399
um and , and the fact that the

38:39.409 --> 38:42.810
stitching together , um it seemed seems

38:42.820 --> 38:45.500
as though things are , are so close to

38:45.510 --> 38:47.732
each other , at least that it's so easy

38:47.732 --> 38:49.843
to jump from one to the other and the

38:49.843 --> 38:51.954
networks can be extremely confident .

38:51.954 --> 38:55.219
Um That , that uh that uh say A

38:55.229 --> 38:58.129
three is actually a seven in the case

38:58.139 --> 39:01.300
of the mists , right ? Um And um or ,

39:01.310 --> 39:04.780
or , or worse now , I think so , iii I

39:04.790 --> 39:08.010
think adversarial examples are a good ,

39:08.050 --> 39:10.320
I I don't think machine learning people

39:10.330 --> 39:12.879
think of them in this way as as though

39:12.889 --> 39:14.939
they are evidence of the lack of

39:15.139 --> 39:17.489
manifold learning in what I think they

39:17.500 --> 39:19.722
are . They're hopeful that there really

39:19.722 --> 39:21.889
is manifold learning um going on . But

39:21.889 --> 39:24.000
I think the adversarial examples show

39:24.000 --> 39:26.790
it's not the case . Um And so it would

39:26.800 --> 39:30.399
be interesting to try your approach

39:30.409 --> 39:33.830
here with AAA vision model , maybe one

39:33.840 --> 39:36.050
not run as a classifier but run as a

39:36.239 --> 39:39.719
autoregressive mode , right tokens

39:39.729 --> 39:42.120
necessarily to benefit from here . So ,

39:42.429 --> 39:44.540
you know , they need to be adapted in

39:44.540 --> 39:46.651
some way . But I'd be curious of your

39:46.651 --> 39:48.929
thoughts on , on that . I , I would be ,

39:48.929 --> 39:48.879
I I would say actually , it would

39:48.889 --> 39:50.778
probably be give you a very clear

39:50.778 --> 39:52.945
answer . Uh So the the sonar case that

39:52.945 --> 39:55.111
I was mentioning actually is a case of

39:55.111 --> 39:56.667
an immersed subman . So the

39:56.667 --> 39:58.810
singularities are not so bad . Um And

39:58.820 --> 40:01.042
the , the reason why that's the case is

40:01.042 --> 40:03.399
because there's physics , whereas in

40:03.409 --> 40:06.199
here language is not physics and that's

40:06.209 --> 40:08.431
why that's why things are complicated .

40:08.639 --> 40:10.806
Um But , but kind of to your point , I

40:10.806 --> 40:13.083
think you're , you're absolutely right .

40:13.083 --> 40:15.083
So the picture that , that I said ,

40:15.083 --> 40:17.250
take this key visual away this knitted

40:17.250 --> 40:19.306
hyperbolic plane . Uh The moment you

40:19.306 --> 40:21.750
try to knit a positively curved two

40:21.760 --> 40:23.427
dimensional manifold in three

40:23.427 --> 40:25.699
dimensions , you end up with lots of

40:25.709 --> 40:27.709
these different parts of it getting

40:27.709 --> 40:29.876
really close to each other . They just

40:29.876 --> 40:31.709
don't fit any other way in three

40:31.709 --> 40:33.653
dimensions . So the possibility if

40:33.653 --> 40:33.580
there's a little bit of error of

40:33.590 --> 40:35.757
falling off of that surface and ending

40:35.757 --> 40:38.399
up who the heck knows where far away in

40:38.409 --> 40:41.080
the manifolds way of measuring and sort

40:41.090 --> 40:43.201
of the intrinsic follow your , follow

40:43.201 --> 40:46.439
the path on the surface that's

40:46.449 --> 40:50.090
becomes unavoidable . So it really does

40:50.100 --> 40:52.322
feel like in fact , actually , that was

40:52.322 --> 40:54.544
part of the motivation for checking the

40:54.544 --> 40:56.656
dimension . I didn't even expect that

40:56.656 --> 40:56.429
this would be a stratified space

40:56.439 --> 40:58.495
because all the literature says it's

40:58.495 --> 41:00.439
manifold . Uh I was just wondering

41:00.439 --> 41:02.606
about , about what does it look like ?

41:02.606 --> 41:04.661
What's the dimension ? Um And then I

41:04.661 --> 41:06.550
was , I was interested in , hey ,

41:06.550 --> 41:06.120
what's the curvature again ? Because

41:06.129 --> 41:08.709
that has I impact on , on the dynamics .

41:09.050 --> 41:11.161
So the fact , the fact that I kind of

41:11.161 --> 41:13.439
stumbled on this caught me by surprise ,

41:13.439 --> 41:15.550
but it makes sense in retrospect . So

41:15.550 --> 41:17.717
let me , let me say to you that , that

41:17.717 --> 41:19.661
the other possibility which hasn't

41:19.661 --> 41:21.828
really played a role in the discussion

41:21.828 --> 41:23.994
today , the fact that the curvature is

41:23.994 --> 41:25.661
negative could also be really

41:25.661 --> 41:27.840
explanatory for adversarial examples .

41:29.629 --> 41:31.796
OK . Let me respond to something I see

41:31.796 --> 41:33.296
in the chat . Do you think

41:33.296 --> 41:33.239
incorporating graph models into

41:33.250 --> 41:35.083
research could provide a broader

41:35.083 --> 41:37.280
perspective ? Actually , graphs are a

41:37.290 --> 41:39.760
fantastic example of a stratified space .

41:39.929 --> 41:42.151
So what I'm talking about actually is a

41:42.151 --> 41:44.373
generalization of graphs . Uh So we can

41:44.373 --> 41:47.489
actually do graph based stuff . Uh a as

41:47.500 --> 41:49.722
a consequence of this , they , they are

41:49.722 --> 41:51.778
one dimensional stratified manifolds

41:52.399 --> 41:53.399
Nathaniel .

41:56.659 --> 41:58.881
Yes . Uh This is really , really cool .

41:58.881 --> 42:01.479
And I was curious , so part of what you

42:01.489 --> 42:03.489
haven't talked about so much as the

42:03.489 --> 42:05.545
dynamical part . OK . So we have the

42:05.545 --> 42:07.656
stratified at all . In fact , yes , I

42:07.656 --> 42:09.711
was wondering if you had anything to

42:09.711 --> 42:09.270
say about it , right ? So it's like we

42:09.280 --> 42:11.224
have this , we have the stratified

42:11.224 --> 42:13.447
manifold . Do , do you see any evidence

42:13.447 --> 42:15.447
that like low dimensional parts are

42:15.447 --> 42:15.379
being matched to low dimensional parts ?

42:15.580 --> 42:18.399
Do you have any interpretation of what

42:18.409 --> 42:21.639
the changes in dimensionality um

42:22.600 --> 42:25.260
with what happens there ? So , so , um

42:25.350 --> 42:26.961
in fact , I can tell you low

42:26.961 --> 42:29.072
dimensional parts are likely not maps

42:29.072 --> 42:31.406
to low dimensional parts and vice versa .

42:31.500 --> 42:33.667
Uh And the reason why I could say this

42:33.667 --> 42:35.667
is I ran an experiment uh just this

42:35.667 --> 42:37.778
week , actually , it was just sort of

42:37.778 --> 42:39.833
noodling around wondering what would

42:39.833 --> 42:41.889
happen . As I , as I noticed many of

42:41.889 --> 42:44.000
the tokens in , in , in Lema happened

42:44.000 --> 42:46.111
to be whole English words , not , not

42:46.111 --> 42:48.278
surprising , it's trained on English .

42:48.278 --> 42:50.389
So what I did is I just said , let me

42:50.389 --> 42:52.333
go ahead and since I don't want to

42:52.333 --> 42:54.222
write a tokenizer right now , and

42:54.222 --> 42:56.278
furthermore , I don't want to try to

42:56.278 --> 42:58.500
yank lemmas tokenizer out of the source

42:58.500 --> 43:00.611
code , let me just grab whole words ,

43:00.611 --> 43:03.179
match them up with the lemma token and

43:03.189 --> 43:06.350
color code free English text

43:07.239 --> 43:10.899
based on its dimension . Um And so you

43:10.939 --> 43:13.106
like take free English text . What I I

43:13.106 --> 43:15.272
grabbed , I think what , what did II I

43:15.272 --> 43:17.439
don't remember . I just grabbed some ,

43:17.439 --> 43:19.328
some English document I had lying

43:19.328 --> 43:18.919
around . I think it was Jane Austen or

43:18.929 --> 43:21.151
something . Uh Because I was doing this

43:21.151 --> 43:25.070
in r sad to say um and merged

43:25.080 --> 43:27.469
up , labeled the text with color based

43:27.479 --> 43:31.080
on dimension . If you are

43:31.090 --> 43:33.146
tend to be low dimensional , stay in

43:33.146 --> 43:35.368
low dimensional . What you would expect

43:35.368 --> 43:37.534
to see is you would expect to see long

43:37.534 --> 43:40.540
runs of all the same color . In fact ,

43:40.550 --> 43:43.229
you tend to see alternating runs low

43:43.239 --> 43:45.128
dimensional per bit and then high

43:45.128 --> 43:47.183
dimensional , low dimensional , high

43:47.183 --> 43:49.239
dimensional , low dimensional , high

43:49.239 --> 43:51.295
dimensional . Uh and more surprising

43:51.295 --> 43:54.010
perhaps is that the syntactic words ,

43:54.459 --> 43:56.459
things like prepositions , articles

43:56.459 --> 43:59.510
that those kind of things uh all color

43:59.520 --> 44:01.631
coded the same , which indicates that

44:01.631 --> 44:03.853
their dimensions are very similar . The

44:03.853 --> 44:07.560
takeaway is if the strata have some

44:07.570 --> 44:09.580
kind of syntactic meaning ,

44:12.050 --> 44:14.550
then this would be what you would

44:14.560 --> 44:17.080
expect to see as in I'm I'm using the

44:17.090 --> 44:19.790
same kinds of words pass them through

44:19.800 --> 44:22.320
the same strata in the same order on my

44:22.330 --> 44:25.479
way to somewhere else . So no , it does

44:25.489 --> 44:27.267
not look like one of these nice

44:27.267 --> 44:28.989
dynamical systems in which the

44:28.989 --> 44:31.156
dimensions of the different pieces fit

44:31.156 --> 44:33.322
together uh in a way that is preserved

44:33.322 --> 44:35.433
by the dynamical system . But this is

44:35.433 --> 44:38.500
like super can could we how about a

44:38.510 --> 44:40.689
hypothesis that if the signal were

44:40.699 --> 44:43.370
physics generated , then the

44:43.379 --> 44:46.659
dimensionality would tend to be . Yeah ,

44:47.300 --> 44:49.800
maybe if you sign physics generated

44:49.810 --> 44:52.179
like your sonar example . Yeah . Yeah .

44:52.189 --> 44:54.467
Yeah . It's entirely possible actually ,

44:54.467 --> 44:56.719
you , you could , you could well uh sit

44:56.729 --> 45:00.659
down and , and try to use uh some of

45:00.669 --> 45:02.669
the estimates in the , in Pepperman

45:02.669 --> 45:04.891
paper , but not necessarily , it's kind

45:04.891 --> 45:07.002
of harder because those , those don't

45:07.002 --> 45:09.113
involve what , what happens under the

45:09.113 --> 45:11.280
action of a , of a continuous map . Um

45:11.280 --> 45:13.558
So we need to have some notion of what ,

45:13.558 --> 45:15.725
what a N hypothesis looks like there .

45:15.725 --> 45:17.836
Um Certainly we can do some empirical

45:17.836 --> 45:20.002
stuff but uh there , there's , there's

45:20.002 --> 45:23.449
some theoretical deep water there . Uh

45:23.459 --> 45:26.879
I'm seeing some other cool stuff here

45:26.889 --> 45:29.229
in the chat with , with low dimensional

45:29.239 --> 45:31.469
tokens related relating to

45:31.479 --> 45:33.479
low-dimensional tokens be kind of a

45:33.479 --> 45:35.701
black hole . So actually , let me , let

45:35.701 --> 45:37.757
me , let me kind of wind this back a

45:37.757 --> 45:39.979
bit . Uh The feeling of a black hole in

45:39.979 --> 45:42.035
in dynamics is of course a , a , the

45:42.035 --> 45:44.146
key thing uh And the way that this is

45:44.146 --> 45:45.979
classically done is you look for

45:45.979 --> 45:48.201
equilibrium points places in your state

45:48.201 --> 45:50.368
space where the dynamical system takes

45:50.368 --> 45:52.423
you back to those points . And the ,

45:52.423 --> 45:54.535
the lesson of smooth dynamics is that

45:54.535 --> 45:57.320
they , they can be characterized by uh

45:57.330 --> 45:59.497
the small number of dimensions based a

45:59.497 --> 46:01.441
small number of degrees of freedom

46:01.441 --> 46:04.020
based on the uh ba based on the Hessian

46:04.030 --> 46:06.260
matrix . Uh And that will tell you

46:06.270 --> 46:08.800
whether you're attracting , repelling

46:08.810 --> 46:11.780
or otherwise . And there's a nice

46:11.790 --> 46:14.120
canonical decomposition of subspaces

46:14.129 --> 46:16.719
this way what this means is some

46:16.729 --> 46:19.610
equilibria could be attracted , some

46:19.620 --> 46:21.842
equilibria can be repelling and there's

46:21.842 --> 46:24.620
certainly trae how many dimensions you

46:24.629 --> 46:26.740
can sort of get sucked into uh really

46:26.740 --> 46:28.796
is something that , that , that's uh

46:28.796 --> 46:31.659
estimable . Now , let me , let me give

46:31.669 --> 46:34.840
you some , some intuitive visceral feel

46:34.850 --> 46:36.906
as to why large language model state

46:36.906 --> 46:40.060
spaces might have attractors that are ,

46:40.070 --> 46:42.639
have a large dimensional attractions .

46:43.489 --> 46:46.169
Part of the reason why GP T three and

46:46.179 --> 46:48.750
whatnot is no longer open source is

46:48.760 --> 46:50.593
because of the potential for bad

46:50.593 --> 46:52.816
behavior . What does that mean ? Well ,

46:52.816 --> 46:54.927
one of the observations has been that

46:54.927 --> 46:56.982
uh these large language models being

46:56.982 --> 46:59.204
trained on the internet , which has all

46:59.204 --> 47:01.204
of the , the wonderful things about

47:01.204 --> 47:03.371
human existence as well . Some of them

47:03.371 --> 47:03.030
that are not so wonderful . Uh The

47:03.040 --> 47:05.770
models like GP T three have a tendency

47:05.780 --> 47:07.780
of getting sucked into a trap where

47:07.780 --> 47:09.724
they become jerks and never end up

47:09.724 --> 47:12.780
being nice again . That sure feels like

47:12.790 --> 47:14.979
a black hole , an attractor that has a

47:14.989 --> 47:17.939
very large number of attracting

47:17.949 --> 47:21.739
dimensions . I don't know , is that

47:21.750 --> 47:23.917
really the case that this is something

47:23.917 --> 47:26.083
that could be tested ? And part of the

47:26.083 --> 47:28.194
reason why I went down the , the path

47:28.194 --> 47:30.028
of trying to estimate dimensions

47:30.028 --> 47:32.250
because I wanted to know the size of an

47:32.250 --> 47:34.472
attractor like that . Uh And I actually

47:34.472 --> 47:36.194
wanted to run some of the cool

47:36.194 --> 47:38.250
computational dynamical system tools

47:38.250 --> 47:40.583
that are out there for estimating these .

47:40.583 --> 47:42.694
But I know they scale and dimension .

47:42.694 --> 47:44.750
So I wanted to have an idea what the

47:44.750 --> 47:46.583
dimension was left is what we're

47:46.583 --> 47:50.459
talking about . I see some

47:50.469 --> 47:52.691
other , other things in the chat here .

47:52.691 --> 47:54.802
It says , could you use this to model

47:54.802 --> 47:56.802
things ? A camera can't be based on

47:56.802 --> 47:58.858
information and input ? Oh , maybe ,

47:58.858 --> 48:02.780
maybe you can . Um , OK . Yeah ,

48:02.790 --> 48:05.012
that's quite possible because you model

48:05.012 --> 48:07.068
things that , that a camera couldn't

48:07.068 --> 48:08.957
see on , based on information and

48:08.957 --> 48:11.290
inputs . So short answer is potentially .

48:11.290 --> 48:13.889
Um And one of the lessons that , that I

48:13.899 --> 48:16.010
haven't really seen people write this

48:16.010 --> 48:18.177
down , but it kind of tends to be true

48:18.177 --> 48:20.121
is that if you have a good physics

48:20.121 --> 48:22.177
based model , that's probably better

48:22.177 --> 48:24.288
than one that you've learned from the

48:24.288 --> 48:26.455
data . But if you have a physics based

48:26.455 --> 48:26.209
model , it probably has a lot of blind

48:26.219 --> 48:28.330
spots and those can be well supported

48:28.330 --> 48:31.310
by the data . So kind of some of the

48:31.320 --> 48:33.479
other topological machine learning

48:33.489 --> 48:35.489
tools that I've been playing around

48:35.489 --> 48:38.290
with are built around uh solving data

48:38.300 --> 48:41.489
fusion problems that in a way that is

48:41.500 --> 48:44.110
uh physics model based that way I know ,

48:44.120 --> 48:46.287
hey , I'm pretty confident the laws of

48:46.287 --> 48:48.300
physics are right ? And I'm pretty

48:48.310 --> 48:50.199
confident that I can measure some

48:50.199 --> 48:52.366
things some of the time . And then I ,

48:52.366 --> 48:54.532
I have these grand blind spots where I

48:54.532 --> 48:56.860
can't see anything uh in the , in the

48:56.870 --> 48:59.060
classical that , well , this is like

48:59.070 --> 49:00.959
the problem of tracking and stuff

49:00.959 --> 49:03.014
Talman filters were built for that .

49:03.260 --> 49:05.482
The question is , can we build , can we

49:05.482 --> 49:08.239
build common filters that are more

49:08.250 --> 49:10.361
general that will interpolate over in

49:10.361 --> 49:12.583
the right way ? Maybe . Uh this is kind

49:12.583 --> 49:14.472
of what , what , what these large

49:14.472 --> 49:16.694
language models being autoregressive uh

49:16.694 --> 49:18.861
feel like to make . But I don't know .

49:25.439 --> 49:28.520
That's so cool . Thank you . I , I ,

49:28.530 --> 49:30.969
I've , I've been very jazzed about this .

49:31.169 --> 49:33.391
Uh , since I've fallen into what this ,

49:33.391 --> 49:35.370
this is unexpected . Of course , I

49:35.379 --> 49:37.610
don't know what anything is going on in

49:37.620 --> 49:39.842
terms of the strata . I don't know what

49:39.842 --> 49:41.953
they mean . I don't know why they are

49:41.953 --> 49:44.176
the way they are . But of course , that

49:44.176 --> 49:46.287
that's the uh the the joy of the hunt

49:46.287 --> 49:46.129
right there .

49:53.379 --> 49:56.320
Other questions and discussion just to

49:56.330 --> 49:58.441
go back to the dynamics and thank you

49:58.441 --> 50:00.441
for your answer . So in one level ,

50:00.441 --> 50:02.679
right ? Like we can , we think about

50:02.689 --> 50:04.949
the strata almost , you know , because

50:04.959 --> 50:06.681
the , because the transformers

50:06.681 --> 50:08.681
continuous , right ? And what we're

50:08.681 --> 50:10.848
doing is we're picking out words after

50:10.848 --> 50:13.070
words , after words , then what's going

50:13.070 --> 50:12.989
on with the strata , right ? On some

50:13.000 --> 50:15.000
level is that it's uh points in the

50:15.000 --> 50:17.222
strata are being mapped to other points

50:17.222 --> 50:19.333
close to each other , on the strata ,

50:19.333 --> 50:21.500
at least at some local level , right ?

50:21.500 --> 50:24.510
So in some sense , then are the strata

50:24.520 --> 50:26.810
giving us degrees of freedom of

50:26.820 --> 50:29.520
similarity there ? Yes , that's a good

50:29.530 --> 50:31.697
way to think about it . So let me give

50:31.697 --> 50:33.752
you an example . Uh 01 of the things

50:33.752 --> 50:35.752
that , that my colleagues at Galois

50:35.752 --> 50:37.919
have been doing is they've been trying

50:37.919 --> 50:40.252
to do systematic poking at these models .

50:40.252 --> 50:42.586
Uh And , and , and it really feels like ,

50:42.586 --> 50:42.560
like , uh what happened ? What's the

50:42.570 --> 50:44.570
degree of degrees of freedom I have

50:44.570 --> 50:46.903
when I play Mad Libs with , with an LLM ?

50:47.040 --> 50:49.489
So you say here , here , here's a query

50:49.500 --> 50:51.667
and I've got a bunch of slots that I'm

50:51.667 --> 50:53.667
gonna drop in . Uh And then you see

50:53.667 --> 50:55.722
what happens on the output is I vary

50:55.722 --> 50:57.778
those slots . And one of the sets of

50:57.778 --> 50:59.833
queries they've been playing with is

50:59.833 --> 51:01.944
they've been saying , you know , it's

51:01.944 --> 51:03.889
sort of well understood that these

51:03.889 --> 51:03.840
large language models are really bad at

51:03.850 --> 51:07.439
arithmetic . So let's make a mad lib

51:07.449 --> 51:09.770
query where we ask it to do some math

51:10.080 --> 51:12.270
like basic arithmetic math , we know

51:12.280 --> 51:14.502
what the answer should be . So we could

51:14.502 --> 51:16.613
even check how , how accurate it is .

51:16.613 --> 51:18.669
We can also see how on topic it is .

51:18.669 --> 51:21.399
Here's an interesting thing , Minstrel

51:21.409 --> 51:25.280
and lemma and Pythia tend to stay on

51:25.290 --> 51:27.568
topic when you ask it about arithmetic .

51:27.590 --> 51:29.701
But every now and again , when you're

51:29.701 --> 51:31.868
talking to GP T two , so you give it a

51:31.868 --> 51:34.034
query , same clue you just gave to MSL

51:34.034 --> 51:35.868
and Lemon . They gave you back a

51:35.868 --> 51:38.034
numerical answer that was wrong . They

51:38.034 --> 51:40.146
gave you back a numerical answer . Uh

51:40.146 --> 51:42.312
GP T two runs off the rails and starts

51:42.312 --> 51:44.929
talking about current events . What ,

51:44.939 --> 51:48.209
how did that happen if you look into

51:48.219 --> 51:50.530
that numerical archipelago that I had

51:50.540 --> 51:52.429
shown you where all the numerical

51:52.429 --> 51:55.050
tokens in GP T two live . Uh It's a ,

51:55.060 --> 51:57.959
it's a stratum that's mostly flat ie

51:57.969 --> 52:00.679
the curvature is , is well compared to

52:00.689 --> 52:02.889
the rest of it , it's fairly flat . Uh

52:02.899 --> 52:06.810
And its dimension is 5 to

52:06.820 --> 52:09.110
10 ish , basically the same as human

52:09.120 --> 52:11.231
language in most other contexts , not

52:11.231 --> 52:13.620
GP T for whatever reason . But if you

52:13.629 --> 52:15.685
look into there and you say , let me

52:15.685 --> 52:17.851
look at what else is in that stratum ,

52:17.889 --> 52:19.833
you find out that I mentioned this

52:19.833 --> 52:22.530
through our dates . And so what tends

52:22.540 --> 52:25.310
to happen is GP T has a tendency of

52:25.320 --> 52:28.020
getting confused when you ask it ,

52:28.030 --> 52:30.310
numerical queries , the numbers are

52:30.320 --> 52:33.570
kind of nearby the dates . And so it

52:33.580 --> 52:36.709
gets sucked into an attractor involving

52:36.719 --> 52:38.929
current events . So it was kind of

52:38.939 --> 52:41.659
explanatory in that way . Now , if you

52:41.669 --> 52:43.725
look at where the numeric tokens are

52:43.725 --> 52:45.447
for lemma and for Mistral , in

52:45.447 --> 52:47.558
particular , uh the other recognition

52:47.560 --> 52:49.850
is that there are far fewer of them .

52:49.860 --> 52:52.889
So there's a few dozen numerical tokens

52:53.429 --> 52:55.600
in lemma and MSL , there is about 100

52:55.610 --> 52:59.610
of them in , in GP T um factor two

52:59.620 --> 53:02.649
or three more . Um GP TS numerical

53:02.659 --> 53:04.992
tokens are all cut up in weird ways too .

53:04.992 --> 53:07.215
And as I said , many of them are cut up

53:07.215 --> 53:09.381
that way because they're reflective of

53:09.381 --> 53:11.437
dates . And so as a result , you see

53:11.437 --> 53:14.239
the sort of behavioral difference and

53:14.250 --> 53:17.320
the way that these models interact with

53:17.330 --> 53:19.108
something as basic as numbers .

53:25.750 --> 53:28.100
So we've got 55 minutes left . I see

53:28.110 --> 53:31.429
the , the chat is still going strong .

53:31.500 --> 53:35.429
Um , in terms of where we go as a , as

53:35.439 --> 53:37.550
a group . Next Patrick , do you think

53:37.550 --> 53:39.328
the , the content that you were

53:39.328 --> 53:41.328
thinking about with your colleagues

53:41.328 --> 53:43.328
would be good in some of the models

53:43.328 --> 53:42.600
you've been working with , or do you

53:42.610 --> 53:44.721
want to follow up later ? Yeah . It ,

53:44.721 --> 53:46.943
it could be good . Yeah , it could be ,

53:46.943 --> 53:48.999
it could be nice segue . Yeah , that

53:48.999 --> 53:51.221
might be nice to , we can continue this

53:51.221 --> 53:53.332
discussion that way next week . And ,

53:53.332 --> 53:56.979
um , yeah , I don't know . Um , Michael ,

53:56.989 --> 53:59.211
you follow in the , the chat ? Yeah , I

53:59.211 --> 54:01.580
am actually . So , so you're , you're

54:01.590 --> 54:03.812
absolutely right . You could actually ,

54:03.812 --> 54:06.034
and there's been some , some discussion

54:06.034 --> 54:08.034
in the literature of trying to , to

54:08.034 --> 54:10.146
hand recode parts of lom so that they

54:10.146 --> 54:13.500
behave in different ways . Um , so from

54:13.510 --> 54:15.566
the standpoint of , if you know some

54:15.566 --> 54:17.677
physics , well , all right , let's be

54:17.677 --> 54:19.788
honest . A transformer is a fancy way

54:19.788 --> 54:22.889
to do , uh , spla interpolation , not a

54:22.899 --> 54:25.121
great way to do it , but it a fancy way

54:25.121 --> 54:27.343
to spl interpolation . So you can , you

54:27.343 --> 54:29.177
could build a physics model as a

54:29.177 --> 54:31.177
transformer by hand in a way that's

54:31.177 --> 54:33.288
reasonably principled and splice that

54:33.288 --> 54:35.729
in . That would be kind of neat . Uh ,

54:35.739 --> 54:38.270
w whether that's easy or not , I don't

54:38.280 --> 54:40.689
have a good sense . Uh , now , as for

54:40.699 --> 54:43.032
maybe some of the empty psychology , uh ,

54:43.032 --> 54:45.088
uh , uh , was dysfunctional traits .

54:45.088 --> 54:48.250
Absolutely . That's entirely possible .

54:48.370 --> 54:52.209
Uh And I've noticed in Lema and

54:52.219 --> 54:55.239
Mistral things that look like artifacts

54:55.250 --> 54:57.306
that are due to how the training was

54:57.306 --> 54:59.379
done , but not so much in GP t , I

54:59.389 --> 55:01.649
haven't seen them there . Uh , but I ,

55:01.659 --> 55:03.659
but I've seen some things that look

55:03.659 --> 55:06.010
very much like , uh , forced

55:06.030 --> 55:09.010
normalization in weird places as in it

55:09.020 --> 55:10.909
and all this is from plotting the

55:10.909 --> 55:13.131
volume versus radius curve . That's all

55:13.131 --> 55:15.353
I'm doing , looking at those curves and

55:15.353 --> 55:17.409
saying they should be straight lines

55:17.409 --> 55:19.409
and huh , there's a weird kink in a

55:19.409 --> 55:19.379
weird spot and it's like all of them ,

55:19.389 --> 55:21.510
what's going on there uh That , that

55:21.520 --> 55:23.631
suggest that there's been either some

55:23.631 --> 55:25.949
unlearning fine tuning or some other

55:25.959 --> 55:28.350
kind of uh heavy handed modification .

55:31.110 --> 55:33.221
And , and yeah , I definitely would ,

55:33.221 --> 55:35.166
would second the idea of , of , of

55:35.166 --> 55:37.110
hearing about some computer vision

55:37.110 --> 55:39.277
ideas here . But I think that would be

55:39.277 --> 55:38.770
really handy

55:43.139 --> 55:45.195
ted it looked like he might have had

55:45.195 --> 55:48.899
something to add . Um Well , I , I

55:48.909 --> 55:51.860
liked your response . Um It , it's , I

55:51.870 --> 55:54.530
mean , ii , I imagine that we are

55:54.540 --> 55:57.780
creating new mathematics um around this

55:57.790 --> 56:00.409
stuff , but I also was just playing

56:00.419 --> 56:02.530
around in my mind of you take this uh

56:02.530 --> 56:04.550
these analyses and you take a

56:04.560 --> 56:07.820
particular political uh uh

56:07.830 --> 56:11.419
theory or per person's position and

56:11.429 --> 56:13.707
you'll find that they're gonna be very ,

56:13.707 --> 56:15.818
you're gonna , the manifolds of these

56:15.818 --> 56:17.762
different people are gonna be very

56:17.762 --> 56:19.873
different or the stratification . And

56:19.873 --> 56:22.120
that would be just a fun , fun I mean ,

56:22.129 --> 56:24.296
that's , that's definitely uh an op ed

56:24.296 --> 56:26.462
piece in New York Times to take Kamala

56:26.462 --> 56:28.820
and , uh , and Donald and , and , and

56:28.830 --> 56:30.830
show the way that they , uh their ,

56:30.830 --> 56:33.159
their , their statements uh vary . Uh

56:34.919 --> 56:36.863
That , that , that , that's , uh ,

56:36.863 --> 56:38.975
that's actually an interesting idea .

56:38.975 --> 56:41.086
We , we've been discussing uh amongst

56:41.086 --> 56:43.030
my collaborators and I , comparing

56:43.030 --> 56:44.975
models using that . I , I've got a

56:44.975 --> 56:47.086
bunch of different models . They're ,

56:47.086 --> 56:49.141
they're different . Um , they really

56:49.141 --> 56:51.308
seem to have different personalities .

56:51.308 --> 56:53.363
Can we quantify what that means ? Uh

56:53.363 --> 56:55.586
And , and potentially , I mean , once ,

56:55.586 --> 56:57.808
once these things are into the world of

56:57.808 --> 56:59.919
math , we can do the math on them and

56:59.919 --> 57:02.030
do some stat on them . I think that's

57:02.030 --> 57:04.252
where your graphs are , right ? They're

57:04.252 --> 57:06.141
showing that personality . Yeah ,

57:06.141 --> 57:08.308
absolutely . Think there's some really

57:08.308 --> 57:10.252
good theory here today , Michael .

57:10.252 --> 57:12.500
Thank you . You're welcome . We talk

57:12.510 --> 57:15.350
for just a second . What will we do

57:15.360 --> 57:17.416
next ? So we could look at this in a

57:17.416 --> 57:21.239
real applied way . It's great .

57:21.250 --> 57:23.350
Everybody loves Siri , but then you

57:23.360 --> 57:26.260
have to switch gears and show me who's

57:26.270 --> 57:28.350
using it . Does it really work for

57:28.360 --> 57:31.679
something Xy or Z ? Like John , John

57:31.689 --> 57:34.870
Luga said , maybe hybrid multi model

57:34.879 --> 57:37.489
transformers , right ? John , I'm

57:37.500 --> 57:40.020
looking at you . Yep . Yep . The other ,

57:40.030 --> 57:42.141
the other thought I , I had just from

57:42.141 --> 57:44.308
what you just said , Michael was , can

57:44.308 --> 57:46.308
this be used in the spirit of water

57:46.308 --> 57:48.641
marking output ? Oh , impossibly . Yeah .

57:48.919 --> 57:52.639
Yeah . Yeah . OK . So we're , well , go

57:52.649 --> 57:54.816
ahead if you want to respond Michael .

57:54.816 --> 57:56.927
But I just want to remind everyone we

57:56.927 --> 57:59.093
continue these conversations via email

57:59.093 --> 57:58.850
throughout the week . And so please

57:58.860 --> 58:01.082
don't let this be the end . Of course ,

58:01.082 --> 58:03.249
we're gonna continue this here at this

58:03.249 --> 58:05.138
meeting next week but with , with

58:05.138 --> 58:07.082
Patrick presenting . But um please

58:07.082 --> 58:09.304
follow these up with email threads . Um

58:09.304 --> 58:09.270
Sorry , I didn't mean to cut you off

58:09.280 --> 58:11.280
Michael . No , that , that , that's

58:11.280 --> 58:14.399
fine . Yeah , I , I second the new math

58:14.409 --> 58:17.169
or , or even new applications of math

58:17.179 --> 58:19.290
that have been formally non applied .

58:19.929 --> 58:22.262
Uh , I mean , the fact the fact is , II ,

58:22.360 --> 58:24.582
I never thought I would be picking up a

58:24.582 --> 58:26.949
book on surgery theory , not like

58:27.260 --> 58:29.260
biology , surgery theory , but look

58:29.260 --> 58:31.389
math , surgery theory and trying to

58:31.399 --> 58:33.566
find applications for it and realizing

58:33.566 --> 58:35.510
this is probably a tool we need to

58:35.510 --> 58:35.310
understand .