In Canada, all three major AI research centers use clusters created with public money. These clusters receive regular additional hardware as new generations of GPUs become available. Considering how these institutions work, I'm pretty confident they've considered the alternatives (renting, AWS, etc). So that's one data point.
The intent behind media matters but isn't all that matters. How people might interpret something is important (albeit often unpredictable).
I think the symbolism of "let's crush all remaining vestiges of creative culture" is a pretty obvious _potential_ interpretation from a _non-trivial amount_ of people. In that sense it is an interpretation that matters for our present discourse, even if it isn't the interpretation that the creator of the ad intended.
> How people might interpret something is important (albeit often unpredictable).
It's a big world out there. There are literally billions of possible ways that people can interpret whatever you put out in the ether, and many of them are...precious...to the extreme. Worry too much what any one of them is going to think, and you won't do anything.
The obvious conclusion, to quote every influencer on the internet, is: "Haters gonna hate", but admittedly, I don't work in Apple PR.
One of HNs failure-modes is inaccuracies get voted to the top if they seem correct to the majority of voters whose biases resonate with poster's.
Aside: thank you for asking. When I previously encountered incorrect top-level comments that I knew to be wrong (insider information), I'd simply ignore and move on. You've inspired me to push back more often.
But not always! There are those among us who like nothing better than to double down in a flame war. One nice thing about having visited often over the past 7 years is that I know whom to avoid responding to (for the most part).
If you've worked at a large organization, you'll know the news can paint a cartoonishly distorted picture largely informed by the perspective of the anonymous sources, journalist and news organization.
The WSJ article expressly considers that factor and goes into detail on what's under the surface.
> The end of the long-running negotiations, which hasn’t previously been reported, is the latest example of how Google and other tech giants are trying to strengthen their control over the study and advancement of artificial intelligence.
Fighting between Deepmind and Google leadership over autonomy doesn't really directly support that Google Brain employees and Deepmind had infighting. They seem to me to be quite different things.
It seems like a big leap to take these articles as support the statement:
> Demis now has a load of people reporting to him who previously were rooting for his failure
It certainly might be true, but I'm missing the connection between these articles and the statement.
How are "Google Brain employees" distinct from "Google leadership with Google Brain personnel in their respective reporting line?" What is the criteria for that distinction?
Good managers insulate reports from the politics, if you weren’t plugged into it it’s either your manager did a good job or it’s the only part of Google that isn’t 90% politics.
Signed, “didn’t work at brain or dm but was involved in a lot of alphabet level decision making”.
I never like the word “politics.” It carries the association of a bunch of people just playing backstabbing games to further themselves.
While this does occur, in general what I see is that with any large-enough group of people, there will be strong differences of opinions on how to steer the project to success.
In fact, I don’t think I can remember a single “political battle” that didn’t stem from a legitimate concern in how some project was being run and what they had decided to focus on.
Not really no, there's been a bump which studies seem to agree is an aftereffect of COVID, but the media is also paying more attention to it. There's always been some amount of shootings and stabbings in Montreal, and 99% of Montreal is safe during night time.
euh thanks for proving my point lol have you read the report ?
"À Montréal, en 2021, ceux-ci ont augmenté de 17,3 % par rapport à la moyenne
de 2016 à 2020. Les homicides et les tentatives de meurtre ont augmenté de façon importante et la problématique
de la violence armée contribue très certainement à ce portrait. En effet, la moitié des homicides et des tentatives
de meurtre commis sur le territoire du SPVM en 2021 impliquait la présence ou l’utilisation d’une arme à feu
(voir tableau Armes à feu). La problématique de la violence armée demeure au centre des préoccupations du SPVM. "
Wow, what a take. I seriously doubt that people in Montreal/Quebec are more or less xenophobic than the average North American; if you have data supporting this I'd be happy to change my mind.
Also please be more skeptical of thoughts of the form "X is Y because of hot take Z". Reality is complex and things have many causes. Montreal has a long history of doing city planning differently than other cities in Canada.
Having been closely tied to Real Estate and landlords, this is the current understanding of most realtors that deal in English Montreal and its surrounding areas. Sure it may not be the only reason, but it is a definite factor.
As for examples of xenophobia, I don't know of any other places where they have a separate language police, I know personally of non-white owned stores being harassed and fined for having the wrong accent on their french sign. There are countless politicians who've blamed immigrants for their woes, Jacques Parizeau being a famous one for blaming the Ethnic votes for their loss. CAQ just took a majority, look up Bill 21.
While I don't think rent and xenophobia are related, it really is palpable here specially when you're immigrant and specially now during election (Quebec election are happening next Monday)
I know, the CAQ is full of xenophobes, and it can suck not looking white in Quebec (although much less so in Montreal). I'm skeptical that it drives rent prices in any detectable way.
Having lived in Montreal it's probably in the top end of the most inclusive place I have been.
The only thing i feel like parent could be referring to is the language laws, but the reasoning here is that they are necessary when surrounded by english (and has nothing to do with race). Plus language laws are barely respected already.
This is definitely the case in DL (and I'm assuming elsewhere too but I wouldn't know).
I've lost count honestly, running 1-2 year old paper github repos with some detail missing (like the Python version!) that make it non-trivial to run as is. Libraries make undocumented breaking changes, wrong pickle format, authors used a nightly version which didn't make it to a tagged version, and so on.
This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.
> This perhaps says also something about the CS (versus software eng) background that most people engaging in DL publishing have.
Are those things enjoyable? Or is hacking and playing with ideas enjoyable?
Huge portions of PhD students spent time as software engineers prior to starting their programs. It's not about know-how. It's about not being paid to engineer systems in addition to doing research.
Fewer than 1 in 100 labs have dedicated software engineers, and PhD students are paid $30K/yr. There's no way in hell most of them are going to spend their time doing dependency management or setting up CI/CD pipelines for that salary. If they wanted to spend their time doing software engineering, then can (and would) move to an industry SWE job at 10x the total comp.
Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function).
It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361
I don't know much about theorem proving though. Can some value be attributed to partial proofs?
Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.
I'm still wrapping my head around this (it seems rather magical; can it really be just that?)... but I find the idea of "sampling from a distribution of trajectories that reach a terminal state whose probability is proportional to a positive reward function, by minimizing the difference between the flow coming into the trajectories and the flow coming out of them, which by construction must be equal" to be both beautiful and elegant -- like all great ideas.
How did you and your coauthors come up with this? Trial and error? Or was there a moment of serendipitous creative insight?
--
To the moderators: My comment is now at the top of the page, but manux's comment above is more deserving of the top spot. I just upvoted it. Please consider pinning it at the top.
I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state.
But, yes in the moment it felt like some very serendipitous insight!
Thank you. It's nice to hear that you had one of those shower/bathtub Eureka moments!
> ...we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state
Makes sense. FWIW, to me this looks like a Conservation Law -- as in Physics. I mean, it's not that the flows "must be" conserved, but that they are conserved (or go into sinks). Any physicists interested in AI should be all over this; it's right up their alley.
Could you highlight the difference between this and training a permutation invariant or equivariant policy network using standard supervised or RL methods? Assuming I also have a way of having an invariant/equivariant loss function
What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward:
- RL says, give me a reward and I'll give you its max.
- GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)).
Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
Yeah the "net" in GFlowNet refers to how the underlying state is interpreted, not to an architecture. It is a way to train generative models, on any kind of discrete data (like graphs).
Source: am first author of original GFlowNet paper.
Thanks for your comment! I originally finished my PhD in computational neuroscience and have been riding the deep learning wave for the past five years. I listened to the audio Machine Learning Street Talk episode on GFlowNets, and that was my general introduction. It wasn't until I looked at the image of the GFlowNet on your blog post a connection to actually biological plausible architectures became apparent. The individual firing of a neuron can be interpreted as a Poisson process, but with populations of neurons we often would approximate the firing rate of the entire population as a Gaussian distribution. That would be one patch of cortical column, but these populations are linked together to form larger scale functional brain networks. Anyway, thinking about that scale you could imagine the neural activity communicated between these patches of activity as a type of flow from one region to another. I think your paper raises interesting questions about how this inference engine in our brain might be doing some sort of MCMC like sampling and constructing different belief hypothesis.
Yeah! This kind of reasoning is why it's exciting to think that we could use the GFlowNet machinery to construct latent representations--ones that map more directly to the notion of "beliefs" that we're used to think about as humans, something "discrete" and relational.
Concretely what this could mean is using these tools to generate causal hypotheses, like what's been done here: https://arxiv.org/abs/2202.13903
It's quite likely some people will come out of reading this thinking we should stop sending people to grad school. Shouldn't we instead take this as a signal that it needs improvement?
Grad students are one of the very few subpopulations of humanity allowed to take on extraordinary epistemological risks; a kind of immune system of our civilization. I'm not even talking about some elusive notion of "progress", just [intellectual] societal health. It would feel to me like a tremendous loss if we let go of such a component of society.
You are assuming that the institutions are making worthwhile "epistemological risks"; when, clearly, there is a lack of the facilitation of the knowledge of Fourier Mathematics (e.g. the Fourier Transform) in virtually every scientific discipline. Yet this is what most of these researchers involve themselves with everyday with their digital computations. Isn't it ironic, don't you think?
