At this year’s Microsoft Build conference, Microsoft CEO Satya Nadella’s keynote speech sparked various discussions, but in addition to product demonstrations, the on-site conversation between OpenAI President Greg Brockman and Microsoft CTO Kevin Scott is also worthy of attention.
Greg Brockman is one of the core builders of ChatGPT, and Kevin Scott is directly responsible for the integration of ChatGPT and Microsoft.
Like many Silicon Valley legends, OpenAI’s other “father” Greg Brockman is also a dropout from a prestigious school. He dropped out of Harvard and MIT successively, and joined the payment software company Stripe before founding OpenAI. It is said that most of the talents of the OpenAI founding team were recruited by Greg Brockman, and he has also led a series of projects.
Microsoft CTO Kevin Scott is also a person with great enthusiasm for technology. He debuted at Google and joined LinkedIn. He was ordered to help the company successfully complete the platform growth and expansion on the eve of the IPO, and finally successfully went public. He was known as the “engineer who saved LinkedIn”. Shortly after Microsoft acquired LinkedIn, he was quickly promoted to CTO.
In the eyes of most people, Kevin Scott and Greg Brockman, who are pivotal in the AI circle, more or less represent “technical development” and “application platform”. influence the future of the industry. This conversation may be the last thing anyone who cares about ChatGPT technology and engineering should not miss. The conversations are organized below for readers’ reference.
Kevin Scott：Thank you so much for coming to Build. I want to start with the ChatGPT experience because it really amazes everyone. There is as much interest in the ChatGPT app as there is excitement about it, and building something like this is a huge engineering challenge indeed. Can you share your opinion.
Greg Brockman：In terms of infrastructure and aspects, ChatGPT is a very interesting process. We have been studying how to build a natural chat system for many years, and we also launched a demo called Web GPT before, which is a very interesting demo. We found hundreds of testers, paid them to use the system, and the feedback was: “This (Web GPT) is useful, and you can write programs.”
But for me, the moment that really shines is when I have GPT-4. There was a set of familiar processes before. For example, GPT-3 only deployed the basic model and pre-trained without any direction fine-tuning. With GPT-3.5, we started to make it operate according to the instructions. The tester has a series of steps to train, and when I get to GPT-4, I conduct a small experiment. If the model generates some content and then gives it a second instruction, what happens? The model has a perfect answer, which is the fusion of old answers and new instructions.
So once I realize how powerful the model is, it really sums it up: “Well, when you want me to follow instructions and give me new instructions, maybe you just want to chat with me.”
For me, that was my “knowledge” moment: well, we have the infrastructure, and it’s already working great on early models. Although this model is not designed for chatting, it does. So this is a real “aha!” moment. From then on we thought, this thing has to be pushed out, because it can do great things.
Kevin Scott：Yes, this did surprise me a lot. I remember when Sam Altman called me and said “We’re planning to release ChatGPT, it’s going to take a few weeks,” and I was like, why not? I didn’t realize at the time that this technology would be so technically efficient, or that it would be such an insane success. I know that you are one of the main architects of the infrastructure of GPT-4, which powered the development of ChatGPT. This is an inspiration for everyone working in the field of AI. So I was wondering if you could share something interesting.
Greg Brockman：To a large extent, the GPT-4 project has made us “laborers of love”, which is hard work but worth it. In fact, after GPT-3, we tried several times to surpass the performance of this model, but all ended in failure. This is not an easy task. We eventually decided to go back to square one and rebuild the entire infrastructure. We take many approaches and work on every detail.
I believe that even now, we may still find more bugs. But Yaakov, one of the project leaders, once used a good analogy when he said that it’s almost like building a rocket, you want every engineering tolerance to be as small as possible. For example, we once had a bug with checkpoint reads where if you stopped the job at the wrong time, when the job restarted, you might have a mix of new weights and old weights. This is actually harmless, because machine learning can recover from it. But every time you see some weird fluctuations in your graph, you wonder what could be causing it. So, I go back and revisit every detail, and these seemingly tedious engineering jobs are my main responsibility.
Kevin Scott：Those seemingly “boring engineering jobs” you do have grown to an unbelievably staggering scale. I do think it’s a great inspiration for everyone in the room, that sometimes it’s the seemingly mundane groundwork that really leads to success.
Satya Nadella mentioned in his speech the shared cheats we are developing. The idea is that we’re going to empower everyone in the room to write software that extends the capabilities of ChatGPT, and all these Copilots that we’re building. It’s also an interesting technical challenge, we haven’t solved all the technical issues yet, there’s still a lot of work to do to get it to our final target state. So I’d love to know if you have some thoughts you’d like to share.
Greg Brockman：I love hacks! I think this is a really great opportunity for every developer to take advantage of this technology and make the system better for everyone, right? Part of the reason we designed it as an open standard at the time was that, as a developer, you build it once and any AI can use it. That’s a really good idea, isn’t it?
It’s like the main reason the Internet drives development: You can build a website and everyone can access it. Then you open up an API and anyone can take advantage of it. I think this kind of core design principle is great, it allows anyone to attach and obtain system functions, and it is able to expand the functions of various fields into ChatGPT itself.
Kevin Scott：One thing I really like about the plugin is how simple it is conceptually. This reminds me of the first HTTP server I wrote. Once you understand the core concepts, you’ll be able to quickly build something powerful. I think that’s a great thing, so in your role at OpenAI, you’re always thinking about how to push the limits of technology. One of the really amazing things about our partnership is that we seem to be able to see further because of you. So I’d love to know if you can share some apps or models that excite you.
Greg Brockman：What’s also interesting to me is that we’re pretty much in a loop like the “tick tock model” that Intel used in its early years, where you come up with an innovation and then really push it. (Note: “Tick-Tock” is a strategic model proposed by Intel in 2007 for the development of microprocessor chip design and manufacturing business. This model staggers the update of processor micro-architecture and chip manufacturing process to improve efficiency. Every A Tick represents a micro-architecture chip process update, aimed at reducing the chip area, reducing energy consumption and heat generation; each Tock represents updating the microprocessor architecture on the basis of the previous Tick to improve performance. The cycle of this mode is Two years, one of which is Tick and the other is Tock.)
Just like GPT4, we are still in the early stages of the push, right? We’ve announced the vision capability, but it’s still in production. I believe this will change how and how these systems work and feel, and the various applications built on top of them. So I’m really excited about that. Looking back at the history of the past few years, I think two years ago we cut the price by 70%. In the past year, we have cut prices by another 90%. This looks pretty crazy, doesn’t it? I believe we will be able to repeat things like this with new models. Now, while GPT4 is expensive and not fully available, I think this is one of the things that will change.
Kevin Scott：And that’s one thing I want to leave with all of you here: “What is expensive today won’t be expensive tomorrow, because technological progress is so amazing.” We still have time to talk about the last topic – you have provided developers here A series of very good suggestions, is there anything else you want to share with you?
Greg Brockman：I think in this field, the technical route is getting clearer and the technology is getting better and better. But I believe one thing that every developer can do that is difficult even for a big company like Microsoft is to really go deep into a particular area and figure out how to make this technology work in that area effect. So I really appreciate companies that are working in areas like the legal space, they’ve gained expertise, talking to a lot of lawyers, understanding their pain points with this technology. I believe everyone’s efforts can add tremendous value to this technology.
Kevin Scott：marvelous. As Greg said, you are all the people who make AI great. Greg, thank you so much for spending time with us today!
(This article is reprinted with authorization from Pinwan; source of the first picture: screenshot of the video)