Googles ALPHACODE-2 Just SHOCKED The ENTIRE INDUSTRY! Full Breakdown + Technical Report
Google’s AlphaCode-2 has recently shocked the entire industry with its impressive coding capabilities. Built on a fine-tuned version of Gemini Pro, AlphaCode-2 is a separate system from the current Bod coding capabilities. In a technical report, it was highlighted that AlphaCode-2 outperforms its predecessor by solving 1.7 times more problems and performing better than 85% of the competition participants.
The system generates up to a million code samples per problem using a randomized temperature parameter to encourage diversity. It then filters out bad ideas, groups similar solutions, and selects the best ones for solving the coding problem. While AlphaCode-2 excels in competitive programming, it is still too costly to operate at scale.
Moving forward, the future of AlphaCode-2 looks promising as improvements in the foundation model, Gemini Pro, could lead to further enhancements in the system’s performance. With the potential for increased efficiency and scalability, AlphaCode-2 could revolutionize the coding assistance provided by AI in the future. This groundbreaking system offers a glimpse into the future of AI-assisted programming, showcasing the potential for collaboration between human coders and advanced AI models.
Watch the video by TheAIGRID
Video Transcript
A quick announcement I did recently make a patreon and that’s for those of you who want to just give some extra support because sometimes the onean operation can be a bit daunting so if you do want to join the Discord join the patreon get featured in some videos for your
Comments vote on what videos I should do next don’t forget to check out the link in the description otherwise enjoy the rest of the video so Alpha code 2 by Gemini this was something that I wanted to cover in yesterday’s video but I’m sure it deserved its own video because
It was a bombshell that was dropped that unfortunately went under the radar and kind of slipped by in terms of people missing this crazy announcement so what we have here guys is a very impressive coding system and this is built on top of a fine-tune version of Gemini fro so
Alpha code 2 is essentially built on a fine tuned version of Gemini Pro and what you need to understand about Alpha code 2 is that Alpha code 2 is an entirely separate system and currently isn’t available in bod so as you know many people are using bod and looking at
Their Advanced coding capabilities but please understand that this is not Alpha code 2 as Alpha code 2 has not yet been released and they are working on doing that so even if right now you do think B’s coding capabilities have increased please understand that you’re not seeing
The capabilities of alpha code 2 as it is a separate system so I want to quickly dive into the technical reports first page then I want to show you the actual demo from Google and then I’m going to go into some further details about Alpha code 2 that I think you
Should know the first paragraph of the alpha code 2 technical report actually houses some very fascinating stuff so it actually does mention the fact that their previous version of alpha code if you didn’t know Alpha code was essentially the first AI system to perform at the level of the median
Competitor in competitive programming a difficult reasoning task involving Advanced maths logic and computer science and essentially what they’ve introduced is their second iteration of this model and essentially what they’ve introduced was was Alpha code 2 their second iteration of this model and it’s a new and enhanced system with massively
Improved performance powered by Gemini Alpha go 2 relies on the combination of powerful language models and a bespoke search and reranking mechanism which is really cool and we’re going to get into that later and when evaluated on the same platform as the original Alpha code we found the alpha code 2 solved 1.7
Times more problems and performed better than 85% of competition participants so let’s take a look at that Google demo and then I’m going to go back and do a deep dive into the paper because there’s some really cool things that you should know about Alpha code 2 in terms of
Where Google is going to take the system we build Gemini from the ground up to be natively multimodel including something quite important for both of us programming code Gemini is able to consistently understand explain and generate code that is correct and well written in most programming languages that includes includes python Java C++
And go it substantially improves coding abilities over previous Palm two models from a benchmark around 200 programming functions in Python it consistently solves about 75% of them in the first try versus around 45% on Palm two if you allow Gemini to check and repair its own answers this number jumps to over 90%
Which is a huge step forward it can help you create and prototype new ideas in seconds let’s give it a try I really like trains and if I wanted to create a transporting location web app I can simply ask and get a working prototype in less than a
Minute while the code isn’t perfect it’s really helpful to have a first draft Gemini on its own has the ability to transform software development as we understand it but it can also be deployed as a key component of more sophisticated systems Gemini is great at coding but we’ve been able to take it
Even further creating a specialized version that performs remarkably well at competitive programming now why do we care about competitive programming well it is one of the ultimate litmus tests of algorithmic coding abilities so we have thousands of talented programmers from all over the world that come together to compete and try to solve
Incredibly complex problems that require not only coding but also math and reasoning two years ago we presented Alpha code and it was the first AI system that could compete roughly at the level of the average human competitor today I’m delighted to introduce Alpha good 2 a new and enhanced system with
Massively improved performance powered by Gemini when we evaluate Alpha code 2 on the same platform as the original Alpha code we solve almost twice as many problems while Alpha code broke through the top half of human competitors on average we estimate that Alpha code 2 performs better than 85% of competition
Participants let’s have a look at our model in action on one of the hardest problems that we faced and I say hard because in the original contest in which the problem appeared less than 0 2% of participants actually solved it the problem is is quite difficult it’s very
Abstract so I can’t get into too many details but the basic gist of it is that we are tasked with Computing aggregate statistics that account for what appears to be an impossibly large amount of random arrays the really cool thing is that to solve it Alpha c 2 makes use of
Dynamic programming dynamic programming is an advanced algorithmic technique which basically simplifies a complicated Problem by breaking it down into easier sub problems again and again and what’s really impressive is that not only alphaa 2 knows how to properly implement this strategy but also when and where to
Use it what the example shows us is that competitive programming is not just about implementation it’s also about understanding maths computer science and indeed coding and that makes it an extremely hard reasoning task so it’s not very surprising that up till now generally available large language models have scored very poorly on this
Benchmark these models are really really good at following instructions but Alpha code needs to do more than that it needs to show some level of understanding some level of reasoning designing of code Solutions before it can actually get to the actual implementation to solve the problem and it does all that on problems
That it’s never seen before another thing that is great about Alpha code is that it performs even better when it collaborates with human coders who can provide grounding basically developers can specify properties that the code samples have to obey and when we do that we see performance increase
Significantly we think of this this kind of interaction between uh programmers and AIS as the future of programming where coders will not just give instructions but actually collaborate with highly capable AI models that can reason about their problems that can propose code designs and that can even help with the actual implementation Alpa
2 was built for competitive programming but we’re already working on bringing some of its unique capabilities right into the General Gemini models as a first step towards making this new programming Paradigm available for everyone so that was a very nice comprehensive overview of the alpha code 2 capabilities and I found it really
Fascinating how well it does on coding and just how much it seems to understand the task due to the fact that it is on a fine Tred version of Gemini so one thing I also did find was the interesting overall system so essentially what we
Have here is the alpha code 2 system and it’s quite different to how normal systems are done so essentially Alpha code 2 has a team of mini brains or model and each one is really good at coming up with different ways to solve the coding problems and essentially they
All work together to create tons and tons of possible solutions in effect I’m pretty sure later on in the technical report they state that it is actually millions of possible solution solutions that they do create now essentially when they create many many different of these Solutions in this sampling mechanism
What they do next is they then essentially get rid of the bad ideas and this is of course the filtering mechanism so some of the solutions that they do come up with obviously won’t work so they have a special Checker that throws out any ideas that don’t actually
Solve the puzzle then of course we have a clustering SLG grouping algorithm where they notice which ideas are kind of similar and to make things easier they group the similar ideas together this way they don’t waste time on ideas that are almost the same then of course
What they do is they essentially for each group of ideas they then pick the very best one and they have a special way to score each idea to figure out which one is the most likely to solve each puzzle SL coding problem and overall what we have here is a very
Effective system so the one thing I wanted to expand a bit upon was their sampling system because I found out that this is where things do get interesting so essentially Alpha codes 2 sampling you can see that it says we generate up to a million code samples per problem
Using a randomized temperature parameter for each sample to encourage diversity so essentially they create 1 million different solutions for one single coding problem and of course to make sure that each solution is different from the last they use a special trick so essentially they adjust the robots in
Imagination to come up with a wide range of ideas and of course this is done by using something called a randomized temperature parameter so think of it like a turning D that changes how creative or different each solution is then of course we have adding more details so they add and change small
Details about each solution based on things like how hard the puzzle is or what kind of puzzle is and that makes each solution even more unique then of course they have the fine tuned models or their brains essentially that are really good at coming up with the
Solutions and it uses all of them equally to get a good mix of ideas so although Alpha code used to think in both Python and C++ languages it now only uses C++ because it makes better quality Solutions in this language then of course we have the picking SL
Filtering and because they have so many solutions they have to pick the best ones it looks at all the solutions throws out the best ones and then ranks them to find the the top 10 and in the end it chooses up the best top 10 solutions to try and solve this problem
So it’s a really really interesting Deep dive to see exactly how the model thinks its entire process and how it gets from problem to solution now with that I did want to look at the future of alpha code because although the system was just released it is interesting to see where
This could go now one of the problems that Alpha code does suffer from just like the other advanced systems like GPT 4 it does suffer from cost to run so it says our system requires a lot of trial and error and remains too costly to operate at scale further it relies
Heavily on being able to filter out obviously bad code samples so it seems that right now I don’t think we’re going to be getting Alpha code because it’s too expensive to run and too costly to operate at the scale that they would need to if they wanted to deploy it
Worldwide and of course they also do state that a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders so it seems like we are moving towards a future where coders aren’t going to be replaced but more so we’re
Going to get coding assistance in the form of AI rather than complete Replacements so one thing I did really want to include was of course the evaluation because it’s really important to see how it scores up compared to Alpha code 1 so essentially alpha alpha code 2 was tested on code forces the
Same platform that used the original Alpha code they chose 12 recent contests with over 8,000 participants from two difficulty levels division two the harder division one and two covering 77 problems in total so for each problem Alpha c 2 generated 1 million potential Solutions which were their candidates
And then submitted up to 10 Solutions per problem following a specific selection process this continued until it found the correct solution or ran out of candidates and in terms of the performance results Alpha code 2 solved 43% of these competition problems this is almost double the performance of the
Original Alpha code which solved 25% in terms of competition rankings Alpha code 2 is estimated to be in the 85th percentile performing better than 85% of the participants in comparison with human competitors it ranks between expert and candidate Master categories on code forces and the estimated ranking
Does take into account a simulated time penalty in efficiency problem solving in some contestants Alpha code 2 performs better than 99.5% of the participants and it was found to be 10,000 times more sample efficient compared to the original Alpha code and it requires about 100 samples
To reach what Alpha code achieved with a million samples in addition it does seem like Alpha code 2 could get even better because in addition it does seem like Alpha code could actually get quite better because they do use use Gemini Pro as their Foundation model and using
This Foundation model led to significant increases in performance on two critical components of the system the policy models generating the code samples and the scoring model used to select them so essentially they basically say that we suspect using geminite Ultra as the foundation model instead with its improved coding and reasoning
Capabilities would lead to further improvements in the overall Alpha code 2 approach so now it’s simply a waiting game to see whether or not Alpha code 2 is going to be made more efficiently so that it can scaled up and deployed globally or they’re going to just simply
Improve on it like they did with Alpha go type systems and see if they can potentially get 100% on some of these benchmarks
Video “Googles ALPHACODE-2 Just SHOCKED The ENTIRE INDUSTRY! Full Breakdown + Technical Report” was uploaded on 12/07/2023 to Youtube Channel TheAIGRID