Back in the days before I worked for Tabnine I was tasked with various ML problems. From industrial product defect review, CCAI solutions to fraud detection. Nearly all of these tasks utilized some version of deep learning via neural networks, with various levels of complexity. As any reader versed in the current state of machine learning might be aware, natural language processing is where the most difficult and simultaneously intriguing problems lie. It’s even become a bit of an arms race to see who can train the most cutting edge NL model. From BERT to XLNet and GPT-3 and numerous derivations of each, natural language models are the definition of large, computationally intensive “artificial intelligence”.
So where does a programming language fit in the machine learning space? Programming languages are “languages”, it says so right there in the name. But are they really in the same problem space as actual spoken or written languages? Famously the phrase, “Sam was found by the stream” is linguistically ambiguous without surrounding context in which to place the various actors. Of course we as humans know that a stream can’t really “find” anyone but that’s only because we are in possession of a fantastically well trained neural network of gray matter in between our ears. For a NL algorithm to learn ambiguity, cultural turns of phrase or contextual summary is an incredibly difficult problem.
However, programming languages follow much stricter guidelines. They have to if we hope to have it compile and run. Ambiguity is not an option. In the bias/variance tradeoff we can err on the side of overfitting without serious deleterious consequences. There are, of course, multiple ways to write any given function in programming language that will accomplish a defined task. So if we are to use an AI code completion tool like Tabnine, the model running on that code base can be quite specialized.
And this brings us to the question of parameter metrics. Many of the NL models tout the number of parameters used to build the underlying model. We are now well into the hundreds of billions of parameters for the leading models. But does this necessarily mean that using one of these models as the basis for programming language transfer learning will give us better suggestions in the IDE? The answer is quite clearly no, it does not. Training a specialized model for a particular task utilizing specific gold standard code and libraries will give the developer some really amazing in-line suggestions, full code snippets and contextual awareness.
So beware of AI based code tools that purport to have XXXM more parameters than some other tool. Just because there are more parameters does not mean that it will work better for a specific programming task. As in life, so goes AI assisted pair programming. Balance is key. A specific model trained for the front-end devs in your company will likely show some significant ROI if it was well considered and deployed. But you would not want to hand that model over to the data science group as it is the equivalent of two left shoes.
A properly done based model for your organization’s developer teams is ideal. Tabnine’s team learning model automatically learns from the group to constantly improve and focus the model on what the team is developing. Metrics surrounding the team’s usage and model effectiveness should be tracked and changes made if those metrics begin to slip. Like any properly managed MLOps process, AI pair programming is a powerful tool when done right.
In conclusion, while NL model parameter count can tell us something about the inherent applicability to our everyday communication, it isn’t the right metric to judge the effectiveness as applied to programming language models. Choose wisely and keep vigilant on your organizations model library and the returns will be significant.
Readers of a particular age and background might be familiar with the American radio variety show “A Prairie Home Companion”. Every week the show’s host Garrison Keillor tells a rambling and amusing ongoing story of a fictional town in northern Minnesota named Lake Wobegon. It’s a great listen and truthfully a “podcast” ahead of its time. But for us, the most interesting part of the tale is the way Keillor signs off from the story every week. “…this is the news from Lake Wobegon, where all the women are strong, the men are good looking and all the children are above average.”
It’s humorous to be sure, but the pedantic among us might recognise the tongue in cheek observation that everyone can be “above average”. Certainly any group of people fall in a distribution around an average, but ask any particular individual if they are better than average at any given task and likely you’ll get an enthusiastic yes! Of course I’m a better driver than most folks out there. And of course you are as well, dear reader. The natural human tendency to overestimate one’s capabilities is so well known among social scientists that it has, in fact, been labeled, “The Lake Wobegon Effect” (Go ahead, wiki it…)
So what about coding? If I’m truly honest, I’m not a great coder. Years ago, before I found Tabnine and came to work for them, I worked for Google. While I was interviewing for Google I was trying to write an image classification algorithm under a time constraint. My python linter had enough fluff to fill a few throw pillows at least. I was able to go back and debug, re-write, fix the issues and get hired for the job…but it wasn’t elegant. I’ve met folks in my life that write code like they were born to it. It is a special person that sees code like music, or a novelist sees their language.
Fast forward to my interview with Tabnine and I faced the same issues. Live demo of code I wrote showing capabilities of the tools Tabnine provided. Fear, sweaty palms as I worked through the code in my head. But this is where things were a whole lot different. I had dropped in the Tabnine extension on VSCode and followed the simple directions. After consolidating the ML libraries and example repos into a root directory, Tabnine’s local model provided me with a highly specialized set of suggestions that were tailored exactly to what I wanted to do in my image classification demo. Drop a layer from the CNN to improve regularization? I commented my intentions and Tabnine pulled the correct TF function with suggested boolean values to make it happen. In short, Tabnine made me a better programmer.
Tabnine’s toolset will be different things to different people. Folks like myself will find the suggestions helpful and will lower time spent debugging and asking senior devs for review. It helps me get done what I need to get done faster and with less pain. Junior devs might find it even more helpful. Senior folks will find value in Tabnine guiding their teams with best practices, lowering tech debt and allowing them to focus on research or new innovation.
Over the coming months I’ll be writing a series of posts focusing on the value that Tabnine can bring to the spectrum of developers. I hope you’ll continue to join me.
That’s the news from Tabnine, where all the managers are hands-off, the devs are geniuses, and all the zoom meetings are above average.
Artificial Intelligence is rapidly changing nearly all aspects of life and business, and software development is no exception. We started Tabnine with a single goal in mind, to put the power of AI in the hands of developers and teams. Our top priority since day one has been to help developers become better with intelligent tools that fit seamlessly into their workflow. The rapid adoption of Tabnine and the consistently enthusiastic feedback from developers has already made it clear that we’re onto something huge.
The announcement of GitHub Copilot marks a milestone for AI-Assisted Software Development in general and for Tabnine in particular. As pioneers in the AI-Assisted Software Development space, we’re excited to see giants like Microsoft, OpenAI, and GitHub recognize this incredible opportunity.
At Tabnine, we believe in putting the developer at the center, using AI as smart tooling that augments and assists developers, making them better. We will continue to prioritize making developers successful above anything else. In particular, we’ll invest in key factors that we believe make Tabnine so appealing to developers using all IDEs and programming languages:
I am certain that AI will play an incredibly important role in modern software development. At Tabnine we’re proud to be a part of bringing this future to life and leveraging the amazing technology efficiently and responsibly for the benefit of developers and teams.