Seven steps to choosing the right AI code assistant

The arrival of AI is changing how developers build software, morphing the SDLC in what feels almost like real time. Everything we do, from designing apps to debugging them, we now will almost certainly do with AI involvement — unless we already are.

This is why finding the right AI code assistant is crucial. AI has become a fundamental technology in the software development stack, and it touches every part of the SDLC. But it’s not a magical panacea. The current state of AI is that AI can help greatly in performing straightforward coding tasks, but it can’t (yet) handle complex logic and system design decisions. Long-term dependencies like choosing appropriate architecture patterns, planning scalable system infrastructures, or ensuring that distributed codebase components work together cohesively all still require human input and intelligence. As does choosing the right tool in the first place. 

As the human in charge of finding it, here are the non-negotiable features and capabilities that should be on your AI code assistant shopping list.

Four basic AI code assistant features

The first four features to look for are “no-brainer” criteria: mandatory capabilities that any reasonably mature AI software development solution should be expected to offer. 

1. Code completion and generation

AI code assistants incorporate AI into our developer toolkit by producing high-quality code based on the inputs we give them. There are the features to look for: 

• The AI should autogenerate context-aware code as you type. This includes generating full code snippets and functions as well as completing lines of code for you as you work in your IDE. Beyond completing lines of code, the tool should also be able to create new code blocks or functions from plain language comments or other prompts that you give it.

2. AI chat assistance

AI chat agent functionality is a separate and distinct feature from AI code completion and generation. Stellar natural language chat is a critical feature because the sad reality is that many of us spend far more work hours on all the tasks around writing code than on writing the code itself. The core capability of chat is to assist with all these other tasks, like debugging and writing tests and documentation, so we can get back to the fun stuff.

You’ll want to look for an AI coding tool with a chat agent that:

• • Allows developers to ​​ask general coding questions and get relevant recommendations inside their IDE. The chat agent lets them use natural language to ask general coding questions and request recommendations for improving, testing, debugging, and documenting code. 
  • Can assist with modernizing code, debugging code, and code refactoring. 
  • Helps with writing tests, improving test coverage, and generating test data. You want an AI assistant that can analyze code and generate tests that cover a range of scenarios, edge cases, and boundary conditions.
  • Helps with writing solid documentation and pull request descriptions for all of the above activities. 

The benefit of an intelligent chat agent inside your IDE is immediate: Stack Overflow’s 2023 survey shows that 63% of developers spend more than 30 minutes a day searching for solutions to coding problems. Being able to find answers without ever leaving the IDE lets us stay focused and keep our flow going.

3. Supports your IDE and other tools important to your work

Development teams work with many different programming languages and frameworks. And getting a developer to change IDEs is akin to getting a kid to leave their favorite playground — it’s hypothetically possible, but they’ll protest loudly and resist every step of the way. So it’s important to realize that the quality of support for the tools that matter to you can vary widely between different AI code assistants.

• Ensure that the AI tool offers a high degree of integration. Any AI tool should simply just work with your existing source code management tools and IDEs. As developers, we should not need to learn any new tools or add new dependencies to use an AI code assistant. 

4. Language support

Some AI code assistants are limited to a handful of the most popular languages and frameworks. Others claim to support a large number of languages, but their LLMs may not be trained with equal depth on each of them — meaning that the AI will return lower quality results for, say, Lisp and Erlang than for JS and Python.

• Look for an AI code assistant that utilizes a foundational model or models (LLMs) specifically trained in high-quality code. “Garbage in, garbage out” still applies here in the Age of AI. Code assistants excel at generating code based on common patterns and best practices for widely adopted programming languages, but we all know there’s a lot of cruft out there. Developers using more obscure or highly specialized languages may struggle with getting an AI trained on generalized data (cough ChatGPT cough) to return even moderately useful suggestions.

Three advanced features to look for in an AI code assistant

Our final three categories dig into the more sophisticated characteristics of AI tools that not only developers but also other teams in your org (security, legal, operations) will likely care about. These include key questions regarding privacy around your code and data usage; security of the platform and protection from legal liability; and how finely you’re able to personalize and tune the tool to your unique codebase and the way you work.

5. Privacy

Some AI tools can take your code, data, and usage patterns to feed their platform’s general models, making your information available to anyone (and everyone) using that platform. “Leakage” of proprietary information and data (including code) into LLMs is a major concern for AI researchers and security firms. Even if the data itself is not sensitive information, it can still contain business logic or knowledge that you as a developer don’t want to share with others.

To make sure this isn’t you, here are the privacy features to look for when you’re evaluating an AI coding assistant:

• The solution commits to zero data retention for your codebase as well as data around your usage. Review the AI assistant’s policies around data retention and ask to see how data is managed from the IDE to the inference server and LLM and beyond. Query the specific process for data transfer and management when third-party models and APIs are involved. 

If you’re privacy conscious, it’s crucial to review not only the license agreements and terms of service for the AI code assistant itself, but also the license agreement and TOS of the LLMs used behind the tool. Even if the AI code assistant doesn’t collect or save your data itself, it could still be passing some or all of it through to the foundational model it sits on top of.

• You can control the assistant’s deployment location (i.e., SaaS, VPC, on-premises). Some AI software development tools are only available on the vendor’s cloud platform. If your organization requires — or even simply prefers — private deployments, the AI code assistant should be able to be deployed on-premises for you to maximize control, deployed as single-tenant SaaS for convenience, or deployed on a VPC as a balance of the two.
  

• Fully air-gapped, “closed circuit” deployments are supported. In a truly private AI assistant, there’s no network path outside your environment. The only way to guarantee absolute privacy for the data used to interact with any AI platform is for there to be no way for that data to leave your perimeter.

6. Personalization

Personalization is how we evaluate a code assistant’s ability to “learn” as it’s being used, plus its capacity for fine-tuning and optimization according to your specific needs. Studies show that personalization features are the number one feature desired by most developers. 

However, personalization capability varies widely between different AI code assistants. Key factors to look for include local and global context and the ability to automatically review code to enforce compliance to internal standards. Look for:

• Local code awareness (local context): For utmost personalization, an AI code assistant needs to access the code available locally from the developer’s machine. This includes data from a developer’s IDE for information like variable types used near the completion point in the code, comments they’ve added, open files they’ve interacted with, any imported packages and libraries, open projects, and more.
• Organizational-level code awareness (global context): Not all information is available locally to provide the AI with the personalization context it needs. This is why an AI code assistant should allow administrators to connect it with organizational-level sources of information (like code repositories, design documents, Jira tickets, and more) to generate recommendations more aligned with a team’s way of working. 

Beyond investigating an AI assistant’s capacities for local and global context, evaluate the tool’s ability to perform automatic code review to enforce coding standards and guidelines. With this feature, the AI automatically parses any generated code, tests, or documentation to ensure code is compliant. 

Automated AI code review can take multiple forms: review is triggered as code generates, so only compliant recommendations are created. It can also be triggered at the pull request to automatically review for compliance. Finally, an advanced AI assistance tool can perform constant scans of code open in the IDE or project to spot outdated libraries or code that’s against the established rules of the company.

7. Protection: Security, compliance, and indemnification

We tend to focus on the really cool stuff that AI can do, and can help us do, because this stuff really is cool. But we can’t overlook the less flashy but even more important features that an AI software development tool needs to offer: security, compliance, and indemnification. 

Adopting AI into your organization affects others beyond just the team using the tool. Security, legal, and operations/IT teams also have requirements around how a code assistance tool protects and serves your org. Arguably, then, the most critical criteria to examine when evaluating an AI code assistance is how well it protects the user — you and your org — in terms of security, compliance, and legal indemnification. Here’s what to look for.

• Security: Built-in enterprise-grade security that’s confirmed through industry certifications. The tool must come with certifications and practices in place to show that it’s undergone rigorous testing and adheres to high standards of protection and security. For starters, SOC 2 compliance is an important baseline indicator that the vendor tool has established and follows strict information security policies and procedures, encompassing the processing, integrity, and confidentiality of customer data. There may also be security certifications specific to your industry: for example, ISO/IEC 27001, GDPR compliance, and PCI DSS if your application processes, stores, or transmits credit card information.
• Compliance: The code assistant is trained on legally obtained data and licensed code. Although advanced AI models don’t typically hold IP rights over their output, the output from the AI model and the use of the underlying code sources and datasets used to train them could potentially violate copyright protections and cause breach of license agreements. Keep in mind that “open source” does not equal “unrestricted.” Verify that an AI code assistant is trained solely on legally obtained data or has licenses for all copyrighted material and licensed code used in their model(s). 

Compliance is never sexy. But if portions of your application’s source code have been generated by an AI assistant that scraped its source material from copyrighted sources, you become vulnerable to potential legal actions. If a generative AI platform customer deploys code that matches the code the AI model trained on, the original creators and license holders may be able to bring suit. Fortunately, you can avoid these pitfalls simply and entirely by choosing an AI code assistant that’s transparently trained only on legally obtained data sources.

• Protection: Indemnification for infringement. Much of the law around AI is still evolving and unsettled, and this cumulative legal uncertainty and mutability means future risks — issues we currently can’t even anticipate — could arise. Since all of this is beyond our ability to control or do anything about, another important criteria for choosing an AI code assistant that protects your interests is indemnification. 

Indemnification is the vendor’s commitment to compensate your enterprise for expenses arising from any lawsuits or claims related to their AI tool. Look for guaranteed coverage of legal defense costs, financial damages and settlements, and protection against disruption. This last one covers scenarios where your software and development processes would be disrupted by legal challenges related to the AI tool — challenges that might result in you being audited, required to remove any affected code, or forced to redevelop elements of your software.

We live in interesting times. As AI code assistants mature from an emerging tool to a critical technology at the center of software development, the pressure is intense to add it to the software developer toolkit. Realistically, most developers have been at least playing around with AI code generation for a while now while trying to figure out exactly what we want a code assistant to do for us, and how that translates into specific features.

Evaluating AI is really not all that different from evaluating any other tech tool. We care about the same outcomes here: how this tool works, how it will fit into our SDLC, and whether it’s easily integrated with all the other tools we care about. Now, understanding the seven key factors that differentiate a high-quality tool from a lesser, possibly even risk-inducing, offering — paying special attention to privacy, personalization, and protection features — choosing the right AI code assistant is as straightforward as vetting any other software tool.