Designing an Autocomplete Editor from Scratch for a New Programming Language

Suchit Sahoo

11 min read

·

Nov 11, 2025

--

Listen

Share

How many times have you thought, “Damn, what’s the correct spelling of that word?” or “What was that word starting with ‘ah’…?” More often than not, even before the thought fully registers, your device is already offering the answer through autocomplete. What began as a simple convenience for everyday writing has evolved far beyond just word suggestions — it now plays a vital role in more technical domains, such as coding. I’ve yet to meet a developer who hasn’t relied on autocomplete, and with the rise of modern editors like Cursor and Kiro, it’s now supercharged with AI, delivering context-aware, intelligent suggestions that feel almost magical. But what if you had to design an autocomplete system for an editor that supports a completely new language? Sure, you could Google it — or, as the kids now say, ask ChatGPT — but in this article, I’ll break down all the key components and walk you through how to design, build, and refine such a system from scratch.

Before we dive into building our autocomplete system, a quick note on where my perspective comes from. I’ve worked extensively on OpenSearch Dashboards, an open-source data analytics suite, where I developed an autocomplete feature for the OpenSearch PPL query editor. Throughout this article, I’ll draw on that experience to share practical insights and real-world examples.

Technologies Used

• ANTLR Engine: Creatively named ANother Tool for Language Recognition is a very popular open-source tool for processing and building grammar.
• Monaco Editor: The open-source editor that powers VsCode and derived projects like Cursor, Windsurf etc.

Prerequisites

Before we can start building our autocomplete system, lets address the elephant in the room “Grammar”. You might be wondering, “Why do I need grammar” or “How do I get grammar” — depending on whether you’re newbie or seasoned professional.

To answer both questions: grammar is a set of rules that defines how a language is structured. It’s what transforms a random blurb of words into a proper sentence in English. In the world of programming languages, grammar refers to elements such as how expressions are constructed, which keywords are reserved, and so on.

So, if we want to build an autocomplete editor for a new language, we need to understand that language’s grammar. Without this, our suggestions will be inconsistent and unhelpful, and users might avoid using the editor altogether.

Now, how do we define grammar? There are different standards and tools available. For this discussion, we’ll focus on ANTLR-based grammar. Usually, an ANTLR grammar consists of two files: a lexer and a parser. The lexer defines the tokens used in the language, while the parser defines the grammar rules that dictate how those tokens combine.

Check out the following files for examples of lexer and parser definitions used in OpenSearch Dashboards:

Requirements

As with all software engineering projects, our journey begins with good old requirements gathering.

Problem Statement: We want to build a Code Editor that supports a autocomplete for OpenSearch PPL. (a pipe-based query language used for data processing.)

Is it enough? Absolutely NOT !!!.

We will need a lot more details, what feature do we want to support, Do we want syntax highlighting, how about errors , Do we want to showcase documentations and billion more questions. So we need a more detailed version of the requirements.

Detailed Requirements:

• As an user whenever I type in a PPL query , I should be able to differentiate between keywords, Values, Column Name etc.
• As an user when I type in PPL query, The editor should be able to highlight errors in the query with red squiggly lines and give me a reason why is the query invalid
• As an user, the editor should suggest me valid PPL Keywords along with their associated documentation

As an user, the editor should suggest valid datasource name. For eg: for PPL command source has syntax. source = <datasourcename>

PPL command source has syntax. source = <datasourcename> . The editor should be able to identify the valid datasources and suggest me when I type source =

• As a user, the editor should suggest the column name present in the current datasource

PPL command where has syntax. source = <datasourcename> | where <columnName>. The editor should be able to identify the valid columnName and suggest me when I type source = <datasourcename> | where

• As a user, the editor should suggest values for a given columnName

source = <datasourcename> | where <columnName> = .
The editor should be able to identify the values available for columnName and suggest me when I type source = <datasourcename> | where columnName =

Design

Press enter or click to view image in full size

This is a simplified design of the system.

Key parts

1. Editor — The user-facing code editor interface.
2. Suggestion Service — Acts as a proxy between the editor and the rest of the system.
3. Parser — Translates the user’s input into grammar tokens for the language.
4. Suggestion Engine — Uses the parsed grammar tokens to determine the current context and suggest the next valid tokens.
5. Data Source Connections — Provide access to dataset schemas and values for context-aware suggestions.

For our build, the Monaco Editor will handle the front-end editing experience, ANTLR c3 will serve as both the parser and suggestion engine, and the connection object in OpenSearch Dashboards will provide seamless access to our data sources.

Implementation

Instead of building the entire system all at once like a mad genius we will follow the boring way of building the system brick by brick and finally unveil the entire working system.

Setting up Engine for AutoComplete

The basics of setting up a parser are straightforward. With the ANTLR C3 engine, the steps are simple:

Step 1:Setting up the Parser

// Setting up the Parser
const parser = new Parser(new CommonTokenStream(new Lexer(CharStream.fromString(query))));
parser.removeErrorListeners(); // Remove any default Listeners so we can add more 
parser.interpreter.predictionMode = PredictionMode.LL; // Handle more ambigious queries

That’s really all it takes to get a parser up and running. The two caveats to watch out for are:

• Error Listeners — by default, ANTLR attaches its own, but we will need our custom listeners for things we want to do (Kudos if you have already guessed Error highlighting)
• Prediction Mode — more often than not, you would want to use the default, but depending on the accuracy/speed, we want to customise it. The available options are SLL (Single Lookahead Parsing Default) and LL(*). LL algorithm takes into account the entire context and produces better results when faced with ambiguous scenarios as compared to the SLL; however, it loses out to others in terms of speed.

References:

Step 2: Setting up the Code Completion Engine

With our parser in order, we can now start working towards initializing our suggestion engine and using it to power our autocomplete.

// Setting up the Code Completion Engine
const core = new CodeCompletionCore(parser);
core.ignoredTokens = ignoredTokens;
core.preferredRules = rulesToVisit;
const cursorTokenIndex = findCursorTokenIndex(tokenStream, cursor, tokenDictionary.SPACE);

// Gettting Suggest
const suggestKeywords: KeywordSuggestion[] = [];
const { tokens, rules } = core.collectCandidates(cursorTokenIndex, context);

The above snippet looks innocous, but trust me, it is a whole can of worms to unpack.

Let's address the obvious questions:

• Why do we need ignored tokens, and why not consider all the tokens that we have?
• Why do we have preferred rules? Also, if we have preferred rules, why not pass all the rules that we have? Surely that would give us better results, wouldn’t it?
• There seems to be a custom cursor token index. Why is it needed?

With the questions out of the way, let's start the hard part of actually trying to understand each of them.

Q1: Why do we need ignored tokens, and why not consider all the tokens that we have?

Depending on your grammar, the no of tokens that we can vary usually ranges from a few hundred to tens of thousands. Now would it be great if we could process all of them? Yes, but consider the time it would take to do so. Also, at the end user can only act upon a smaller subset of tokens and not all the tokens available in the grammar. So limiting the search space to a set of tokens helps address the above.

Q2. Why do we have preferred rules? Also, if we have preferred rules, why not pass all the rules that we have? Surely that would give us better results, wouldn’t it?

Grammar tokens without associated rules are meaningless. ANTLR C3 allows us to define a preferred set of rules that we want to evaluate during completion.

The natural question is: why restrict ourselves to a subset instead of passing every rule to the engine when processing a query?

To answer that, we need to understand how ANTLR C3 evaluates rules and constructs the Abstract Syntax Tree (AST). ANTLR builds the AST in a top-down, recursive manner. During this process, the C3 engine stops and returns the rule path as soon as it encounters the first matching preferred rule.

If we include too many high-level rules in the preferredRules set, these rules will always match early in the traversal, preventing deeper, context-specific rules from being evaluated. As a result, we lose the ability to generate precise, context-aware suggestions.

Therefore, it is always helpful to use lower-level rules that are actually relevant for suggestions.

For example:

As per the PPL grammar, every statement falls under the broad umbrella of logicalExpression, so if I use logicalExpression as a preferred rule along with tableIdent. For the query
source =
Instead of hitting tableIdent, it will hit the rule logicalExpression, and it will be difficult, actually, to suggest the tableName in the above scenario.

const preferredRules = new Set([
  OpenSearchPPLParser.RULE_statsFunctionName,
  OpenSearchPPLParser.RULE_percentileAggFunction,
  OpenSearchPPLParser.RULE_takeAggFunction,
  OpenSearchPPLParser.RULE_getFormatFunction,
  OpenSearchPPLParser.RULE_tableIdent,
  OpenSearchPPLParser.RULE_positionFunctionName,
  OpenSearchPPLParser.RULE_evalFunctionName,
  OpenSearchPPLParser.RULE_literalValue,
  OpenSearchPPLParser.RULE_integerLiteral,
  OpenSearchPPLParser.RULE_decimalLiteral,
  OpenSearchPPLParser.RULE_keywordsCanBeId,
  OpenSearchPPLParser.RULE_renameClasue,
  OpenSearchPPLParser.RULE_qualifiedName,
  OpenSearchPPLParser.RULE_tableQualifiedName,
]);

Q3. There seems to be a custom cursor token index. Why is it needed?

The reason why we need to have a custom cursor token depends a lot on our tokenisation strategy. Do we want to consider whitespaces as a token or not? Well, it is not that easy; it really depends a lot on the grammar that we have on hand and our use case. For example, for the PPL grammar, I wanted to consider whitespace as a token, since that helps me easily track where I am in the query and appropriately either suggest some suggestions or not.

const { tokens, rules } = core.collectCandidates(cursorTokenIndex, context);

With this setup, we obtain two outputs: tokens and rules.

• Tokens represent grammar token suggestions generated by the C3 engine during its forward traversal of the ATN. However, tokens alone provide limited contextual insight.
• Rules, on the other hand, identify the currently matched preferred rule and the complete rule stack leading to it.

This rule information enables much deeper contextual understanding and opens the door to more intelligent, context-aware suggestions. For example revisiting our query

source =

With my preferred rule of RULE_tableIdent, now at this point it returns me the preferred Rule of tableIdent along with the rule stack. I can have an enrichment layer which upon encountering the above rule uses my operating environment to fetch and showcase all of the available tables at this point.
Below is an example of the rule-based orchestration layer that I have set up. It may appear more complex, but it tries to handle smaller edge cases, hence it is just bookkeeping.

export function processVisitedRules(
  rules: c3.CandidatesCollection['rules'],
  cursorTokenIndex: number,
  tokenStream: TokenStream
): ProcessVisitedRulesResult<OpenSearchPplAutocompleteResult> {
  let suggestSourcesOrTables: OpenSearchPplAutocompleteResult['suggestSourcesOrTables'];
  let suggestAggregateFunctions = false;
  let shouldSuggestColumns = false;
  let suggestFieldsInAggregateFunction = false;
  let suggestValuesForColumn: string | undefined;
  let suggestRenameAs: boolean = false;
  let suggestSingleQuotes: boolean = false;
  const rerunWithoutRules: number[] = [];

  const lastNonOperatorToken = findLastNonSpaceOperatorToken(tokenStream, cursorTokenIndex);

  for (const [ruleId, rule] of rules) {
    const parentRuleList = rule.ruleList;
    switch (ruleId) {
      case OpenSearchPPLParser.RULE_sqlLikeJoinType:
      case OpenSearchPPLParser.RULE_positionFunctionName:
      case OpenSearchPPLParser.RULE_integerLiteral:
      case OpenSearchPPLParser.RULE_decimalLiteral:
      case OpenSearchPPLParser.RULE_searchableKeyWord:
      case OpenSearchPPLParser.RULE_keywordsCanBeId: {
        break;
      }
      case OpenSearchPPLParser.RULE_statsFunctionName: {
        // suggests aggregate functions like avg, count etc after stats command , eg: source = abc | stats
        suggestAggregateFunctions = true;
        break;
      }
      case OpenSearchPPLParser.RULE_comparisonOperator:
      case OpenSearchPPLParser.RULE_searchComparisonOperator: {
        const lastToken = findLastNonSpaceToken(tokenStream, cursorTokenIndex);
        if (lastToken?.token && isAValidExpressionEndToken(lastToken?.token)) {
          rerunWithoutRules.push(OpenSearchPPLParser.RULE_searchComparisonOperator);
          rerunWithoutRules.push(OpenSearchPPLParser.RULE_comparisonOperator);
          rerunWithoutRules.push(OpenSearchPPLParser.RULE_searchCommand); // Since this is the parent of Search Comparison
        }
        break;
      }
      case OpenSearchPPLParser.RULE_searchCommand: {
        const firstTokenAfterPipe = findFirstNonSpaceTokenAfterPipe(tokenStream, cursorTokenIndex);
        if (
          !firstTokenAfterPipe ||
          ![OpenSearchPPLParser.DESCRIBE, OpenSearchPPLParser.SHOW].includes(
            firstTokenAfterPipe.token.type
          )
        ) {
          rerunWithoutRules.push(ruleId);
        }
        break;
      }
      case OpenSearchPPLParser.RULE_wcQualifiedName:
      case OpenSearchPPLParser.RULE_qualifiedName: {
        // Check if we're in a stats function context
        const isInStatsFunction = (parentRuleList ?? []).includes(
          OpenSearchPPLParser.RULE_statsFunction
        );

        if (isInStatsFunction) {
          suggestFieldsInAggregateFunction = true;
        }

        // Avoids suggestion fieldNames when last token is source. eg: source = , should suggest only tableName and not fieldname
        const lastTokenResult = findLastNonSpaceOperatorToken(tokenStream, cursorTokenIndex);
        if (lastTokenResult?.token.type === tokenDictionary.SOURCE) {
          break;
        }
        // In case we have a command with a field list for example source = abc | fields field1, field2, ... .
        // Suggest a fieldname only if the lastCharacter is not a fieldName. eg: source = abc | fields field1 -> should suggest | and , but source = abc | fields field1, -> should suggest fields
        if ((parentRuleList ?? []).some((parentRule) => fieldRuleList.includes(parentRule))) {
          const lastNonSpaceToken = findLastNonSpaceToken(tokenStream, cursorTokenIndex);
          if (lastNonSpaceToken?.token.type === tokenDictionary.ID) {
            break;
          } else {
            shouldSuggestColumns = true;
            break;
          }
        }

        // handling the scenario if the last Token is ID, we Don't suggest Column except in the case the second last token is Source to handle the scenario source = tablename fieldname suggestions
        if (lastNonOperatorToken && lastNonOperatorToken?.token.type === tokenDictionary.ID) {
          const secondLastTokenResult = findLastNonSpaceOperatorToken(
            tokenStream,
            lastNonOperatorToken.index
          );
          if (secondLastTokenResult?.token.type !== tokenDictionary.SOURCE) {
            break;
          }
        }
        shouldSuggestColumns = true;
        break;
      }
      case OpenSearchPPLParser.RULE_tableQualifiedName: {
        const lastToken = findLastNonSpaceToken(tokenStream, cursorTokenIndex);
        if (![tokenDictionary.ID, tokenDictionary.BQUOTA_STRING].includes(lastToken?.token.type)) {
          suggestSourcesOrTables = SourceOrTableSuggestion.TABLES;
        }
        break;
      }
      case OpenSearchPPLParser.RULE_renameClasue: {
        // if we're in the rename rule, we're either suggesting
        // field first token
        // 'as' second token
        // nothing third token, because it should be user specified

        const expressionStart = rule.startTokenIndex;
        if (expressionStart === cursorTokenIndex) {
          shouldSuggestColumns = true;
          break;
        }

        if (expressionStart + 2 === cursorTokenIndex) {
          suggestRenameAs = true;
          break;
        }

        break;
      }
      case OpenSearchPPLParser.RULE_stringLiteral: {
        // on its own, this rule would be triggered for relevance expressions and span. span
        // has its own rule, and relevance ....
        // todo: create span rule
        // todo: check if relevance expressions have incorrect behavior here
        if (cursorTokenIndex < 2) break; // should not happen due to grammar

        suggestSingleQuotes = true;

        let currentIndex = cursorTokenIndex - 1;
        // get the last token that appears other than whitespace
        const lastToken =
          tokenStream.get(currentIndex).type === tokenDictionary.SPACE
            ? tokenStream.get(currentIndex - 1)
            : tokenStream.get(currentIndex);
        // we only want to get the value if the very last token before WS is =, or
        // if its paren/comma and we pass by IN later. we don't need to check that we pass IN
        // because there is no valid syntax that will encounter the literal value rule with the
        // tokens '(' or ','
        if (
          ![tokenDictionary.EQUAL, tokenDictionary.OPENING_BRACKET, tokenDictionary.COMMA].includes(
            lastToken.type
          )
        ) {
          break;
        }

        const validIDToken = (token: Token) => {
          return token.type === tokenDictionary.ID || token.type === tokenDictionary.BACKTICK_QUOTE;
        };

        while (currentIndex > -1) {
          const token = tokenStream.get(currentIndex);
          if (
            !token ||
            token.type === tokenDictionary.PIPE ||
            token.type === tokenDictionary.SOURCE
          ) {
            break;
          }

          // NOTE: according to grammar, backticks in PPL are only possible for fields
          if (validIDToken(token)) {
            let combinedText = removePotentialBackticks(token.text ?? '');

            // stitch together IDs separated by DOTs
            let lookBehindIndex = currentIndex;
            while (lookBehindIndex > -1) {
              lookBehindIndex--;
              const prevToken = tokenStream.get(lookBehindIndex);
              if (!prevToken || prevToken.type !== tokenDictionary.DOT) {
                break;
              }
              lookBehindIndex--;
              combinedText = `${removePotentialBackticks(
                tokenStream.get(lookBehindIndex).text ?? ''
              )}.${combinedText}`;
            }

            suggestValuesForColumn = removePotentialBackticks(combinedText);
            break;
          }
          currentIndex--;
        }
        break;
      }
    }
  }

  return {
    suggestSourcesOrTables,
    suggestAggregateFunctions,
    shouldSuggestColumns,
    suggestFieldsInAggregateFunction,
    suggestValuesForColumn,
    suggestRenameAs,
    rerunWithoutRules,
    suggestSingleQuotes,
  };
}

With the orchestration layer, we now have a fully functioning intelligent grammar-based solution for us. However, there a lot of avenues of improvement. Tokens the ones we just took directly from the engine, we can add a context-aware filtering layer that provides a tighter set of more relevant tokens alone. This layer can range from being a set of regex to an LLM that actively tries to learn and rerank the grammar-relevant tokens. These small tweaks and refinements can make an ordinary autocomplete system extraordinary.

Syntax highlighting

Example