AI Native Module

Overview

Starting from OpenSumi 3.0, it supports customizing AI capabilities through the integration of AI Native Module, including but not limited to:

  • Built-in AI Chat assistant
  • Providing Agent open capabilities and quick command capabilities registration and expansion
  • Inline Chat capability is open, allowing for rich interaction and AI capabilities to generate or understand code within the editor
  • Code completion capability is open, including block completion, inline completion, and other basic capabilities
  • Problem diagnosis capability is open, providing detection capabilities for program runtime errors or static syntax issues
  • Intelligent terminal capability is open
  • Intelligent conflict resolution capability is open
  • Intelligent renaming capability is open
  • ...

More IDE functionalities' AI capabilities will continue to be opened up in the future.

How to Use

Step 1: Import the Module

Firstly, import the AI Native module in both the browser and server layers. It is recommended to use it together with the DesignModule!

// Browser layer
import { AINativeModule } from '@opensumi/ide-ai-native/lib/browser';
import { DesignModule } from '@opensumi/ide-design/lib/browser';

renderApp({
  modules: [...CommonBrowserModules, DesignModule, AINativeModule]
});

// Server layer
import { AINativeModule } from '@opensumi/ide-ai-native/lib/node';

startServer({
  modules: [...CommonNodeModules, AINativeModule]
});

Step 2: Contribution

The purpose of this step is to register various AI capabilities.

  1. Create a new contribution file and implement the AINativeCoreContribution interface.
import { AINativeCoreContribution } from '@opensumi/ide-ai-native/lib/browser/types';

@Domain(AINativeCoreContribution)
export class AiNativeContribution implements AINativeCoreContribution {
  // Register various AI capabilities here
}
  1. Inject it into the DI's Providers list or the providers configuration of a custom module.
const injector = new Injector();
injector.addProviders(AiNativeContribution);

opts.injector = injector;
const app = new ClientApp(opts);

Step 3: Register Backend Service

The purpose of this step is to enable the front-end AI interactions to request services from the backend interface.

Complete example code can be found in ai.back.service.ts.

  1. Create a new backend service file and inherit from the BaseAIBackService service.
@Injectable()
export class AiBackService implements IAIBackService<ReqeustResponse, ChatReadableStream> {
  // Here you can interact with any large model API interface
}
  1. Dependency injection of AIBackSerivceToken.

Provide the newly created AiBackService file to the Provider through dependency injection.

import { AIBackSerivceToken } from '@opensumi/ide-core-common';

const injector = new Injector();
injector.addProviders({
  token: AIBackSerivceToken,
  useClass: AiBackService
});

opts.injector = injector;
const serverApp = new ServerApp(opts);

Now that the module introduction phase is complete, the AI capabilities are missing (for example, inline chat is not triggered, etc.).

So, it is necessary to provide various provider capabilities in AiNativeContribution.

Contribution

Taking inline chat as an example, implement the registerInlineChatFeature method in AiNativeContribution.

@Domain(AINativeCoreContribution)
export class AiNativeContribution implements AINativeCoreContribution {
  // Obtain the registered backend service through AIBackSerivcePath
  // At this point, you can directly RPC call the functions provided by the backend service
  @Autowired(AIBackSerivcePath)
  private readonly aiBackService: IAIBackService;

  registerInlineChatFeature(registry: IInlineChatFeatureRegistry) {
    registry.registerEditorInlineChat(
      {
        id: 'ai-comments',
        name: 'Comments',
        title: 'add comments',
        renderType: 'button',
        codeAction: {}
      },
      {
        execute: async (editor: IEditor) => {
          // Handle any interaction after the inline chat button is clicked here
        }
      }
    );
  }
}

At this point, selecting a piece of code in the editor will display the just registered inline chat button in the appropriate area.

Other capabilities' Provider API documentation is as follows:

For more detailed and complete interface definitions, see: types.ts

Method NameDescriptionParameter TypeReturn Type
registerInlineChatFeatureRegisters inline chat related featuresIInlineChatFeatureRegistryvoid
registerChatFeatureRegisters chat panel related featuresIChatFeatureRegistryvoid
registerChatRenderRegisters chat panel related rendering layers, can customize renderIChatRenderRegistryvoid
registerResolveConflictFeatureRegisters intelligent conflict resolution related featuresIResolveConflictRegistryvoid
registerRenameProviderRegisters intelligent renaming related featuresIRenameCandidatesProviderRegistryvoid
registerProblemFixFeature
Register smart repair related functionsIProblemFixProviderRegistryvoid
registerIntelligentCompletionFeatureRegister for smart code completion related functionsIIntelligentCompletionsRegistryvoid

IInlineChatFeatureRegistry

Method NameDescriptionParameter TypeReturn Type
registerEditorInlineChatRegisters inline chat functionality in the editorAIActionItem, IEditorInlineChatHandlerIDisposable
registerTerminalInlineChatRegisters inline chat functionality in the terminalAIActionItem, ITerminalInlineChatHandlerIDisposable

IEditorInlineChatHandler

Method NameDescriptionParameter TypeReturn Type
executeDirectly executes the action's operation, and the inline chat disappears after clickingICodeEditor, ...any[]void
providerDiffPreviewStrategyProvides a preview strategy for the diff editorICodeEditor, CancellationTokenMaybePromise

ITerminalInlineChatHandler

Method NameDescriptionParameter TypeReturn Type
triggerRulesDefines trigger rules'selection' or BaseTerminalDetectionLineMatcher[]void
executeExecutes the action's operationstring (stdout), string (stdin), BaseTerminalDetectionLineMatcher? (rule)void

MaybePromise is a type alias indicating that the method may return a Promise or directly return a value of the corresponding type.

ChatResponse is a union type, indicating that the response can be any of ReplyResponse, ErrorResponse, or CancelResponse.

export class ReplyResponse {
  constructor(readonly message: string) {}

  static is(response: any): boolean {
    return (
      response instanceof ReplyResponse ||
      (typeof response === 'object' && response.message !== undefined)
    );
  }
}

export class ErrorResponse {
  constructor(readonly error: any, readonly message?: string) {}

  static is(response: any): boolean {
    return (
      response instanceof ErrorResponse ||
      (typeof response === 'object' && response.error !== undefined)
    );
  }
}

export class CancelResponse {
  readonly cancellation: boolean = true;

  constructor(readonly message?: string) {}

  static is(response: any): boolean {
    return (
      response instanceof CancelResponse ||
      (typeof response === 'object' && response.cancellation !== undefined)
    );
  }
}

export type ChatResponse = ReplyResponse | ErrorResponse | CancelResponse;

Example:

// Register inline chat in the editor
registerInlineChatFeature(registry: IInlineChatFeatureRegistry) {
  registry.registerEditorInlineChat(
    {
      id: 'comments',
      name: 'Comments',
    },
    {
      providerDiffPreviewStrategy: async (editor: ICodeEditor, token) => {
        const crossCode = this.getCrossCode(editor);
        const prompt = `Comment the code: \`\`\`${crossCode}\`\`\`. It is required to return only the code results without explanation.`;
      },
    },
  );
}

// Register inline chat in the terminal area
registry.registerTerminalInlineChat(
  {
    id: 'terminal',
    name: 'terminal',
  },
  {
    triggerRules: 'selection',
    execute: async (stdout: string) => {},
  },
);

IChatFeatureRegistry

Method NameDescriptionParameter TypeReturn Type
registerWelcomeRegisters the welcome message for the Chat panelIChatWelcomeMessageContent or React.ReactNode, ISampleQuestionsvoid
registerSlashCommandRegisters the quick command for the Chat panelIChatSlashCommandItem, IChatSlashCommandHandlervoid

Example:

registerChatFeature(registry: IChatFeatureRegistry): void {
  registry.registerWelcome(
    new MarkdownString(`Hello, I am your dedicated AI assistant, here to answer questions about code and help you think. You can ask me some questions about code.`),
    [
      {
        icon: getIcon('send-hollow'),
        title: 'Generate a Java Quicksort Algorithm',
        message: 'Generate a Java Quicksort Algorithm',
      },
    ],
  );

  registry.registerSlashCommand(
    {
      name: 'Explain',
      description: 'Explain',
      isShortcut: true,
      tooltip: 'Explain',
    },
    {
      providerRender: CustomSlashCommand,
      providerInputPlaceholder(value, editor) {
        return 'Please enter or paste the code.';
      },
      providerPrompt(value, editor) {
        return `Explain code: \`\`\`\n ${value}\`\`\``;
      },
      execute: (value: string, send: TChatSlashCommandSend, editor: ICodeEditor) => {
        send(value);
      },
    },
  );
}

IChatRenderRegistry

Method NameDescriptionParameter TypeReturn Type
registerWelcomeRenderCustomizes the rendering of the welcome message viewChatWelcomeRendervoid
registerAIRoleRenderCustomizes the rendering of the AI session viewChatAIRoleRendervoid
registerUserRoleRenderCustomizes the rendering of the user session viewChatUserRoleRendervoid
registerThinkingRenderCustomizes the rendering of the loading viewChatThinkingRendervoid
registerInputRenderCustomizes the rendering of the input box viewChatInputRendervoid

Example:

registerChatRender(registry: IChatRenderRegistry): void {
  // Directly inject React components
  registry.registerWelcomeRender(CustomWelcomeComponents);
}

IResolveConflictRegistry

Method NameDescriptionParameter TypeReturn Type
registerResolveConflictProviderRegisters the model-问答 service for resolving conflictskeyof typeof MergeConflictEditorMode, IResolveConflictHandlervoid

Example:

registerResolveConflictFeature(registry: IResolveConflictRegistry): void {
  registry.registerResolveConflictProvider('traditional', {
    providerRequest: async (contentMetadata, options, token) => {
      return new ReplyResponse('Resolved successfully!');
    },
  });
}

IRenameCandidatesProviderRegistry

Method NameDescriptionParameter TypeReturn Type
registerRenameSuggestionsProviderRegisters the provider for renaming suggestionsNewSymbolNamesProviderFnvoid

Example:

registerRenameProvider(registry: IRenameCandidatesProviderRegistry): void {
  registry.registerRenameSuggestionsProvider(async (model, range, token): Promise<NewSymbolName[] | undefined> => {
    return {
      newSymbolName: 'ai rename',
      tags: [NewSymbolNameTag.AIGenerated],
    };
  });
}

IProblemFixProviderRegistry

Method NameDescriptionParameter TypeReturn Type
registerHoverFixProviderRegister a provider for problem diagnosisIHoverFixHandlervoid

Example:

registerProblemFixFeature(registry: IProblemFixProviderRegistry): void {
  registry.registerHoverFixProvider({
    provideFix: async (
      editor: ICodeEditor,
      context: IProblemFixContext,
      token: CancellationToken,
    ): Promise<ChatResponse | InlineChatController> => {
      const { marker, editRange } = context;
      const prompt = 'Self-assembled prompts for problem diagnosis';

      const controller = new InlineChatController({ enableCodeblockRender: true });
      const stream = await this.aiBackService.requestStream(prompt, {}, token);
      controller.mountReadable(stream);

      return controller;
    },
  });
}

IIntelligentCompletionsRegistry

Method NameDescriptionParameter TypeReturn Type
registerIntelligentCompletionProviderRegister a smart completion providerIIntelligentCompletionProvidervoid

Note: Configuring the enableMultiLine field in the returned completion list can enable multi-line completion.

Example:

registerIntelligentCompletionFeature(registry: IIntelligentCompletionsRegistry): void {
  registry.registerIntelligentCompletionProvider(async (editor, position, bean, token) => {
    const model = editor.getModel()!;
    const value = model.getValueInRange({
      startLineNumber: position.lineNumber,
      startColumn: 1,
      endLineNumber: position.lineNumber + 3,
      endColumn: model?.getLineMaxColumn(position.lineNumber + 3),
    });

    const cancelController = new AbortController();
    const { signal } = cancelController;

    token.onCancellationRequested(() => {
      cancelController.abort();
    });

    const getRandomString = (length) => {
      const characters = 'opensumi';
      let result = '';
      for (let i = 0; i < length; i++) {
        result += characters.charAt(Math.floor(Math.random() * characters.length));
      }
      return result;
    };

    /**
     * 随机增删字符
     */
    const insertRandomStrings = (originalString) => {
      const minChanges = 2;
      const maxChanges = 5;
      const changesCount = Math.floor(Math.random() * (maxChanges - minChanges + 1)) + minChanges;
      let modifiedString = originalString;
      for (let i = 0; i < changesCount; i++) {
        const randomIndex = Math.floor(Math.random() * originalString.length);
        const operation = Math.random() < 0.5 ? 'delete' : 'insert';
        if (operation === 'delete') {
          modifiedString = modifiedString.slice(0, randomIndex) + modifiedString.slice(randomIndex + 1);
        } else {
          const randomChar = getRandomString(1);
          modifiedString = modifiedString.slice(0, randomIndex) + randomChar + modifiedString.slice(randomIndex);
        }
      }
      return modifiedString;
    };

    try {
      await new Promise((resolve, reject) => {
        const timeout = setTimeout(resolve, 1000);

        signal.addEventListener('abort', () => {
          clearTimeout(timeout);
          reject(new DOMException('Aborted', 'AbortError'));
        });
      });

      return {
        items: [
          {
            insertText: insertRandomStrings(value),
            range: {
              startLineNumber: position.lineNumber,
              startColumn: 1,
              endLineNumber: position.lineNumber + 3,
              endColumn: model?.getLineMaxColumn(position.lineNumber + 3),
            },
          },
        ],
        // Whether to enable multi-line completion
        enableMultiLine: true,
      };
    } catch (error) {
      if (error.name === 'AbortError') {
        return { items: [] };
      }
      throw error;
    }
  });
}

**Complete example code can be found in [ai-native.contribution.ts](https://github.com/opensumi/core/blob/main/packages/startup/entry/sample-modules/ai-native/ai-native.contribution.ts).**

## Related Configuration

The AI Native Config-related configuration parameters can control the on/off state of all AI capabilities.

### IAINativeConfig

| Property Name | Type                  | Description                                |
| ------------- | --------------------- | ------------------------------------------ |
| capabilities  | IAINativeCapabilities | Configuration items for AI functionalities |
| layout        | IDesignLayoutConfig   | Configuration items for layout design      |

### IAINativeCapabilities

> All the following capabilities are enabled by default.

| Property Name                  | Type    | Description                                                      |
| ------------------------------ | ------- | ---------------------------------------------------------------- |
| supportsMarkers                | boolean | Whether to enable AI capabilities to handle the problem panel    |
| supportsChatAssistant          | boolean | Whether to enable the AI chat assistant feature                  |
| supportsInlineChat             | boolean | Whether to enable the Inline Chat feature                        |
| supportsInlineCompletion       | boolean | Whether to enable the code intelligent completion feature        |
| supportsConflictResolve        | boolean | Whether to enable the AI intelligent conflict resolution feature |
| supportsRenameSuggestions      | boolean | Whether to enable the AI to provide renaming suggestions feature |
| supportsProblemFix             | boolean | Whether to enable AI problem diagnosis capability       |
| supportsTerminalDetection      | boolean | Whether to enable the AI terminal detection feature              |
| supportsTerminalCommandSuggest | boolean | Whether to enable the AI terminal command suggestion feature     |

### IDesignLayoutConfig

| Property Name              | Type    | Description                                                     |
| -------------------------- | ------- | --------------------------------------------------------------- |
| useMergeRightWithLeftPanel | boolean | Whether to merge the right panel with the left panel            |
| useMenubarView             | boolean | Whether to use the new menu bar view                            |
| menubarLogo                | string  | Sets the icon for the menu bar, which can be a path to an image |