Protobuf

Protobuf (Protocol Buffer) is a protocol open-sourced by Google that is language- and platform-independent, extensible, and used for serializing structured data.

It can be used for data communication and data storage: you only need to define the data structure once, and then you can use the generated source code to write and read structured data from various data streams or languages.

Protobuf is used as the communication protocol for microservices.

Basic syntax

Basic framework

syntax = "proto3";

package xxx.xxx;

option gp_package = "./xxx/xxx/xxx; xxx";

// 一共有三个部分
// 第一行是版本声明
// 第二个是包的声明，防止不同项目同名消息的冲突
// 第三个手生成文件路径和生成的Go的包名

Defining a service

Notes

• service indicates that this is a service (interface). As for its naming rules, given that Golang distinguishes private from public by the case of the first letter, it must be uppercase so that other files can call it, and it should end with the suffix Service to indicate that it is a service

• rpc must be written, and this is the only one allowed; it indicates the use of rpc remote procedure calls

• stream is streaming, used for large file transfer, chat scenarios, typewriter effects, and so on

  • One question, multiple answers: after the question is asked, the answers are returned in strict order

  • Multiple questions, one answer: requests are input in strict order, then wait for one response

  • Multiple questions, multiple answers: two completely asynchronous pipes that do not block each other

    • Mode 1, one question and one answer, like playing ping-pong

    • Mode 2, like chatting with someone you’re flirting with: you send 10 messages and they reply with one, or the other way around — either is possible; requests and responses go through different channels

    • In that case, it needs to be implemented with goroutines in Golang

service TestService {
        rpc IndexDocument (IndexRequest) returns (IndexResponse)
        rpc Chat (ChatRequest) returns (stream ChatReponse) // 一问多答
        rpc UploadFile (stream FileChunk) returns (UploadStatus) // 多问一答
        rpc LiveChat (stream VoiceData) returns (stream VoiceData) // 变问边答
}

Defining messages

message is equivalent to a struct in Golang

Basic syntax format: type field name = unique number;

Below is a demo

message IndexRequest {
        string library_id = 1; // 字符串
        int32 file_size = 2; // 整数类型int32
        bytes file_content = 3; // 文件内容使用二进制
        bool is_public = 4; // 布尔值
}

A note on the unique number: it is like an identity for the field. During RPC, to save traffic, the original variable name is not transmitted; the number is transmitted directly. To prevent the numbers from getting confused, do not modify existing numbers after going live. The correct approach: once a number is deprecated, leave it empty and stop using it, and use a new value for new fields

Comparison between message and Go types

More advanced data structures in messages

Arrays (slice)

Use the repeated keyword

message ChatResponse {
        repeated string sources = 1;
}

Dictionaries (map)

A map is a key-value pair

Use map as the keyword

message IndexRequest {
        // Metadata map[string]string
        map<string, string> metasata = 2;
}

Enums (enum)

Use enum as the keyword; an enum must start from 0, which is usually the default value or the unknown value

An enum can only be used as part of a message and cannot be transmitted on its own

// 定义枚举 (必须从 0 开始)
enum DocType {
  UNKNOWN = 0; // 规范：第一个必须是 0，通常作为默认/未知值
  PDF = 1;
  IMAGE = 2;
  WORD = 3;
}

message Document {
  string name = 1;
  DocType type = 2; // 使用上面定义的枚举
}

Nested messages (nested)

message Author {
        string name = 1;
}

message Book {
        string title = 1;
        Author author = 2;
}

Syntax conventions

• Message Name: use UpperCamelCase (ChatRequest).

• Field Name: use snake_case with underscores (library_id, file_name).

  • Why? Because the generated Go code automatically converts them to camel case (LibraryId, FileName), but underscores are clearer to read in the Proto file.

• Service Name: use UpperCamelCase (OmniRAGService).

Installing protobuf

Installing on Mac

Just install it with the following commands

brew install protobuf

protoc --version

# 1. 安装生成数据结构的插件
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

# 2. 安装生成 gRPC 服务的插件
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Installing on Windows

Installing Protobuf and the Go plugins on Windows mainly consists of two steps: installing the compiler core (protoc) and installing the Go language plugins.

Since Windows does not have a unified default package manager like Homebrew, we need to download and configure the environment variables manually. Please follow the steps below:

Step 1: Install the compiler core (protoc)

This is the core tool provided by Google, responsible for reading .proto files.

1. Download the archive:

   • Visit the GitHub release page: Protocol Buffers Releases

   • Find the latest version (for example v25.x or higher).

   • In the “Assets” list, download protoc-xx.x-win64.zip (be sure to choose win64).

2. Extract:

   • Extract the archive to a place you won’t delete; C:\protoc is recommended.

   • After extracting, you should see a bin folder containing a protoc.exe.

3. Configure the environment variable (Path) — the most critical step:

   • Press the Win key, search for “Edit system environment variables”, and open it.

   • Click the “Environment Variables” button in the lower-right corner.

   • In the “System variables” section below, find Path, select it, and click “Edit”.

   • Click “New” on the right.

   • Enter the path of the bin directory you just extracted, for example: C:\protoc\bin.

   • Click “OK” all the way through to save.

4. Verify:

   • Open a new CMD or PowerShell window.

   • Enter: protoc —version

   • If a version number is displayed (such as libprotoc 25.1), the core was installed successfully!

Step 2: Install the Go language plugins

You need to install two Go plugins so that protoc can generate Go code.

1. Open PowerShell or CMD and run the following two commands:

# 1. 安装生成数据结构的插件
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

# 2. 安装生成 gRPC 服务的插件
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

1. Check Go’s environment variables:

   • go install downloads the plugins to your $GOPATH/bin directory (usually C:\Users\your-username\go\bin).

   • You need to make sure this path is also in your system environment variable Path.

   • How to verify: enter protoc-gen-go —version in the terminal.

     • If it reports “is not recognized as an internal or external command”, repeat the environment variable configuration steps in “Step 1” and add %USERPROFILE%\go\bin (that is, your go bin directory) to Path.

Using protobuf

Usage example

protoc --proto_path=. \
       --go_out=paths=source_relative:. \
       --go-grpc_out=paths=source_relative:. \
       api/omnirag/v1/omnirag.proto

Usage notes

• —proto_path=.: tells the compiler to start looking for imported files from the current directory (the root directory).

• —go_out=paths=source_relative=.:

  • Generates .pb.go files (data structures).

  • paths=source_relative: means the generated file path is at the same level as the proto file (that is, generated under api/omnirag/v1/), rather than being scattered around according to the package name.

  • :.: outputs to the current directory.

• —go-grpc_out=paths=source_relative=.:

  • Generates _grpc.pb.go files (service interfaces).

  • Same configuration as above.

• api/omnirag/v1/omnirag.proto: the path of the target file.