GoLink · oranhe.com

daily

A simple and practical short-link project. Project repository

Introduction to short links

What it is

In plain terms, a short link is a relatively long URL that is converted by programmatic computation into a simple, short web address.

The short links I receive most often are the ones sent by JD.com. Here is an example:

[JD] Beijing national subsidy carnival, limited-time big news: the 2026 national subsidy is officially live! Go to JD to claim your subsidy and save 15% instantly. Limited-time grab, while supplies last! Grab 3.cn/v/1-FT59nd. Reply R to opt out

The URL we get after clicking is:

https://pro.m.jd.com/mall/active/4YSZ3d72rtEfxiWiDVcWRBEjCbSd/index.html?babelChannel=ttt58

This is exactly how a short link replaces an originally long address with a short one that people cannot decipher.

Many companies also offer short-link services:

Baidu: dwz.cn
Weibo: t.cn
…

Why

A company usually has many scenarios that require sending links, and the business-side details are very granular, which makes the links very long. This has many disadvantages in scenarios such as sending SMS messages, sending messages through IM tools, and push notifications, for example:

SMS content may exceed the length limit; truncation costs more money and the information is also incomplete
Character limits on social platforms
Some platforms also have relatively strict moderation mechanisms that can recognize keywords and rule the content a violation, for example Xiao-mou-shu, Mou-hong-shu, and Xiao-hong-mou
URLs can be converted into QR codes; if a URL is too long, the resulting QR code becomes very complex, whereas a QR code generated from a short link is clearer

At this point, converting long links into short links becomes very important.

How it works

daily

A browser cannot convert a short link into a long link by itself; this involves the concept of redirection.

The browser receives the address of the short link and sends a request to the company’s short-link server. The company returns data and a 302 redirect status code, after which the browser follows the redirect and sends a request to the actual business server.

daily

Project design

Requirements analysis

Background

The company’s internal business needs to send a large number of marketing SMS messages and notification SMS messages; after customer service makes sales calls, supporting marketing SMS messages need to be sent.

A short-link service is needed to serve the various business lines. At this stage it provides interfaces for converting links; later it can provide data report and analysis interfaces for link-click statistics.

Description

Given a long URL, get a unique short URL
When a user clicks the short link, they can be redirected to the corresponding URL
To ensure business continuity, short URLs remain valid long term

Positioning

It is a short-link service used only by the company’s internal business; it accepts only internal requests to convert long links into short links and is not provided as an external service
It is used mainly within China
Later, short-link access data report interfaces may be provided according to demand

Scale

How many business lines within the company it serves
What order of magnitude the user scale is
What the QPS is
1. Note: QPS stands for Queries Per Second. It is an important metric for measuring system performance and indicates the number of query requests the system can handle per second. In areas such as databases, networks, and servers, QPS is a common performance metric; generally, the higher it is, the better the system performance.

Technical targets

Request latency: how many milliseconds the whole process must complete within
Reliability: ensure that 99.99% of redirects work normally
Security: no data leaks and no attacks

Functional module analysis

Based on the requirements analysis, the requirements can be split into a link-conversion module, a storage module, and a link-viewing module, as well as an extensible data-statistics module.

Link-conversion module
1. The same long link is converted into the same short link (it can also be converted into different ones based on personal information, for example under a promotion mechanism, down to the individual)
2. The generated short link should contain as few characters as possible
3. It is necessary to avoid some inappropriate words and some words with special meanings
4. It is necessary to avoid circular conversion (taking something that is already a short link and converting it into a short link again)
Storage module
1. Save the correspondence between the original long link and the short link
2. Be able to look up the original long link from the short link
Link-viewing module
1. After finding the long link from the short link, return a redirect response
2. Later data-report requirements may require collecting and aggregating request-header data

System design

Given the actual business scenario, we can see that few links are stored while many links are accessed; this is a typical read-heavy, write-light system.

Another characteristic is that the data basically does not need to change after it is written, so data consistency issues can actually be ignored, and a cache system can be used to improve read efficiency.

Overall architecture

daily

Long link to short link
- Deployed separately as a microservice (the conversion service)
- Provides conversion services to other services and requires authentication
  - Call our conversion endpoint through a RESTful API
  - Call our conversion method through RPC (implement an RPC version of the conversion yourself)
Viewing short links
- Deployed separately as a service (the short-link viewing service)
- nginx forwards view requests, /[0-9+a-z+A-Z]* —> forwarded to our link-lookup service
- Collect via access.log (EFK) and aggregate access data

Technical approach

Short-link generation methods

Hash
- Use a hash function to hash the long link and use the resulting hash value as the short-link identifier
- The advantage of doing this is simplicity, but hash collisions occur when the data volume is large
- MurmurHash is a non-cryptographic hash function. Compared with other popular hash functions, it performs better at randomly distributing keys with strong regularity; this algorithm has good random-distribution characteristics and is fast.
Ticket server / auto-increment sequence
- After each request to convert a long link into a short link is received, the ticket server generates an incrementing number, then that sequence number is converted to base 62 (10+26*2), and finally appended to the short domain to obtain the final short link.
- The ticket server can generate auto-incrementing ids, and in theory its capacity is sufficient to meet the implementation requirements. However, under high concurrency the design of the ticket server is relatively difficult, because reasonable auto-increment must be achieved.
- Ways to implement a ticket server
  - Based on uuid
    - Advantages: no duplicates, good performance
    - Disadvantages: the number is too large—a 32-digit hexadecimal number
  - Ticket server based on redis
    - Advantages: high performance
    - Disadvantages: requires a high-availability architecture and consideration of persistence
  - Distributed ID generator based on the Snowflake algorithm
    - Advantages: high performance, high availability
    - Disadvantages: complex to implement, depends on the clock
  - Ticket server based on a MySQL auto-increment primary key
    - Advantages: simple and reliable
    - Disadvantages: depends on MySQL, and performance becomes a bottleneck, but availability can be scaled through sharding
Implementing the ticket server based on a MySQL auto-increment primary key
- Our project adopts the solution of using a MySQL database primary key as the ticket server
- Build a table with a relatively simple structure that must not contain other redundant data fields
- For every link-conversion request, we insert a record into the table, and we can use the table’s auto-increment ID as the generated number
- Use MySQL replace: if the old row and the new row in the table have the same value on a primary key or unique index, the old row is deleted before the new row.
- To avoid a single point of failure, we split the ID generator into odd and even parts, deployed on two MySQL servers respectively.
```
server1:
auto-increment-increment = 2
auto-increment-offset = 1

server2:
auto-increment-increment = 2
auto-increment-offset = 2
```
- Below are the table creation statement and related SQL statements
```
/* 序号表的设计 */

CREATE TABLE `sequence` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    `stub` varchar(1) NOT NULL,
    `timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`),
    UNIQUE KEY `idx_uniq_stub` (`stub`)

) ENGINE=MyISAM DEFAULT CHARSET=utf8;

REPLACE INTO sequence (stub) VALUES ('a');
SELECT LAST_INSERT_ID();
```

A simple test

daily

Tables

Table design

lurl and surl are both unique; neither long links nor short links may be duplicated
Long links have a large data volume and are unsuitable for indexing, so the md5 value is used as the index here
is_del is the soft-delete flag
A read-write separation model can be adopted: write to the primary database and read from replicas

CREATE TABLE `short_url_map` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT '主键',
    `create_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `create_by` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '创建者',
    `is_del` tinyint UNSIGNED NOT NULL DEFAULT '0' COMMENT '是否删除：0正常 1删除',
    
    `lurl` varchar(2048) DEFAULT NULL COMMENT '长链接',
    `md5` char(32) DEFAULT NULL COMMENT '长链接MD5',
    `surl` varchar(11) DEFAULT NULL COMMENT '短链接',
    PRIMARY KEY(`id`),
    INDEX(`is_del`),
    UNIQUE(`md5`),
    UNIQUE(`surl`)
) ENGINE=INNODB DEFAULT CHARSET=utf8mb4;

Data volume
- The maximum URL length differs across browsers
- Assume the storage space required for one long link plus short link is 200 bytes per record
- Assume each business line writes 100 records per second; 10 lines means 1,000 records per second
- One primary and two backups make three copies of the data, and the redundancy factor for indexes and the like is 1.5
- Thus 200*1000*3600*24*365*3*1.5 is roughly equal to 20T
Data deletion
- This project does not involve configuring expiration times for links; all deletions are soft deletions, implemented by marking records manually

Attack prevention

External service
- IP request limits
- Per-user limits on conversion quota
- Cache the URLs that have already been converted to prevent IDs from being exhausted; this applies to the scenario where the same URL can be converted into different short links
  - LRU cache of URLs
  - Bloom filter
- Validate whether the link is valid
Internal service
- Authorization and authentication
- Daily account quotas
- Validate whether the link is valid

Accessing short links

Use Redis as cache and MySQL for persistent storage
When a short-link request comes in, first query the cache by identifier; if it is there, return it; if not, query the MySQL database; if it is found, return the redirect response and also write it into the cache
About cache settings
- Use the LRU approach
- Deploy Redis as a cluster
- Design the memory size according to the data volume, use the LRU eviction policy, and remove the least recently used keys
Handling the cache breakdown problem
- Increase expiration times
- Add locking
- Use singleflight to merge requests
Solving the cache penetration problem
- With a Bloom filter that does not store the original data: if the filter cannot find it, it definitely does not exist; if it does find it, query the database, where it may not exist
- Cuckoo filter, which supports deletion; a cuckoo filter is relatively suitable for this project

Deployment

Our short-link service is deployed behind Nginx. In this way, access data can be aggregated through Nginx’s access log (access.log). (For example, collect logs through EFK and produce statistical reports.)

Project management

Use git

Technology stack summary

Golang Go-Zero MySQL Redis Git Nginx EFK

Project implementation

1. Setting up the project skeleton

1.1 Creating the database and tables

Using the MySQL container created with Docker, create the database and tables inside it; the specific process is shown below:

daily

1.2 Building the skeleton of the go-zero framework

We use goctl to generate the framework code

daily

api file

syntax = "v1"

// 进行项目的简单说明
info (
    title:   "短链接项目"
    author:  "aorangehc"
    date:    "2025年 8 月 5 日"
    version: "v1"
)

type ConvertReq {
    LongUrl string `json:"longUrl"`
}

type ConvertResp {
    ShortUrl string `json:"shortUrl"`
}

type ShowReq {
    ShortUrl string `json:"shortUrl"`
}

type ShowResp {
    LongUrl string `json:"longUrl"`
}

@server (
    prefix: "/api/shortener"
)
service shortener_api {
    @handler ConvertHandler
    post /convert (ConvertReq) returns (ConvertResp)

    @handler ShowHandler
    get /:shortUrl (ShowReq) returns (ShowResp)
}

model file

CREATE TABLE `sequence` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    `stub` varchar(1) NOT NULL,
    `timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`),
    UNIQUE KEY `idx_uniq_stub` (`stub`)

) ENGINE=MyISAM DEFAULT CHARSET=utf8;

CREATE TABLE `short_url_map` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT '主键',
    `create_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
    `create_by` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '创建者',
    `is_del` tinyint UNSIGNED NOT NULL DEFAULT '0' COMMENT '是否删除：0正常 1删除',
    
    `lurl` varchar(2048) DEFAULT NULL COMMENT '长链接',
    `md5` char(32) DEFAULT NULL COMMENT '长链接MD5',
    `surl` varchar(11) DEFAULT NULL COMMENT '短链接',
    PRIMARY KEY(`id`),
    INDEX(`is_del`),
    UNIQUE(`md5`),
    UNIQUE(`surl`)
) ENGINE=INNODB DEFAULT CHARSET=utf8mb4;

1.3 Downloading project dependencies

# 在根目录下使用
go mod tidy

1.4 Running the project

It runs normally

daily

1.5 Modifying the configuration file and config struct

Add the database configuration to the yaml file and add the corresponding parameters in the config file.

ShortUrlDB:
  DataSource: "user:password@tcp(127.0.0.1:3307)/golink?charset=utf8mb4&parseTime=True&loc=Local"

SequenceDB:
  DataSource: "user:password@tcp(127.0.0.1:3307)/golink?charset=utf8mb4&parseTime=True&loc=Local"
  
  type Config struct {
    rest.RestConf

    ShortUrlDB struct {
        DataSource string
    }

    SequenceDB struct {
        DataSource string
    }
}

2. Link conversion

Long link -> short link

2.1 Long-link validation

Data cannot be empty

Use the validator library for parameter validation

go get github.com/go-playground/validator/v10

Then add the validate parameter and setting after the parameters in the api file

type ConvertReq {
    LongUrl string `json:"longUrl" validate:"required"`
}

type ConvertResp {
    ShortUrl string `json:"shortUrl"`
}

type ShowReq {
    ShortUrl string `json:"shortUrl" validate:"required"`
}

type ShowResp {
    LongUrl string `json:"longUrl"`
}

After regenerating the code, modify the context file and test. The data validation setting works, and empty data reports an error directly.

// 参数校验
if err := validator.New().StructCtx(r.Context(), &req); err != nil {
    httpx.ErrorCtx(r.Context(), w, err)
    return
}

daily

The long link can be requested

Create an HTTP client using the standard http library to probe liveness

daily

Checking whether the link has already been converted

The database uses md5 as the primary key

First generate the md5

Then check whether that md5 exists in the database

md5Str := md5.Sum([]byte(req.LongUrl))
u, err := l.svcCtx.ShortUrlModel.FindOneByMd5(l.ctx, sql.NullString{String: md5Str, Valid: true})
if err != sqlx.ErrNotFound {
    if err == nil {
        return nil, errors.New("该长链接已转链，短链接为：" + u.Surl.String)
    }
    logx.Errorw("ShortUrlModel.FindOneByMd5 error", logx.Field("err", err), logx.Field("md5", md5Str))
    return nil, err
}

// Sum 计算数据的 MD5 值，返回十六进制字符串表示
func Sum(data []byte) string {
    h := md5.New()
    h.Write(data)
    return hex.EncodeToString(h.Sum(nil))
}

Avoiding circular link conversion

Parse the base path of the link, then query the database to check whether it is a short link that has already been converted

basePath, err := urltools.GetBasePath(req.LongUrl)
if err != nil {
    logx.Errorw("GetBasePath error", logx.Field("err", err))
    return nil, err
}
u, err = l.svcCtx.ShortUrlModel.FindOneBySurl(l.ctx, sql.NullString{String: basePath, Valid: true})
if err != sqlx.ErrNotFound {
    if err == nil {
        return nil, errors.New("该链接已转链，短链接为：" + u.Surl.String)
    }
    logx.Errorw("ShortUrlModel.FindOneBySurl error", logx.Field("err", err), logx.Field("basePath", basePath))
    return nil, err
}

// GetBasePath 获取 url 的 base path
func GetBasePath(baseUrl string) (string, error) {
    myUrl, err := url.Parse(baseUrl)

    if err != nil {
        return "", err
    }

    basePath := path.Base(myUrl.Path)
    return basePath, nil
}

2.2 Implementing the ticket server

For each incoming conversion request, use a replace into statement to insert a row into the sequence table, and take the id as the number.

We add a new method to the model that implements the replace method and returns the latest id

// Replace 实现 repalce into语句，返回最后插入数据的 id
func (m *defaultSequenceModel) Replace(ctx context.Context, newData *Sequence) (int64, error) {
    query := fmt.Sprintf("replace into %s (%s) values (?, ?)", m.table, sequenceRowsExpectAutoSet)
    ret, err := m.conn.ExecCtx(ctx, query, newData.Stub, newData.Timestamp)
    if err != nil {
        return 0, err
    }
    id, err := ret.LastInsertId()
    if err != nil {
        return 0, err
    }
    return id, nil
}

The test results are normal.

daily

2.3 Base 62 link conversion

package base62

import (
    "math"
    "strings"
)

// 62进制转换
// 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
// 0-9：0-9
// a-z：10-35
// A-Z：36-61

// 为了避免安全性，打乱顺序
var (
    baseStr    string
    baseStrLen uint64
)

// BaseInit 初始化
func BaseInit(bs string) {
    if len(bs) == 0 {
        panic("baseStr is empty")
    }
    baseStr = bs
    baseStrLen = uint64(len(bs))
}

// Base62Encode 将数字转换为62进制
func Base62Encode(num uint64) string {
    if num == 0 {
        return string(baseStr[0])
    }
    var res []byte
    for num > 0 {
        res = append(res, baseStr[num%baseStrLen])
        num /= baseStrLen
    }

    return string(reverse(res))
}

// Base62Decode 将62进制转换为数字
func Base62Decode(str string) (seq uint64) {
    bl := []byte(str)
    bl = reverse(bl)

    for idx, b := range bl {
        base := math.Pow(float64(baseStrLen), float64(idx))
        seq += uint64(strings.Index(baseStr, string(b))) * uint64(base)
    }

    return seq
}

func reverse(s []byte) []byte {
    for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
        s[i], s[j] = s[j], s[i]
    }
    return s
}

Security is simply ensured by shuffling the global base-62 string

Certain words are prevented from appearing by configuring a global short-link blacklist

2.4 Storage and testing

The result is stored in the database

// 4.存储长链接和短链接的映射关系
    if _, err := l.svcCtx.ShortUrlModel.Insert(l.ctx, &model.ShortUrlMap{
        Lurl: sql.NullString{String: req.LongUrl, Valid: true},
        Surl: sql.NullString{String: short, Valid: true},
        Md5:  sql.NullString{String: md5Str, Valid: true},
    }); err != nil {
        logx.Errorw("ShortUrlModel.Insert error", logx.Field("err", err))
        return nil, err
    }

    shortUrl := l.svcCtx.Config.ShortDoamin + "/" + short // shortUrl = 'http://localhost:8080/' short
    resp = &types.ConvertResp{
        ShortUrl: shortUrl,
    }

    return resp, nil

daily

3. Viewing short links

3.1 Implementing the short-link redirect endpoint

After receiving the short-link request, redirect to the real link, and test it at the same time

func (l *ShowLogic) Show(req *types.ShowReq) (resp *types.ShowResp, err error) {
    // 根据短链接查询长链接
    u, err := l.svcCtx.ShortUrlModel.FindOneBySurl(l.ctx, sql.NullString{String: req.ShortUrl, Valid: true})
    if err != nil {
        if err == sql.ErrNoRows {
            return nil, errors.New("404 Not Found")
        }
        logx.Errorw("ShortUrlModel.FindOneBySurl error", logx.Field("err", err))
        return nil, err
    }
    resp = &types.ShowResp{
        LongUrl: u.Lurl.String,
    }

    // 返回查询到的长链接，在 handler 层进行重定向返回

    return resp, nil
}

daily

To improve performance, add a Redis cache to speed things up

daily

3.2 singleflight

It provides a mechanism for suppressing duplicate function calls, which can be used to avoid making the same function call at the same time. While the first call is not yet complete, subsequent duplicate calls wait; when the first call completes, the result is shared with them. In this way, although the function call is executed only once, all calls obtain the final result.

The go-zero framework uses singleflight by default