Logs are everywhere in software development. Without them there’d be no relational databases, git version control, or most analytics platforms.

日志在软件开发中无处不在。 没有它们，就不会有关系数据库，git版本控制或大多数分析平台。

For many developers that’s the end of our relationship with logs. We use tools that rely on logs. We turn to logs to understand why something went wrong. And up until fairly recently, you could probably go an entire career without thinking much more deeply about them than that.

对于许多开发人员而言，这就是我们与日志的关系的终点。 我们使用依赖日志的工具。 我们转向日志以了解为什么出了问题。 直到最近，您可能在整个职业生涯中都没有对它们进行更深入的思考。

Commit logs, in particular, have come out of the wings and into the spotlight. At the heart of tools such as Apache Kafka, commit logs are essential to handling increased volumes of data, bringing coherence to distributed systems, and providing a common source of truth for microservices architectures.

尤其是提交日志，已脱颖而出，成为人们关注的焦点。 作为Apache Kafka之类工具的核心，提交日志对于处理增加的数据量，使分布式系统具有连贯性以及为微服务架构提供通用的真理来源至关重要。

Understanding how commit logs work will help you to build more resilient systems using tools such as Kafka.

了解提交日志的工作方式将帮助您使用诸如Kafka之类的工具构建更具弹性的系统。

这是一个日志，仅此而已 (It’s a log and that’s about it)

The good news is that commit logs are about as straightforward as it gets. They are a sequence of records, each with its own unique identifier. Want to add a record? No problem, it gets appended at the end of the log. Want to change an existing record? That you can’t do; once written, records are immutable.

好消息是，提交日志与获得日志一样简单。 它们是一系列记录，每个记录都有自己的唯一标识符。 要添加记录吗？ 没问题，它会附加在日志末尾。 要更改现有记录吗？ 你做不到； 一旦写入，记录是不可变的。

What about reading? That always happens from left to right. While there’s no query, as such, you can use offsets to specify the start and end points of your read.

读书呢？ 这总是从左到右发生。 这样，虽然没有查询，但可以使用偏移量来指定读取的起点和终点。

And that’s it. We can all go home. Well, perhaps not just yet. As often, the raw “what” is far less interesting than the “why”.

就是这样。 我们都可以回家。 好吧，也许还没有。 通常，原始的“什么”远没有“为什么”那么有趣。

即使其他人都离开了您，日志也在那里 (Logs are there even when everyone else has left you)

Commit logs have been around for a long while and that’s because they solve a core problem in software development. They provide a source of truth for what has happened in a system and in what order.

提交日志已经存在很长时间了，这是因为它们解决了软件开发中的核心问题。 它们为系统中发生的事情和以什么顺序提供了真理来源。

Let’s think about a relational database such as PostgreSQL. In Postgres, commit logs are called write ahead logs. Each write to a Postgres database must first be recorded in the write ahead log before the data is changed in either a table or an index. The first benefit is that it speeds up database writes.

让我们考虑一下诸如PostgreSQL之类的关系数据库。 在Postgres中，提交日志称为预写日志。 在表或索引中更改数据之前，必须首先在预写日志中记录对Postgres数据库的每次写入。 第一个好处是它可以加快数据库写入速度。

Writing to a commit log is relatively fast, even on disk. Writing to more complex data structures, such as a relational table, is necessarily slower. So long as the transaction is recorded in the write ahead log, the changes to indexes and tables can happen in memory, with a page to disk later on. That way, the slow part of writing to the database happens asynchronously. Even if the plug is pulled from the server before the table and index changes are written to disk, the database can be recreated by following the story of what happened and in what order as recorded in the write ahead log.

即使在磁盘上，写入提交日志的速度也相对较快。 写入更复杂的数据结构(例如关系表)的速度一定较慢。 只要将事务记录在预写日志中，对索引和表的更改就可以在内存中发生，稍后再将页面分页到磁盘。 这样，写入数据库的较慢部分将异步发生。 即使在将表和索引更改写入磁盘之前从服务器上拔下插头，也可以通过按照事前写入日志中记录的情况和顺序来重新创建数据库。

The bigger benefit, perhaps, is that any database can be recreated from scratch simply by following the write ahead log. Whether that’s as part of disaster recovery or for the purpose of streaming live changes to a read-only replica.

也许更大的好处是，只要遵循预写日志，就可以从头开始重新创建任何数据库。 无论是作为灾难恢复的一部分，还是将实时更改流式传输到只读副本的目的。

Now, what if that commit log could serve the same purpose and not just for a database but, instead, an entire software architecture?

现在，如果该提交日志不仅可以用于数据库，还可以用于整个软件体系结构，可以达到相同的目的呢？

日志可以处理音量 (Logs can handle volume)

Let’s say you’re building an ecommerce application. You want to better understand how customers make their way around the site and so you record every click, every search term, every page view.

假设您正在构建一个电子商务应用程序。 您想更好地了解客户如何在网站上购物，并记录每次点击，每个搜索字词和每个页面浏览量。

Regardless of how you later process that data, you first need to capture it. Perhaps the quickest solution to develop would be to store each event in your operational database. But that puts you on the wrong side of a trade-off. The operational database — most likely a relational database — has a relatively slow write time. That’s fine because it gives you benefits such as rich query, transactionality, and mutability. But right now all you want to do is capture the data and worry about what to do with it later.

无论您以后如何处理该数据，都首先需要捕获它。 最快的解决方案也许是将每个事件存储在您的运营数据库中。 但这使您处于权衡的错误方面。 运营数据库(很可能是关系数据库)的写入时间相对较慢。 很好，因为它为您带来了诸如丰富的查询，事务性和可变性之类的好处。 但是现在，您要做的就是捕获数据，并担心以后如何处理。

One option would be to put Redis in front of your operational database, so that it can soak up the volume and release slowly at a rate suited to the main database. However, that’s just offsetting the problem and leaves your relatively expensive operational database full of semi-structured data. The visitor data is coming through in huge volumes and there’s no point in increasing the cost and complexity of your operational database in order to handle “nice to have” data such as this.

一种选择是将Redis放在您的操作数据库的前面，以便它可以吸收卷并以适合主数据库的速率缓慢释放。 但是，这只是在解决问题，并且使相对昂贵的运营数据库中充满了半结构化数据。 访客数据正在大量涌入，而为了处理诸如此类的“精打细算”的数据而增加运营数据库的成本和复杂性毫无意义。

A commit log is ideal, though. The data you need to store is made up of discrete, ordered events. And the simplicity of commit logs means that they can easily handle far larger volumes of data than a typical relational database.

不过，提交日志是理想的。 您需要存储的数据由离散的，有序的事件组成。 而且，提交日志的简单性意味着与典型的关系数据库相比，它们可以轻松处理大量的数据。

In fact, this is a typical use case for Apache Kafka, which puts a commit log at the heart of its data model.

实际上，这是Apache Kafka的典型用例，它将提交日志置于其数据模型的核心。

日志是真理的源头 (Logs are a source of truth)

Logs are fast, they are simple, they handle large volumes of data, and they are a source of truth. This makes them ideal for situations where multiple autonomous components form a single system. That could be a full blown microservices architecture or a monolith where you’ve hived off one or two processes.

日志速度快，简单，可以处理大量数据，并且是事实的来源。 这使它们成为多个自治组件组成一个系统的情况的理想选择。 那可能是成熟的微服务架构，也可能是您淘汰了一个或两个进程的整体。

In these situations, a log does two jobs. One is that it provides a single place for all parts of the system to find changes. The other is that it imposes an order on those changes simply by the fact of how it works. Let’s go back to ecommerce to explore what this means.

在这些情况下，日志执行两项工作。 一个是它为系统的所有部分提供了查找更改的单一位置。 另一个是仅仅通过其工作原理就对这些更改施加了命令。 让我们回到电子商务来探索这意味着什么。

Say we have a simple ecommerce application made up of five distinct services: catalog, payments, orders, customers, and shipping. In this situation, a customer order might take this path:

假设我们有一个简单的电子商务应用程序，由五个不同的服务组成：目录，付款，订单，客户和运输。 在这种情况下，客户订单可能采用以下路径：

1. The order system writes the customer’s order to the log.

   订单系统将客户的订单写入日志。

2. The catalog system reads the order from the log and checks that there is sufficient stock to fulfill it. It finds there is and so temporarily reduces the stock count accordingly and writes a new record to the log indicating that the order can be fulfilled.

   目录系统从日志中读取订单，并检查是否有足够的库存来履行订单。 它找到了，因此暂时减少了库存数量，并向日志中写入了一条新记录，表明可以履行订单。

3. The payment system reads the record showing that the order can be fulfilled and attempts to take payment from the customer’s card. The payment is successful and the payment system writes back to the log that the order can progress.

   付款系统读取表明可以履行订单的记录，并尝试从客户的卡中付款。 付款成功，付款系统将写回订单可以继续进行的日志。

4. The shipping system reads the log and prepares the order for dispatch.

   运送系统读取日志并准备要下达的订单。

5. The customer system reads the same entry and updates the customer’s record with details of their new order.

   客户系统读取相同的条目，并使用其新订单的详细信息更新客户的记录。

6. A dispatcher in the warehouse marks the package as sent in the shipping system. The shipping system writes to the log that the package has shipped.

   仓库中的调度员将包裹标记为已在运输系统中发送。 运输系统将包裹已运输的日志写入日志。

7. The customer system sees that status change from the shipping system and updates the customer’s tracking information accordingly.

   客户系统从运输系统看到状态更改，并相应地更新客户的跟踪信息。

8. The catalog confirms the earlier temporary reduction in stock level.

   目录确认了库存水平的较早暂时减少。

In reality, there would be many more steps. For example, there might be an entire microservice that kicks into life when a stock reduction message appears in the log and reduces the stock level in the operational database by the right amount. The point is, though, that each of these systems never interacts directly. Each one writes to and reads from the log. That has several benefits.

实际上，将有更多步骤。 例如，当日志中出现库存减少消息并将操作数据库中的库存水平降低适当的数量时，可能会有整个微服务投入使用。 关键是，这些系统中的每一个都从不直接交互。 每个人都写入和读取日志。 这有几个好处。

With the log as the source of truth, there’s no danger that one service might accidentally pre-empt or overrule another because everything happens in order. Another benefit is that, so long as the log is available, the system can theoretically continue even when some components are offline.

有了日志作为事实的来源，就不会有一项服务可能会意外抢占或推翻另一项服务的危险，因为一切都会按顺序进行。 另一个好处是，只要日志可用，理论上即使某些组件处于脱机状态，系统也可以继续运行。

提交日志 (Committing to logs)

You could build your own log but you’re likely to get a better return on your time using something like Apache Kafka.

您可以构建自己的日志，但是使用Apache Kafka之类的东西可能会在时间上获得更好的回报。

Kafka gives you the central benefit of a commit log — the immutable, ordered record of events — with the benefits of integrating with common data sources, writing data out to other systems such as Postgres, and the ability to act on and transform the data it processes. There are SDKs for most common languages and if you use a hosted service you don’t need to worry about administering your own Kafka cluster. And, believe me, that’s no small task; take a look at how Heroku manages their huge fleet of Kafka instances.

Kafka为您提供了提交日志的主要好处-事件的不可变，有序记录-并具有与通用数据源集成，将数据写到其他系统(例如Postgres)以及对其进行操作和转换的能力流程。 有适用于大多数常见语言的SDK，如果您使用托管服务，则无需担心管理自己的Kafka集群。 而且，相信我，这不是一件容易的事。 看看Heroku如何管理他们庞大的Kafka实例队伍。

Logs photo by Oliver Paaske
Ringbinders photo by Viktor Talashuk
Truth photo by Magda Ehlers

原木照片由Oliver Paaske Ringbinders摄影： Viktor Talashuk 真相由Magda Ehlers摄