# legume 

[![Build status](https://builds.sr.ht/~ser/legume/.build.yml.svg)](https://builds.sr.ht/~ser/legume/.build.yml?) ★ [Releases](https://downloads.ser1.net/software/legume/) ★ [Changelog](https://hg.sr.ht/~ser/legume/browse/CHANGELOG.md?rev=tip) ★ [Issues](https://todo.sr.ht/~ser/legume) ★ [![Go Report Card](https://goreportcard.com/badge/ser1.net/legume)](https://goreportcard.com/report/ser1.net/legume) 

Legume is a distributed issue tracker CLI tool based on developer code comments
such as TODO and FIXME. It has no separate metadata or database and understands
several programming languages. Legume follows in the footsteps of other
distributed ticket tracking systems such as lentil, BugsEverywhere,
and ditz, but in all cases is an improvement in: speed; being VCS agnostic;
and/or requiring no external database and relying _only_ on code comments. It
also in many cases has features such as understanding unified diffs and
generating structured output, which allow it to better interoperate with other
tooling for e.g. generating changelogs.

Keeping notes in code is a venerable programmer tradition.  Notes of this sort
retain locality; they are reminders and warnings for code browsers, and retain
important meta-data about where to start. They integrate naturally with whatever
VCS the developer is using, and make it easy for a developer to remember to
remove the comment when the topic is addressed. They require no extra external
database.

Legume can be used alongside other, more formal, issue trackers which might
allow non-developers to submit tickets. I also use it, rather than a web-based
ticket tracker, on small, single-file scripts.

[The project home is here](https://sr.ht/~ser/legume/). [File bugs here](https://todo.sr.ht/~ser/legume). Send patches, or any other comments or questions, to [~ser/legume@lists.sr.ht](mailto:~ser/legume@lists.sr.ht)  There is a public chat available at [\#legume:matrix.org](https://app.element.io/#/room/#legume:matrix.org) ![](https://img.shields.io/matrix/gotop:matrix.org) ([matrix clients](https://matrix.to/#/#legume:matrix.org)).

Is Legume using the Sourceforge issue tracker ironic? Legume is for developers, and the only way to add issues is to add code comments. It doesn't have a public interface, so the issue tracker is for non-contributers. If you are interested in the history of Legume -- why I wrote it, the value of DITs -- I wrote a [blog about it](https://www.ser1.net/post/legume/) (also at gemini://ser1.net/post/legume.gmi).


# Features

The biggest selling feature of legume is that issues are embedded in code.
This has a number of benefits:

-  Issues can be place-marks for where developers think things
   are happening.
-  Commenting code with **TODO** and **FIXME** comments is a
   decades-old traditional developer habit
-  Integrated with version control at an intuitive level, reducing
   cognitive load. No external integration with VCS necessary.
-  No external DB for tickets. TODOs and FIXMEs are still relevant even
   if you stop using legume. Which also means...
-  legume is not strictly necessary for working with issues. It's a
   convenience tool. This means you don't have to build your workflow
   around legume, and nobody in your team *needs* to install legume. You
   can `find` and `grep` your way through issues.

Legume will also address other, non-developer, use cases by recognizing a
`todo.txt` file in the directory of execution. Legume will assume every line is
a separate TODO.  This allows easy integration with `todo.txt`, but also
provides a place for sticking issues that either aren't linked to source, or
come from a source other than a developer (e.g., web app). `todo.txt` syntax is
also supported in comments.

-   *(A)* priorities
-   *2017-03-20* "created" dates
-   *due:2017-03-20* "tagged" dates
-   *@contexts_
-   *+projects*
-   *key:value* tags
-   Multi-line issues (new issue, blank line, or non-comment breaks issue)

The list of programming languages that legume *should* understand is [here](https://hg.sr.ht/~ser/legume/browse/parse/syntaxes.go?rev=tip).

# Install

## Alpine & Arch

A package for Legume is in AUR; the binary installed is called `legume`. If you aren't also using the `peg` package (a parser generator for C), then I recommend aliasing `legume` to `leg`, since that's supposed to be the actual executable name.

```sh
yay -S legume
echo 'alias leg=legume' >> ~/.bashrc    # or .profile, or .cshrc, or .zshrc
```

If you want to build it yourself, you can either `go install` the package, or clone it and compile it manually. Check [the log](https://hg.sr.ht/~ser/legume/log) for the most recent bookmark (tag) and use it instead of `v1.3.3` in the example:

```sh
go install ser1.net/legume/cmd/leg@v1.3.3
```

**Note** "latest" doesn't work at the moment. This is an older project, and the combination of Mercurial bookmarks and tags confuses `go install`. Use a version number. This method doesn't install a manpage, but the `--help` documentation is pretty complete.

## Binares

[Binaries for OSX, Linux, and Windows are here](https://downloads.ser1.net/software/), and are signed with the GPG key [5E0D7ABD6668FDD1](https://meta.sr.ht/~ser.pgp) (available from hkp://hkps.pool.sks-keyservers.net). The Windows archives are zips; everyone else gets tarballs. The archives contain:

- The `leg` executable
- A manpage (`leg.1`, uncompressed)
- The plugins for [Kakoune](https://kakoune.org/) and [nvim](https://neovim.io/) (see the sections [kakoune](#kakoune) and [vim](#vim) for more info); there is no plugin for Helix because Helix sadly doesn't have a plugin system yet.
- This README.md
- The LICENSE
- leg-fzf, which is a bash helper script that allows you to work with legume-style tickets via fzf. Run it with `-h` at least once -- it tells you what key bindings are available, but also lists a couple caveats you should definitely read before using the script.

The tarballs extract into a directory called `legume`; the Windows zips just explode into the current directory, as do most Windows zips.

Here are the shell steps for DIY installation (adjust for the current release version):

```sh
# Set this to the appropriate OS, architecture, and version you want to fetch
TARBALL=leg_linux_amd64_v1.3.3.tar.zst
curl -L https://downloads.ser1.net/software/legume/${TARBALL}
tar -xf ${TARBALL}
cd legume
gzip leg.1
sudo install -Dm755 leg leg-fzf -t /usr/local/bin/
sudo install -Dm644 leg.1.gz -t /usr/local/share/man/man1/
sudo install -Dm644 LICENSE -t /usr/share/licenses/legume/
sudo install -Dm644 README.md -t /usr/share/doc/legume/
```

And then, if you're using Kakoune or NeoVim, follow the instructions in [kakoune](#kakoune) or [vim](#vim) as desired.

## Building

Here's a little install copy/paste snippet that should work on all Unix-ish OSes (including OSX); you'll need to have [Mercurial](https://www.mercurial-scm.org/) and [Go](https://golang.org) installed:

```sh
hg clone https://hg.sr.ht/~ser/legume && cd legume
VERSION=$(hg log -r tip --template '{bookmarks}')
go build -ldflags "-Xmain.Version=v${VERSION:-dev}" -o leg ./cmd/leg
gzip leg.1
sudo install -Dm755 leg leg-fzf -t /usr/local/bin/
sudo install -Dm644 leg.1.gz -t /usr/local/share/man/man1/
sudo install -Dm644 LICENSE -t /usr/share/licenses/legume/
sudo install -Dm644 README.md -t /usr/share/doc/legume/
```

## vim

**Note** I haven't used vim, except in brief encounters with servers, since early 2020. I have not kept up with vimscript or whether there's now different ways of installing things. YMMV.

A super-simple vim script is available; to install, copy the contents of
`legume.vim` into your vimrc.  It binds `<leader>lg` to execute `legume` with
the `-f vim` option and open the quickfix buffer.

## kakoune

**Note** I now haven't used kakoune _at all_ since mid-2023. Scripting, especially, was subject to frequent changes at the time I switched to Helix. YMMV.

[<img src="https://asciinema.org/a/J16Ca5N0iAOzMdyFE6VhyeV7I.png" alt="drawing" width="50%"/>](https://asciinema.org/a/J16Ca5N0iAOzMdyFE6VhyeV7I)

The [kakfile](https://hg.sr.ht/~ser/legume/raw/legume.kak) can be put in your `$XDG_CONFIG_HOME/kak/autoload` directory, and exposes some make-like functions & functionality. Kakoune's CWD should be the project directory.

- `leg` opens a buffer list of the issues
- `leg-next-todo` jumps to the next issue
- `leg-previous-todo` jumps to the previous

If you want, map these to shortcuts as you would for make, for example:

```
declare-user-mode leg
map global leg n ": leg-next-todo<ret>" -docstring "go to next issue"
map global leg p ": leg-previous-todo<ret>" -docstring "go to previous issue"
hook global BufCreate \*leg\* %{
    map global user g ": enter-user-mode leg<ret>" -docstring "legume issues"
}
hook global BufClose \*leg\* %{
    unmap global user g 
}
```

# Usage 

There's the usual `-h`/`--help`, but if you installed with a package
manager there should also be a more detailed man page (`man leg`, or `man legume`). If
you installed from source, there's a manpage in the top directory.

Comments must always start with a keyword, `TODO`, `FIXME`, `BUG`, or
`XXX`.  These key words are case sensitive, and they may optionally
be followed by a `:` (colon) which, if present, must be followed by a
space. After that, [todo.txt](http://todotxt.org/) formats are allowed. Example:

```
    // TODO 2017-03-20 (A) @con_example +proj_example due:2018-03-22 key:value Priorities, contexts, projects, created & due dates, and key/value pairs
```

Use the built-in help for options, such as filtering.

```
$ leg       # to list all todo/fixmes in a project
$ leg -d 5  # to list the details of item # 5
$ leg 5     # Same as above: for convenience, the `-d` may be ommitted
```

Filtering changes indexing. Because there is no canonical item numbering,
any filters that produce a list must be **also** used when using `-d`
(details).  For example,

```
$ leg -P test       # to ignore all files in the test/ directory
$ leg -P test 3     # use the same filter when accessing tickets
```

If the `--diff` flag is used, Legume reads from the stdin and processes it
as a diff.  In this mode, Legume counts any add (+) tickets as new, and
any removed (-) tickets as closed, and reports them this way.  This can
be used in conjunction with version control systems that support unified
diff output; For example, on the Legume repository itself:

```
➜  hg diff -r 4:98 | leg --diff
  1 REQ    NEW Include the time stamp in the report; removed use old version, add use new
  2 REQ    NEW filter on priority, category, and project; use meta-tags and -t
  3 BUG    NEW catch string lit escapes
  4 REQ CLOSED Support for STDIN
  5 REQ CLOSED Config file
  6 REQ CLOSED Support for unified diff
  7 REQ CLOSED Add test cases.
  8 REQ CLOSED Add test cases.
  9 REQ CLOSED Add test cases.
 10 REQ    NEW Add unit tests for Alias [component:ui]
 11 REQ    NEW Implement & add unit tests for Keywords [component:ui]
 12 REQ CLOSED Add test cases.
 13 REQ CLOSED Add test cases.
 14 REQ CLOSED Add test cases.
 15 REQ    NEW Parsing diffs seems to be broken
 16 REQ CLOSED Add test cases.
 17 INF    NEW refactor parsing to "consume-to-end"
```

This can be helpful when building change logs for a change set.

Obvious limitations result from how diff reports information; a changed
line is reported as a combination delete + add, which looks to Legume like
a close + open. In practice and over short spans, it works pretty well.

If you have commonly used arguments, you can put them in a file named
`.legrc` and `leg` will load them from there:

```
-P test
-P data
-f detail
```

`leg` looks for this file in the directory from which it is run.

## Performance samples

Legume has already satisfied the basic performance requirement; small projects
have sub-second parse times.  A future version may implement caching to bring
larger projects to this benchmark.

Project with 108 files, 14k lines, 15 todos:
```
leg .  0.04s user 0.03s system 81% cpu 0.090 total
```

Project with 992 files, 564k lines, 244 todos:
```
leg .  1.34s user 0.07s system 95% cpu 1.464 total
```

The Linux 5.7 kernel source tree, 64,309 files, 28,136,537 lines, 9,697 todos:
```
leg .  49.60s user 0.96s system 99% cpu 50.713 total
```

Legume was not designed for large projects such as the Linux kernel;
at that scale, there are likely many users and developers, and source
code comments won't be sufficient and a separate ticketing system is
probably more appropriate. Consequently, the current performance is
quite acceptable. Please report if the performance is impeding you;
I'd be interested to see a project that large where legume is still
providing value.

### Tree-sitter

This is an aside about alternative options for parsing; it's only
interesting as a data point about sourcecode parsing.

I attempted in one branch to use a "proper" multi-syntax parser, tree-sitter, via
the github.com/smacker/go-tree-sitter project. The 129,354 line
[bash/parser.c](https://github.com/smacker/go-tree-sitter/blob/master/bash/parser.c)
file, in the go-tree-sitter library itself, took several minutes to
walk. FFI was certainly a factor, but tree-sitter itself was pretty
slow parsing this file: the Rust text editor Helix also uses
tree-sitter, and is extremely laggy on even a decently fast
CPU. My bespoke parser processes the file in less than 0.08s (on my
CPU). Working with a cut-down file of 27kLOC (the 129kLOC file took
too long to run for me to want to benchmark) I got the
following results:

| Version              | Min    | Max    | Mean   | Runs |
| -------              | ---    | ---    | ----   | ---- |
| Bespoke              | 23.9ms | 24.9ms | 24.3ms | 113  |
| tree-sitter, visitor | 2.523s | 2.561s | 2.540s | 10   |
| tree-sitter, query   | 1.178s | 1.232s | 1.209s | 10   |

Notably, the performance of tree-sitter (or the Go binding) does not
scale linearly, either by walking the parse tree or using the query
API. Tree-parser parses a file 1/5th the size in 1/348th of the time. For
comparison:

|         | Size (lines)   | Bespoke time (ms)   | tree-sitter time (ms)  |
| ------- | -------------: | ------------------: | ---------------------: |
|         |         26,837 |                  24 |                  2,540 |
|         |        129,354 |                  74 |                421,040 |
| Factor  |          4.81x |               3.08x |                   168x |


Even if Go FFI were faster, tree-sitter is doing a lot more work parsing
and tokenizing, while my bespoke parser is only scanning for and extracting
comments -- it's doing **far** less work. Since the tree-sitter time
cost is prohibitive, legume sacrifices potentially better correctness
for speed.


# Caveats

**Note** Legume's executable is called `leg`. However, there's an
older project called [peg](http://piumarta.com/software/peg/) (a C
parser generator) that also has a `leg` command; to avoid coflicts in
distributions, I've named the executable `legume` in package managers. In
this document I use `leg`.

Legume is intended for a narrow problem space.  Large projects, such as the
Linux kernel (and certainly the boundary is much lower than that) are
certainly better served by a "real" issue tracking system.  For large code
bases, any tool that parses the entire code base on every invocation will be
too slow, even if a tool like legume didn't lack most of the features of a
sophisticated isuse tracking system.

Testing on languages other than Go has been limited.  Testing on Windows is
non-existent.

The diff feature absolutely has a number of limitations; diffs containing
multi-line TODOs where a line other than the first changed will be missed;
similarly, lines where only the first line changed will report partial
descriptions.  There's simply a limit to how much meta-information can be
interpreted from a diff.

At the time of this writing, despite the version number this legume
is young, and I'm the only user I know of.  There will be bugs.
Sorry about that.

## Limitations

The trade-offs of using a tool like Legume for ticket tracking, which
may or may not be considered limitations, include:

-   No unique IDs, which can make referencing tickets difficult or cumbersome
-   Very simple tickets. No attachments, no cross-references or dependencies,
    no robust commenting; history tracking limited to what's available in the
    VCS.
-   Efficiency. In a pure state, collating tickets requires walking directory
    trees and parsing source files. At present, small and even
    medium-sized projects have sub-second processing, and this is faster
    than making calls to a remote web service to fetch tickets. At some
    size, Legume will take too long to parse the source tree for it to
    be a pragmatic tool.

# Prior Art

Legume was inspired by
[lentil](http://ariis.it/static/articles/lentil/page.html) and follows
the philosophy of [todo.txt](http://todotxt.org/). Legume has goals
beyond merely duplicating lentil:

-   Accept diffs & STDIN. This allows some ability to track history by using
    the output of a VCS. E.g., `git diff | leg` would produce a
    list of all issues added and resolved, using diff markup (+/-) to indicate
    opened/closed
-   Improve performance
-   Support todo.txt syntax (priorities, dates, etc)

Below are some other distributed ticketing systems, many of which I've tried.
If legume isn't your cup of tea, maybe one of the ones listed below  will be
more suitable for your workflow.

-   [lentil](https://matej.ceplovi.cz/blog/current-state-of-the-distributed-issue-tracking.html),
    discussed below.
-   [Artemis](https://matej.ceplovi.cz/blog/current-state-of-the-distributed-issue-tracking.html).
    Separate bug DB (maildir); integrated with VCS (Mercurial, beta git
    support). Supports more traditional ticketing features, such as attachments
    and comments.  Only slightly younger than BugsEverywhere, the first commit
    was in 2007.
-   [b](https://matej.ceplovi.cz/blog/current-state-of-the-distributed-issue-tracking.html).
    Separate bug DB. Tightly coupled with Mercurial -- implemented as an
    HG extension. Supports more traditional ticketing features, such as
    assigning tickets to people.
-   [BugsEverywhere](http://bugseverywhere.org/).  Separate bug DB. Lots of
    features, including a web interface, email ticket communication, supporting
    multiple VSes, and most traditional ticketing features.  The grand-daddy of
    distributed ticketing systems; first commit was 2005!
-   [git-bug](https://github.com/MichaelMure/git-bug), tightly coupled to git.
-   The one with my favorite name, [ScmBug]()
-   [Fossil](), which is a kitchen-sink project; the VCS & bug tracker are very
    tightly coupled, but it _does_ have a distributed ticketing system, so it
    makes the list.
-   [SD](http://syncwith.us/sd/), which has integration with git and darcs, and
    keeps bugs in a SQLite database.
-   [ditz](https://github.com/jashmenn/ditz), which I used quite a bit way back when
    I was using [Darcs](http://darcs.net/) (a predecessor to both git & Mercurial).
-   [ticgit](https://github.com/schacon/ticgit/wiki), now unmaintained.
-   [ditrack](https://pypi.org/project/DITrack/); the web site appears to have
    reverted to the domain name provider.

Distributed ticket tracking has been discussed in numerous places:

-   [One discussion](http://adam.schmideg.net/2010/10/integrated-and-distributed-issue-tracking/)
    about distributed & integrated bug tracking.
-   [Another discussion](http://www.spmbook.com/tools/issue_tracking-cli.html)
    (and a list) of distributed issue tracking solutions.
-   A [2008 LWN article](https://lwn.net/Articles/281849/) about distributed
    issue tracking
-   A 2009 article about [DVCS and bug
    tracking](https://ericsink.com/entries/dbts_fossil.html), but which is
    really about Fossil.
-   A 2010 "[State of
    Affairs](https://matej.ceplovi.cz/blog/current-state-of-the-distributed-issue-tracking.html)"
    of distributed issue tracking.
-   A 2013 "[State of Affairs](https://www.jelmer.uk/distributed-bug-trackers.html)" of
    distributed bug trackers.

Most of these keep bug data in a separate database that's version tracked
alongside the code.  One of my requirements is to leverage code comments, and
Lentil is the only tracker I found that does that.  Some trackers keep bugs in
a database that are one or more text files, which is OK because the VCS will be
able to diff them efficiently; any tracker that keeps bugs in a binary DB (like
SQLite) that requires checking into the VCS is, IMO, a non-starter.

Lentil is written in Haskell, and although I do like Haskell, I wanted
something lower-impact to compile than Haskell. The core Haskell stack on Arch
Linux, including common libraries, is 183 packages and 1.9GB.  This is a heavy
lift for small systems.

Additionally, I'm less fluent these days in Haskell than Go, and one of the
issues I have with Lentil is performance.  I'm not advanced enough with Haskell
to be able to easily performance tune somebody else's code, and getting better
at performance tuning Haskell was not my main objective for this project;
I just wanted better speed and some additional functionality, and to get to the
functionality I'd have to have solved the performance issues first... it was
simply easier to implement a new tool.
