Blog
About Me
Todo aggregation with R4d
Simon Creek
2022-07-06 14:08

Todo aggregation for too many projects

When you have too many projects with plenty of todos. You often spend time shifting between those projects in between thinking which one I should implement first. Now the task itself is somewhat interesting experience, but wasting strokes just to remind the list that I saw half a minute ago is not a constructive behaviour.

I wanted to aggregate all todo information into a single interface that I don't have to type more than one command line. And here's how I did it with r4d.

R4d comes to rescue

Now r4d is not necessarily a sole solution for a text file aggregation. But I have no patience to tackle down a task into a unix-able chunks and throw each into a built-in core utils. Actually, I made r4d because I needed a thick swiss army knife for any text manipulation.

You can download r4d from github repository, and I highly recommend installing a 3.0 version because there has been lots of good improvement for 3.0 version. And this post is based on 3.0 version with many breaking changes from 2.0.

To install a r4d simply execute a cargo install command.

# In case you don't have "cargo" installed,
# go visit https://www.rust-lang.org/tools/install for related toolchains

# With github repository
cargo install r4d --git https://github.com/simhyeon/r4d --features \
binary --locked

# Or simply with crates.io registry
cargo install r4d --features binary --locked

Basic syntax of r4d

Now I doubt many would know what is r4d and how r4d's syntax looks like. So I'll introduce a r4d briefly.

Basically, r4d is a text oriented macro processor. R4d provides handful of built-in macros with ability to extensively define user macros.

You can create a macro with define macro.

% Define a macro with two arguments and body of $a_first() + $a_second()
$define(macro_name,a_first a_second=$a_first() + $a_second())

% Or simply define an empty macro like
$define(macro_name=)
$static(macro_name,)

After a macro is created, you can call a macro by simply typing a macro name followed by macro arguments if any.

$macro_with_args(1,2)
$macro_name()

I'll try to illustrate what each macro do as detailed as possible but you can always refer a repository for a detailed usage. Or use a --man flag to read a manual of a macro.

Now we will get into our usage without further ado.

Deciding information I need

All of my projects have a meta.md file inside docs directory which includes various scribbles, quick thoughts or todos. I need to extract only a todo part from the files. There could be several ways to do so but I chose relaying mechanics for easier configuration. I'll explain what a relaying is later.

My meta file had been looked like this,

### Header with some texts

* [ ] Do something
* [ ] Do this thing too
    * [ ] Some sub-todos for the upper todo
    - Some necessary descriptions for the status
* [ ] Some bug: I need this to be fixed

which I modified to as following


### Header with some texts

$define(todo_content=)
$relay(macro,todo_content)
* [ ] Do something
* [ ] Do this thing too
    * [ ] Some sub-todos for the upper todo
    - Some necessary descriptions for the status
* [ ] Some bug: I need this to be fixed
$halt()
% Macro attribute "^" trims newlines from expanded output
$tempout($todo_content^()$nl())
$exit()

The newer version of meta file includes pair of relay and halt macros. The relay macro redirects following texts into a relay target which is a macro called todo_content in this case. And relaying stops when halt macro is executed. And finally, saved content is written to a temporary file with tempout macro. Practically, my macro codes have following logics.

  • Create a container macro
  • Tell a macro processor to relay(redirect) all texts into the container macro
  • Stop relaying
  • Print saved texts into a temporary file
  • Exit processing

Now I can aggregate todos into a temporary file easily with the command

# We use lenient to enable macro override
# and also open file out permission.
# Lastly enable comment for better understanding of macro executions
rad meta.md --lenient -a fout --comment

Aggregated todos were saved in a temporary file. Which is /tmp/rad.txt in *nix OS. The result looks like a following

<!-- /tmp/rad.txt -->
* [ ] Do something
* [ ] Do this thing too
    * [ ] Some sub-todos for the upper todo
    - Some necessary descriptions for the status
* [ ] Some bug: I need this to be fixed

Wrapper macro to achieve expandability

Now "that" was what I want. However, using a raw macro code is not so ideal in terms of expandability. Let's define a wrapper macro that automates relaying and writing.

% File name : mac.r4d
% Create an empty container beforehand
% Define a pair to start macro and end macro

$define(cont=)
$define(todo_start=
    $repl(cont,$empty())
    $relay(macro,cont)
)
$define(todo_end=
    $halt()
    $tempout($cont^()$nl())
    $exit()
)

Previously "cont" macro was simply defined every time it was executed. Although this might be ok with file based executions or a lenient flag, but it is better to avoid macro override if possible. By defining a "cont" macro beforehand and replacing its content when macro starts, will prevent errors on strict mode.

Updated todo files look like this

$todo_start()
* [ ] Do something
* [ ] Do this thing too
    * [ ] Some sub-todos for the upper todo
    - Some necessary descriptions for the status
* [ ] Some bug: I need this to be fixed
$todo_end()

Now we can run a command like a following

# mac.r4d should come before meta.md 
# I used discard flag in this case, because the processing doesn't print to console
rad mac.r4d meta.md --lenient -a fout --discard

The result is exactly the same but now we don't have to edit all meta.md file but only a source file and changes will be applied to all todo files.

Now let's try to really aggregate multiple todos into a temporary file with multiple arguments. The result looks...

* [ ] New command
    * [ ] Import-option would be useful in execute context
    * [-] Drop : Not necessarily needed, THis is handled by command loop logic
    * [-] Add column loop : Not necessarily needed
    - Possibly...
        * [ ] Script command
        * [ ] Trim all + Strip Double quote ( This requires newer dcsv version )
        * [ ] Undo/Redo multiple times ( This needs cursorvec... maybe? )
* [ ] Make OPERATE order consistent
    - To work lv and rv's order doesn't matter

* [ ] Join functionality
* [ ] <OR> variant for predicate
- Currently all predicate are AND variant
* [ ] Evaluation process would be useful
* [ ] Count, average, sum
    * [ ] This is technically a sql function support
* [ ] Index method to query with indexes
    e.g. ) SELECT IN(2)

Utilizing input and "path" macros for headers

Welp. I couldn't know what belongs to what! I need to make a separator or better, a header for each todos! Let's add a header to macro with some optional codes.

% File name : mac.r4d
$define(cont=)
$define(todo_start=
    $repl(cont,$empty())
    $relay(macro,cont)
    $let(file_path,$input(true))
    $let(
        pr_name,
        $cut(/,-3,$file_path())
    )
    % This is markdown's link syntax
    $nl()# [$pr_name()]($file_path())$nl()
)
% todo_end has no changes

I used handful of new macros. Let me split them and give a brief explanation.

  • Let macro binds local macro. Local macros only span until outmost macro execution ends.
  • Input macro prints current processing input's name. It is either stdin or file name. Give optional true argument to print absolute path.
  • cut splits a text and cut a indexed item from it. In this case, input_path is "project_name/docs/meta.md" thus I needed to extract only the project's name part. Which is index by number -2.
  • nl macro prints newline. You can always use literal newline but nl macro makes it explicit and easy to configure formatting.

Essentially newly added codes extract project's name from current input's path and construct a markdown link from it.

Now a processed file looks like the following.

# [ced](/home/simon/programming/rust/ced/docs/meta.md)

* [ ] New command
	* [ ] Import-option would be useful in execute context
	* [-] Drop : Not necessarily needed, THis is handled by command loop logic
	* [-] Add column loop : Not necessarily needed
	- Possibly...
		* [ ] Script command
		* [ ] Trim all + Strip Double quote ( This requires newer dcsv version )
		* [ ] Undo/Redo multiple times ( This needs cursorvec... maybe? )

# [cindex](/home/simon/programming/rust/cindex/docs/meta.md)

* [ ] Make OPERATE order consistent
	- To work lv and rv's order doesn't matter

* [ ] Join functionality
* [ ] <OR> variant for predicate
- Currently all predicate are AND variant
* [ ] Evaluation process would be useful
* [ ] Count, average, sum
	* [ ] This is technically a sql function support
* [ ] Index method to query with indexes
	e.g. ) SELECT IN(2)

The file looks pretty good enough. I know where the todos belong and can go to original todos with vim's go-to-file command. But can we make it better?

Add a finished rate to a file, with regex of course

Now I want to see the total percentage of the todos per project. Let's get going.

% File name : mac.r4d
% ...
% todo_start is same
$define(todo_end=
    $halt()
    $let(
        total,
        $countl($grepl(^\\* \[.\], $cont^()))
    )
    $let(
        finished,
        $countl($grepl(^\\* \[x\], $cont^()))
    )
    $let(
        result,
        $if=(
            $total(),
            Finished : $finished() / $total() [$prec($eval($finished().0 / $total().0 * 100),0)%]
        )
    )
    % A little bit of newlines with horizontal line is better for readability
    $tempout($cont^()$nl(2)$result^()$nl(2)---$nl(2))
    $exit()
)
  • Grepl is like a coreutil grep. It finds regular expression from given content and prints matched lines back. In this case I matched either * [ ] or * [x] that starts from the start of the line. Because \* is part of a literal syntax, I escaped as \\* .
  • countl counts lines from given input.
  • if macro executes following expressions only when the condition is true
  • prec changes the precision of a number

Phew, that was mouthful. What these codes do is

  • Find all lines that is a first level todo and bind the counts into a total macro
  • Find all lines that is a first level finished todo and bind the counts into a finished macro
  • If total counts are not 0(false), then print the finished todos with its ratio from total todos

Now let's mark some todos as checked and see a result.

# [ced](/home/simon/programming/rust/ced/docs/meta.md)

* [ ] New command
	* [ ] Import-option would be useful in execute context
	* [-] Drop : Not necessarily needed, THis is handled by command loop logic
	* [-] Add column loop : Not necessarily needed
	- Possibly...
		* [ ] Script command
		* [ ] Trim all + Strip Double quote ( This requires newer dcsv version )
		* [ ] Undo/Redo multiple times ( This needs cursorvec... maybe? )

Finished : 0 / 1 [0%]

---

# [cindex](/home/simon/programming/rust/cindex/docs/meta.md)

* [ ] Make OPERATE order consistent
	- To work lv and rv's order doesn't matter

* [x] Join functionality
* [x] <OR> variant for predicate
- Currently all predicate are AND variant
* [ ] Evaluation process would be useful
* [ ] Count, average, sum
	* [ ] This is technically a sql function support
* [ ] Index method to query with indexes
	e.g. ) SELECT IN(2)

Finished : 2 / 6 [33%]

---

Now I can see how lazy I'm and I am happy with it.

Creating a TOC for aggregated file

For the last part, I want to make a toc for a file. Because there are several of projects, I want to get information about file's composition. Dont' worry, we will not do a fancy hardcore stuff but some naive text editing.

$define(format_toc=# TOC$nl(2))
$define(cont=)
% Use trim input attribute to apply triml
$define=(todo_start=
    $repl(cont,$empty())
    $relay(macro,cont)
    $let(file_path,$input(true))
    $let(
        pr_name,
        $cut(/,-3,$file_path())
    )
    % Append a formatted toc list into a "format_toc" macro
    % [$pr_name()](#$pr_name()) is markdown's toc syntax
    $append(format_toc,* [$pr_name()](#$pr_name())$nl())
    # [$pr_name()]($file_path())
)
% ---
% todo_end macro is same
% ---
$tempout(\*$format_toc()$escape()*\$nl())

I also changed several things. First, I made a new container called format_toc and added a new code which appends toc list item into the container.

After macro definitions the script writes \*$format_toc()$escape()*\ into a temporary file. Now the argument text is wrapped in literal syntax which means it is not expanded but will be put as literal form inside temporary file. Temp file will have a line $format_toc()$escape() before every todo content pasted into.

And we will make a new script called agg.r4d with following single line.

$readto($temp(),final.md)

What this does is simple, read from temporary file and put into a final.md. And finally if we run a command, when a processor re-reads temporary file, format_toc will be expanded into a proper table of contents. Processor will stop expanding after escape macro. We don't need to expand macros after format_toc because everything is already expanded.

# the order is important
# - definition file : mac.r4d
# - source files    : multiple docs/meta.md file
# - aggregator      : mac.r4d
# We need to give fin permission because we are now reading from temporary file
rad mac.r4d $SOME_PATH $OTHER_PATH agg.r4d -a fout+fin --lenient

final.md will be 

# TOC

* [ced](#ced)
* [cindex](#cindex)

# [ced](/home/simon/programming/rust/ced/docs/meta.md)

* [ ] New command
	* [ ] Import-option would be useful in execute context
	* [-] Drop : Not necessarily needed, THis is handled by command loop logic
	* [-] Add column loop : Not necessarily needed
	- Possibly...
		* [ ] Script command
		* [ ] Trim all + Strip Double quote ( This requires newer dcsv version )
		* [ ] Undo/Redo multiple times ( This needs cursorvec... maybe? )

Finished : 0 / 1 [0%]

---

# [cindex](/home/simon/programming/rust/cindex/docs/meta.md)

* [ ] Make OPERATE order consistent
	- To work lv and rv's order doesn't matter

* [ ] Join functionality
* [ ] <OR> variant for predicate
- Currently all predicate are AND variant
* [ ] Evaluation process would be useful
* [ ] Count, average, sum
	* [ ] This is technically a sql function support
* [ ] Index method to query with indexes
	e.g. ) SELECT IN(2)

Finished : 0 / 6 [0%]

---

The long journey is finally over. Hurray!

Conclusion

R4d provides very powerful text manipulation macros. From tedious tasks such as aggregating todo objects to automating documents creation, r4d can tackle a complex job with a single contained script. Although the syntax or all kinds of macros look overwhelming at the first time; you'll love expressiveness the r4d provides at the end.

My real script

Ok the previous examples were simplified versions of my real script. Because I wanted to make the progress more linear rather than a macro bombarding. If you want to see how a real script looks like, go see my gist