Maximilian Scholz

Edge Cases

Mon, 13 Apr 2020 19:04:01 +0200

Some edge cases and cautionary examples on using Markdown for writing content using this theme. In particular, list syntax can really knot things up.

Mathjax improperly parsing greater and less than and ampersands inside blocks

The mathjax HTML <head> scripts have been modified to properly render block style mathjax expressions inside a $$ ... $$ set of character pairs, using the standard Kramdown parser. Some examples sent to me by Quxiaofeng are now parsing correctly, I believe.

This code:

$$
  D = \left(\begin{matrix}
  1 & -1 & & & & \\
  &    & \cdots &   & \\
  &    &        & 1 & -1
 \end{matrix}
 \right)
$$

yields this:

\[D = \left(\begin{matrix} 1 & -1 & & & & \\ & & \cdots & & \\ & & & 1 & -1 \end{matrix} \right)\]

Other examples from the wikia Tex reference:

\[\begin{matrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{matrix}\] \[\begin{bmatrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{bmatrix}\] \[\begin{Bmatrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{Bmatrix}\] \[\begin{vmatrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{vmatrix}\] \[\begin{Vmatrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{Vmatrix}\] \[\begin{Vmatrix} \alpha& \beta^{*}\\ \gamma^{*}& \delta \end{Vmatrix}\]

However, a problem still exists for inline matrix notation, from an example here:

A matrix in text must be set smaller: $\bigl(\begin{smallmatrix}a & b \\ c & d\end{smallmatrix} \bigr)$ to not increase leading in a portion of text. The way this inline matrix is written is: $$ \bigl(\begin{smallmatrix}a & b \\ c & d\end{smallmatrix} \bigr) $$

Edge Case 1 from Quxiaofeng:

No blank lines between Markdown list items

The issue arises when sidenotes and marginnotes are put into list items. As mentioned in the main documentation page, lists can be problematic not only for semantic clarity, but also because they can creating formatting issues. For example:

Split Bregman iteration Goldstein, T. and Osher, S. (2009). The split Bregman method for l1-regularized problems. SIAM J. Img. Sci., 2:323-343.
Dykstra’s alternating projection algorithm Dykstra, R. L. (1983). An algorithm for restricted least squares regression. J. Amer. Statist. Assoc., 78(384):837-842.
Proximal point algorithm applied to the dual
Numerous applications in statistics and machine learning: lasso, gen. lasso, graphical lasso, (overlapping) group lasso, …
Embraces distributed computing for big data Boyd, S., Parikh, N., Chu, E., Peleato, B., and Eckstein, J. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. learn., 3(1):1-122.

Why this matters

Notice how the sidenotes display properly, but the fact that sidenotes have more display ‘volume’ than the list items themselves causes a horizontal mismatch between the sidenote item’s number and its corresponding list item.

Please note that there must be no blank lines between your list items. This is due to a really strange thing about the Jekyll Markdown engine I have never noticed before. If you have a list, and you put a blank line between the items like this:

  + list item 1

  + list item 2

It will create an html tag structure like this:

<ul>
   <li>
      <p>list item 1</p>
  </li>
  <li>
      <p>list item 2</p>
   </li>
</ul>

Which totally goofs up the layout CSS.

However, if your Markdown is this:

    + list item 1
    + list item 2

It will create a tag structure like this:

<ul>
   <li>list item 1</li>
   <li>list item 2</li>
</ul>

Here is the same content as above, with a blank line separating the list items. Notice how the sidenotes get squashed into the main content area:

Remarks on ADMM version 2 - one blank line between Markdown list items

Related algorithms

Split Bregman iteration Goldstein, T. and Osher, S. (2009). The split Bregman method for l1-regularized problems. SIAM J. Img. Sci., 2:323-343.
Dykstra’s alternating projection algorithm Dykstra, R. L. (1983). An algorithm for restricted least squares regression. J. Amer. Statist. Assoc., 78(384):837-842.
Proximal point algorithm applied to the dual
Numerous applications in statistics and machine learning: lasso, gen. lasso, graphical lasso, (overlapping) group lasso, …

Liquid tag parsing strangeness

Example of the proper way to write an url inside a Liquid full-width image tag.

This code: {% fullwidth "assets/img/rhino.png" "Tuftes pet rhino (via <a href=\"//www.edwardtufte.com/tufte/\">Edward Tufte</a>)" %}

produces the following image with a title. Notice that I have had to escape the double quotes in the HTML with a backslash. Also, the example above leaves out the single quote in ‘Tufte’s” because of the topsy-turvy way that you have to escape the escapes in code sections that are used for illustrative purposes in this text. Bottom line is that there are occasionally some odd interactions between the Markdown parser, custom Liquid tags and HTML.

Tufte’s pet rhino (via Edward Tufte)

Tufte-style Jekyll blog

Mon, 13 Apr 2020 11:46:04 +0200

The Tufte Jekyll theme is an attempt to create a website design with the look and feel of Edward Tufte’s books and handouts. Tufte’s style is known for its extensive use of sidenotes, tight integration of graphics with text, and well-set typography. The idea for this project is essentially cribbed wholesale from Tufte and R Markdown’s Tufte Handout formatSee tufte-latex.github.io/tufte-latex/ and rmarkdown.rstudio.com/tufte_handout_format This page is an adaptation of the Tufte Handout PDF.

Jekyll customizations

This Jekyll blog theme is based on the github repository by Edward Tufte here, which was orginally created by Dave Leipmann, but is now labeled under Edward Tufte’s moniker. I borrowed freely from the Tufte-CSS repo and have transformed many of the typographic and page-structural features into a set of custom Liquid tags that make creating content using this style much easier than writing straight HTML. Essentially, if you know markdown, and mix in a few custom Liquid tags, you can be creating a website with this document style in short order.

The remainder of this sample post is a self-documenting survey of the features of the Tufte-Jekyll theme. I have taken almost all of the sample content from the Tufte-css repo and embedded it here to illustrate the parity in appearence between the two. The additional verbiage and commentary I have added is to document the custom Liquid markup tags and other features that are bundled with this theme.

The SASS settings file

I have taken much of the actual Tufte-css files and modified them as necessary to accomodate the needs inherent in creating a Jekyll theme that has additional writing aids such as the Liquid tags. I have also turned the CSS file into a SASS file (the .scss type). This means that you can alter things like font choices, text color, background color, and underlining style by changing values in this file. When the Jekyll site is built using jekyll build the settings in this file will be compiled into the customized CSS file that the site uses.

This file looks like this:

/* This file contains all the constants for colors and font styles */

$body-font:   ETBembo, Palatino, "Palatino Linotype", "Palatino LT STD", "Book Antiqua", Georgia, serif;
// Note that Gill Sans is the top of the stack and corresponds to what is used in Tufte's books
// However, it is not a free font, so if it is not present on the computer that is viewing the webpage
// The free Google 'Lato' font is used instead. It is similar.
$sans-font:  "Gill Sans", "Gill Sans MT", "Lato", Calibri, sans-serif;
$code-font: Consolas, "Liberation Mono", Menlo, Courier, monospace;
$url-font: "Lucida Console", "Lucida Sans Typewriter", Monaco, "Bitstream Vera Sans Mono", monospace;
$text-color: #111;
$bg-color: #fffff8;
$contrast-color: #a00000;
$border-color: #333333;
$link-style: color; // choices are 'color' or 'underline'. Default is color using $contrast-color set above

Any of these values can be changed in the _sass/_settings.scss file before the site is built. The default values are the ones from tufte-css.

Fundamentals

Color

Although paper handouts obviously have a pure white background, the web is better served by the use of slightly off-white and off-black colors. I picked #fffff8 and #111111 because they are nearly indistinguishable from their ‘pure’ cousins, but dial down the harsh contrast. Tufte’s books are a study in spare, minimalist design. In his book The Visual Display of Quantitative Information, he uses a red ink to add some visual punctuation to the buff colored paper and dark ink. In that spirit, links are styled using a similar red color.

Headings

Tufte CSS uses <h1> for the document title, <p> with class code for the document subtitle, <h2> for section headings, and <h3> for low-level headings. More specific headings are not encouraged. If you feel the urge to reach for a heading of level 4 or greater, consider redesigning your document:

[It is] notable that the Feynman lectures (3 volumes) write about all of physics in 1800 pages, using only 2 levels of hierarchical headings: chapters and A-level heads in the text. It also uses the methodology of sentences which then cumulate sequentially into paragraphs, rather than the grunts of bullet points. Undergraduate Caltech physics is very complicated material, but it didn’t require an elaborate hierarchy to organize.

http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0000hB

As a bonus, this excerpt regarding the use of headings provides an example of using block quotes. Markdown does not have a native <cite> shorthand, but real html can be sprinkled in with the Markdown text. In the previous example, the <cite> was preceded with a single return after the quotation itself. The previous blockquote was written in Markdown thusly:

[It is] notable that the Feynman lectures (3 volumes) write about all of physics in 1800 pages, using only 2 levels of hierarchical headings: chapters and A-level heads in the text. It also uses the methodology of *sentences* which then cumulate sequentially into *paragraphs*, rather than the grunts of bullet points. Undergraduate Caltech physics is very complicated material, but it didn’t require an elaborate hierarchy to organize.
<cite>[http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0000hB](http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0000hB)</cite>

In his later books http://www.edwardtufte.com/tufte/books_be , Tufte starts each section with a bit of vertical space, a non-indented paragraph, and sets the first few words of the sentence in small caps. To accomplish this using this style, enclose the sentence fragment you want styled with small caps in a custom Liquid tag called ‘newthought’ like so:

{% newthought 'In his later books' %}

Text

In print, Tufte uses the proprietary Monotype BemboSee Tufte’s comment in the Tufte book fonts thread. font. A similar effect is achieved in digital formats with the now open-source ETBembo, which Tufte-Jekyll supplies with a @font-face reference to a .ttf file. Thanks to Linjie Ding, italicized text uses the ETBembo Italic font instead of mechanically skewing the characters. In case ETBembo somehow doesn’t work, Tufte CSS degrades gracefully to other serif fonts like Palatino and Georgia. Notice that Tufte CSS includes separate font files for bold (strong) and italic (emphasis), instead of relying on the browser to mechanically transform the text. This is typographic best practice. It’s also really important. Thus concludes my unnecessary use of em and strong for the purpose of example.

Code snippets ape GitHub’s font selection using Microsoft’s Consolas and the sans-serif font uses Tufte’s choice of Gill Sans. Since this is not a free font, and some systems will not have it installed, the free google font Lato is designated as a fallback.

Epigraphs

The English language . . . becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts.
George Orwell, "Politics and the English Language"

For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.
Richard P. Feynman, “What Do You Care What Other People Think?”

If you’d like to introduce your page or a section of your page with some quotes, use epigraphs. The two examples above show how they are styled. Epigraph elements are modeled after chapter epigraphs in Tufte’s books (particularly Beautiful Evidence). The Tufte-css gitub repository has detailed instructions on how to achieve this using HTML elements. As an easier alternative, the Tufte-jekyll theme uses custom Liquid tag pairs that allow the writer to embed elements such as epigraphs in the middle of the regular Markdown text being edited.

In order to use an epigraph in a page or section, type this:

{% epigraph 'text of citation' 'author of citation' 'citation source' %}

to produce this:

I do not paint things, I paint only the differences between things.
Henri Matisse, Henri Matisse Dessins: thèmes et variations, 1943

"How did you go bankrupt?" Two ways. Gradually, then suddenly.
Ernest Hemingway, "The Sun Also Rises"

Lists

Tufte points out that while lists have valid uses, they tend to promote ineffective writing habits due to their “lack of syntactic and intellectual discipline”. He is particularly critical of hierarchical and bullet-pointed lists. So before reaching for an HTML list element, ask yourself:

Does this list actually have to be represented using an HTML ul or ol element?
Would my idea be better expressed as sentences in paragraphs?
Is my message causally complex enough to warrant a flow diagram instead?

This is but a small subset of a proper overview of the topic of lists in communication. A better way to understand Tufte’s thoughts on lists would be to read “The Cognitive Style of PowerPoint: Pitching Out Corrupts Within,” a chapter in Tufte’s book Beautiful Evidence, excerpted at some length by Tufte himself on his website. The whole piece is information-dense and therefore difficult to summarize. He speaks to web design specifically, but in terms of examples and principles rather than as a set of simple do-this, don’t-do-that prescriptions. It is well worth reading in full for that reason alone.

For these reasons, Tufte CSS encourages caution before reaching for a list element, and by default removes the bullet points from unordered lists.

Figures

Margin Figures

⊕
F.J. Cole, “The History of Albrecht Dürer’s Rhinoceros in Zoological Literature,” Science, Medicine, and History: Essays on the Evolution of Scientific Thought and Medical Practice (London, 1953), ed. E. Ashworth Underwood, 337-356. From page 71 of Edward Tufte’s Visual Explanations.

Images and graphics play an integral role in Tufte’s work. To place figures in the margin, use the custom margin figure liquid tag included in the _plugins directory like so:

{% marginfigure 'mf-id-whatever' 'assets/img/rhino.png' 'F.J. Cole, “The History of Albrecht Dürer’s Rhinoceros in Zoological Literature,” *Science, Medicine, and History: Essays on the Evolution of Scientific Thought and Medical Practice* (London, 1953), ed. E. Ashworth Underwood, 337-356. From page 71 of Edward Tufte’s *Visual Explanations*.' %}.

Note that this tag has three parameters. The first is an arbitrary id. This parameter can be named anything as long as it is unique to this post. The second parameter is the path to the image. And the final parameter is whatever caption you want to be displayed with the figure. All parameters must be enclosed in quotes for this simple liquid tag to work!

In this example, the Liquid marginfigure tag was inserted before the paragraph so that it aligns with the beginning of the paragraph. On small screens, the image will collapse into a small symbol: ⊕ at the location it has been inserted in the manuscript. Clicking on it will open the image.

Full Width Figures

If you need a full-width image or figure, another custom liquid tag is available to use. Oddly enough, it is named ‘fullwidth’, and this markup:

{% fullwidth 'assets/img/napoleons-march.png' 'Napoleon's March *(Edward Tufte’s English translation)*' %}

Yields this:

Napoleon’s March (Edward Tufte’s English translation)

Main Column Figures

Besides margin and full width figures, you can of course also include figures constrained to the main column. Yes, you guessed it, a custom liquid tag rides to the rescue once again:

{% maincolumn 'assets/img/export-imports.png' 'From Edward Tufte, *Visual Display of Quantitative Information*, page 92' %}

yields this:

From Edward Tufte, Visual Display of Quantitative Information, page 92

Sidenotes and Margin notes

One of the most prominent and distinctive features of Tufte’s style is the extensive use of sidenotes and margin notes. Perhaps you have noticed their use in this document already. You are very astute.

There is a wide margin to provide ample room for sidenotes and small figures. There exists a slight semantic distinction between sidenotes and marginnotes.

Sidenotes

SidenotesThis is a sidenote and displays a superscript display a superscript. The sidenote Liquid tag contains two components. The first is an identifier allowing the sidenote to be targeted by the twitchy index fingers of mobile device users so that all the yummy sidenote goodness is revealed when the superscript is tapped. The second components is the actual content of the sidenote. Both of these components should be enclosed in single quotes. Note that we are using the CSS ‘counter’ trick to automagically keep track of the number sequence on each page or post. On small screens, the sidenotes disappear and when the superscript is clicked, a side note will open below the content, which can then be closed with a similar click. Here is the markup for the sidenote at the beginning of this paragraph:

{% sidenote 'sn-id-whatever' 'This is a sidenote and *displays a superscript*'%}

Margin notes

Margin notes ⊕This is a margin note without a superscript are similar to sidenotes, but do not display a superscript. The marginnnote Liquid tags has the same two components as the sidenote tag. Anything can be placed in a margin note. Well, anything that is an inline element. Block level elements can make the Kramdown parser do strange things. On small screens, the margin notes disappear and this symbol: ⊕ pops up. When clicked, it will open the margin note below the content, which can then be closed with a similar click. The Markdown content has a similar sort of markup as a sidenote, but without a number involved:

{% marginnote 'mn-id-whatever' 'This is a margin note *without* a superscript' %}

Equations

The Markdown parser being used by this Jekyll theme is Kramdown, which contains some built-in Mathjax support. Both inline and block-level mathematical figures can be added to the content.

For instance, the following inline sequence:

When $a \ne 0$, there are two solutions to $ax^2 + bx + c = 0$

is written by enclosing a Mathjax expression within a matching pair of double dollar signs: $$:

When $$ a \ne 0 $$, there are two solutions to $$ ax^2 + bx + c = 0 $$

Similarly, this block-level Mathjax expression:

\[x = {-b \pm \sqrt{b^2-4ac} \over 2a}\]

is written by enclosing the expression within a pair of $$ with an empty line above and below:

$$ x = {-b \pm \sqrt{b^2-4ac} \over 2a} $$

You can get pretty fancy, for instance, the wave equation’s nabla is no big thing:

\[\frac{\partial^2 y}{\partial t^2}= c^2\nabla^2u\]

All of the standard L^aT_eX equation markup is available to use inside these block tags.

Please note that the block-level Mathjax expressions must be on their own line, separated from content above and below the block by a blank line for the Kramdown parser and the Mathjax javascript to play nicely with one another.

The Mathjax integration is tricky, and some things such as the inline matrix notation simply do not work well unless allowances are made for using the notation for a small matrix. Bottom line: If you are using this to document mathematics, be super careful to isolate your L^aT_eX blocks by blank lines!

Tables

Tables are, frankly, a pain in the ass to create. That said, they often are one of the best methods for presenting data. Tabular data are normally presented with right-aligned numbers, left-aligned text, and minimal grid lines.

Note that when writing Jekyll Markdown content, there will often be a need to get some dirt under your fingernails and stoop to writing a little honest-to-god html. Yes, all that hideous <table>..<thead>..<th> nonsense. And you must wrap the unholy mess in a <div class="table-wrapper"> tag to ensure that the table stays centered in the main content column.

Tables are designed with an overflow:scroll property to create slider bars when the viewport is narrow. This is so that you do not collapse all your beautiful data into a jumble of letters and numbers when you view it on your smartphone.

⊕Table 1: A table with default style formatting

Content and tone of front-page articles in 94 U.S. newspapers, October and November, 1974	Number of articles	Percent of articles with negative criticism of specific person or policy
Watergate: defendants and prosecutors, Ford’s pardon of Nixon	537	49%
Inflation, high cost of living	415	28%
Government competence: costs, quality, salaries of public employees	322	30%
Confidence in government: power of special interests, trust in political leaders, dishonesty in politics	266	52%
Government power: regulation of business, secrecy, control of CIA and FBI	154	42%
Crime	123	30%
Race	103	25%
Unemployment	100	13%
Shortages: energy, food	68	16%

This is not the One True Table. Such a style does not exist. One must craft each data table with custom care to the narrative one is telling with that specific data. So take this not as “the table style to use”, but rather as “a table style to start from”. From here, use principles to guide you: avoid chartjunk, optimize the data-ink ratio (“within reason”, as Tufte says), and “mobilize every graphical element, perhaps several times over, to show the data.Page 139, The Visual Display of Quantitative Information, Edward Tufte 2001. Furthermore, one must know when to reach for more complex data presentation tools, like a custom graphic or a JavaScript charting library.

As an example of alternative table styles, academic publications written in L^aT_eX often rely on the booktabs package to produce clean, clear tables. Similar results can be achieved in Tufte CSS with the booktabs class, as demonstrated in this table:

⊕Table 2: A table with booktabs style formatting

Items
Animal	Description	Price ($)
Gnat	per gram	13.65
	each	0.01
Gnu	stuffed	92.50
Emu	stuffed	33.33
Armadillo	frozen	8.99

The table above was written in HTML as follows:

<div class="table-wrapper">
<table class="booktabs">
          <thead>
            <tr><th colspan="2" class="cmid">Items</th><th class="nocmid"></th></tr>
            <tr><th class="l">Animal</th><th>Description</th class="r"><th>Price ($)</th></tr>
          </thead>
          <tbody>
            <tr><td>Gnat</td>     <td>per gram</td><td class="r">13.65</td></tr>
            <tr><td></td>         <td>each</td>    <td class="r">0.01</td></tr>
            <tr><td>Gnu</td>      <td>stuffed</td> <td class="r">92.50</td></tr>
            <tr><td>Emu</td>      <td>stuffed</td> <td class="r">33.33</td></tr>
            <tr><td>Armadillo</td><td>frozen</td>  <td class="r">8.99</td></tr>
          </tbody>
</table>
</div>

I like this style of table, so I have made it the default for unstyled tables. This allows use of the Markdown Extra features built into the Kramdown parser. Here is a table created using the Markdown Extra table syntax to make a nice table which has the side benefit of being human readable in the raw Markdown file:

⊕Table 3: a table created with Markdown Extra markup using only the default table styling

	mpg	cyl	disp	hp	drat	wt
Mazda RX4	21	6	160	110	3.90	2.62
Mazda RX4 Wag	21	6	160	110	3.90	2.88
Datsun 710	22.8	4	108	93	3.85	2.32
Hornet 4 Drive	21.4	6	258	110	3.08	3.21
Hornet Sportabout	18.7	8	360	175	3.15	3.44
Valiant	18.1	6	160	105	2.76	3.46

Using the following Markdown formatting:

|                 |mpg  | cyl  |  disp  |   hp   |  drat  | wt  |
|:----------------|----:|-----:|-------:|-------:|-------:|----:|
|Mazda RX4        |21   |6     |160     |110     |3.90    |2.62 |
|Mazda RX4 Wag    |21   |6     |160     |110     |3.90    |2.88 |
|Datsun 710       |22.8 |4     |108     |93      |3.85    |2.32 |
etc...

The following is a more simple table, showing the Markdown-style table markup. Remember to label the table with a marginnote Liquid tag, and you must separate the label from the table with a single blank line. This markup:

{% marginnote 'Table-ID4' 'Table 4: a simple table showing left, center, and right alignment of table headings and data'  %}

|**Left** |**Center**|**Right**|
|:--------|:--------:|--------:|
 Aardvarks|         1|$3.50
       Cat|   5      |$4.23
  Dogs    |3         |$5.29

Yields this table:

⊕Table 4: a simple table showing left, center, and right alignment of table headings and data

Left	Center	Right
Aardvarks	1	$3.50
Cat	5	$4.23
Dogs	3	$5.29

Code

Code samples use a monospace font using the ‘code’ class. The Kramdown parser has the ‘GFM’ option enabled, which stands for ‘Github Flavored Markdown’, and this means that both inline code such as #include <stdio.h> and blocks of code can be delimited by surrounding them with 3 backticks:

(map tufte-style all-the-things)

is created by the following markup:

```(map tufte-style all-the-things)```

To get the code highlighted in the language of your choice like so:

module Jekyll
  class RenderFullWidthTag < Liquid::Tag
  require "shellwords"

    def initialize(tag_name, text, tokens)
      super
      @text = text.shellsplit
    end

    def render(context)
      "<div><img class='fullwidth' src='#{@text[0]}'/></div> " +
      "<p><span class='marginnote'>#{@text[1]}</span></p>"
    end
  end
end

Liquid::Template.register_tag('fullwidth', Jekyll::RenderFullWidthTag)

Enclose the code block in three backticks, followed by a space and then the language name, like this:

 ``` ruby
    module Jekyll
    blah, blah...
   ```

GSoC 2017 Postmortem Part 2: Org Application

Mon, 13 Nov 2017 00:00:00 +0100

In the beginning, there was an organization application. And with it, the need for someone responsible. That one was me. I always wanted to help with GSoC but felt that my coding skills and codebase knowledge were not good enough to be a mentor as good as I’d like to be. So I followed the call for main admin and on the story went.

Google Application Form

The most obvious piece of the application process is the application form every organization has to fill on the GSoC page. The problems I encountered while drafting the first version of our org profile and “Ten Reasons Why You Should Take coala”-text were essentially the same as for every application I had ever written. How does one sell themselves without making things up.

Although coala had two years of GSoC experience, those were as a sub org under the PSF umbrella. This year should be the first under our own flag, so everything had to be perfect. Again and again I asked for feedback on my drafts and input on things where I didn’t know enough about. Both Lasse and John were very helpful, both with plenty more GSoC experience than I have. We felt that we were in a pretty good position. Our community has a strong connection with the GSoC program, many of our long time members were students once upon a time. We had a total of 10 students over the two years with PSF, and from what I have heard, they were more than pleased with our performance as an organization, both the students and the PSF. The only thing that we felt went against us was us being a novice independent org, and us not having failed any of our students so far. Yup, this is a thing. From what I understood, Google expects around 20% failure rate across all organizations. If an org deviates from that too much for too long, then they are either too strict with their students, not strict enough, or there is another problem with their process. We actually were quite worried about this point but from what I have learned, there is no real reason to worry about this. Document what you and your students are doing and as long as you have good reasons for the decisions you are making, everything should be fine in the end. About the novice part, there wasn’t really anything we could do besides pointing to our past GSoC experience and hoping that Terri Oda, a PSF GSoC admin, would vouch for us.

Projects Page

Another, big, part of the application is the projects page. It is a list of project ideas that students can apply for. Although it is encouraged that students come up with their own project, most applications will come from the project list. And a nice project list is a must have, from what we were told, to get into GSoC. We felt that the project page should be the most well fleshed out part of our application, as it is actually useful during the following parts of the program and offers a lot of value to the students and us when the student application phase comes around. This lead to our web frontend hero Hemang building our new and awesome Project Page. Before this, we just used a GitHub wiki page with some project drafts but the new page was much easier to use, gave a better overview about the projects and it gives us the possibility to add features to it in the future. During this early phase, we included the old projects from the wiki and asked mentors and community members for other projects ideas they might be interested in mentoring. For those who haven’t read any of the project ideas on our page, the structure always is something like this: Introduction and motivation, sometimes followed by some additional information about how to solve the problem if we already had some idea about it. Then we show a time line following the GSoC 4 phase model, that holds some milestones and gives a rough idea about when to deliver the single parts. Finally, we link all additional information we have about the project, like issues or cEPs, add mentors as contact persons, and some more meta information. In addition to the project ideas the page also holds a faq section, where we started to collect questions we received again and again or just felt were important. We actually tried to let the student’s add the questions and answers to the faq themselves to include them into our process and also to make sure they actually understood our answer and thus to double check if our answer was good enough or if maybe we need a better explanation. We had close to twenty project idea drafts at this point and felt that we were more than ready to hand everything in to Google and be accepted. Before we move on to the next part, I want to point out that we also built the projects page so other orgs could use it and we have received some good feedback on it.

FOSSDEM and Feedback from Stephanie

In February, a few weeks before the application deadline, we went to FOSSDEM with coala. We had a stand where we promoted what we did but most important, our stand was right next to the GSoC stand. A great opportunity to get feedback and ask some questions we figured, and so I walked over in a calmer minute to show Stephanie what we had so far. Starting with the projects page, she liked it a lot. She liked the idea of it and how it was structured and could help to introduce students even easier. The only improvement she asked for with it were the project descriptions. Flesh out the motivation for most projects and just write a little more to make it easier for students was what I took away for the projects age. But overall this part gave me a very good feeling!

Then we talked about our chances to get into the program with the experience we had under our belt and our expectations and wishes for slots we would get. As I said before, we had 10 students over the last two seasons. First 2, then 8. We felt, that we could probably grow some more with the mentor power we had amassed but also didn’t expect too much as we had some challenges coming up, that we didn’t have the last two times. What I asked was probably something along the lines of “how high are our chances to get around 12 students this year with what I just showed you + improvements I’ll implement?”. And here is where the not-so-fun part started. She essentially looked at me as if I asked for a million dollars out of nowhere. Turns out, that first time organizations usually get 1 or 2 students. And even with our past experience as a sub org, we kind of counted as a first time org. I pointed out that we had 8 students under PSF last year and received great feedback from their admins for our work and Stephanie couldn’t believe that we did get such an enormous number of students as such a young org. I came out of that part of our conversation with the feeling that we’d probably not get more than 4 slots no matter what we did. Not a great takeaway.

Implementing Feedback

I rewrote a lot of our application texts to focus more on the things Stephanie pointed out as important factors after we returned but the main part of work I did was essentially rewriting all twenty something project proposals we had. Although I wrote some of them myself, most of them were actually written by students or mentors and I did all the review work. The reason or this was that even though I probably would have been faster just writing everything myself, I wouldn’t be able to get enough reviews to merge everything in time. So the solution was to poke students that were interested in the projects and mentors to rewrite it and me reviewing it.

During this time I also struggled with the expected slots we would get. While yes, we would finally be our own organization, with all the pros that would bring with it long term, it also felt like a step backwards to loose students slots compared to last year. But the fact that we couldn’t grow further under the PSF and had plans to act as an umbrella for smaller linter projects made the investment in our long term growth worth it in our eyes. I should probably add that we had more than 5 newcomers per week during this time that were interested in GSoC with us. This meant that the newcomers from one week might be more than we would get. And there were quite some students that came to us end of last year and had contributed a lot. Hard time deciding whom to take on the horizon already.

After completing the projects rewrite we called it done and handed everything in for the final time. Now all that was left was to wait and get to know the possible students.

Result Day, Slots and Thoughts

We were accepted. And although it doesn’t quite fit here chronologically, we received 10 slots. We were happy when we were accepted and a huge weight left my shoulders. But the real joy came when we received those 10 slots. Apparently Terri gave us one hell of a recommendation and all the mentors we had collected and project ideas we had prepared were worth something after all. John, one of the other mentors, also met Stephanie at FOSSASIA and kind of implied that getting only two slots wouldn’t work for us so maybe that also helped. Regardless of how we did it, yeay for the slots!

So what do I think about how everything went down? I feel like the projects page was a great idea and will serve us (and maybe other orgs) well in the future. There are some downsides to it in my opinion that I will cover in the next part more in depth but the main thing is that all those super defined projects left the students with nothing to think about in some cases. In the future, we will focus more on presenting the problem and why we’d like to fix it and less about the how to fix it in detail. This way we give students a chance to shine during their application phase by working on the specifics with the respective mentors and take some of the work off of us.

The part where we included students in writing the projects and faq entries on the other hand was great. It is definitely more work than just doing it yourself but it works as a first identifier for very good students. They will be happy to help defining projects, already starting to do research on it and discuss the details with possible mentors. Doing this in a more structured way will probably something we will try to do in future years.

About the meeting with Stephanie at FOSSDEM, I would recommend that to everyone! Stephanie is a lovely person and helped us a lot with her feedback I feel. She is also just nice to talk to :).

Overall I feel like this part of GSoC went pretty well and without real problems beside our panic of getting too few slots. Things won’t stay this rosy in the next phase sadly so if you want some drama and more interesting “things we could have done better” things, make sure to come back. I’m not sure when I will find the time to finish the next part as I have some university to catch up on and it takes me a good 2 hours to write one of these, at least so far, but I’ll try to keep at least a weekly schedule for this series until I am done.

Thanks again to Terri Oda from the PSF and Stephanie Taylor from Google for your support during this phase that definitely helped us to get into GSoC and learn a lot as an org.

GSoC 2017 Postmortem Part 1: Overview

Thu, 02 Nov 2017 00:00:00 +0100

The 2017 GSoC is over and it was awesome. We had 10 great students (Yes all of them!) and a tremendous experience for all of us. In this series I’ll try to reflect on how the summer went from the point of view of the main admin for coala. As I both feel that reading very long blog posts is hard and that I won’t be able to write everything down in one go, I will split this up in multiple parts. How many, I don’t know at this point.

The Admin Role, GSoC and Me

To give some perspective to the things I write, here some background about what my role and background were:

My role in this year’s Google Summer of Code for coala was being the main org admin. This meant, that I (mainly) was responsible for everything coala should deliver during those months. I had two backup admins with @sils and @jayvdb, who both were very helpful. And I want to highlight that a lot of the time when I write “I” that is actually all three of us (and sometimes also mentors). I just didn’t think about adding “me and all the lovely people that helped me” but it is what I mean most of the time. You can replace most “I”s and “me”s in this text with “we”.

Org Application

As an admin, my job is not to do everything myself, but to make sure everything gets done. And, if no one else will do it, then it becomes my job. This started with the org application to the program itself. Writing the application and reviewing what input others had given, gathering information from past years and recruiting mentors. During that phase we also met Stephanie Taylor at FOSSDEM and got some feedback on our application and our newly created projects page. Again, I was responsible for all the feedback being implemented, not necessarily doing so myself.

Student Application

After coala was accepted into the program, the student application phase soon opened. During this phase, my main work consisted of figuring out how we should cope with the insane amount of applications we received, building a process that would somehow be able to handle them (hint: we kind of failed with this), review applications and support mentors and students on their way to getting to know each other and writing applications. I am pretty sure that this part will get it’s own post and will probably be the largest one of them all. We learned a lot in this area this year but it wasn’t without growing pains sadly. At the end of the student application phase came the time to choose which student’s we would accept. With the process we had in place this year, I had to collect feedback from the mentors, form my own opinion about the students and in the end make the decision, which students we wanted to accept. And finally during this phase my job was to listen to all those, whose dreams I crushed by not choosing them.

Bonding and Coding Phases

When the actual part of GSoC started, I could actually start to relax a little bit. Most of the work now lay in the hands of the students, mentors and co-mentors. During this time I answered a lot of “how are we handling this thing” questions from students and mentors. I usually sent one or two mails per phase to both students and mentors, describing what they had to do during the current phase and reminding them of any upcoming deadlines. If any questions or problems came up and I felt the answer might help the other participants, I also added those things. Finally, I did some checkups on mentors and students, just to make sure that there no hidden problems or friction between anyone we didn’t know about.

Evaluations

The last part of GSoC (at least what I have in my head right now) were the evaluations. I will group them, as they essentially are the same every time in terms of what I had to do. I reminded the mentors in the phase starting mail that there would be evaluations and when our internal and when Google’ deadline was. Then I gave another reminder a few days before the evaluation opened, highlighting what mentors should focus on and some instructions about what those evaluations are for and about. During the evaluation phase I essentially just pinged every mentor that I felt might be running a little late and made sure that all evaluations were filled on time. In one case, by filling it myself. More about that when we arrive in that chapter.

Compensation (and other money things)

I think that it is important to be transparent, especially in an open source project like coala and when handling something where so many people have worked on as GSoC. This is why I’d like to at least mention, that I received compensation from coala for the admin work I did. 1500€ to be precise. This was decided by a vote (c-level decision) by the coala maintainers and taken from the money coala as an organization received from Google. The amount was roughly estimated using the german (where I am from) minimum wage of 8.50€ and using 8h/week as the work hours. Make of this what you will. I just felt it noteworthy.

I guess I might use this as a final chapter to talk about the money we received, and what we plan on doing with it. And in that way also shine a light on the growing coala association

I will probably write the second part on my flight back on Tuesday so stay tuned for that.

Cheers!

April Community Team Update

Wed, 19 Apr 2017 00:00:00 +0200

The last time you heard something from the community team was almost 3 month ago. Long long time ago…

So what have we been up to? And where were all those monthly updates?

GSoC 2017

So yea. coala is in. We are our own organization and might even serve as an umbrella for some smaller projects.

How awesome is that!?!

We received 54 applications and the average quality is very high compared to what other orgs get. At least that is what I hear from admins and mentors. We have 22 Mentors and three admins, two of whom would also be willing to mentor students.

79 people who would like to GSoC at coala. Wow!

I won’t mentor anyone as we feel like a designated admin with no student ties might be a good idea if any conflict come up. This also keeps me available as a backup mentor if everything else fails and generally let’s me keep eyes on the hoard with less distraction.

Projects Page

So I talked about the project page in the first post. Go visit projects.coala.io and see for yourself. It is absolutely gorgeous. Again designed by Hemang and deployed by Kaisar (our dev - ops dream team, praise be, praise be).

After it was done, we had to fill it with projects. There was an old wiki page with rough idea outlines and as a starter, we mainly just copy-pasted the ideas over onto the new page. At FOSDEM we met Stephanie from Google and she really liked the page but told us, that the descriptions were meh for some projects. This lead to us rewriting ~20 project descriptions over two weeks. A lot of students that were already preparing for their applications helped with the new descriptions and reviews which probably helped them a lot with understanding the projects better.

Although we received a lot of positive feedback for the projects page, we might not do it as in depth the next time. Although it helped students to understand the projects better, we feel like it also makes it harder for them to distinguish themselves in through the applications. It is hard to come up with your own solutions if you have a set path laid out in front of you. This is something for us to keep in mind and I bet that there will be a lot more of these ideas in the resume post of GSoC that surely will come.

Members cEP and Updates to the Newcomer Guide

The rewrite is essentially done but blocked by the additions to the newcomer guide that need to happen. The main additions are rules about how to remove ranks/rights from members of the community. This is mainly needed due to the high number of inactive maintainers from our early days. I also clarified some of the thoughts and ideas behind the rights tied to the ranks, for example the idea that non code contributors could become maintainers.

The part that blocks the PR are the rules for becoming a developer. The newcomer guide is supposed to guide everyone to the point, where we consider them a part of our developing community no longer in the learning phase of basics. To make things easier, I combined this with a lot of changes I wanted to do to the guide and so the PR is now huge and will probably take another few weeks to get merged.

Why the silence?

Basically GSoC happened.

I totally underestimated the amount of time the GSoC preparation would ask from me. Not only in a productive “I have to review 20 PRs a day to get the project descriptions right” way, but also in the “Are we missing anything? Could we do something better” mind hogging kind of way. The preparation of GSoC just took all of my awareness hostage and so it almost became the sole thing I did over the last months.

This also led to most people from the community team being entangled in the preparation. Hemang and Kaisar basically were on standby 24/7 just waiting for us to ask for the next addition or fix we desperately needed before some deadline, Lasse was part of the never sleeping admin team, Mariatta is now a python core dev and Udayan is working and researching. Basically we were all either working on GSoC or something outside of coala. I feel like I should add, that they all worked on coala a lot more than I did during the last few month. Probably.

Essentially we all got eaten by GSoC and now that I have some breathing room again, I try to collect everyone to get rolling again.

Outlook

So what is coming up?

There will be a lot of GSoC work for sure.
I just wrote the intro for our Call for Chefs cooking competition, that essentially was born over a ramen pep talk with Asnel half a year ago.
I am updating our ideas page and will try to get more stuff into the works
We held our first maintainer meeting today. The idea is quite old but we never came around doing it. I might write something about it as we want them to be public anyway but maybe I can get someone else to do it :).
I will kick everyone to write more blog posts although I feel like we are improving on the consistency of our blog entries.

And I think that’s it from the top of my head.

coala's Call for Chefs

Sat, 15 Apr 2017 00:00:00 +0200

A story I like to tell to showcase the welcoming atmosphere at coala is one about ramen. Some time last year I was talking with Asnel about how to improve a bowl of instant ramen until you get a healthy (and also very tasty) meal. Because it was a rather quiet evening we did so in our main channel, when suddenly a wild newcomer appeared (I sadly don’t remember the name). While we helped him get started, we also continued our discussion on how adding some vegetables and a soft-boiled egg could make all the difference between a cheap snack and a fulfilling meal.

Later that evening, the newcomer just found their way into our offtopic channel, they voiced their amazement about how we, “the gods of programming, the wise elders of this awesome open source project” could discuss such profanities as upgrading your ramen game. Surely we had more important, ground breaking, work to do. I am pretty sure those weren’t the exact words but the idea was the same. Even more amazing, the newcomer was able to participate (who has never eaten a bowl of instant ramen?) and contribute. The newcomer loved the casual atmosphere and being included, who would have guessed.

This taught me some things.

The welcoming culture in other open source projects must be horrible if such a simple thing would blow someone’s mind. From my point of view, we didn’t do anything impressive. We talked about food…
A lot of people don’t know how to cook nice meals. Or even just how to get a little more out of your instant ramen.
Including newcomers in your offtopic fun is a way to help them get accustomed to the new environment they ventured into.

It was the last realization that made me embrace the “Secretary of Offtopic” title (which I gave to myself to be honest). Offtopic interaction seems to be something people love and it can connect complete strangers over something all sides know about.

In comes:

Call for Chefs

Right. In an attempt to get more offtopic fun going and to get more people to cook real food, we will start a cooking competition. Probably with more focus on the cooking and less focus on the competition. The goal after all lies on the fun of experimenting in the kitchen, trying out new meals and get becoming your own chef.

The Format

At this point I am not entirely sure how the format will go. We probably will do a 2 week run, where everyone can enter with a dish they cooked themselves and the recipe and then there will be a vote. I guess we’d invite the winner(s) to showcase their dish on the coala blog (maybe we can get our own category so we don’t spam it too hard) and maybe we can find someone crazy enough to get some form of prizes, who knows. As long as there are no real prizes involved, I don’t think any real rules will be written as the whole thing is just for fun and giggles.

Any Timeline?

Just to get rolling, the first round will start now and end in two weeks (April 29.). If you want to participate, just cook something, make nice pictures of the result (maybe also during the cooking part), ~~write a blog post and get it onto the coala blog.~~
Update: Create a PR to coala/recipes with your entry as a .md file and blog post format.
The first round will mostly be an idea finding time to figure out how we can organize this and how it is supposed to work. :)

Finally, have fun cooking!

The Birth of a coala

Mon, 30 Jan 2017 00:00:00 +0100

This is my first entry on the coala blog. Welcome everyone. My name is Maximilian and I am the community team leader for coala since December. In this post I want to talk about the teams in coala and especially about the community team, its members and our work so far.

Teams in coala

I had the idea for the teams around EuroPython 2016 as a way to take part of the organisatory burden from Lasse. At that time he was making almost all the meaningful decisions in coala, not because he wanted to, but because everyone asked him. This meant that Lasse had almost no time to do anything besides organization and management stuff. Considering that he is one of our more experienced developers that sounds like a waste of potential. Not to say that he wasn’t doing an awesome job managing everything, but maybe someone else could take on the management role and free some of Lasse’s time up for developing again. Thus the teams idea was born. Having team leaders that take over parts of the management burden associated with their team’s scope would leave Lasse more free time, add more voices and brains to decisions and diversify our “leadership rank”. In the end Monolythic management can only get you so far. If you want to read more about the structure of teams in coala you can take a look at the teams cEP

Meet the community Team

I started recruiting members right after the election was over and found a number of like-minded coalians that were interested in community work in the broader sense. We have Lasse, who has his fingers everywhere but especially wanted to help with the GSoC again. Mariatta, yeay an imdb star!, who is interested in the wellbeing of the community and the newcomer experience, Hemang, our frontend guru, Kaisar the backend equivalent and Udayan who will work on administering RGSoC (besides being the Research team leader, awesome!).

Together we have many ideas on how to improve and grow a healthy community. Creating a welcoming environment where everyone feels welcome, a place where we can teach and learn from each other, a place where we can exchange mutual interests, a place to grow as professionals and humans.

What we achieved so far

This section is part of a activity report we wanted to implement for all teams. Partly to create a mechanism of control for the rest of the coala community but mainly to give the teams a place and platform to talk about the awesome stuff they create and work on.

The first thing that I would claim for the community team would be the creation of the teams cEP, election for team leaders and all together the creation of the coala teams. While the cEP was written by me I had great reviewers. I wanted to list them here but looking at the PRs, that would be too many names. We had 7 leader applications for 6 teams and recently we added the review team with another Max as leader.

After that Mariatta added the Code of Conduct which was lacking until then. This enabled Udayan to apply coala for the RGSoC, which just got accepted. YEAY!

After that we finally got our call for participation out so all the coalians who want to become team members knew it was possible. We got a lot of applications and so far it looks promising.

And finally Hemang and Kaisar wove their web-dev magic (dark arts if you ask me) and created not only a new landing page but also this awesome projects page. Although evil tongues claim the landing page was not only a community team effort I claim that all your page are belong to us.

You should definitely check out all the cool stuff created and give some love to the members of this awesome team.

What’s coming up next

We are in the process of finalizing our GSoC application but there will be a blog post about that in the future I guess. Let’s just say that the new projects page will hopefully help us a lot (and everyone else who decides to use it. Open source baby!) and we have some exciting projects and collaborations in the pipeline.

Besides the huge GSoC effort, I am in the process of rewriting the members cEP, which includes some major changes to our newcomer guide and we are working on improving the communication for maintainers. Hopefully this will be the done until our next post.

If you have ideas, questions, comments or other input, visit us in our gitter channel or drop us an email under community at coala.io.

Until next time,

Your Community Team

Hello World!

Tue, 17 Jan 2017 00:00:00 +0100

So it begins :)