Documentation and diagrams
Hi, I asked thoughts on Spectrum documentation improvements some time back on #spectrum. I'd like to improve documentation in terms of architecture (views, description, diagrams) to support communication with people adopting Spectrum in our research/engineering project(s). Of course I can create presentations for such purposes but those are not supporting the project. There's nothing wrong with asciidoc but as of now it lacks support for diagrams. Another view to diagrams was how we want to create diagrams - writing dsl(diagram-specific-language)-to-diagrams or drawing. Based on our #spectrum discussion I was guided to asciidoc diagram extension - https://docs.asciidoctor.org/diagram-extension/latest/ - so I finally gave it a try last Friday with ditaa - an ascii-to-png-conversion. Long story short - I got it working by adding dependencies to jdk8 etc. but I'm not happy with the result for following reasons: Limitations: - ditaa (and other dsl-to-diagrams) have limited support in either diagram dsl comprehension or available editors. Namely, ascii-based box drawing programs exist (e.g. https://asciiflow.com/#/) but do not directly support ditaa. If ascii-drawings can be modified to ditaa, it's slow, error-prone and worst of all - importing version controlled ditaa-ascii-graph text files back to editors do not work as object imports so in worst case the objects need to be drawn again when one wants to modify the diagrams. - dsls have learning curves with different slopes. Some are easy, some take domain level understanding (like UML) in addition to dsl but it's there. - layout algorithms - many cases go quite ok with this but then there's points where the layout algorithms fail miserably with solutions beyond reasonable effort. - dsls are usually not readable in source format but beyond trivial diagrams always require validation visually Benefits: + Some asciidoc diagram-extension dsl:s (like plantuml, graphviz) work quite ok for both visualizing UML, graphs, dependencies, trees. + Focus on content, not presentation - kind of like LaTeX. Though this is limited by tools like ditaa which can get worst of both worlds. + dsls can be great for code level views. I've used tool where dsl has been combined into generating the diagrams from code or tree like structures. In fact, there's quite a few such tools to do this with nix-store dependency trees and have been useful in spectrum architecture comprehension. What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license. I'd like to hear thoughts/decisions on this before sending any patches to contribute to spectrum architecture documentation. Best regards, -Ville
What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams.
I’ve had good experiences with Mermaid: https://mermaid-js.github.io/mermaid/#/
Hi Ville, thanks for looking into this! On Tue, May 17, 2022 at 05:52:35AM -0000, ville.ilvonen@unikie.com wrote:
What I'd like to propose that in addition to aiming at text-to-diagrams we support diagram drawing tool(s) and agree on either png or svg with diagrams. With diagram drawing tools, we should store the diagrams in editable format in version control (not only png and svg). My 2 cents would go to https://github.com/jgraph/drawio but I'm fine with others as well - as long as they do not require commercial license.
I've heard very good things about draw.io. I agree with your assessment that having a graphical diagramming tool would be useful. draw.io being FOSS is a big plus for it as well — I think it's desirable that contributing to Spectrum does not depend on proprietary software. One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?
Hi all,
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?
drawio supports being called on the command-line like this: drawio in.drawio --export --output out.png # or svg but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…
On Tue, May 17, 2022 at 10:00 PM Paco <paco@ecomail.io> wrote:
Hi all,
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?
drawio supports being called on the command-line like this:
drawio in.drawio --export --output out.png # or svg
Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg" However, [vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum
but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…
This could be managed with caching but that's not the issue. People seem to have worked around the issue in CI with solutions like https://hub.docker.com/r/tomkludy/drawio-renderer I did not test it yet but would this be ok for a CI pipeline? Best, -Ville
On Wed, May 18, 2022 at 03:44:45PM +0300, Ville Ilvonen wrote:
On Tue, May 17, 2022 at 10:00 PM Paco <paco@ecomail.io> wrote:
Hi all,
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?
drawio supports being called on the command-line like this:
drawio in.drawio --export --output out.png # or svg
Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg"
However,
[vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg
Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum
Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.
but it's a huge dependency, as it's an Electron app, installing it requires ~400 MB…
This could be managed with caching but that's not the issue.
People seem to have worked around the issue in CI with solutions like https://hub.docker.com/r/tomkludy/drawio-renderer
I did not test it yet but would this be ok for a CI pipeline?
If we can get it running in a Nix build, I don't think closure size will be an issue. If the size becomes an issue in future, we could try running the rendering code without Electron, by running the browser version in whatever the modern equivalent of PhantomJS is, or just trying to extract the renderer code and run it in Node directly. But if we can get the Electron version running, it should be fine to use that.
Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.
Dummy X is pretty easy. I use it. https://github.com/7c6f434c/lang-os/blob/master/wrapped-firefox-launcher.nix...
On Wed, May 18, 2022 at 12:53:53PM +0000, Alyssa Ross wrote:
On Wed, May 18, 2022 at 03:44:45PM +0300, Ville Ilvonen wrote:
On Tue, May 17, 2022 at 10:00 PM Paco <paco@ecomail.io> wrote:
Hi all,
One question about draw.io: is there a tool we could use in the build process of our documentation to render the .drawio files to SVG, so we don't have to check in the SVG files?
drawio supports being called on the command-line like this:
drawio in.drawio --export --output out.png # or svg
Yes, and no. Yes, command line but no in a headless environment. It was easy to test this before "jekyll build" with "drawio in.drawio --export --output test.svg"
However,
[vilvo@blop:~/dev/spectrum/Documentation]$ nix-build -I nixpkgs=../../nixpkgs-spectrum ... error: builder for '/nix/store/r8xf2kpba673ngi2fck4xsyvyc6pd3j1-spectrum-docs.drv' failed with exit code 133; last 10 log lines: > at new ElectronStore (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/node_modules/electron-store/index.js:48:32) > at Object.<anonymous> (/nix/store/iqinmbk2jzl8nay42hqkbd32mbj1jzq7-drawio-18.0.6/share/drawio/resources/app.asar/electron.js:15:15) > at Module._compile (node:internal/modules/cjs/loader:1116:14) > at Object.Module._extensions..js (node:internal/modules/cjs/loader:1169:10) > at Module.load (node:internal/modules/cjs/loader:988:32) > at Module._load (node:internal/modules/cjs/loader:829:12) > at Function.c._load (node:electron/js2c/asar_bundle:5:13343) > at Object.<anonymous> (node:electron/js2c/browser_init:185:3104) > at Object../lib/browser/init.ts (node:electron/js2c/browser_init:185:3308) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1: 5 Trace/breakpoint trap (core dumped) drawio in.drawio --export --output test.svg
Issue is the same with drawio versions: - version 15.7.3 from nixpkgs - version 17.4.2 from nixpkgs-spectrum - latest rpm available - 18.0.6 from updated nixpkgs-spectrum
Does it maybe require an X server / Wayland compositor to talk to? That would be a pain to do in the build but maybe not impossible. I know there are programs for dummy compositors out there but don't know how easy they are to use.
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like export XDG_CONFIG_HOME="$(mktemp -d)"
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross <hi@alyssa.is> wrote:
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like
export XDG_CONFIG_HOME="$(mktemp -d)"
I think it was a good idea to test but unfortunately it did not work out: '' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out '' $ nix-build this derivation will be built: /nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv building '/nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv'... /build/tmp.mOBtFDpZlO Fontconfig error: Cannot load default config file: No such file: (null) /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 3: 4 Segmentation fault (core dumped) drawio in.drawio --export --output test.svg error: builder for '/nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv' failed with exit code 139; last 3 log lines: > /build/tmp.mOBtFDpZlO > Fontconfig error: Cannot load default config file: No such file: (null) > /build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 3: 4 Segmentation fault (core dumped) drawio in.drawio --export --output test.svg For full logs, run 'nix log /nix/store/k2z4xgs1dpp5p52sds98vl46x7myl924-spectrum-docs.drv'. Best, -Ville
On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross <hi@alyssa.is> wrote:
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like
export XDG_CONFIG_HOME="$(mktemp -d)"
I think it was a good idea to test but unfortunately it did not work out:
'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''
I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something. [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...
On Thu, May 19, 2022 at 10:35 AM Alyssa Ross <hi@alyssa.is> wrote:
On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross <hi@alyssa.is> wrote:
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like
export XDG_CONFIG_HOME="$(mktemp -d)"
I think it was a good idea to test but unfortunately it did not work out:
'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''
I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something.
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...
I was also starting to think that the X env is required. X virtual framebuffer is a great idea. Good catch with pandoc approach to generate drawio to pdf in xvfb.
On Thu, May 19, 2022 at 11:33:24AM +0300, Ville Ilvonen wrote:
On Thu, May 19, 2022 at 10:35 AM Alyssa Ross <hi@alyssa.is> wrote:
On Thu, May 19, 2022 at 10:25:16AM +0300, Ville Ilvonen wrote:
On Wed, May 18, 2022 at 4:11 PM Alyssa Ross <hi@alyssa.is> wrote:
Actually, this looking more closely at the stack trace, this is more likely to be it not having a writeable XDG_CONFIG_HOME or HOME directory. So you could try setting those to temporary directories in the build. Something like
export XDG_CONFIG_HOME="$(mktemp -d)"
I think it was a good idea to test but unfortunately it did not work out:
'' export XDG_CONFIG_HOME="$(mktemp -d)" echo $XDG_CONFIG_HOME drawio in.drawio --export --output test.svg jekyll build --disable-disk-cache -s $src -d $out ''
I was just looking at the draw.io packaging in Nixpkgs, and I discovered that there's already code in there for running headless draw.io! [1] But it's not exposed outside of this pandoc filter package. I'm going to see if it works, and if it does I'll see if we can pull it out to the top level, as a drawio-headless package or something.
[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/pandoc-drawio-f...
I was also starting to think that the X env is required. X virtual framebuffer is a great idea. Good catch with pandoc approach to generate drawio to pdf in xvfb.
participants (6)
-
Alyssa Ross -
Mark Stosberg -
Michael Raskin -
Paco -
Ville Ilvonen -
ville.ilvonen＠unikie.com