[Contents]

Copyright © 2014 jsd

Graphics Notes ... Especially Graphics in the Browser
John Denker

*   Contents

1  Low-Level Stuff
2  Higher-Level Languages
3  Tools and Mid-Level Libraries
4  Screencasting
5  Shader Programs, Running on the Graphics Card
6  Video Navigation
7  Browser Video Players
8  Top-Level Applications
9  Subtitles
10  Some Observations and Howtos
10.1  Miscellaneous
10.2  Workflow
11  Vector Field Visualization

1  Low-Level Stuff

1.
HTML5 adds many new syntactic features. These include the new <video>, <audio> and <canvas> elements, as well as the integration of scalable vector graphics (SVG) content (that replaces the uses of generic <object> tags) and MathML for mathematical formulas. These features are designed to make it easy to include and handle multimedia and graphical content on the web without having to resort to proprietary plugins and APIs. Other new elements, such as <section>, <article>, <header> and <nav>, are designed to enrich the semantic content of documents. New attributes have been introduced for the same purpose, while some elements and attributes have been removed. Some elements, such as <a>, <cite> and <menu> have been changed, redefined or standardized. The APIs and Document Object Model (DOM) are no longer afterthoughts, but are fundamental parts of the HTML5 specification. HTML5 also defines in some detail the required processing for invalid documents so that syntax errors will be treated uniformly by all conforming browsers and other user agents.

http://en.wikipedia.org/wiki/HTML5

2.
HTML5 provides for 2D graphics using the low-level <canvas> element. If you want 3D graphics, consider WebGL (item 3), which is layered on top of <canvas>.

3.
WebGL is a cross-platform, royalty-free web standard for a low-level 3D graphics API based on OpenGL ES 2.0 (item 4), layered on top of the HTML5 Canvas element as Document Object Model interfaces. Developers familiar with OpenGL ES 2.0 will recognize WebGL as a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES 2.0 API. It stays very close to the OpenGL ES 2.0 specification, with some concessions made for what developers expect out of memory-managed languages such as JavaScript.

http://www.khronos.org/webgl/

WebGL (Web Graphics Library) operates at a very low level.

It is a JavaScript API for rendering interactive 3D graphics and 2D graphics within any compatible web browser without the use of plug-ins. WebGL is integrated completely into all the web standards of the browser allowing GPU accelerated usage of physics and image processing and effects as part of the web page canvas. WebGL elements can be mixed with other HTML elements and composited with other parts of the page or page background. WebGL programs consist of control code written in JavaScript and shader code that is executed on a computer’s Graphics Processing Unit (GPU). WebGL is designed and maintained by the non-profit Khronos Group.

http://en.wikipedia.org/wiki/Webgl

4.
OpenGL (Open Graphics Library) also operates at a low level. It has been around for a long time, and is widely used. WebGL (item 3) is practically synonymous with a major subset of OpenGL.

It is a cross-language, multi-platform application programming interface (API) for rendering 2D and 3D vector graphics. The API is typically used to interact with a Graphics processing unit (GPU), to achieve hardware-accelerated rendering.

OpenGL was developed by Silicon Graphics Inc. (SGI) from 1991 and released in January 1992 and is widely used in CAD, virtual reality, scientific visualization, information visualization, flight simulation, and video games. OpenGL is managed by the non-profit technology consortium Khronos Group.

http://en.wikipedia.org/wiki/Opengl

5.
Higher-level graphics libraries are discussed in section~3.

2  Higher-Level Languages

6.
Emscripten takes as its input LLVM bitcode, typically created by compiling from C or C++. As output it emits a file in the programming language JavaScript which can run in web browsers.

It can emit asm.js (item 9).

There is a brief tutorial that shows how to use clang + emscripten to compile C++ programs into javascript bytecode.

A working setup is installed locally; see

        nodejs /usrsrc/emscripten/a.out.js

and also colored square test and dropping cube demo, which uses the “bullet” physics engine.

Note that there is emscripten support for OpenGL.

In particular, ammo.js, which is a “Direct port of the Bullet physics engine to JavaScript using Emscripten”. https://github.com/kripken/ammo.js

7.
There also exists an “emcc” compiler which is an alternative to using clang+emscripten. It compiles C++ code in to asm.js (item 9). The emcc compiler itself appears to be written in Python.

8.
Python is another high-level language.

It’s hard to know what glowscript is, because the name "glowscript" gets used in at least four different ways. Let me define the -a -b -c and -d suffixes as follows:

The glowscript-d runtime library is loosely based on Vpython, and its black-box specification is reasonably well thought out. The implementation leaves something to be desired, especially in terms of the quantity and quality of the documentation.

I cannot recommend writing anything in glowscript-b. It is a low-level language, retaining all the worst features of javascript, and then breaking some of the more desirable features.

Constructive suggestion: There are newly available tools that let you write in Python and compile that to asm.js bytecode (item 9) that runs in the browser, using glowscript-d as the runtime graphics library. I haven’t explored this myself, but from a distance it seems like a non-ridiculous possibility.

Reference: http://permalink.gmane.org/gmane.comp.python.idle/2469

9.
asm.js is an intermediate programming language consisting of a strict subset of the JavaScript language. It allows browser-based web applications to be written in languages other than JavaScript, such as C or C++, where developers can take advantage of the features and tools that have already been developed for those ecosystems.

The Emscripten project (item 6) provides tools which can be used to compile stuff into asm.js.

http://en.wikipedia.org/wiki/Asm.js

3  Tools and Mid-Level Libraries

10.
Cubicvr looks very attractive. It appears to have a complete toolchain that runs on linux and provides for writing the main program in C++.

11.
Another higher-level graphics library is three.js. It can handle things like extrusions. That would be fine if you want to write the main program in javascript, which I don’t. Conversely I haven’t figured out to call three.js from a C++ main program.

12.
There is a graphics library that goes by the name of glowscript, although the name is ambiguous; see item 8. This would be fine if I wanted to write the main program in javascript, which I don’t.

13.
Question: Can someone explain how to extrude objects in opengl?

Answer: You don’t. OpenGL is a drawing API, not a geometry processing library. Look at libraries like GTS http://gts.sourceforge.net/

14.
A list of dozens and dozens of game engines:
http://en.wikipedia.org/wiki/List_of_game_engines

If you restrict it to open-source, lightweight, C++, opengl ... then there aren’t very many still standing.

15.
OpenSceneGraph is an open source high performance 3D graphics toolkit, used by application developers in fields such as visual simulation, games, virtual reality, scientific visualization and modelling. Written entirely in Standard C++ and OpenGL it runs on all Windows platforms, OSX, GNU/Linux, IRIX, Solaris, HP-Ux, AIX and FreeBSD operating systems. The OpenSceneGraph is now well established as the world leading scene graph technology, used widely in the vis-sim, space, scientific, oil-gas, games and virtual reality industries.

See file:///home/jsd/hack/osg-hello.c

There has been some effort to get openscenegraph to compile via emscripten: http://www.sim-ai.org/blog/?p=2419 and especially Kurdakov Emscripten OSG Introduction.

Note there exists osg.js ... although that seems like a step in the wrong direction.

4  Screencasting

   : 1; avconv -f x11grab -s 528x384 -r 30 -c rawvideo -i :0.0+20+100 -y ~/video/out.mkv
   : 2; time avconv -i out.mkv -qscale 5  -y out.mp4

Better, proof of concept:
  : 1; avconv -f x11grab -s 528x384 -r 25 -i :0.0
        -qscale 0 -vcodec huffyuv ~/video/grab.avi
  : 1; avconv -f x11grab -show_region 1 -s 528x384 -r 30 -i :0.0+50,175 \
        -qscale 0 -vcodec huffyuv ~/video/grab.avi
  : 1; avconv -f x11grab -show_region 1 -s 1100x800 -r 30 -i :0.0+50,175 \
        -qscale 0 -vcodec huffyuv ~/video/grab.avi

Correct aspect ratio:
  : 1; avconv -f x11grab -show_region 1 -s 854x480 -r 30 -i :0.0+50,175 \
        -qscale 0 -vcodec huffyuv ~/video/grab.avi
  : 2; time avconv -i grab.avi -qscale 5 grab2.flv
  : 2; time avconv -i grab.avi -qscale 4 ./grab4.ogg

With audio
  : 1; avconv -f x11grab -show_region 1 -r 30 \
              -s 854x480 -aspect 16:9                   \
              -i :0.0+85,218 -qscale 0 -vcodec huffyuv  \
              -f alsa -i hw:0,0 -acodec flac            \
              ~/video/grab.avi

        screenres=$(xrandr | grep '*' | cut -c 4-15)
        ffmpeg -f x11grab -r 25 -s $screenres -i :0.0
                -vcodec libx264
                -preset ultrafast -pix_fmt yuv420p -crf 0
                -y $DESTINATION.mp4

On my laptop, encoding with good quality (libx264 -preset veryslow) takes about 1 minute of real time for each minute of video, so it would be risky at best to compress while recording anything that places nontrivial demands on the computer.

The solution is to record using libx264 -preset ultrafast and then re-encode later to get more compression.

Rejected alternative: The lossless codec (huffyuv) gobbles up disk at the rate of 2 gigabytes per minute, which is not very nice.

HOWTO: Proper Screencasting on Linux ... two-step process: raw video now, convert later.

Good mp4 encoding with avconv

Miscellaneous lore: For using the Wacom tablet: Mention wacom in /etc/modules. Also make sure your kernel was compiled to support the wacom driver; hint: "locate wacom.ko". You will also need to install the window-system support. On non-Ubuntu systems it is called the "xf86-input-wacom" but on ubuntu the same functionality is provided by the "xserver-xorg-input-wacom" package. Don’t ask me to explain the names.

When the tablet works, it’s super-easy to use; most applications see it as another mouse. Some of them recognize it as being pressure-sensitive.

Here are some basic checks you can make, along with the output you should see:

lsmod | grep wacom
wacom                  68652  0
hid                   102587  2 wacom,usbhid

dmesg | grep -i wacom
[   23.776786] input: Wacom Intuos5 touch M Pen as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.0/0003:056A:0027.0001/input/input14
[   23.776893] input: Wacom Intuos5 touch M Pad as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.0/0003:056A:0027.0001/input/input15
[   23.778221] wacom 0003:056A:0027.0001: hidraw0: USB HID v1.10 Mouse [Wacom Co.,Ltd. Intuos5 touch M] on usb-0000:00:1a.0-1/input0
[   23.780219] input: Wacom Intuos5 touch M Finger as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.1/0003:056A:0027.0002/input/input16
[   23.780365] wacom 0003:056A:0027.0002: hidraw1: USB HID v1.10 Device [Wacom Co.,Ltd. Intuos5 touch M] on usb-0000:00:1a.0-1/input1


Jul  6 16:02:27 asclepias kernel: [1512378.654720] wacom: disagrees about version of symbol dev_warn
Jul  6 16:02:27 asclepias kernel: [1512378.654724] wacom: Unknown symbol dev_warn (err -22)
Jul  6 16:02:27 asclepias kernel: [1512378.654748] wacom: disagrees about version of symbol _dev_info
Jul  6 16:02:27 asclepias kernel: [1512378.654749] wacom: Unknown symbol _dev_info (err -22)
Jul  6 16:02:27 asclepias kernel: [1512378.654752] wacom: disagrees about version of symbol device_create_file
Jul  6 16:02:27 asclepias kernel: [1512378.654753] wacom: Unknown symbol device_create_file (err -22)
Jul  6 16:02:27 asclepias kernel: [1512378.654764] wacom: disagrees about version of symbol dev_err
Jul  6 16:02:27 asclepias kernel: [1512378.654766] wacom: Unknown symbol dev_err (err -22)
Jul  6 16:02:27 asclepias kernel: [1512378.654781] wacom: disagrees about version of symbol device_remove_file
Jul  6 16:02:27 asclepias kernel: [1512378.654782] wacom: Unknown symbol device_remove_file (err -22)
Jul  6 16:04:23 asclepias kernel: [   23.776786] input: Wacom Intuos5 touch M Pen as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.0/0003:056A:0027.0001/input/input14
Jul  6 16:04:23 asclepias kernel: [   23.776893] input: Wacom Intuos5 touch M Pad as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.0/0003:056A:0027.0001/input/input15
Jul  6 16:04:23 asclepias kernel: [   23.778221] wacom 0003:056A:0027.0001: hidraw0: USB HID v1.10 Mouse [Wacom Co.,Ltd. Intuos5 touch M] on usb-0000:00:1a.0-1/input0
Jul  6 16:04:23 asclepias kernel: [   23.780219] input: Wacom Intuos5 touch M Finger as /devices/pci0000:00/0000:00:1a.0/usb3/3-1/3-1:1.1/0003:056A:0027.0002/input/input16
Jul  6 16:04:23 asclepias kernel: [   23.780365] wacom 0003:056A:0027.0002: hidraw1: USB HID v1.10 Device [Wacom Co.,Ltd. Intuos5 touch M] on usb-0000:00:1a.0-1/input1

In contrast, if the system recognized the wacom devices as nothing more than a "core HID device" you’ve got a problem. The device is not going to be usable as a tablet.

5  Shader Programs, Running on the Graphics Card

The stuff in this section is for mad scientists only. Suppose you are crazy enough, and suppose you want really high performance – hundreds of gigaflops – running on your desktop, and portable to somebody else’s desktop as well. Then you can run physics programs on the graphics card!

A particularly physicsy example calculates the probability density for the hydrogenic wavefunctions. I recommend taking a look at the following: http://www.ibiblio.org/e-notes/webgl/chem/vb/ao_p_col.html

Note that the Schrödinger solution is calculated not by javascript code, but rather by a gnarly shader program that runs in the GPU.

I suspect we will be seeing more of this sort of thing in the future. Note that cryptologists have been running codebreaking code in GPUs for a long time. It’s a cheap way of getting lots of parallel processing power.

The language (OpenGL) is reasonably portable.

I think the atomic example and numerous others come from http://www.ipm.sci-nnov.ru/~Demidov/ and I think the code is in the public domain.

The author obviously knows a lot about physics, and the code is in some ways clever ... but IMHO in other ways it is off-scale terrible code that should never be shown to students. In particular, there are no comments and no documentation of any kind AFAICT. The code contains some tricky optimizations that you can figure out if you do enough reverse engineering, but that’s not how things are supposed to work. Open code is supposed to be open in the practical sense, not just in the legal sense. That is, it should be possible to understand, test, maintain, and extend the code without having to reverse-engineer it.

6  Video Navigation

7  Browser Video Players

In firefox, the html5 <video> element cannot natively play .flv or .mp4 files; .ogg is the only supported format. Conversely, Microsoft Explorer 9 will not natively play .ogg files; .mp4 is the only option.

By far the most capable .swf video players I’ve seen are from flv-player.net:

http://flv-player.net/players/js/ http://flv-player.net/players/js/documentation http://flv-player.net/players/maxi/ http://flv-player.net/players/maxi/documentation/

Compared to the js player, the maxi player is somewhat easier to use. It may be somewhat less capable, although it has a nontrivial javascript interface of its own.

8  Top-Level Applications

16.
Developed in France,TVPaint Animation is the leading paperless drawing and animation tool, able to imitate traditional techniques such as Pen Brush, Gouache, Watercolour, Pencil, Felt-tip and Airbrush, and animate them over frames.

Using only TVPaint Animation, you can create an animated movie from start to finish, including storyboarding and all the steps (rough, clean-up, keyframes and in-betweening) through to final compositing.

17.
Review of 10 "top html5 games"
http://www.creativebloq.com/html5/top-html5-games-12122765

C++ games include “Dune 2 Online” and “BananaBread”.

18.
For screencasting, you don’t need a special-purpose application; there are a lot of advantages to using whatever applications you normally use, and capturing the video and audio with a general-purpose tool such as avconv.

9  Subtitles

There are lots of good reasons for including subtitles:

The avconv tool is supposed to know about this. It supports three streams: v (for video), a (for audio) and s (for subtitles).

A widely used format for subtitle info (when it is separate from the video) is .srt which stands for SubRip Text format ... or perhaps Subtitle Resource Tracks.

HTML5 has a way of delivering a subtitle file separately from the video file, namely WebVTT.

10  Some Observations and Howtos

10.1  Miscellaneous

  1. Note: There is no such thing as a transparent jpeg image.
  2. Note: Many of the “inkscape preferences” do not take effect until you restart inkscape.
  3. Screen grabbing in preparation for screencasting: see section~4.
  4. Good way to get information about a video file:
            mplayer -nolirc -frames 0 -vo null -ao null -identify "$@" \
                    | grep ^ID_
    

    Similar, but output harder to parse:

          avconv -i asdf.flv -f null -
    
  5. Rotate a video 90 degrees (ref):
            avconv -i video.mp4 -vf transpose=1 -y out.mp4
    
    
  6. To transfer file(s) (video or otherwise) to/from the phone: over bluetooth (ref):
            cd destination
            obexpushd -B -n
    
  7. According to “Advanced encoding settings” youtube has standardized on a 16x9 aspect ratio.
       2160p: 3840x2160
       1440p: 2560x1440
       1080p: 1920x1080
       720p: 1280x720
       480p: 854x480
       360p: 640x360
       240p: 426x240
    
  8. Note: Khan Academy youtube videos are small: 426x240p
  9. When using avconv:
    1. Apparently avconv can handle gif but not png on input.
    2. If you want to make a movie from a sequence of cels, avconv can handle a bundle of individual gifs. However, it cannot hande one big animated gif.
    3. When numbering the cells, the numbers need to be fixed width (e.g. %03d) ... not variable width (e.g. %d).
    4. Apparently -vf transpose=clock is not the same as -vf transpose=1.
    5. The HTC Sensation produces variable-frame-rate video. This confuses avconv greatly. However, it seems -vf transpose=clock fixes this somehow. If necesary, -vf transpose=clock,transpose=cclock fixes it with no obvious side-effects.

10.2  Workflow

To rebuild a video from scratch, do something like this:

# for file in $(make show-raw-2) ; do vtrim $file ; done && make
\end{itemize}

Workflow, in somewhat more detail:
\begin{itemize}
  cd ~/education/howtalk
  slideshow list job long-div   # create the .d file
  cat long-div.d                # see the list of slides
  cd video
  to set the mixer controls
  vgrab -y some-temporary-junk.mp4
  then use vu-level to double-check that the
  audio-levels are reasonable:
  arecord -f S16_LE -c1 -r44100 | vu-level > /dev/null

  Now get down to business:
  slideshow present job long-div
     You may have to hit "4" "1" to persuade inkscape to
     position the screen just right (100% zoom, centered)
  vgrab -y take1-raw.mp4    # for this scene, do as many takes as necessary
  mv take1-raw.mp4 long-div-title-raw.mp4   # when you are happy with it
  mv take1-raw.mp4 long-div-def-raw.mp4     # next scene
   ... et cetera, one raw video for each slide
  make long-div.mp4     # concatenates all the slide-videos
                        #  to make one big video
  make long-div.flv     # convert format

11  Vector Field Visualization

For tracing field lines:

*) https://graphics.ethz.ch/teaching/former/scivis_07/Notes/stuff/StuttgartCourse/VIS-Modules-07-Vector_Field_Visualization.pdf

High-level discussion of Line Integral Convolution, among other things. Not a lot of detail.

Discussion of Eulerian versus Lagrangian viewpoints. Stream lines (aka streak lines) versus time lines.

*) http://web.cse.ohio-state.edu/~hwshen/788/Site/Slides_files/vectorViz.pdf

High level, no details, no references.

*) http://vis.cs.ucdavis.edu/papers/pg2011paper.pdf

Stream tapes. Scholarly paper.

*) http://web.mit.edu/8.02t/www/802TEAL3D/visualizations/guidedtour/Tour.htm

Visual tour of electromagnetism. Some nice animations.

[Contents]

Copyright © 2014 jsd