Passive aggressive Easter egg of the week

import seaborn as sns
import numpy as np
data = np.random.randint(0,10,(50,5))
sns.violinplot(data,palette='jet')

An ugly method to get matplotlib to add transparency to streamplots

Well, they said in the API docs: "This container will probably change in the future to allow changes to the colormap, alpha, etc. for both lines and arrows, but these changes should be backward compatible." but it looks like there's still a lot broken in the backend (see e.g. this GitHub issue).

So here's a very ugly workaround, based off one of the official streamplot examples. 
We plot some kind of velocity field, with the absolute speed in a background colour map, and alpha, width and colour mapping of the streamlines set by the speed. To get the alpha, we use an intermediate greyscale colourmap, and retroactively feed those grey values into the colour map and alpha values we want to actually use.  (TL;DR: I don't understand all the details, but it works)

from matplotlib import pyplot as plt
from matplotlib import cm
import numpy as np

fig,ax=plt.subplots(1)

w = 3
Y, X = np.mgrid[-w:w:100j, -w:w:100j]
U = -1 - X**2 + Y
V = 1 + X - Y**2
speed = np.sqrt(U**2 + V**2)
alph = (speed-speed.min())/(speed.max()-speed.min())
lw=4*alph
speed = np.sqrt(U**2 + V**2)
im=ax.imshow(speed,extent=[-w,w,-w,w],cmap=cm.bone_r)
stream=ax.streamplot(X, Y, U, V, density=0.6,linewidth=lw,color=alph,cmap=cm.gray)

#populate the colours list via draw()
fig.canvas.draw()

cols=stream.lines.get_colors()
alphas=cols[:,0] #copy alphas
cols=cm.afmhot(cols[:,0])#apply colour map
cols[:,3]=alphas #apply alphas
stream.lines.set_colors(cols)

#since the 'arrows' collection of the streamplot doesn't work,
#we access the arrow props via the axes' patch list.
for p in ax.patches:
    c=p.get_ec()#copy edge colour
    col=list(cm.afmhot(c[0]))[:3]+[c[0]]
    p.set_ec(col)
    p.set_fc(col)

plt.colorbar(im,label='speed')
plt.savefig('myfigure.png')
plt.show()

Customized matplotlib styles where python can find them.

 Bugged me for a while. Turns out matplotlib is quite nitpicky about exact locations and file extensions. We assume the style file is called 'mystyle.mplstyle'.

import matplotlib as mpl
import matplotlib.pyplot as plt

Find the configdir of your matplotlib:

mpl.get_configdir()

Usually it's ~/.config/matplotlib. Put the style file in a subfolder named stylelib.

plt.style.use('mystyle') should work now.

How to create your own style?

The syntax is the same as in a matplotlibrc file, so find one (e.g. via mpl.matplotlib_fname()), copy and paste. 

How to get a matplotlib colormap into inkscape.

Inelegant, but works.

TL;DR: let python create an SVG file with a rectangle that uses the colormap as a gradient. Open, import or copy in inkscape.

Python code (this one gives coolwarm012.svg in your working directory):

ncol=12 #number of stops
grad='coolwarm' #name of matplotlib gradient
from matplotlib import colors, cm
import numpy as np

stopstr="""  <stop
    style="stop-color:%s;stop-opacity:1"
    offset="%.4f"
    id="stop%03d" />"""

init="""<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
   width="1cm"
   height="1cm"
   viewBox="0 0 1 1"
   id="svg">   
  <defs id="defs">
    <linearGradient
       id="%s%03d">
"""%(grad,ncol)

exit="""
</linearGradient>
</defs>
<rect
       style="fill:url(#%s%03d)"
       id="rectangle"
       width="1"
       height="1"
       x="0"
       y="0"/>
</svg>"""%(grad,ncol)

cmap = cm.get_cmap(grad, ncol)   
spac=np.linspace(0,1,ncol)

#saves  <gradientname+numberofstops>.svg with a 1x1cm^2 rectangle using the gradient 
with open('%s%03d.svg'%(grad,ncol),'w') as f:
    f.write(init)
    for i in range(cmap.N):
        rgba = cmap(i)
        hexcol = colors.rgb2hex(rgba)
        f.write(stopstr%(hexcol,spac[i],i))
    f.write(exit)

Some code taken from https://stackoverflow.com/questions/33596491/extract-matplotlib-colormap-in-hex-format

How to convert a LaTeX beamer presentation with videos to PPTX using Python

Update: The crash issue appears to be fixed in recent LibreOffice. So there is really no reason to do all of the below any more, hooray. See updated howto.

First, why on earth would you want to do that?

The American Physical Society insists that you upload PPTX slides to their conference server and LibreOffice crashes every time I try to import a movie. The developers on the LO forums say it's not their problem because it works with ODP. Glad you asked.

You'll need pdftoppm (or similar, comes with poppler) and the python-pptx and PIL modules. Also, FFMPEG if your movies still need conversion.

First, create page images, in bash:
pdftoppm mypresentation.pdf img -png -r 250

The output is a series of img-1.png, ...,img-n.png. Resolution is here 250.

If necessary, convert all movies to something PowerPoint will digest. Other containers and codecs might work as well, but my Windows failsafe is WMV2.

ffmpeg -i input.avi -codec:v wmv2 -b:v 2000k output.wmv

Batch conversion:

for m in *.avi; do ffmpeg -i $m -codec:v wmv2 -b:v 2000k ${m%avi}wmv; done

We'll assume that we have movie1.wmv and movie2.wmv on slide 2 and movie3.wmv on slide 5.

Now for the tedious bit - determine the bounding box (left, top, width, height, in px) for each movie on the img-1.png and img-5.png slide images. I just used gwenview's crop tool.

Now for the python script:

from pptx import Presentation
from pptx.util import Inches
from glob import glob
from PIL import Image

prs = Presentation()
prs.slide_width = Inches(16) #assuming 16:9 aspect ratio
prs.slide_height = Inches(9)

blank = prs.slide_layouts[6]

imgs = sorted(glob("img*.png"))
w,h=Image.open(imgs[0]).size
# conversion factor for pixel values to inches.
pxtoin=16./w
slides={}

# did the movie-slide attribution via some kind of messy dictionary/list construction. Use the top,left,width,height values determined above.
movies={4:[['movie3.wmv',[626,323,1263,708]]],
        1:[['movie1.wmv',[155,324,710,710]],['movie2.wmv',[1163,324,710,710]]]}

for i in range(len(imgs)):
   slides[i] = prs.slides.add_slide(blank)
   # add the slide contents as bitmaps
   pic = slides[i].shapes.add_picture(imgs[i], 0, 0, height=prs.slide_height)
   try:
      for it in movies[i]:
         movie = slides[i].shapes.add_movie(it[0],*[Inches(r*pxtoin) for r in it[1]])
   except KeyError: pass

prs.save("test.pptx")


This got me a minimum working presentation when I tested it on a remote Windows machine, apart from the fact that the movies had ugly speaker icons instead of poster images. There's a poster_frame_image keyword in add_movie where you can provide an image filename, but it didn't seem to change anything.

Sources:

The python-pptx documentation has a lot of basic use cases. https://python-pptx.readthedocs.io/

Installing external python modules for Blender

Blender comes with its own Python, and it's not necessarily your system's, so just symlinking system packages like matplotlib is risky and cumbersome.
Some Googling Stackoverflow provided me with the following approach:

• create a (temporary) virtual environment (VE) linking against Blender's python executable
• within that VE, pip install the packages you need, but into a folder in Blender's Python path (e.g. the one in ~/.config)
• I'd keep the VE, but Blender doesn't need it to be active to run.

In code (I keep Blender in ~/.local/bin, adapt to your setup), for matplotlib (You might have to run blender once at first to create the file structure in .config):
virtualenv --python=~/.local/bin/blender-2.81a/2.81/python/bin/python3.7m ve-blender
source ve-blender/bin/activate
mkdir -p ~/.config/blender/2.81/scripts/modules/
pip install --upgrade -t ~/.config/blender/2.81/scripts/modules/ matplotlib
deactivate

Run Blender.

Update: When I tried to do this with pickle5,  gcc failed via "Python.h: No such file or directory".

Fix: within the active VE, determine the Blender Python version (python -V).
Download the respective sources (e.g. 3.7.7) from www.python.org. and unpack.
Copy the contents of the Include directory to ~/.local/bin/blender-2.81/2.81/python/include/python3.7m/ (adapt versions and paths).

Next update:

Broken pip after initializing the virtual environment. Fixable via pip's bootstrap installer:
curl -sS https://bootstrap.pypa.io/get-pip.py | python3

Sources:
https://stackoverflow.com/questions/1534210/use-different-python-version-with-virtualenv
https://blender.stackexchange.com/questions/132278/how-to-make-use-of-custom-external-python-modules-in-blender-or-an-add-on-on-lin

https://blender.stackexchange.com/questions/81740/python-h-missing-in-blender-python 

https://stackoverflow.com/questions/49478573/pip3-install-not-working-no-module-named-pip-vendor-pkg-resources

do-release-upgrade or don't

I couldn't resist the lure of the Dingo, but it took me some time to get there. Long story short: do-release-upgrade is a picky bitch. It kept and kept and kept coming up with a "Your python3 install is corrupted. Please fix the '/usr/bin/python3' symlink." message. That message is produced by a script downloaded into /tmp during the installation process and IMO *should* just make sure that the /usr/bin/python3 symlink points to a correct python binary (might have to be 3.6). After all, the update scripts have a "#!/usr/bin/python3" header.
However, over the years, people have had to resort to the weirdest workarounds:

• Make sure /usr/bin/python and /usr/bin/python3 point to python3.6
• Reinstall default python3 
• Purge anything done by update-alternatives (sudo update-alternatives --remove-all python)
• Purge python2.x

Reader, I tried all of them.
update-alternatives --display python
sudo update-alternatives --remove-all python

sudo apt install --reinstall ubuntu-release-upgrader-core

sudo apt install --reinstall python3
sudo ln -sf /usr/bin/python3.6 /usr/bin/python
sudo ln -sf /usr/bin/python3.6 /usr/bin/python3
sudo apt-get remove --purge python2.7-minimal
In the end, purging python2.7 did the trick. That involved uninstalling inkscape and bits of TeXlive, but it was a small list of dependencies that could be reinstalled after the update. Still, a mess. Serves me right for not bothering to do a clean reinstall.

Spyder 3.2.8, pip3 and and PyQt5

I recently ran into segfaults on all my computers after upgrading spyder via pip3. Lots of Qt5 error messages. The trouble seems to be PyQt5-5.10.1; spyder 3.2.8 requires at least 5.10.0 (which happens to work). Thus pip3 install --upgrade PyQt5==5.10.0 spyder 
solved my issue.
Note to self: pip3 install PyQt5== is a handy way to list available versions. Sadly, it's deprecated (--use-deprecated=legacy-resolver).
Second note to self: There's also spyder==4.0.0b1 which seems to work with newer PyQts. Good thing I switched to virtual environments.

Adding matplotlib to locally installed blender

Update: Unless you hate virtual environments, I found a better way:
https://eumenidae.blogspot.com/2020/01/installing-external-python-modules-for.html
I had some trouble getting a local blender 2.75 in my home directory to use matplotlib - using the system wide python3.4 for some reason failed, so in the end I just copied over files to the local version until blender stopped complaining.
Required packages for Ubuntu 14.04: python3-pyparsing, python3-dateutil¸ python3-matplotlib, python3-cycler, python3-six. Using the following destination root directory ~/bin/blender/2.75/python/lib, I copied these files and folders:

source	destination
/usr/lib/python3/dist-packages/pyparsing.py	python3.4/site-packages/
/usr/lib/python3/dist-packages/dateutil	python3.4/site-packages/
/usr/lib/python3/dist-packages/matplotlib	python3.4/site-packages/
/usr/lib/python3/dist-packages/mpl_toolkits	python3.4/site-packages/
/usr/lib/python3/dist-packages/pylab.py	python3.4/site-packages/
/usr/lib/python3/dist-packages/cycler.py	python3.4/site-packages/
/usr/lib/python3/dist-packages/six.py	python3.4/site-packages/
/usr/lib/python3.4/distutils	python3.4/

A bit ridiculous for just wanting to use their colour maps, but there you go… Note: Still works for blender 2.77/Python 3.5/Kubuntu 16.04. Replace Python3.4 with Python3.5.

pylibtiff fails on Ubuntu 14.10

I ran into this while setting up trackpy - after installing a recent git clone and trying to import the module in an ipython console, I got the following message:
Failed to find TIFF header file (may be need to run: sudo apt-get install libtiff4-dev)
A bit of a problem, as recent Ubuntus come with libtiff5-dev and it doesn't resolve the issue.
The message is actually libtiff's as can be easily checked with an 'import libtiff' statement.
According to bug discussions on Launchpad and Google Code this is due to Ubuntu providing a pylibtiff that's too old for its libtiff, so it hasn't caught on to version 5 yet.
The quick fix is to get the missing tiff_h_4_0_3.py and saving it in /usr/lib/python2.7/dist-packages/libtiff/ (sudo that). Haven't put the module through its paces yet, but at least it loads without complaints now.
Also, for recent git versions of trackpy, Ubuntu's python-six is not recent enough (needs to be >=1.8).  Fixable with sudo pip install --upgrade six, if you happen to have pip around.

Quick 'n dirty wxPython plot digitiser

There are a number of digitisers out there, but I wasn't happy with any of them - they either gave me weird results or didn't work on Linux. Also I hadn't played with wxPython in a long while.

Dependencies: wxpython, PIL, numpy, scipy.

What it does:

  - manual or automatic (scatter or line) detection of plot points by colour. 

  - axis calibrated ascii (x,y) data export.

Download

  

Resuscitating Nanoengineer on Kubuntu

Nanoengineer has been unsupported for at least five years, and it shows. The 1.1.12 source zip file apparently contains corrupted files, and the code relies heavily on the deprecated python-Numeric.

Movie playback in Latex/Beamer: the current situation with Adobe Reader, Okular and Impress

A few years ago, the only option to include multimedia content in Linux presentations was linking to an external player. While the situation is still far from ideal, we have a few options now:

• Okular has been able to play back movies via mplayer for quite some time now. Features: pause and seek. Downside: poster images appear all black (fixed in KDE 4.10), control bar spacing.
• Adobe Reader can embed external flash players and videos. Features: pause. Downside: Flash, works only with old acroread.
• Open/Libre Office Impress movie playback finally works. Features: presenter console. Downside: plays immediately, pause and seek only outside presentation mode. Also, the LaTeX PDF has to be converted into page images.

HTML directory of PDF files

The file names for my PDF presentations contain only my name and the presentation date. Having to sort through a hundred meeting write-ups when looking for a specific bit of data is annoying, so I wrote a python script to put title, author, modification date and a link to the file into a HTML table.

Keynotifying PDF presentations

Update: (03/01/12) My version of pdf2odp.py now converts movies and Impress can actually play them. See this post for details.
Confession time: I did my defence on daWuzzzz's MacBook. The first reason was that I already had an Apple remote, the second was Keynote's excellent presenter console.
The downside was having to import my LaTeX/Beamer PDF presentation as images into Keynote and fiddling around with movie positioning afterwards.
You can do something like that purely on Linux - excluding the Apple remote :-) - so this post covers 3 topics:

• the PDF presenter console
• a harangue on crappy vector graphics import in OpenOffice and its presenter console
• using python scripts to convert PDF into ODF with page images

Parallel Python Client with matplotlib and pylab trouble

Problem: use OpenSuSE 11.1 as parallel python client with numpy, scipy and matplotlib, which starts automatically at boot time without the need to have anyone logged in.

I love disper!

...and, if you use a laptop with nVidia graphics and want to extend/clone your screen output to an external monitor or projector, you will, too.
Disper is a python-based command line app and can be downloaded from here. I don't think there are any prerequisites except the nVidia packages and basic Python, and potentially xrandr. Install with make /sudo make install.
The beauty of disper is that it automatically and quite intelligently fits your screen resolution and panning to the detected displays.

State of Pymorph Address

Friends, Romans, countrymen, lend me your ears:
Ahem.

Actually, I was just looking for a quick 'n dirty Python morphology library to do a kind of string length image analysis and pymorph looked promising. Sadly, a major overhaul seems to be going on there right now, so while the actual code I ended up with is nice, short and robust, getting there was quite a pain.
Pymorph's old (<= 0.8) version has some very helpful demonstration pages, which have to be adapted for the new version (0.92 - and I found nothing but win32 packages for 0.8). Some hints are given on the developer's page, but quite a lot one has to figure out for oneself - mostly formatting issues.

Tu quoque, Python?

Ah, well, it's nothing compared to certain Excel bugs…

More fun with Frescobaldi: KDE updates

On Tisiphone, Frescobaldi had a segmentation fault immediately at startup after a KDE 4.3 update; the next update corrected that, luckily (OK, score wizard is still broken, there seem to be major overhauls somewhere). Now the problem has diffused to Archimedes' 10.3 / KDE 4.2 setup, and here it's rather sticky.
Frescobaldi is written in Python, so I tried running frescobaldi.py line-by-line from the build directory. Turns out that the PyKDE4.kdecore module is broken, it kills python as soon as you try importing anything from it. You can't use a different version, because it has to be in sync with your KDE build. I tried building the source RPM, however, it crashed building libakonadipart1.cpp :-(

Hell, I don't even use akonadi on 10.3 because it breaks down anyway...

Update: As of today, (July 6th), the python bindings from the KDE:42 repository seem to work, as well as frescobaldi. Frescobaldi 0.7.13 in combination with KDE 4.3 even displays the score wizard.