Today I sat down and tried to refactor satyr once more after dinner. This time I was trying to decouple the functionality related to multi-collection playlists from PlayListModel while moving it to the default skin. The idea was to be able to create another skin which used a QTableView, which in turn would be a trampoline for implementing tag editing and writing. But I started to stall a little, and that normally provokes me to defocus, to dezone. When that happens, I go and read some piled up posts in akregator[1].

This time I came around a post by Martin Michlmayr (who I read through Planet Debian) from 10 days ago which talks about lessons learned about free software projects. Actually the post is just a resume of 4 posts from the FreeDOS founder Jim Hall. At some point he writes «releases are important».

Bing! goes my head[2]. satyr is already 3 months, 12 days or 117 revisions old and I hadn't released it yet, even after I promissed to do so almost a month ago! The problem is that I kept adding features (and squashing bugs in Phonon[3]) and completely forgot about releasing. Just one semi-colon before he also writes «initial users of the software should be recruited as developers»... which users? If one doesn't release, one might never have users to recruit!

So instead of the pharaonic refactor I had in mind (an in another branch, blessed be bazaar) I wrote a setup.py script in 15 minutes, massaged a little the files (I had to create a package and modify almost all the files to reflect this), tested a little, and produced a nice triplet of files:

satyr-0.1-beta1.tar.bz2
satyr-0.1-beta1.tar.gz
satyr-0.1-beta1.zip

So there you have it, a realease! Ok, it's a 'beta1', but it's out. Go grab it, test it, complain about bugs, tell us you like it, suggest improvements, whatever! And tell your best friend to use it, even if you don't like it! Where to get it? Why, from the project's download's page, of course!

satyr pykde python

[1] Yes, once upon a time I developed my own feed reader called kReiSSy, but it's implemented in PyKDE3 and I don't plan to port it yet, even if somehow is better for me than akregator. A shame, reallly...

[2] in the same way that a µ-wave oven goes “bing”, not in an “Eureka!” way...

[3] I even fixed the need for the 'file' scheme in the Gstreamer backend.

One of the things I had to while developing satyr is building a model for a QListViewer. It should be straighforward from qt's documentation, but I found a couple of things that I would like to put in a post, specially because there doesn't seem to be much models in PyQt4 easily found in the web.

According to its description, a subclass of QAbstractListModel as this one should mostly implement the data() and rowCount() methods, which is true. This example creates a read-only model, so no need to implement setData(), but given the simplicity of data(), it doesn't seem too difficult to do. I also wanted it to react when more Songs were added on the fly[1].

The method data() is the most important one. It is not only used for retrieving the data itself, but also some metadata useful for showing the data, like icons and other stuff. For selecting what the caller wants, it refers a Qt.ItemDataRole. The role for the data itself is Qt.DisplayRole. One of the particularities of this method is that it could be called with any vegetable as input; namely, it can refer to a row that does not exist anymore or for metadata that you don't care about. In those cases you must return an empty QVariant, not None. So, a first implementation is:

def data (self, modelIndex, role):
    if modelIndex.isValid () and modelIndex.row ()<self.count and role==Qt.DisplayRole:
        # songForIndex() returns the Song corresponding to the row
        song= self.songForIndex (modelIndex.row ())
        # formatSong() returns a QString with the data to show
        data= QVariant (self.formatSong (song))
    else:
        data= QVariant ()

    return data

This method, together with a rowCount() that simply returns self.count, is enough for showing data that is already there. Notice that the QModelIndex can be not valid, and in this case we only care about its row because we're a list.

But then I wanted my QListViewer to show songs progresively as they are loaded/scanned[2] and also as they are found as new. But then a problem arises: the view is like a table of only one column. The width of this colunm at the begining is the same width as the QListView itself. But what happens when the string shown is too big? What happens is that it gets chopped. We must inform the view that some of the rows are bigger. That's where the metadata comes into play.

Another possible role is Qt.SizeHintRole. If we return a size instead of an empty QVariant, that size will be used to expand the column as needed, even giving us a scrollbar if it's wider that the view.

Now, we're supposed to show the tags for the Song (that's what formatSong() does if possible; if not, it simply returns the filepath), so this width should be calculated based on the length of the string that represents the song[3]. But if we try to read the tags for all the songs as we load the Collection, we end up with too much disk activity before you can show anything to the user, which is unacceptable[4]. So instead we calculate based on the filepath, which is used for Songs with too few tags anyways. Here's the hacky code:

...
# FIXME: kinda hacky
self.fontMetrics= QFontMetrics (KGlobalSettings.generalFont ())
...
def data (self, modelIndex, role):
    if modelIndex.isValid () and modelIndex.row ()<self.count:
        song= self.songForIndex (modelIndex.row ())

        if role==Qt.DisplayRole:
            data= QVariant (self.formatSong (song))
        elif role==Qt.SizeHintRole:
            # calculate something based on the filepath
            data= QVariant (self.fontMetrics.size (Qt.TextSingleLine, song.filepath))
        else:
            data= QVariant ()
    else:
        data= QVariant ()

    return data

The last point then is reacting to Songs are added on the fly. This is also easy: you tell the views you're about to insert rows, you insert them, tell the views you finished, and then emit dataChanged():

def addSong (self):
    # lastIndex keeps track of the last index used.
    row= self.lastIndex
    self.lastIndex+= 1

    self.beginInsertRows (QModelIndex (), row, row)
    # actually the Song has already been added to the Collection[5]
    # so I don't do anything here,
    # but if you keep your rows in this model you should do something here
    self.endInsertRows ()

    self.count+= 1

    modelIndex= self.index (row, 0)
    self.dataChanged.emit (modelIndex, modelIndex)

Later I'll post any peculiarities I find porting all this stuff to a read/write QTableModel.

satyr pykde python

[1] That's material for another post :)

[2] This feature can be said to be a little too much. Actually, I get a flicker when scanning.

[3] Of course the next step is to use a table view and make a model for it.

[4] Right now the load time for a Collection of ~6.5k songs is quite long as it is.

[5] This is a design decision which is not relevant to this example.

One of the features I planned for satyr almost since the begining was the possibility to have 'skins'. In this context, a skin would not only implement the look and feel, but also could implement features the weren't available in the shipped classes. I also planned to implement this feature after I had most of the others one already done. But then I was bored this weekend with nothing to do and I decided to set off to at least investigate how to do it. Of course, what happened was that I implemented it almost completely.

Up to now, satyr's user interface was implemented in two files: default.ui, which was compiled with pyuic4 into default.py, and some code in satyr.py itself. This of course would not scale, and I always had the idea of moving the behaviour implemented in satyr.py to a file called default.py and load the ui directly from the default.ui file without compiling, getting rid of the need for a compilation at the same time. This also meant that then a skin would consist of a .py file and possibly a .ui file. There are three problems to solve for this: getting the local-to-the-user's skin directory, loading the skin and loading the correspondant .ui file.

The first part is simple from the PyKDE4 point of view:

# get the app's dir; don't forget the trailing '/'!
appDir= KStandardDirs.locateLocal ('data', 'satyr/')

I'll first explain the other two parts, loading the skin and the .ui file, before returning more deeply to the consequences of this solution.

I put all the skins in a skins subdirectory. To make it a proper python module I added an empty __init__.py file. Now, I could simply import skins.<skinName> and possibly instantiate some class in it, but of course one cannot write that. I could resort to eval ('import skins.'+skinName), but we know that eval() has the most long-standing typo in the history of computer languages, and it's actually called evil().

What we can do is resort to __import__() instead. This little function does approximately what we want. I say approx because it has some surprises in the sleeves of its sleeveless code. I suggest you to go read carefully its documentation. Meanwhile, the magic itself:

mod= __import__ ('skins.'+skinName, globals(), locals(), 'MainWindow')
mw= mod.MainWindow ()

Loading the .ui file in the skin's code is rather simple: just get the skin module's filepath, replace .py with .ui, and load it with PyQt4.uic.loadUiType()[1]. This function returns a generated class for the topmost widget and its Qt base class. This generated class has a setupUi() method that is the one that actually builds de UI[2]. So, we just instantiate the main window's class and call its setupUi() method:

from PyQt4 import uic

# !!! __file__ might end with .py[co]!
uipath= __file__[:__file__.rfind ('.')]+'.ui'
# I don't care about the base class
(UIMainWindow, buh)= uic.loadUiType (uipath)

self.ui= UIMainWindow ()
self.ui.setupUi (self)

Note the comment about the __file__ attribute of a module.

Now, and back to the first part, finding the local-to-the-user's skin directory is the easiest part. From there, things get a little bit more complicated:

• The skins subdirectory might not exist.
• If you create it, you gotta make sure to also throw in a __init__.py file.
• Once you've done it, you also need to add the local-to-the-user's app directory to the path. It's easy, just prepend it to sys.path, so it's used before any system-wide directory.
• The last problem that remains is exactly that: once the __init__.py is there and the user's dir is prepended to sys.path, the user's local skin directory is always used when importing anything from the skins module, so if a skin is not there it is not loadable. All skins distributed with satyr will be inaccesible!

So I'm in a kind of dead alley here. I have a couple of ideas on how to work-around this, but they're at best hacky, and I don't want to implement them until I'm sure that it's inevitable.

satyr pykde python

[1] Not a very happy name, if you ask me.

[2] Very similar to what you get if you compile the .ui with pyuic4.

I certainly hope this is the last post in the Phonon-and-badly-encoded/mixed-encodings-filenames saga, but I know is just wishful thinking: as all encoding-related problems they never really dissapear, it's just that you hadn't hit the right wrong stone yet. In any case, I fixed all my later problems wherever they where, and now I can answer this question: how to play files whose filenames are badly encoded and/or have mixed encodings, all this in Phonon?

Right now the answer is: you have to provide a properly encoded QUrl. How, you might ask, can I get one of those? Are they selled in the same odly-looking places where you can buy cigarretes, or even marihuana[1]? The answer, luckly, is way more simple.

Putting together all the code I've been showing about Python, PyQt4/PyKDE4 and Phonon recently, it comes down to this[5]:

# path is a str()
qba= QByteArray (path)
# the exceptions are not needed,
# but is cleaner if you print the outcome of this
qu= QUrl.fromEncoded (qba.toPercentEncoding ("/ "))
# this is needed by the gstreamer backend[3],
# and the xine backend doesn't complain
qu.setScheme ('file')

... and that's it. You can now create a MediaSource with this qu.

There are a couple of ideas that I want to express as conclusion to all this:

• In an ideal world these things should not happen. But this is one of the lesser problems with this non-ideal world, so bear with it.
• Paths should not be stored in QStrings, even if they can (and they do) store this kind of pathnames, because if you try to 'encode' its contents (in the Unicode sense; that is, convert it to an encoding like UTF-8[4]) you get farts or barks at best. Yes, you always have constData() but from QString's class reference there is no warranty that this will keep being the case[6].
• In fact, QString's class reference says at some point: «[one case] where QByteArray is appropriate are when you need to store raw binary data...», and as I already wrote, «[t]his would be the case for paths; you need the bytes».
• QFile and QDir can only be created from QStrings. I'm not sure if, given all I wrote, that's right.

The good news is that satyr now can play any file that the backends can whatever their filename-as-string-of-bytes is, I'm a little bit happier about it, I got another contribution to KDE and might even have to close a lot of bugs!

satyr pykde python phonon

[1] That question is only legal in Nederlands[2] and very few others cities in the planet.

[2] Actually is not legal. See this wikipedia article.

[3] I might pull up my sleeves again and fix that.

[4] You might have already know this, but if you not: you cannot print Unicode, because Unicode is not and encoding. You have to encode it first. Hence, the toLatin1(), toUtf8() and similar QString methods, and also the inverse from*().

[5] Of course the equivalent C++ code also works, with path being a char *.

[6] And in the case of PyQt4, that method is not even available. But I already globed about it.

More than two months ago I globed about QStrings and paths. The problem was this: my app accepts paths via command line, which are processed via KCmdLineOptions; which in turn converts everything to QStrings. What I wanted were paths, which are more like QByteArrays, not QStrings (because the latter have internally an unicode representation; more on that later). Including PyQt4 in the equation forced me to resort to QByteArray to get the path as a str instead of using QString.constData() (PyQt4 doesn't export that function). But that's only the beginning of the problem.

Take for instance this situation. I have a music collection that I've been building for years now (more that 10, I think). In the old times of this collection the filenames were encoded in iso-8859-1. Then the future came and converted all my machines to utf-8. But only the software; the filesystems were in one way or another inherited from system to system, from machine to machine. So I ended with a mixture of utf and iso filenames, to the point where I have a file whose filename is in iso, but the directory where it is is in utf. Yes, I know, it is a mess. But if I take any decent media player, I can play the file allright. That's because the filesystem knows nothing of encodings (otherwise it would reject badly encoded filenames).

I just spent last saturday making sure that satyr only stored filepaths in strs, not unicodes or QStrings. It took concentration, but having just a bunch of classes and only 3 or 4 points where the filepaths are managed it wasn't that difficult. Still, it took a day. But then, as I mentioned in that post, Phonon the is not able to play such files... or so I thought.

If you run satyr after executing export PHONON_XINE_DEBUG=1 you'll see a lot of Phonon debug info in the console (not that there is another way to run satyr right now anyways). Among all that info you'll see lines such as these two:

void Phonon::Xine::XineStream::setMrl(const QByteArray&, Phonon::Xine::XineStream::StateForNewMrl) ...
bool Phonon::Xine::XineStream::xineOpen(Phonon::State) xine_open succeeded for m_mrl = ...

If you're sharp enough (I'm not; sandsmark from #phonon had to tell me) you'll note the mention of MRL's. MRL's are xine's URL for media. As any URL, they can (and most of the time must) encode 'strange' characters with the so-called "percent encoding". This means that no matter what encodings the different parts of a filepath is in, I just add file:// at the beginning and then I can safely encode it scaping non-ascii characters to %xx representations... or that's what the theory says. One thing to note is that the file:// part must not be scaped; xine complains that the file does not exist in that case.

Looking for help in Qt's classes one can find QUrl and the already known QByteArray. I can call QByteArray.toPercentEnconding() from my str and feed that to QUrl.fromPercentEncoding() (which strangely returns a QString, which is exactly what we're avoiding) or QUrl.fromEncoded(). But then the first function encodes too much, replacing :// with %3A%2F%2F. No fun.

Ok, let's try creating a QByteArray with only the file:// and then append() the toPercentEncoding() of the path only. It works:

PyQt4.QtCore.QByteArray('file://%2Fhome%2Fmdione...%2F%C3%9Altimo%20bondi%20a%20Finisterre%2F07-%20La%20peque%F1a%20novia%20del%20carioca.wav')

But then calling QUrl.fromEncoded() gives:

PyQt4.QtCore.QUrl("file://xn--/home/mdione.../ltimo bondi a finisterre/07- la pequea novia del carioca-wkmz60758d.wav")

The URL got somehow puny-encoded, which of course xine doesn't recognize for local files.

Another option is to create an empty QUrl, call setEncodedUrl() with the ParsingMode to QUrl.StrictMode so we avoid 50 lines of code that start here[1] that try to escape everything all over again (and I already had some double-or-even-triple-enconding nightmares parsing RSS/Atom feeds last year, thank you), but we get puny-encoded again (maybe it is 'pwny-encoded'?).

Last resort: backtrack to the point were we created only one QByteArray with the path and call toPercentEncoding(); feed that to the method setEncodedPath() of an empty QUrl. Then we add the last piece calling setScheme('file') and we're ready! Of course we're not:

PyQt4.QtCore.QByteArray('file:%2Fhome%2Fmdione...%2F%C3%9Altimo%20bondi%20a%20Finisterre%2F07-%20La%20peque%F1a%20novia%20del%20carioca.wav')

Notice the lack of the two // after file:? xine doesn't like it; hence, I don't either.

Ok, this post got too long. I hope I can resolve this soon, I already spent too much time on it. At least a good part of it was expaining it, so others don't have to suffer the same as I did.

BTW, satyr will shortly be released, whether I fix this bug or not.

satyr pykde phonon

[1] Look at the size of that file! 6k lines to handle URL's! Who would say it was so difficult... Once more I'm remembered of how lucky I am to have this libraries at the tips of my fingers, yay!

For a couple of months I've been globbing about PyKDE4 stuff, and laterally talking about my last project: satyr. satyr (it's name should always be written in lowercase) should have the following features:

• The PlayList and the Collection(s)[0] are the same thing.
• Yours is a Collection of Albums, nothing else[3].
• Some Albums are from the same artists and some are compilations[3].
• If you want an ephemeral playlist you could queue songs[1].
• If you want non-ephemeral playlists, then this player is not for you.
• Ability to search à la xmms, but in the same interface[2]
• Tag reading and writing[3].
• Order you collection based on the tags[3].
• The collection discovers new files and adds them to the playlist on the fly[4].
• Be able to use all the program only with your keyboard (die, mouse, die!)

This and other features should be available soon. The coding has been fast lately, mainly because the Qt/KDE libs are fantastic to work with. The only thing I couldn't do was to read the tags before playing them, so I relied into the kaa libraries.

The project is hosted in savannah, and right now there is no tarball (It's marked as alpha state because I sent a couple of tarballs to some friends who asked for them), so the only way right now is to branch anonimously the bazaar repo. I hope you download and enjoy it as much as I do.

satyr pykde

[0] The support for several collections is not complete yet.

[1] Functionality available via dbus only at the moment.

[2] This is with the current GUI. I'm also thinking in several/pluginable GUI's

[3] Not yet available.

[4] Of course this only works if it's running. Otherwise, you can always ask for a rescanning[1].

There's a problem with Phonon. If you try to play a file in a format that it doesn't support, instead of failing or simply sending the finished() signal it just does nothing. I'll have to look more deeply into this and figure out if it's a bug or feature or how to work-around it.

For now I just decided to use python-magic to decide if the file is playable or not. This module uses libmagic, the same library that the file utility uses. But I hit a problem with it. One of its ussages is to call magic.file() with the filename, but if the filename contains a non-ascii character you get something like this:

Traceback (most recent call last):
  File "satyr.py", line 173, in next
    self.play ()
  File "satyr.py", line 128, in play
    mimetype_enc= self.magic.file (self.filename)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 37: ordinal not in range(128)

I found a workaround. There's another usage, which is to call magic.buffer() with a piece of data to recognize. So I just simply open() the file, read() 4KiB and pass the data to that function.

f= file (self.filename)
data= f.read (4096)
mimetype_enc= self.magic.buffer (data)

Now it works properly. I will file a bug report to the file package once I resort out my mail setup (I cannot send bugs with reportbug).

python satyr

Satyr handles paths. There are some problems with paths and (sigh) encondings. Of those, here are two: there's no way to know in which encoding the filenames in a filesystem are enconded (f.i., there's no way to ask the filesystem), and even if that were possible, the filenames might not even be enconded in that enconding. In these (still!) transitioning times, lots and lots and shitloads of filesystems are used in UTF-8 environments, but some filenames are still in old ISO-8859-1 or whatever the system was using before.

Then comes QString. I'm taking a path from the command line; this path is the location of the (right now only) Collection for the player. I'm handling the command line using KCmdLineOptions, which returns QStrings. As we all know, QString, just like the unicode type in Python, handles all the data internally as Unicode, which is The Right Thing™. If you really need the internal data, say as bytes, you can always call the constData() method and be happy with it[1]. This would be the case for paths; you need the bytes.

Then comes PyQt4. For some reason, which maybe I will ask in the pyqt devel ML[2], constData() is not available. What to do? Well, that's what this post is about. What you're about to read is hacky as it can be, but then it works. I might feel dirty, but I can live with it. As long as I mark it as a utter/über hack and promise to revert it once that's possible...

# path is a QString
qba= QByteArray ()
qba.append (path)
path= str (qba)
# now path is a list of bytes/string.

Even if this part of the bug is fixed, then Phonon.MediaSource or Phonon.MediaObject.play() fails when feeded that same path with this message:

ERROR: backend MediaObject reached ErrorState after  1 . It seems a KioMediaStream will not help here, trying anyway.

or simply refusing to continue. Sure, in my case I should simply ignore the filename and inform the user what's going on, but sometimes you can't be so gentle.

satyr pykde python phonon

[1] blah.

[2] but then it doesn't make much sense now since Phil wants to get rid of QString (for several reasons, which might most possibly include this one).

In my last post I said «The next step is to make my Player class to export its methods via DBus and that's it!». Well, tell you what: is not that easy. If you try to inherit from QObject and dbus.service.Object you get this error:

In [3]: class Klass (QtCore.QObject, dbus.service.Object): pass
TypeError: Error when calling the metaclass bases
    metaclass conflict: the metaclass of a derived class must be a
    (non-strict) subclass of the metaclasses of all its bases

This occurs when both ancestors have their own metaclasses. Unluckily Python doesn't resolve it for you. The answer is to create a intermediate metaclass which inherits from both metaclasses (which we can obtain with type()) and make it the metaclass of our class. In code:

class MetaPlayer (type (QObject), type (dbus.service.Object)):
    """Dummy metaclass that allows us to inherit from both QObject and
    d.s.Object"""
    pass

class Player (QObject, dbus.service.Object):
    __metaclass__= MetaPlayer
    [...]

Is that it now? Can I go and do my code? Unfortunately no. See this:

qdbus org.kde.satyr
/
/player
Cannot introspect object /player at org.kde.satyr:
org.freedesktop.DBus.Python.KeyError (Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.5/dbus/service.py", line 702, in _message_cb
    retval = candidate_method(self, *args, **keywords)
  File "/usr/lib/pymodules/python2.5/dbus/service.py", line 759, in Introspect
    interfaces = self._dbus_class_table[self.__class__.__module__ + '.' + self.__class__.__name__]
KeyError: '__main__.Player'
)

This is the class dbus.service.Object complaining something else. It's getting late here and I'm tired, so I'll continue tomorrow.

dbus python pykde satyr

Continuing with the development of Satyr, which doesn't have any GUI. I thought it would be faster to make a DBus interface that a good GUI, not to mention more interesting.

The code snippet of today is this:

class DBusServer (dbus.service.Object):
    def __init__ (self):
        bus= dbus.SessionBus ()
        bus_name= dbus.service.BusName ('org.kde.satyr.dbus.test', bus=bus)
        dbus.service.Object.__init__ (self, bus_name, "/server")

    @dbus.service.method('org.kde.satyr.dbus.test', in_signature='', out_signature='')
    def bing (self):
        print "bing!"

    @dbus.service.method('org.kde.satyr.dbus.test', in_signature='', out_signature='')
    def quit (self):
        app.quit ()

dbus.mainloop.qt.DBusQtMainLoop (set_as_default=True)
dbs= DBusServer ()

sys.exit (app.exec_ ())

This simply defines a class which registers itself with the session bus under the name org.kde.satyr.dbus.test, exporting itself under the path /server and then defining a method that goes bing! :) and another one that quits the app. Note the decorator for the methods.

You might notice the dbus.mainloop.qt.DBusQtMainLoop (set_as_default=True) call. This is needed because both Qt and DBus in asyncronous mode (which is the one we're using and the only one that works under Qt or Gtk, AFAIK) both have their own event loops, and that makes some kind of magic that let both loops coexist without blocking the other. This must be called before connecting to the bus; otherwise, you get this error:

RuntimeError: To make asynchronous calls, receive signals or export objects, 
D-Bus connections must be attached to a main loop by passing mainloop=... 
to the constructor or calling dbus.set_default_main_loop(...)

So, let's test the beast. We run the script in one terminal and in the other:

mdione@mustang:~/src/projects/satyr/live$ qdbus | grep satyr
 org.kde.satyr.py-10154
 org.kde.satyr.dbus.test
mdione@mustang:~/src/projects/satyr/live$ qdbus org.kde.satyr.dbus.test
/
/server
mdione@mustang:~/src/projects/satyr/live$ qdbus org.kde.satyr.dbus.test /server
method void org.kde.satyr.dbus.test.bing()
method QString org.freedesktop.DBus.Introspectable.Introspect()
mdione@mustang:~/src/projects/satyr/live$ qdbus org.kde.satyr.dbus.test /server org.kde.satyr.dbus.test.bing
mdione@mustang:~/src/projects/satyr/live$ qdbus org.kde.satyr.dbus.test /server org.kde.satyr.dbus.test.quit

And in the other console:

mdione@mustang:~/src/projects/satyr/live$ python dbus_test.py
bing!

It goes bing!... and then finishes. The next step is to make my Player class to export its methods via DBus and that's it!. More info in the Python DBus tutorial.

dbus python pykde satyr