So, after finally getting around to fixing that annoying Plasma Task Manager bug (and I must say, I can’t wait until that applet is ported to to QML, the animation/state change code in there is a mess), I noticed a discussion on IRC about file notification APIs and how there seems to be no good in-kernel solutions at the moment. And after reading Vishesh’ blog post I got an idea for how to maybe solve it using LD_PRELOAD.
The idea is that we have a itsy bitsy library that we inject into every process, by setting LD_PRELOAD in the startkde script. This way the dynamic linker uses our custom functions instead of the normal ones. So when we LD_PRELOAD in a library with a custom rename() function (the standard C library rename() function is used for moving files), it calls our library instead of glibc. Our library in turn tries to connect over a local Unix socket to a central process, and reporting in the old and new filename/path, as well as the current working directory (in case we are working with relative paths). Finally we call the normal glibc rename() function, which does what it does best, actually move the file.
So, I thought why not, and quickly wrote a quick and dirty proof of concept. It is available in git: http://quickgit.kde.org/index.php?p=scratch%2Fsandsmark%2Fkfilemon.git&a=summary
While it might initially seem like a fragile solution, I think it should be pretty reliable in practice (funny pictures aside). It might be crazy enough to actually work. The only thing that won’t work is if applications launched outside of a KDE session move files around, but they have no business moving files around in users home folders anyways (and a solution to this is to set LD_PRELOAD even earlier, for example in /etc/profile.d/ somewhere).
So I thought I’d blog about it, so I can get some useful criticism and learn why it shouldn’t be done this way.
sandsmark's blog 
Recently, there was a thread about recursive monitoring on the Qt mailing list, too: http://lists.qt-project.org/pipermail/development/2012-July/005279.html
Using LD_PRELOAD might work as a workaround but I think this should really be fixed in the kernel – KDE is not the only software that might want to use it.
fanotify seems to provide recursive monitoring but does not notify userspace in case of moves / renames. But how hard can it be to add support for that? The hooks for fsnotify_rename are already in vfs_rename because of inotify – fanotify only has to forward the notification to userspace…
Apart from that: What was the reason again not to use xattr to store an uuid for each file if the file system supports it?
That way we could identify the file even after being moved when we missed the move notification. This can easily happen when the user accesses the files from a non-KDE session, e. g. from the command line or when they are on a removable device.
Well, KDE isn’t Linux-only, so we can’t really rely on the kernel, imho. And this need has been identified for a long time, and communicated with the kernel people, and nothing has happened (well, people develop patches, but nothing useful gets merged to mainline, it seems).
As for xattr, I’m not sure why it isn’t used. Maybe it doesn’t have widespread support?
Extended Attributes are a kind of safety net that can help us, but they are not a solution to our problem. Here is why –
Lets say we store some unique identifier in the file’s extended attributes. Now when the file is moved, someone needs to inform the applications that the file has moved, or a new file has been created. Suppose we get a notification that a new file has been created, then we can just look at its extended attributes and discover it was actually that file.
Without any kind of notification, our database (in Nepomuk) will assume that the data it has is correct, and it’s only when it tries to list the file, it will discover that the file no longer doesn’t exist at that location. At that point we could do an entire file system scan to discover any new files, but that is not practical.
Also, before you point out that we can do system wide notifications via fanotify – Yes, we can. But move events are totally ignored by fanotify. Even if we got a simple – “hey! this file just appeared over here”, we could make it work.
> Also, before you point out that we can do system wide notifications via fanotify – Yes, we can. But move events are totally ignored by fanotify. Even if we got a simple – “hey! this file just appeared over here”, we could make it work.
Sure. That is why I said support for that should be added to fanotify.
I know that the maintainer (Eric Paris i think) doses not seem to have the time to work on this. And I am afraid that if no other kernel developer is willing to spend time to implement this feature, someone from the KDE or Qt community will have to look into this… At least if we do not want to use some hack.
> Sure. That is why I said support for that should be added to fanotify.
I know that the maintainer (Eric Paris i think) doses not seem to have the time to work on this. And I am afraid that if no other kernel developer is willing to spend time to implement this feature, someone from the KDE or Qt community will have to look into this… At least if we do not want to use some hack.
Oops. I meant system wide file creation events do not cover move events.
Anyway, I have been looking at the code a little bit, but I’m fairly new to kernel development so it might take me a while. Would be a lot easier if he did it, since he knows the code base really well. Oh well, lets see if something happens.
Interesting idea, but IMHO it should be made sure it works for additional side logins, as well. For example, if I log into my workplace machine via SSH while in the lab. Back at my workplace (after forgetting about it during lunch) I may want to see notifications. I also know that some groups have special lab user accounts to control special devices which may be used concurrently by different people. Notifications would be really useful for them and annoying if they don’t work as expected.
I recommend to really investigate your /etc/profile.d/ idea.
I think this is the completely wrong way to approach the problem. If anything its an interesting hack, nothing more. The problem is that you’re not guaranteed at all that your library is really preloaded in all processes that need it, since the user may start an app through a script or through another app which sets LD_PRELOAD (IIRC valgrind uses LD_PRELOAD itself). So yes this is fragile, far too fragile to be used as a permanent solution.
You’re saying that KDE isn’t Linux-only, but outside of *nix (Or maybe linux, I don’t know wether all Unices support LD_PRELOAD) you’re back to needing another solution again – neither windows nor Linux support LD_PRELOAD.
LD_PRELOAD isn’t a Linux thing, it’s in the dynamic library loader (glibc on most systems), and I’m pretty sure it is supported on all major unices.
And only exceptionally is LD_PRELOAD replaced (I doubt people usually run their applications in Valgrind). And if something absolutely needs to override it, you can add several of them (space-separated), just like you modify the $PATH variable.
Please do not do this, it’s a very bad solution!
http://blog.tridgell.net/?p=141
Also keep in mind that LD_PRELOAD does not work on static executables nor on suid/sgid executables and that it can be overridden by the user (who wants to use LD_PRELOAD for debugging and sure doesn’t expect it to be already set systemwide! It’s supposed to be a debugging feature). It is not a reliable solution.
How many (completely) static executables (I don’t think there’s any proper static executables around anymore, tbh.) or SUID/GUID executables move files around in users home folders?
And how is it unreliable?
As for the blogpost; I’m not doing anything remotely like it; I just sniff and forward the request, and only a very standard one at that.
Also, if you read the comments to the blogpost you’ll see even that case isn’t as clear cut as tridge wants it to be.
It won’t work for anything that does not use the LD_PRELOAD trick, which may include rather more tools than may seem obvious at first. Aside from the standard terminal utilities (mv) you would need to consider various interpreters/VMs (Python, Perl, Ruby, Java, Mono, SH?) and then in turn you would need to filter out what is not relevant in the KDE context or possibly inaccessible.
The whole point of this is that it works for _everything_, because everything uses glibc’s rename(). This includes terminal utilities like mv, Python, Ruby, Qt, whatever.
I guess I am missing something here but to me there seem to remain at least two issues with startkde: (1) LD_PRELOAD is set very late into the boot up process, so a lot of shared libs which may call rename() have been loaded and linked [perhaps I simply don’t understand the linker very well here] e.g. glib, qt…; (2) programs may run and rename() files without startkde ever having been invoked e.g. SSH sessions or alternative logins…
That’s being facetious, right? Because “mv” has no business moving files around in a user’s home folder, if that user runs it from a konsole?
Wait, what?
Sorry, s/konsole/console/ I actually meant a separate TTY/PTY. Then I went back and edited the comment, thinking I could make it “clever”. And broke it.
Yes, that’s why I thought we could maybe put it in profile.d. Or in ld.so.preload, so it is used for everything everywhere.