CDBS Packaging: package relationship management

2013-05-05T05:19:45-07:00

(It took me a while to come up with new CDBS packaging series post not because I stopped using CDBS just because I was procrastinating myself as busy)

This is the second post in the CDBS packaging series. In this series I'm going to talk about package relationship management.

The better example where this feature is useful is packages where build-depends and run-time dependencies overlap. Most of the Perl modules which have test suites have build-depend and run-time dependency intersection. So let me take example of a Perl module.

First lets see control file of a Perl package which is not using CDBS and then let me explain how CDBS can help you improve the situation. I choose libxml-libxml-perl lets see part of control file which includes Build-Depends Depends Suggests Recommends.

Source: libxml-libxml-perl
Maintainer: Debian Perl Group <pkg-perl-maintainers@lists.alioth.debian.org>
Uploaders: Jonathan Yu <jawnsy@cpan.org>,
 gregor herrmann <gregoa@debian.org>,
 Chris Butler <chrisb@debian.org>
Section: perl
Priority: optional
Build-Depends: perl (>= 5.12),
 debhelper (>= 9.20120312),
 libtest-pod-perl,
 libxml2-dev,
 libxml-namespacesupport-perl,
 libxml-sax-perl,
 zlib1g-dev
Standards-Version: 3.9.4
Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libxml-libxml-perl.git
Vcs-Git: git://anonscm.debian.org/pkg-perl/packages/libxml-libxml-perl.git
Homepage: https://metacpan.org/release/XML-LibXML/

Package: libxml-libxml-perl
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, ${perl:Depends},
 libxml-namespacesupport-perl,
 libxml-sax-perl
Breaks: libxml-libxml-common-perl
Replaces: libxml-libxml-common-perl
Description: Perl interface to the libxml2 library
 XML::LibXML is a Perl interface to the GNOME libxml2 library, which provides
 interfaces for parsing and manipulating XML files. This module allows Perl
 programmers to make use of the highly capable validating XML parser and the
 high performance Document Object Model (DOM) implementation. Additionally, it
 supports using the XML Path Language (XPath) to find and extract information.

So 2 packages are both in Build-Depends and Depends field

libsax-xml-perl
libxml-namespacesupport-perl

So in this situation there is a possibility that we miss to add one or both of these packages into Depends field, I'm not saying we will surely miss but we might after all we are all human beings.

So how can we improve the situation using CDBS? Let me go through step by step on what we need to do.

Create a file called control.in with same above contents but slight modification in Build-Depends and Depends section. I will display the diff below to avoid re-pasting entire file again and again.

--- debian/control      2013-04-28 23:08:11.930082600 +0530
+++ debian/control.in   2013-05-04 20:51:18.849680419 +0530
@@ -5,13 +5,7 @@
  Chris Butler <chrisb@debian.org>
 Section: perl
 Priority: optional
-Build-Depends: perl (>= 5.12),
- debhelper (>= 9.20120312),
- libtest-pod-perl,
- libxml2-dev,
- libxml-namespacesupport-perl,
- libxml-sax-perl,
- zlib1g-dev
+Build-Depends: @cdbs@
 Standards-Version: 3.9.4
 Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libxml-libxml-perl.git
 Vcs-Git: git://anonscm.debian.org/pkg-perl/packages/libxml-libxml-perl.git
@@ -20,8 +14,7 @@
 Package: libxml-libxml-perl
 Architecture: any
 Depends: ${shlibs:Depends}, ${misc:Depends}, ${perl:Depends},
- libxml-namespacesupport-perl,
- libxml-sax-perl
+ ${cdbs:Depends}
 Breaks: libxml-libxml-common-perl
 Replaces: libxml-libxml-common-perl
 Description: Perl interface to the libxml2 library
@@ -30,4 +23,3 @@
  programmers to make use of the highly capable validating XML parser and the
  high performance Document Object Model (DOM) implementation. Additionally, it
  supports using the XML Path Language (XPath) to find and extract information.
-

Next we need to have something like below in the rule files

#!/usr/bin/make -f

include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/rules/utils.mk
include /usr/share/cdbs/1/rules/upstream-tarball.mk
include /usr/share/cdbs/1/class/perl-makemaker.mk

pkg = $(DEB_SOURCE_PACKAGE)

deps = libxml-libxml-perl libxml-sax-perl
deps-test = libtest-pod-perl

CDBS_BUILD_DEPENDS +=, $(deps), $(deps-test)
CDBS_BUILD_DEPENDS +=, zlib1g-dev, libxml2-dev, perl (>= 5.12)

CDBS_DEPENDS_$(pkg) = , $(deps)

So basically we moved all Build-Depends and Depends to rules file. The common ones are placed in deps variable and assigned to both Build-Depends and Depends. CDBS uses following variables for package relationship management.

CDBS_BUILD_DEPENDS: This variable helps you manage Build-Depends field and all you need to do is in your control.in files Build-Depends field put place holder @cdbs@
CDBS_DEPENDS: This field can be used to manage Depends field of binary package for each binary package you need to have one CDBS_DEPENDS_pkgname variable with depends assigned to it. And in your control.in append ${cdbs:Depends} to Depends field.
CDBS_PROVIDES, CDBS_BREAKS, CDBS_RECOMMENDS, CDBS_PREDEPENDS, CDBS_SUGGESTS, CDBS_REPLACES: all these does the job what you think it does :-).

Other than CDBS_BUILD_DEPENDS all other variables work using substvars i.e. CDBS will put the respective substitutions in pkgname.substvars file which will be used during deb creation to replace things in control file.

So to make CDBS generate new control file run the below command

DEB_MAINTAINER_MODE=1 fakeroot debian/rules debian/control

Basically this command needs to be executed before starting build process if you miss your changes will not be reflected into debian/control. Additionally this feature is Maintainer Mode helper tool because Debian policy prohibits change of debian/control during normal package build.

So what is the benefit of using this feature of CDBS? I've listed some of them which I felt are obvious.

When there is intersection in Build-Depends and Depends this feature from CDBS is helpful. As I shown above put all intersecting dependencies in common variable and appropriately assign them wherever it is required. Thus we can avoid possible missing of some of run-time dependencies due to human error.
It is also possible that newer version of package requires specific version of some package (mostly libraries) and we updated build dependencies but we might forget to do the same in Depends, by using this feature we can make sure we will not miss such minute details.

One last thing I want to point out is if you are NMUing a CDBS package

NMUs need not (but are encouraged to) make special use of these tools. In particular, the debian/control.in file can be completely ignored.

Before closing down the post, If you find some mistake in the post please let me know either through comments or through the email.

Soon I will be back with new CDBS recipes till then cya.

Included file 'rss_footer.html' not found in _includes directory

Tribute to Beloved Teacher

2013-04-27T22:05:53-07:00

Today I'm writing this blog with saddened heart. My mentor and a best friend Dr.Ashokkumar is no more. He died yesterday after fighing with Lymph Node cancer.

Ashokkumar or Ashok sir that is how we all students used to address him, was a Professor in Information Science Engineering in NMAM Institue of Technology and recently transferred to Computer Science Engineering Departement. My last meeting him with him was last year December during which he was looking every bit okay other than he had knee pain because of which he couldn't walk freely. But I never imagined that it will be my last meeting with him.

Ashok sir was also behind the FLOSS event that took place in NMAM Institute of Technology including MiniDebconf 2011 which saw 2 of the foreign DD's Christian Perrier and Jonas Smedegaard.

It was because of his first organized FLOSS event which I volunteered called Linux Habba I entered into the FLOSS world. This means its because of him that I started my FLOSS world journey and reached to my current level. Also it was because of his motivation I started writing this blog which I continue till this day.

I whole heartedly thank Ashok sir for teaching me guiding me and motivating me during my difficult times. You will always be remembered through out my life. May your soul Rest In Peace.

Here are the 2 pics of Ashok sir taken during Minidebconf (Credits: Christian Perrier and Kartik Mistry)

Good bye Sir :-(

Included file 'rss_footer.html' not found in _includes directory

Handling C++ symbols that vary across architectures

2013-04-14T06:15:25-07:00

Disclaimer

I'm no symbol expert and dealing with symbol files for first time from the time I started packaging. What I did here is depending on some suggestions I got in #debian-mentors.

If you think what I did was wrong please enlighten me :-).

Problem

Recently 2 of my library packages pugixml and ctpp2 got accepted into Debian archive and when buildd tried to build them on remaining architectures other than one for which I uploaded (amd64) builds failed. This was expected as symbols file which I generated was for amd64. As usual I got 2 serious bug reports #704718 and #705135.

Solution

First I was not sure how to handle this. I read article on symbols files handling by rra [1] and tried to use pkgkde-symbolshelper tool only to quickly figure out that I need to use pkgkde_symbolshelper addon for dh sequencer. But this was not possible for me as I was using CDBS for packaging.

I did a quick chat on #debian-mentors and some one suggested me to tag symbols which vary across architecture with (c++) tag. First I was not sure but after reading dpkg-gensymbols man page understood that I need to replace the entire mangled symbols lines with de-mangled version tagged with (c++) in the beginning.

But this was hectic job searching for each deleted symbols and replacing it. So I thought of writing a script to do the job and after struggling for 3 days (yeah I was bit dumb that I didn't read manual date: "2013-04-14 18:39:20 +0530" pugixml and ctpp2 package using it, which is now waiting for Jonas for uploading it.

Here is the script

#!/bin/bash
set +x

if [ $# -lt 3 ]; then
    echo "Usage: $0 failed_buildlogs_directory symbols_file package_version"
    exit 2
fi
    
BUILD_LOG_DIRECTORY=$1
SYMBOLS_FILE=$2
PACKAGE_VERSION=$3
VERSION_TO_REPLACE=" $PACKAGE_VERSION\""

for LOGFILE in $(ls $BUILD_LOG_DIRECTORY/*.build); do
    for i in $(grep '^-\s_Z' $LOGFILE | perl -pe 's/-//g;'); do
        if [ $i = $PACKAGE_VERSION ];then
            continue
        fi
        demangled_version="\""$(echo $i" "$PACKAGE_VERSION | c++filt)"\""
        tagged_version="(c++)"${demangled_version%$VERSION_TO_REPLACE}"\" "$PACKAGE_VERSION
        escaped_tagged_version=$(echo $tagged_version | sed 's/\&/\\\&/')
        sed -i "s#$i $PACKAGE_VERSION#$escaped_tagged_version#" $SYMBOLS_FILE
   done
done

So basically to make this work we need all buildlogs to be downloaded from buildd's again this was easy thanks to rra for developing pkgkde-getbuildlogs :-).

Once you have build logs directory run the above script as follows

cppsymbol_replace.sh path_to_buildlogs path_to_symbol_file upstream_version

After replacing symbols I tried to build package on i386 chroot and build passed successfully but lintian told me that there are symbols which have Debian version appended to it and this might lead to date: "2013-04-14 18:42:42 +0530" back to mentors :-).

It is this time I really understood concept of mangled names generated by compiler and why it vary across the architecture ;-).

This time some one by nick pochu suggested me to pass option -v with package version to dpkg-gensymbols to make it generate symbols with package version and not Debian version.

The following probably needs to be done if package uses dh sequencer but I'm not sure as I've not tested it. If wrong please correct me.

override_dh_makeshlibs:
    dh_makeshlibs -- -v$PACKAGEVERION #package version needs to be either extracted using parsechangelog or manually fed

If you are using CDBS this is pretty simple. Just add following to rules

DEB_DH_MAKESHLIBS_ARGS_$(pkgname) += -- -v$(DEB_UPSTREAM_VERSION)

I noticed that when I provide (c++) tagged de-mangled name dpkg-gensymbols simply replaces it with proper mangled name but the deletion doesn't trigger an error in dpkg-gensymbols.

This script me allowed to replace 128 symbols which were very tricky and long with de-mangled and tagged version in ctpp2, so I hope it should work across different packages without any problem. Only silly mistake I did was occurrence of & symbol in function making sed go mad which I took one full day to debug :facepalm:.

So that's it folks, if you see something wrong in what I did please let me know through the comments.

[1] http://www.eyrie.org/~eagle/journal/2012-01/008.html

Included file 'rss_footer.html' not found in _includes directory

Multi-Arch lintian check for font packages

2013-03-08T07:14:50-08:00

I've provided a new patch #701061 for lintian to warn about font packages that are not marked as Multi-Arch foreign or allowed. Its already included in the lintian by Neils Thykier and will be part of version 2.5.12. The following tag has been implemented

font-package-not-multi-arch-foreign

A Bit of History for this implementation is as follows:

We got a bug report on one of the fonts in pkg-fonts team that its not been installed in i386 architecture on a amd64 multi-arch system #694864. We were first confused but Daniel Kahn Gillmor pointed that we indeed need to mark all font packages as Multi-Arch foreign. He proposed that we should write a lintian check for this which I volunteered to do and then forgot!. Recently I was checking my QA page and landed into Ubuntu's page of my package where I saw they were patching imported font package and marking them as Multi-Arch: foreign and I suddenly remembered my promise! and this patch was the result of same enlightenment :-).

Since there is huge number of font packages maintained by pkg-fonts devel we targeted this for Jessie release.

Here by I request all font package maintainers to consider marking their packages as Multi-Arch foreign. I also request people to join us on pkg-fonts-devel and help us doing this for all font packages maintained by the team, we really lack people in the team.

Included file 'rss_footer.html' not found in _includes directory

CDBS Packaging : copyright file support

2013-02-11T01:57:49-08:00

One of the thing CDBS provides is support for verifying / generating debian/copyright file. This feature will be help full if new files show up in newer releases. It can also be helpful to generate the first version of copyright file. But don't blindly follow this feature, you need to cross verify the things it generates.

To use this feature add the following line to your debian/rules file first

include /usr/share/cdbs/1/rules/utils.mk

Now create a file called debian/copyright_hints using touch command

touch debian/copyright_hints

Now run the following command to allow CDBS to do its magic and generate a new file called debian/copyright_newhints

DEB_MAINTAINER_MODE=1 fakeroot debian/rules build

As you see we set the variable DEB_MAINTAINER_MODE which means this is feature to help the maintainer rather than targeted for automatic builds. So if there is no new things added to the new release you should see something like this in the build log.

Scanning upstream source for new/changed copyright notices...


licensecheck -c '.*' -r --copyright -i '^debian/(changelog|copyright(|_hints|_newhints))' -l '99999' * | /usr/lib/cdbs/licensecheck2dep5 > debian/co$
yright_newhints
19 combinations of copyright and licensing found.
No new copyright notices found - assuming no news is good news...

In case there are new files added you should get output similar to below

Scanning upstream source for new/changed copyright notices...

licensecheck -c '.*' -r --copyright -i '^debian/(changelog|copyright(|_hints|_newhints))' -l '99999' * | /usr/lib/cdbs/licensecheck2dep5 > debian/co$
yright_newhints
0 combinations of copyright and licensing found.

Format: http://www.debian.org/doc/packaging-manuals/…
Upstream-Name: FIXME
Upstream-Contact: FIXME
Source: FIXME
Disclaimer: Autogenerated by CDBS

...

To fix the situation please do the following:
  1) Fully compare debian/copyright_hints with debian/copyright_newhints
  2) Update debian/copyright as needed
  3) Replace debian/copyright_hints with debian/copyright_newhints

So as the last 3 points says you need to compare your copyright_hints and the newly generated file by CDBS copyright_newhints for difference and update the debian/copyright as required and replace the existing debian/copyright_hints with debian/copyright_newhints.

Tips

This feature can be used to generate the copyright file for the first time you need a copyright file with only debian/* copyright in it.
If you are using it to generate copyright file for the first time be sure to cross verify the things mentioned in the file than just blindly copy pasting.
If there are binary files like images are involved in your project be sure to clean up those from your copyright_hints before trying to build the package. Because of too much of binary cruft in any files means dpkg-source will be angry at you :-).

Last but not the least, I know CDBS provides some really cool features which we will explore with next few posts but always keep in mind this quote by Jonas. (Of course don't ask me to prove that Jonas did say it as there is no pgp signature of him with quote to verify ;-) )

CDBS is tool to help packagers and does not involve any magic.

Included file 'rss_footer.html' not found in _includes directory

New Series of Posts on CDBS packaging

2013-02-10T23:47:31-08:00

Well this is just an introductory post to set of posts I'm going to write down here on my blog. Idea is to improve the documentation of CDBS but I still don't have any idea on how I can patch the existing documentation, but in the mean time I'm learning new tricks from Jonas which I want to document some where so I thought my blog would be a good place as whole blog is in markdown format and its the same format me and Jonas agreed to convert the existing CDBS documentation.

So this blog will act as a documentation and these post will be then altered modified and put back into a proper CDBS documentation later.

~~I know many people don't like CDBS and most of it because there is no proper documentation, so I'm trying to improve the situation so sit back and enjoy the upcoming posts :-).~~

OK I was sloppy in giving theory on what others think about the CDBS thanks to Phill Hands and Jonas for enlightening me. So with these series I'm just trying to explain CDBS features with real examples and examples how you can tame nasty packaging beasts with CDBS easily.

Included file 'rss_footer.html' not found in _includes directory

Multi-Arch based Cross-Compilation

2013-01-26T07:37:26-08:00

Disclaimer:

I'm not cross-compile expert or have done cross-compilation before this post is not comparison between normal cross compilation and multi-arch based cross compilation, this post is purely about how I achieved cross compilation using Multi-Arch feature introduced by Debian.

Problem

Get a OS ready for the raspberry-pi computer I purchased recently. No I don't want to use Raspbian even though it claims as Debian itself. I want a clean armel based OS with only components what I require. This is also sort of experiment to improve boxer by adding rpi support to it. (Even though Jonas considers rpi as swamp ;-) )

Components need to be compiled:

u-boot
Linux Kernel

Architecture: armel

Solution:

The first thing to do is to add the required foreign architecture in my case armel, this can be done using below command.

dpkg --add-architecture armel

Then update package list with whatever package management utility, in my case I used aptitude. But before doing that we need to add a repository which provides us GCC cross compiler with gnueabi support. Yes it is not yet in Debian main and with my discussion with Jonas what I understood is since Wheezy is first of a kind with multi-arch capability we can expect cross compiler tool only from Jessie.

deb http://emdebian.org/~thibg/repo/ sid main

This repository is by emdebian contributor not completely sure about his name but he hangs in #emdebian with nick ThibG. According to Jonas repository is bit old but its working where as latest repository from wookey is broken, though this was told a couple of week back and I don't know about current status.

Now run aptitude update to get package list refreshed after adding the new repo. Once done we need to install gcc-4.7-arm-linux-gnueabi , run the following command and I suggest you use aptitude because it will help you resolve dependency mess easily. When dependency suggestion is given for first time try pressing e to enter full screen aptitude and navigate through the solution provided using . and , and select appropriate solution using !. Then press g to install the package.

aptitude install gcc-4.7-arm-linux-gnueabi

This scenario showed me real power of the aptitude which till then I was thinking as nutcase :-).

The above package installs all the cross compilation tools required beginning with prefix arm-linux-gnueabi- but it doesn't create a required symlink /usr/bin/arm-linux-gnueabi-gcc pointing to /usr/bin/arm-linux-gnueabi-gcc-4.7 which is required by all tools for cross compiling, add an alternative using following commands to fix the issue in the package tools ~~just create this symlink manually~~ and you should have GNU cross compiler ready on your system for [armel](http://wiki.debian.org/ArmEabiPort architecture.

Thanks to zumbi for pointing out the write way to fix the package

sudo update-alternatives --install /usr/bin/arm-linux-gnueabi-gcc arm-linux-gnueabi-gcc /usr/bin/arm-linux-gnueabi-gcc-4.7 46 
           --slave /usr/bin/arm-linux-gnueabi-g++ arm-linux-gnueabi-g++ /usr/bin/arm-linux-gnueabi-g++-4.7 
           --slave /usr/bin/arm-linux-gnueabi-gcov arm-linux-gnueabi-gcov /usr/bin/arm-linux-gnueabi-gcov-4.7
           --slave /usr/bin/arm-linux-gnueabi-gfortran arm-linux-gnueabi-gfortran /usr/bin/arm-linux-gnueabi-gfortran-4.7

Now to cross compile all you need to do is

export CROSS_COMPILE=arm-linux-gnueabi-

Once this package lands to Debian main, cross compiling will be way more easier.

Included file 'rss_footer.html' not found in _includes directory

Note to Self: Telling apt to download specific architecture information only

2012-12-27T07:02:17-08:00

If you have enabled multi-arch enabled in your system and one of the third party repository you use in /etc/apt/sources.list is not providing the foreign arch you have selected then fear not just do the below for that particular repository line.

deb [arch=arch1] uri distribution component

In my case mozilla.debian.net doesn't provide binary-armel hence I just did this

deb [arch=amd64] http://localhost:9999/mdn experimental iceweasel-aurora

and apt stopped complaining about missing foreign arch. Well this is present in man page of sources.list but I just wrote here so I can find it easily next time :-).

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 17/23 - 112012

2012-11-19T07:07:30-08:00

The last week was quite productive as I was on vacation and at home town but sadly I couldn't complete this post so again merging the work with this week but this week ain't much productive as I was tired from journey back and didn't get enough time to recover. So here it goes.

Debian Related

I did a little more fine tuning of friendica package to enable http://localhost/friendica on installation and shipped virtual host configuration for Apache and Nginx which user can simply copy to setup a proper friendica instance. I'm not sure if this is right to do or not but Drupal package does this but only it doesn't enable it during installation and leaves it up to user to create required symlink to enable http://localhost/drupal7. Well this is my first time in packaging webapp so I'm all open for suggestions
Helped Tobias Diekershoff prepare and upload the friendica package to mentors and file RFS bug. We got bug number #693504 and now we are looking for sponsors, if you are a DD reading this post and interested in federated social networking and can sponsor us it would be great :-)
Started learning CDBS and converted the Kiwix package to CDBS. I'm glad that Jonas is helping me out when I'm struck some where. Also kiwix is now updated to latest version 0.9~rc2 and I need to do some more fine tuning before asking for upload from Jonas.
I started helping Jonas to put up CDBS documentation. He suggested me to take over the documentation from duckcorp and then merge other documentation in CDBS git tree. As of now I only managed to separate out the chapters from docbook to markdown format. The conversion was done using pandoc but code formatting in docbook was not proper I guess so after conversion it needed some manual intervention in fixing and formatting code blocks. The repository for CDBS doc is here and as of now doesn't contain any new things.
Converted one more of my font package fonts-gubbi to CDBS format but I'm now facing a issue that CDBS is including docs inside the generated udeb which is not right, I already posted the question on how to handle this to build-common-hackers and waiting for Jonas to reply back.
I claimed the ITP #651600 on Roboto fonts from Android OS. After a short discussion with Fabian on pkg-fonts-devel I agreed to merge it with existing fonts-droid and renamed the source package to fonts-android which now builds fonts-droid and fonts-roboto. Package is ready but I'm still seeing if I can fix any bugs on fonts-droid before asking for a upload.
Wrote a simple script which was derived from droid fonts package from Fedora to get the android fonts from the upstream repository automatically. Enabled this in get-orig-source target for fonts-android. Since cloning a more than 600M repository is PITA I have a local copy of fonts-android and use it to create new release of fonts-android. Additionally I should say script is not well tested ;-)
suckless-tools has entered into Debian Experimental thanks to Jakub Wilk for a very nice review of the package which went in a long thread of more than 50 mails in debian-mentors. The package itself closed a shit load of bugs, but after upload I noticed there are still some pending bugs which I missed from adding to changelog. So I closed ~~#606148~~ and ~~#689199~~. Still there are 2 more bugs which seems like fixed in newer version but I need to confirm before closing them, thus making suckless-tools a bug free package.

Upstream Related

I raised a pull request #8 on Gubbi fixing the Makefile to more organized and introducing xz compression in it. Additionally I removed distribution specific stuffs from Makefile and made it generic.

That's all for the two weeks. This week there will be foss.in and we will be having some Debian specific mini conference, including some Debian basics to newbies and some bug squashing if any :-). So more to report next week, till then Cya.

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 27/3 - 112012

2012-11-11T00:04:52-08:00

OK I became again lazy to log my works and combined 2 weeks again :-). Most of my work is for Debian for this 2 weeks.

More reviews from Jakub on suckless-tools and more fixing :-). This time Jakub identified that ssid is not at all different the setsid from utils-linux package which resulted in me writing to Anselm Garbe of suckless project, who told that first reason for writing ssid was setsid code during that time[1] which really does suck. He agreed to obsolete the ssid project after some clarifications.
After mail to Conor Lane Smith upstream author of sprop and then writing to Anselm, sprop now has finally is placed under dl.suckless.org/tools which means I don't need to refer to Mercurial tip anymore.
fonts-deva-extra is no more containing the Gargi and Sarai fonts as both fonts have got upstream and now will be part of fonts-gargi and fonts-sarai. I'm now reviewing fonts-gargi which is being packaged by Jishnu. To be frank I'm loving the review process ;-).
After a while I again worked on friendica package for Debian and after fixing a stupid mistake by me I released a version of package for testing as I run my friendica on my laptop itself. I did try piuparts for testing but couldn't get it working. After testing I got positive feedback from the fellow friendicans [2][3]. Note that [2] will be available only if my laptop is online :-). Now I need to try and automate installation of webserver confs I know might not be good idea but my idea is making friendica installation on Freedombox without any hassle. Of course suggestions are welcome :-) package source is at [4]
I got a grave bug filed on suckless-tools #692737 which of course can't be taken as security hole directly in fact it was a feature of slock which later got modified later by user patches. After talking with person who filed bug who actually said he is not interested in getting it for Wheezy I reduced severity to important. To be frank this feature is part of 39 version of suckless-tools but change is too much to include it in Wheezy and I was not interested in making another version of suckless-tools I already made 38-2 for wheezy fixing important bug, filed by Jakub.

This week was bit hectic I couldn't dedicate much time for any programming. This week I'm on vacation and will get more time to work on FLOSS and other coding stuffs. Only that I'm on slow network connection :-). Ok cya next week (hopefully)

PS: My last weekly log never showed up in planet.d.o not sure what happened may be because I played with feed generating jekyll scripts. Hope this will show up in planet.d.o

[1] http://code.metager.de/source/xref/linux/utils/util-linux/sys-utils/setsid.c
[2] http://vasudev.homelinux.net/display/copyninja/34241
[3] https://kakste.com/display/beardyunixer/974150
[4] anonscm.debian.org/gitweb/?p=collab-maint/friendica.git;a=summary

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 20/27-102012

2012-10-14T10:31:58-07:00

This week decided to write the log from the beginning of the week itself as I don't want to miss any tasks done. So here comes the list. Also I merged 2 weeks log together not because I didn't do much but recently I was being bit lazy :-P

Debian Related

Some more fixes done to suckless-tools 38 version as per comments from Jakub Wilk
After some poking from Intrigeri (my AM) I finally placed the new 38-2 related stuffs in Wheezy branch of suckless-tools.git which I originally prepared for 39 version
After few minor fixes Jakub Wilk has uploaded 38-2 and thus I kill one more RC bug for wheeze ~~#685611~~. Well my AM has given me clue that more RC bugs might be coming my way as part of NM process and I'm waiting :-)
More clarifications on my answers on Philosophy and Procedures - 1 were sent back and I need to give some more again ;-). Will do that probably on Sunday
Fixed issues in installing fontconfig files in font packages fonts-beng-extra and fonts-smc no bugs were filed though. fonts-smc can't be uploaded to unstable for sure as it contains a udeb (fonts-mlym-udeb). Kartik told he will discuss with release-team about fonts-beng-extra but I guess he forgot ;-).
Helped new guys to learn Debian packaging on #debian-diaspora, initiative started by Praveen Arimbrathodiyil who actually taught me the packaging. Idea is to package Diaspora. Even though I'm not much interested in Diaspora I'm helping there as Praveen is my good friend and mentor and in a way I want to increase contributors to Debian.
Started working on suckless-tools 39 fixed few minor issues now seeing a hardening issue which I need to fix before each can be safely uploaded to experimental.

Python Based

Fixed the Unicode comparison issue in the charmap.py from the soundex Python module. Basically when both side of == operator are not of Unicode type the Python considers them to be unequal even if they are equal. Well this is the encoding pain in Python versions less than 3. Now this is handled in 0.3 version of soundex.
Spellchecker module is now having Marathi language support but it doesn't seems to be working fine so 0.3 version is not yet completely done.
Guess Language module from SILPA is now separate module and is uploaded to pypi, though its integration with SILPA rewritten in Flask is pending.
Replaced the pyxmpp2 module used in the dictionary bot to sleekxmpp as it already supports XEP's which I plan to use in the bot
Fine tuned the parsing and returning logic of dictionary-bot and integrated language guessing support.
Inspired by Jonas's coding on smsg I also modified the code so that if needed it can be easily packaged and also can be tested without need to actually installing it.
Added logging functionality to the dictionary bot but yet to log actual data from the bot so that can be analysed later

Few more details on dictionary-bot needs to be sorted out to make it production ready and I'm not getting enough time to do so. May be I should write down TODO's some where so I won't loose track of it.

Perl Based

Not much tasks were done on Perl but I managed to help Jonas to sort out an important issue which was a show stopper for using smsg in the performance he is attending in Zurich. Below are tasks done

Moved the Jabber.pm code which was using AnyEvent::XMPP::Client to Client.pm which inherits from Jabber.pm, basically idea is create one ore class Connection which uses AnyEvent::XMPP::Connection which can be used to connect send message and disconnect. Did you see inherit don't be surprised thanks to Jonas I had to learn bit about Moo :-)
Slightly modified HttpSend.pm command to send the message to Jabber chat room, this was the show stopper for smsg. And after this fix Jonas happily used it in his theatre performance in Zurich.

Wildfire Liberation

I struggled almost 3 days to get my HTC Wildfire liberated from HTC stock OS. I got to know about HTC Developers Site with which HTC officially allows the unlocking the bootloader. After unlocking I was not sure on how to get ClockworkMod Recovery in my device and get root the tools mentioned in different places, revolutionary and unrevoked were not happy with my new HBOOT version 1.02.0000 after struggling a bit I managed to manually download the ClockworkMod for Wildfire from this site but then again I was afraid of bricking my phone. But thanks to Jishnu's timely advice I managed to flash my device and install CM7. Now I enjoy my 2 year old Wildfire!

For those who want to flash their Wildfire with HBOOT version 1.02.0000 please follow the Method via HTC website section here.

Misc

Met Kartik on 20th wanted to have a mini BSP but we got lost in talking as we met after one year
Watched few bunch of movie. This week was bit short as in between the week 3 holidays came due to festivals.

Well that is it for the 2 weeks hopefully I will see ya on next week :-) or next to next if I merge my weekly logs again ;-). Till then Cya.

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 06/13-102012

2012-10-13T09:48:48-07:00

Update for 06/10/2012

Well I had not written the weekly work log for the last week that is because the week was short (oh yeah thanks to all these bandhs our week got fluctuated and to be frank the week was only 4 days long) and second was my lazyness. Here goes the update * After a long discussion with Aravinda on "why productivity is reducing these days." We concluded the social network is eating up most of time. So decision was made and I closed all pinned tabs for Twitter Gmail Identi.ca and Friendica on my browser * After the above resolution I finished almost 4 chapters of Moder Perl within 2 hours!. Indeed Social networks kill the productivity.

Update for 13/10/2012

I've really fallen in love with computerised bots, thanks to the wonderful KGB bot :-). So majority of my work on this week is on bots.

Jabber Dictionary Redesign

After thinking for a while I decided to re-write the dictionary bot which when release got an overwhelming response as see in the comments of above link. Few reason for re-write

Generalise the bot framework so single bot can handle multiple languages
Improve the data collection on bot side
Current code base was not very well organised and trying to add more feature it had become messy.
Provide XEP support to help in data collections

Few changes which are already implemented include

New code is now using pyxmpp2 library instead of GPLed xmppy used by current code.
Implemented XEP-0071 extension to properly format the meaning display by the bot
Current implementation was displaying all words in one set without distinguishing between adjectives,verbs proverbs etc. even though wiktionary displays meaning based on this. New implementation gives out meaning in same format as it is displayed on wiktionary.

Things remaining todo

Separating wiktionary parsing logic from bot code by providing some sort of intermediate interface between bot and wiktionary parsers.
Adding more language wiktionary parsers and teaching the bot to become multilingual :-)
Integrating XEP-0004 (Data Forms) for taking the meaning input from user. Current code requires user to enter data in particular format

suckless-tools fix for Wheezy

Jakub Wilk suggested me to prepare a minimal version of suckless-tools for Wheezy which includes a patch to the bug #685611. And few minor changes which involve taking over the package and fixes in copyright file other than above mentioned bug. Hopefully release-team will be okay with these changes. I'm waiting for the upload to file an unblock-request.

I did face a problem I was halfway through wit suckless-tools_39-1 when Jakub asked me about this change and current repository was fresh one prepared for 39-1 and didn't have history for 38 version. First I thought of preparing separate repository for 38 version which was not an correct option, but I even couldn't play with current repository. So finally I renamed current version of repository to suckless-tools-39.git on collab-maint and prepared fresh repository suckless-tools.git basing it on 38-1 version.

From 39 version suckless-tools will be following 3.0 (quilt) source format and will not be working with git-buildpackage as the tool can't handle multi-tarball packages. Yes every tool involved in the package will have separate tarball from 39.

More work on Bots

Well today I again worked on Jabber bot, but not the dictionary bot. This time its an SMS Gateway bot for Jonas Smedegard and the coding was done in Perl. Thanks to jonas I finally could apply what I learnt in Perl. Code may not be very elegant but it works :-).

And after hacking one full day in Perl now I'm not feeling very much interested to go back and hack on my own Python based dictionary bot :-). But I will anyway.

Misc

fonts-kalapi has been uploaded to unstable by Kartik finally :-). Only sad part is he couldn't use the home grown dh-make-font for this purpose.
Finally Kartik is here in Bangalore and I'm just waiting when I can meet him.
I ordered a RaspberryPi once it comes some hacking will be done on boxer again :-)
Watched couple of Star Wars movie :-) (Still have 4 left)
Expendables 2 movie is in watch list after this post.

So that's it folks. Quite longish post hope you are not bored reading it :-). Well time for movie C'ya all with next weeks log.

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 01102012

2012-10-01T09:15:02-07:00

Well this weeks log comes bit late as my week got extended till today ;-). I had working day at office on Saturday and Sunday due to 2nd Quarter end and instead today is declared as compensatory holiday and tomorrow any how is Gandhi Jayanti. So here it goes.

Debian Related

Most of my work this week was done for Debian-IN team

Integrated the KGB bot for Debian-IN team repository. Thanks to gregoa Tincho and dam for extending helping hand in getting it configured
Updated the setup-repository to install a proper post-receive hook when creating new repositories.
Copied the set-git-hook from pkg-ruby-extras team and modified it slightly to install post-receive hook for KGB integration for already existing repository. With this script I fixed few of the packages to use KGB which I own. Waiting for Kartik to fix remaining repositories as he is the DD for our team :-)
Configured my personal instance of KGB which now stands in Debian-IN's channel (#debian-in). But bot is named RAW (Research and Analysis Wing) equivalaent of KGB and CIA in India :-P

Well most of this weeks work involved KGB ;-)

Personal Works

Attended the PyconIndia-2012 on Saturday, Got a NFC card :-) which I can't use as my phone is not NFC capable. Only handful of talks were nice rest were all newbie like talk which even lacked proper presentation skills hence didn't attend on second day. Looks like quality of conference in India is decreasing.
Finally completed answering all questions on Philosophy and Procedures - 1 part of NM process :-). Phew it took me more time than I expected. Waiting for my AM's comment. Hopefully I will get next bunch of questions soon :-)

Misc

Played Cricket after very long time (Guess after 10 long years). Well it was not official match or anything it was just another Gully Cricket. After the match it came to mind that I need regular physical workout :-P.

Well that's it Cya till next week :-)

Included file 'rss_footer.html' not found in _includes directory

Weekly Log - 22092012

2012-09-22T09:57:31-07:00

After thinking a while I thought I should record my weekly FLOSS work and I'm starting by this week. This week I'm on vacation and in home town so I could give bit more time for my FLOSS activity. So here it goes this weeks work

Bug Squashing

Fixed a RC bug filed on Debian-IN team ~~687176~~ thanks to the timely patch from Kartik
Filed an important bug on fonts-taml which was missing a dependency on fonts-samyak-taml and fixed it myself :-) ~~687806~~

Both these packages are uploaded to unstable first RC bug was related to Debian Installer hence was unblocked before we could file unblock-request. Kartik filed an unblock-request and adsb unblocked it happily and both the packages landed in unstable.

New Packages

fonts-lohit-taml-classical is a new font for Tamil community developed by Lohit projects I don't know much about font itself but I cleared a wnpp bug with this package ~~681655~~
fonts-teluguvijayam is new package containing 3 Telugu fonts which is released by Department of Information and Technology Andhra Pradesh. This package is pending upload and will clear wnpp bug 669238 and is a pending upload.
I finally cleared all review comments by jwilk on surf package and now 0.5 version of surf is in experimental.

Both these packages will be backported once Wheezy freezes. One pending wnpp is left on Debian-IN which I've left for Kartik to finish ;-).

Font Package Helper

I bumped version on dh-make-font[1] to 0.3 now it accepts upstream URL with -u, --upstream switch this will be placed in Homepage: field of control file and in Source: field of copyright file. Probably I need to think of introducing xz compression to original source tarball where upstream is not giving any tarball and probably rewrite it using Perl so it can get included dh-make package. Already a bug filed to get it included in dh-make package 658154.

[1] http://anonscm.debian.org/gitweb/?p=debian-in/dh-make-font.git;a=summary

Misc and Learnings

Helped Jonas to fix a minor issue in boxer. I still have to figure out one issue on boxer and add ramfs support to boxer. If you see boxer commit log you can find a few Thanks to Vasudev Kamath but I still didn't commit anything directly which is now in my TODO :-)
Started learning Perl from Moder Perl book thanks to Jaldhar for pointing this book for me.

Well that is it for this week. Hopefully I will have new stuffs to put up here for the next week, till then Cya.

Included file 'rss_footer.html' not found in _includes directory

Logging issue with Flask

2012-08-06T00:00:00-07:00

While enabling logging for release of rewritten version of SILPA described in my previous post I faced a weird problem. I enabled Logging as described in the Flask documentation but SILPA won't log anything to the log even though log file is created in whatever location I gave while configuring the logging.

Flask documentation suggests following way to use logging

import logging
from logging.handlers import RequiredHandler
file_handler = RequiredHandler(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)

After banging my head for some time I again wanted to look at Python's logging tutorials see if I'm doing something wrong and I found this example

import logging

# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)

# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# add formatter to ch
ch.setFormatter(formatter)

# add ch to logger
logger.addHandler(ch)

# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

So there it is I need to set the logging level even on logger to same level as what I set on my handler. Now I didn't take time to read the documentation because I wanted to see if my findings really does the trick so I modified function configuring logging to look something like below (I'm taking the example given in Flask documentation itself real code can be seen in SILPA code on Github)

import logging
from logging.handlers import RequiredHandler

file_handler = RequiredHandler(...)
file_handler.setLevel(logging.WARNING)

app.logger.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)

This actually did the trick and now Flask is happily logging whatever I give to app.logger.levelname So I don't know this was my stupidity or fault in Flask's documentation by forgetting to mention this crucial point. But in the end all is well :-)

Do comment your thoughts.

Included file 'rss_footer.html' not found in _includes directory

Rewriting SILPA with Flask microframework

2012-08-05T00:00:00-07:00

As a introduction on what is SILPA you can read my previous post This application was written by Santhosh ~~about 6-7 years ago~~ in May ~~2010~~ 2009 ( ~~Correct me if I'm wrong Santhosh ;-)~~ Thanks to Santhosh for correcting me ) in Python without using any web framework. He crafted whole system single handed and with very nice ideas. He wrote all the modules available in it for all possible Indian languages even though he is native Malayalam speaker. I joined him 3 years back and I was providing some or other improvements and he ultimately made me as the Co-Developer of SILPA.

Recently I came across the Flask microframework and thought of rewriting SILPA to use Flask. You might ask me why Flask? So here are my reasonings

 1. Why reinvent Python webframework when we can use existing? Besides maintaining a not so well known framework code which is not written by you will be difficult    and time consuming.
 2. All Python modules are useful even outside SILPA project why make them dependent on SILPA code?
 3. Templating system used is very old and can't be ported to new Python version lets use something which is new and portable
 4. If new contributor wanted to add a module he need to understand SILPA framework to write and integrate it. This might be discouraging new
    contributions so lets use pure python module and make integration with SILPA straight forward.
 5. I was overwhelmed by Flask :P

I had previously tried bottle microframework but for some reason I couldn't crack it so choosing of Flask is not because it was better among other available microframework or something like that. I choose it because I could easily understand it and it suited my needs better.

So I proposed about this to Santhosh and he asked me for a POC code. I started working on April 8 2012 and today by August 5 I can say I've first stable version of SILPA rewritten using Flask deployable over WSGI on Apache2. It can also be deployed on other servers supporting WSGI interface.

It took me quite a bit of time but to be frank in the middle I completely had stopped coding because of some other works. Here is the commit graph and code frequency graphs from Github.

Changes in New Version

Here is some major changes in current version.

1. Newly developed framework provides plug and play features for new modules.
2. You need to install the modules which you want to run on it separately using PIP and only minimal code change is required. Only configuration and webbridge.py 
   will be changed. Ofcourse you need to add html file to *templates* folder.
3. This version uses decent routing logic from Flask for page serving.
4. It also provides JSONRPC interface which is compatible with previous version but only with method name changed.
5. This version separates out run time module loading and routing the request to modules.
6. Dynamic module loading now happens only at the beginning of application, so if I want to implement caching for particular services it will be straight forward
   now.
7. Routing requests to proper module is adapted from old version with few modifications
8. All previous pages are now converted to use *Jinja2* template system.

Things Remaining TODO

Below is some stuffs still pending before this version can completely replace http://silpa.org.in

1. Separate all modules from original SILPA and get it uploaded to pypi
2. Provide RESTful interface
3. Provide proper LICENSE, AUTHORS file for all modules 
4. Prepare a Nice UI
5. Find a better name than *SILPA*
...

Update

Forgot to provide the github link and demo link for the new version of SILPA. Thanks to Aravinda for pointing
it out. Here it goes
Github: http://github.com/copyninja/Silpa-Flask
Demo: http://flasksilpa-indic.rhcloud.com

Included file 'rss_footer.html' not found in _includes directory

Compiling Harfbuzz on Debian

2012-07-22T00:00:00-07:00

Harfbuzz (more correctly harfbuzz-ng) is a Text Layout/shaping engine. Mostly supporting rendering of Text including complex scripts. A quote taken from Wikipedia which describes harfbuzz

HarfBuzz (in Persian: حرف‌باز) is a layout/shaping engine for OpenType fonts. Its purpose is to standardize text layout in FOSS; its code originally started as part of the FreeType project, was then developed separately in Qt and Pango, and finally merged back into a common repository. Both Qt and Pango currently use HarfBuzz; other standalone users include Firefox and Chromium, the open source project behind Google Chrome.

Coming directly to the point. You can get Harfbuzz from its Git repository hosted on Freedesktop. It requires following library and binary compilation

    1. ragel: Finite state machine code compiler
    2. libglib2.0-dev: Glib Development files
    3. libcairo2-dev : Cairo 2D graphics library development files
    4. libicu-dev: Development components for International components for Unicode
    5. libgraphite2-dev: Development files for libgraphite2
    6. libfreetype6-dev: Freetype 2 library development files

Once you install all the above package. Follow following commands to build the harfbuzz.

./autogen.sh
./configure --prefix=/usr
make
sudo make install

You can skip make install if you don't like it to be installed on system.I actually wanted to work on the Debian package of Harfbuzz but since there is some one already working on it. Probably I will wait till the guy completes package.

I will write on testing the text rendering of your language using harfbuzz in my next post. Till then Cya.

Included file 'rss_footer.html' not found in _includes directory

Note to Self: Deleting Remote Branches and Tags in Git

2012-07-15T00:00:00-07:00

Its been a while I'm using Git as a version control system for my various work whether it is FOSS related or my professional work. Every time I stumble into a case where I need to delete an accidentally pushed branch or tag on remote and every time I forget how I did it last time. Today I did it again and I had to search it again to find a method. So here it goes this is a note to myself so I won't forget it next time

Deleting a remote branch

git push origin :branchname

Deleting a remote tag

git push origin :refs/tags/tagname

Where origin can be replaced by proper remote if you are using multiple remote for mirroring purpose.

If you want to delete locally use following code

git branch -d branchname
git tag -d tagname

Note: To delete tag remotes you need to delete it locally first and then push it.

That's it hope it helps others too :-)

Included file 'rss_footer.html' not found in _includes directory

Update: Issue in Kannada Dictionary Bot

2012-04-21T00:00:00-07:00

Just a short update for those who are wondering what is Kannada Dictionary bot, it is a small Jabber chat bot written in Python which pulls out meaning for given English or Kannada word from Kannada Wiktionary. More about the usage of this bot can be found in my previous post

Recently I've been getting many queries from users that they are not able to access the bot from their GTalk account. Of course no one contacted me directly they were pinging my friend Omshivaprakash and in turn he was pinging and telling that some one contacted him and told bot is not online. But I have monit service monitoring bot 24x7 additionally I could see it online all the time, even when shiv is telling me its offline!.

Finally figured out the reason when I went to Jabber So here is the reason

As you’ve probably noticed, the jabber.org service has been experiencing connectivity problems with gmail.com and other Google Talk domains (e.g., you might not be able to see if your contacts there are online). The folks at Google are working on a solution and we will post again once they have fixed the problem.

Now folks please have patience till a solution comes up and once its in place you should be able to use the bot without any issue.

Included file 'rss_footer.html' not found in _includes directory

Update: Falcon Debian package Status

2012-04-15T00:00:00-07:00

Recently I wrote about packaging Falcon programming language for Debian. So here is the status of package. Package builds multiple binaries and we have successfully separated modules from core Falcon language component

falconpl (Falcon language itself )
falconpl-dbg (Debug symbols)
libfalconpl-engine1 (Falcon engine library)
libfalconpl-engine1-dbg (Debug symbols for Falcon enging)

Core modules like feathers and modules writtein Falcon itself are part of libfalconpl-engine1. Rest of modules are separated into following packages

falconpl-mongodb: Mongodb connector module
falconpl-dbi : Database Abstraction layer
falconpl-dbi-sqlite3: SQLite3 connector based on DBI module
falconpl-dbi-postgresql: PostgreSQL connector based on DBI module
falconpl-dbi-mysql: MySQL connector based on DBI module
falconpl-dbi-firebird: Firebird DB server connector
falconpl-curl: Curl bindings for Falcon
falconpl-dbus: DBus bindings for Falcon
falconpl-dmtx: Data Matrix reader/writer for Falcon
falconpl-hpdf: Haru based PDF module for Falcon
falconpl-gd2: Graphic manipulation module for Falcon

There are 4 modules which I'm not sure how to build (dependency) or if at all possible to build for Linux they are dynlib, wopi, sdl, conio. I need to consult upstream on these modules status and ability to build.

Modules not built

We didn't build dbi-oracle and dbi-odbc modules as these are proprietary software dependant. Additionally in a conversation with upstream on IRC it was told dynlib module can't be built for 64bit and X86 version of Linux because lack of assembler, but I've not fully confirmed this.

Legal Issues

To enable the curl bindings I added curl dependency which in turn has dependency on libssl1.0 library which is part of openssl source package which is causing lintian thrown GPL violation error. Kartik will be looking into this.

Patches submitted

I've submitted total of 6 patches of which 5 patches are already applied to upstream git repo and 6th one is waiting to be applied. In other words upstream is really helpful to us.

Well thats all for this week more updates next week (or may be even before that ;-) )

Included file 'rss_footer.html' not found in _includes directory