Swimming in bits

Nokogiri & Char encodings

2011-12-26T17:15:00.001+09:00

Personally, Nokogiri is the best library for ruby to look into html structures, especially css-selector support rocks. However, when dealing with i18n webpages, char encoding is always painful for beginners.

for a reminder, mark this page:
http://stackoverflow.com/questions/2572396/nokogiri-open-uri-and-unicode-characters

the following one works

difference is, open-uri read first and then feed into Nokogiri,

if feed directly, will result in hard encoding pain

# encoding: UTF-8
require 'nokogiri'
require 'open-uri'

html = open('http://www.biblegateway.com/passage/?search=Mateo1-2&version=NVI')
doc = Nokogiri::HTML(html.read)
doc.encoding = 'utf-8'
h52 = doc.css('h5')[1].text
puts h52, h52.encoding, h52 == "Genealogía de Jesucristo"
#=> Genealogía de Jesucristo
#=> UTF-8
#=> true

When inserting into mysql without going through rails, but mysql gem, set the UTF8 specifically is mandatory:

this page describes this in detail:
http://wiki.a2hosting.com/index.php/UTF-8_MySQL_connections_with_Ruby

db = Mysql.init
db.options(Mysql::SET_CHARSET_NAME, 'utf8')
db.real_connect('host', 'user', 'password', 'database', 3306, '/usr/local/lib/mysql.sock')
db.query("SET NAMES utf8")

java generic stuff memo

2011-11-08T11:00:00.000+09:00

1. all generic derived classes share one static object

-> as jvm will not separately construct generic classes

e.g. Employee, is the template

Employee, Employee, are the same class Employee in JVM

thus, assuming Employee has a static field, say counter,

then increase the counter in Employee, then Employee will have the updated counter too. (actually, only one counter)

see: http://www.angelikalanger.com/GenericsFAQ/FAQSections/TypeParameters.html#Is there one instances of a static field per instantiation of a parameterized type?

2. need tricks to get the runtime template class object

say Employee has a function, Class getEmployeeType(),

it's not trivial to implement the function, and get the Class.

- one way to walk around is : Class getEmployeeType(Class curr_class)

- another way: http://www.artima.com/weblogs/viewpost.jsp?thread=208860

tricky modulo operator

2011-11-08T10:59:00.001+09:00

in Ruby:

197%5 = 2

-197%5 = 3

in Java:

197%5 = 2

-197%5 = -2

... ...

compile uuid ossp to dll and used it in VS

2010-04-16T12:20:00.005+09:00

Project these days requires cross-platform feature (.Net 4.0, Android 2.1, in house Linux and etc ...). Thus the common used components, we plan to use standard C. Thanks to so many open source libraries, they save us a lot of work. However, compiling and using *nix libraries to windows is a kind of craftsmanship.

In the project, we need a uuid library. The only free & open & reliable solution probably is OSSP UUID.

I started with the newest version uuid-1.6.2.tar.gz with mingw @ a windows xp sp3 pc. I did not use cygwin, due to the license problem.

1. routine job

./configure

make

make check

It does not generate any dynamic library, but a static one, libuuid.a

2. generate dll

well, by reading the stdout make process, i tried

2.1 get dll & def

gcc -shared -o uuid-ossp.dll uuid.o uuid_md5.o uuid_sha1.o uuid_prng.o uuid_mac.o uuid_time.o uuid_ui64.o uuid_ui128.o uuid_str.o -Wl,--output-def,uuid.dll.def

2.2 get lib

dlltool -d uuid.dll.def --dllname uuid-ossp.dll --output-lib uuid-ossp.lib

2.3 test it with the command line tool uuid_cli.c

gcc -I . uuid_cli.c -o uuid.exe uuid_ossp.lib

everything works so well, the command line uuid.exe generated really works.

3. decide how to use and what do we need

sadly, this library, i could not find related API documents.

what i need is just to generate the uuid, by reading uuid_cli.c, i got some hint, here's a simple example.c, after cropping the uuid_cli.c to generate a version 1 uuid.

#include "uuid.h"
#include <stdio.h>

int main(int argc, char *argv[])
{
uuid_t *uuid;
uuid_rc_t rc;
char *vp;
size_t n;
unsigned int version;

version = UUID_MAKE_V1;


if ((rc = uuid_create(&uuid)) != UUID_RC_OK)
 printf("uuid_create error");

if ((rc = uuid_make(uuid, version)) != UUID_RC_OK)
 printf("uuid_make error");

vp = NULL;
if ((rc = uuid_export(uuid, UUID_FMT_STR, &vp, &n)) != UUID_RC_OK)
 printf("uuid_export error");
if (vp)
printf("%s\n", vp);
free(vp);

if ((rc = uuid_destroy(uuid)) != UUID_RC_OK)
 printf("uuid_destroy error");

return 0;
}

The code works well, and can link to the dll well. The only header file we need it uuid.h, no aother uuid*.h needed

4. used in Visual Studio

in the visual studio project, configure the project by using the uuid-ossp.lib, and include the uuid.h in the search path. copy the example.c into the project.

compile and run, oops ...

4.1 uuid.h

the compiler complains the uuid.h include <unistd.h>, it can not find

sure, the file is in mingw include path. i tried, to include related mingw headers into the project. but i failed, with lots of compilation errors.

but you can safely delete this line from the uuid.h, i tried, everything works the same

4.2 heap error

cool, now, the compiler has no complain, but when run, the example.c get a runtime error, in the line

free(vp);

I dig this for a while, and realized that , the vp is malloc-ed and free-ed by different memory malloc & free implementations.

the workable program in mingw, should changed to the following, then, example.c can work fine now:

#include "uuid.h"
#include <stdio.h>

int main(int argc, char *argv[])
{
uuid_t *uuid;
uuid_rc_t rc;
char *vp;
size_t n;
unsigned int version;

version = UUID_MAKE_V1;
n = 37;
vp = (char*)malloc(n);

if ((rc = uuid_create(&uuid)) != UUID_RC_OK)
    printf("uuid_create error");

if ((rc = uuid_make(uuid, version)) != UUID_RC_OK)
    printf("uuid_make error");

vp = NULL;
if ((rc = uuid_export(uuid, UUID_FMT_STR, &vp, &n)) != UUID_RC_OK)
    printf("uuid_export error");
if (vp)
  printf("%s\n", vp);
free(vp);

if ((rc = uuid_destroy(uuid)) != UUID_RC_OK)
    printf("uuid_destroy error");

return 0;
}

cool, everything works now.

compile xerces-c & xalan-c on cygwin

2009-12-09T16:12:00.002+09:00

Yet another cygwin log again. Since the unfamiliar with under surface details of how it works, whenever sometime wrong happens on cygwin, it'll be a nightmare. This time is the xerces-c & xalan-c.

Since recent project requires to use the xpath in c++, the only choice probably is the xalan-c. I did think that if the library of c++ become rich enough, everyone c++ worker will be much easier from a developer coming from enterprise computing language and script language perspective. = =

This time, the story goes like this, i need to compile the xerces-c 2.7 as xalan 1.1 compatibility. The original binary package for cygwin is 3.0.1-1 or 2.8.0-5 under Devel/libxerces-c-devel. The steps of compilation as the Readme.html said:

1. set the XERCESCROOT virable to the directory contains the extracting of xerces. for example ~/dev/xerces-c-src_2_7_0

2. go the the XERCESCROOT/src/xercesc, use ./runConfigure -pcygwin -cgcc -xg++ to make the Makefile (runConfigure is a helper script, we can use ./configure also)

3. if everything's done properly, just make it. and make install

My problem comes at the step 3, the make can not be processed when reaching the 'deprecated DOM part'. The Problem is the exactly the same as this bug report XERCESC-1501.

Creating library file: \ /cygdrive/d/tmp/xerces-c-src_2_7_0/lib/libxerces-depdom27.dll.a \ CYGWIN/depdom/AttrImpl.o(.text+0x4bb):AttrImpl.cpp: undefined reference to \ `xercesc_2_7::XMemory::operator delete(void*)' \ CYGWIN/depdom/AttrImpl.o(.text+0x50d):AttrImpl.cpp: undefined reference to \ `xercesc_2_7::XMemory::

...

...

... ::fgNoString' collect2: ld returned 1 exit status make[1]: *** [/cygdrive/d/tmp/xerces-c-src_2_7_0/lib/cygxerces-depdom27.dll] Error 1 make[1]: Target `ALL' not remade because of errors. make: *** [/cygdrive/d/tmp/xerces-c-src_2_7_0/lib/cygxerces-c27.dll] Error 2

The error seems saying the it can not find certain deprecated dom library. First, I went through the way this post said. After spending 1 hour to find where I can add that link options in the Makefile, the make still not work.

 
Then I thought probably, it's the libxerces-c-devel problem. I uninstalled it. Then everything just works, with some auto-import warning, INFO. 
    
After make install the compiled code, we can do test with the sample. The compilation of the sample works well. Everything works now. 
    
Next, compilation of xalan-c. Nightmare again ...
 
1. set the XALANCROOT virable to the extracted xalan/c , make sure the XERCESCROOT is still there
   2. in the XALANCROOT, run ./runConfigure -p cygwin -c gcc -x g++ -P /usr/local , make sure including the -P in order to use the 'make install'. 
3. type 'make', here comes the problem, exactly the same as this one: http://marc.info/?l=xalan-c-users&m=115953899805021&w=2
    
make -C src/xalanc all make[1]: Entering directory `/usr/src/xml-xalan/c/src/xalanc' Preparing the directory structure for a build ... mkdir -p ../../obj mkdir -p ../../lib mkdir -p ../../bin make -C Utils prepare make[2]: Entering directory `/usr/src/xml-xalan/c/src/xalanc/Utils' Makefile:71: *** multiple target patterns.  Stop. make[2]: Leaving directory `/usr/src/xml-xalan/c/src/xalanc/Utils' make[1]: *** [prepare] Error 2 make[1]: Leaving directory `/usr/src/xml-xalan/c/src/xalanc' make: *** [all] Error 2

The thread is very interesting. The last one solved this problem partly: http://marc.info/?l=xalan-c-users&m=116048865019326&w=2

  I could now compile Xalan. 
   The problem was the empty XLIFF_FILE_NAME. 
I hard coded the Windows form of the file, "C:/cygwin/usr/....", to Utils/Makefile and changed the line 
   $(XSL_NLS_DIR)/include/LocalMsgData.hpp : $(XSL_BIN_DIR)/MsgCreator $(XLIFF_FILE_NAME) to 
   $(XSL_NLS_DIR)/include/LocalMsgData.hpp : $(XSL_BIN_DIR)/MsgCreator 
   But the "make install" fails:

Actually it says the problem is because the XLIFF_FILE_NAME has not been processed successfully, but with an empty string in the real make run. In order to fix that one, we need to set it manually in the Utils/Makefile.

1. In the line 38 of the Utils/Makefile, we can find the XLIFF_FILE_NAME for Cygwin. comment out that line, add add XLIFF_FILENAME=$XALANCROOT/src/xalanc/NLS/en_US/XalanMsg_en_US.xlf

2. In the line 71, i found that keep it the same still can make successful. so what we need is do the step 1.

I said the solution solves the problem partly, because when I make the xalan now, the result is not that stable, sometimes errors, sometimes successful, and everytime the errors are different. In the make output, After lots of g++ -O, it starts to create the .dll file. The problem here is sometimes, it says some *.o can not be found in obj/ .

The time I successfully compiled it is I redirected the make output into a text file. = = Otherwise, different *.o are reported as can  not be found in different run. = = very wired.

Anyway, the compilation seems finished, then comes the 'make install'.

  Remember when doing ./runConfigure, we should specify the -P as the installation directory. I set as /usr/local/ at the beginning.

The make file will try to install the file at //usr/local//bin|lib|include, and // can not be recogonized by bash in cygwin.

The solution is to modify the Makefile.incl at the root directory. Line 40, change 'prefix=/usr/local/' to 'prefix=usr/local'. I tried to rerun the runConfigure with -P usr/local, but it does not work because absolute path checking.

Then the 'make install' works.

Next nightmare to me, is the installation seems having something wrong.

The cygxalan-c110.dll has been installed in /usr/local/lib, not in /usr/local/bin, and the cygxalanMsg110.dll has been kept in the XALANCROOT/lib, but not in /usr/local/bin

I compiled the samples by 'make samples' to see whether everything works. But the result is frustrating. only the MsgCreator.exe works, others said:

   the procedure entry point AddAtomA could not be located in the dynamic link library cygwin1.dll
   Googled the AddAtomA, it said that AddAtomA has been exported by kernal32.dll : http://www.pc1news.com/virus/function-addatoma-1177.html  kernal32.dll is in C:\Windows\system32
  but why cygwin1.dll can not locate that??

Cygwin/X, cygwin with gui : )

2009-08-27T18:07:00.000+09:00

Still using cygwin in the current project. These days the job is setting up a cygwin rails server with RMagick installed. On the RMagick FAQ page I read some information about setting up the cygwin/x, so I started, which makes this rainy afternoon interesting. : )

Cygwin/x makes x server run happily on windows. The install instruction is straightforward:

http://x.cygwin.com/docs/ug/cygwin-x-ug.pdf

Cygwin/x comes with a xwin exe. When xwin started, it'll start a xserver integrated with a windows notification, which means you can use the xserver as other windows apps.

With startx command, we can start xserver using .xinitrc configuration. Using xterm started by xserver, we can use display and animate if we installed ImageMagick. Note: here is xterm, not the original cgwin shell winodw. If we start from there, we'll get an error : display: unable to open X server `'

btw, by installing the procps package, on cygwin, we can use the top on cygwin.

using SWIG + Cygwin + Ruby

2009-08-21T19:39:00.002+09:00

Basiclly using swig+cygwin+ruby is not that hard. After reading first few pages from the manual, we can get up and start writing simple examples. However, the first example spends me a whole afternoon.

At first, SWIG seems a very inviting tool for me, since it provides the unified development process to deal with different languages integration with C/C++. Due to the project constraint, I can only use Cygwin to simulate Unix/Linux environment.

1. get swig from cygwin setup OR the windows version works also)

2. get ruby from cygwin setup, and download rubygems from rubyforge and install it at cygwin

3. write a simple test.c

int add(int a, int b) {

return a + b;

}

4. write a simple test.i

%module test

extern int add(int a, int b);

5. using swig to generate the wrapper, (if win version, add swig.exe to PATH, restart cygwin to read it)

swig -ruby test.i (if using cpp, swig -c++ -ruby test.i)

this will generate the test_wrap.c, don't forget -ruby mark, by default is tcl

6. compile the test.c test_wrap.c

gcc -c test.c test_wrap.c -I/usr/lib/ruby/1.8/i386-cygwin (or the right place having the ruby C h files)

this wil generate test.o, test_wrap.o

7. generate the dynamic library from .o

gcc -shared test.o test_wrap.o /usr/lib/libruby.dll.a -enable-auto-import -o test.so

if not auto import, this will complain about the auto complete config but will generate file correctly. after this, test.so will be generated.

8. use the irb to test it

start irb from the same directory

irb ... > require 'test' # test is .so file name

=> true

irb ... > puts Test.add(100, 101) # Test.add, Test is module name, should be capitalized.

201

=> nil

that's it.

Here's the tricky points:

1. ruby problem, better install the cygwin version ruby, for me the ruby at windows, when compile report error about mismatch versions of ms windows

2. ruby gems problem, if you don't install ruby gem sperately at cygwin, start ruby program, you'll get some error like 'ubygems could not find .. '. That's because the RUBYOPT environment usually is set by one click installer at windows. The cygwin ruby can see it also, it'll try to find the rubygems program at /usr/lib/ruby/..., but at cygwin rubygems is not installed by default.

3. when compiling, the ruby header files locations

the manual said at /usr/local/include/ruby ...

at cygwin, it's at /usr/lib/ruby/1.8/i386-cygwin

4. when linking, the ruby library locations

the manual does not include it at all, if you use that, you'll get lots of errors like '_rb_error undefined ...'

for ruby 1.8.x, the cygwin path of it is /usr/lib/libruby.dll.a

libruby-static.a is not the right one

don't use the windows version linking library, which is at C:\Ruby\lib\msvcrt-ruby18.lib, use it you'll get lots of errors. that's for ms vc linker.

5. most trickiest, when the module name should be capitalized. (spend most of the afternoon here!!)

irb ..> require 'test'

=> true

irb ..> puts test.add(100, 101)

undefined test ...

should be Test.add(100, 101)

at last. everything finally works. = =!!!

The capitalized problem made me confused a lot, since the references for using swig+ruby+cygwin is not that much, you need to have question at each step ...

btw: Ctrl + L at cygwin console can have the 'cls' function. : ) just got to know ...

gcc's [Undefined reference to 'vtable for ...'] error handling

2009-08-05T09:25:00.002+09:00

I did not touch c++ ooad at least 4 years ... after some standization, the language looks good though, still not that comfortable to me so far. due to some project requirement, i'm struggling with c++ ooad these days. after drawing class chart, i spent lots of time to pick up the ooad features, rules and conventions in cpp.

one of the wired thing today, is after i added some virtual function to some class hierachy, gcc linker reports 'Undefined reference to 'vtable for ...' . after googling, i found this is a gcc feature or limitation or specific implementation towards the iso standard.

here's the official solution: http://gcc.gnu.org/faq.html#vtables

well, i do believe after spending enough time on this language, i'll be much more comfortable with it. hope so ~ ^^

fancy auto dependency generation?? = =

2009-08-04T17:56:00.002+09:00

i encountered some error from g++ today said 'undefined reference to xx method'

the compiler actually said, it can not find the implementation of this method, which means the file may not be compiled.

that's true.

i checked the makefile which i adopted from some open source project. it uses advanced auto dependency generation to check needed files .. however, the dependency generator can only finds the .h file, but it'll not compile the .cpp file. ==

i modified the makefile, everything works fine now. = = however, 2hrs just vanished ...

how to deal with bit stream or bit set in cpp

2009-07-31T10:00:00.002+09:00

what's a byte: uint8_t

containers: bitset vs vector<bool>

comparation: http://www.ddj.com/cpp/184401382

how to print: hex, dec, oct, in std namespace. all have lasting effects which is confusing at some time.

note char = 'A' does not mean 0xA

how to set cygwin bash environment

2009-07-24T10:39:00.003+09:00

according to cygwin site : http://www.cygwin.com/cygwin-ug-net/setup-env.html

to set up the environment variables in cygwin, we need to edit the c:\cygwin\Cygwin.bat, not the .bashrc

the cygwin user's guide seems a good resource for working with cygwin:

http://www.cygwin.com/cygwin-ug-net/cygwin-ug-net.html

how to use cygwin to run a helloworld in eclipse 3.5

2009-07-23T16:49:00.002+09:00

== i would say i spend one hour on it.

For the mingw hello world, it's very straightforward. Just follow the wizard, 'hello world' will print on the console happily.

However, when change to cygwin toolchain, some trouble comes, here's a flow about how to run a hello world makefile project.

Create a project c++, hello world makefile
project -> properties -> C/C++ Build -> Discovery Options -> Cygwin C++/C Compiler -> check off the automate discoverey of paths and symbols.
project -> properties -> C/C++ General -> Paths and Symbols -> Includes -> GNU C -> Add ... C:\cygwin\usr\include
project -> properties -> C/C++ General -> Paths and Symbols -> Includes -> GNU C++ -> Add ... C:\cygwin\lib\gcc\i686-pc-cygwin\3.4.4\include\c++
build the project
project -> Run/Debug Settings -> Select *.exe -> Edit ... -> Environment -> Add PATH = C:\cygwin\bin;

Something is worth noticing

-mno-cygwin flag seems not workable in gcc now, we should put cygwin dll into the path maybe .. still figuring how to do that ... if there is, post a new post.
in the 6th step, add PATH into the Run/Debug Settings instead of system environment is better, since if use mingw on the same machine, mingw sometimes will mess up due to cygwin bin in the PATH variable.
in the 2nd step, since cygwin is not gnu compiler, the automatic discovery of path and symbols is not workable, disable it will remove some warnings.
in 3rd and 4th step, to specify the include path variables.

== not that trivial, really ...

how to build exiv2 on windows and run samples

2009-07-23T15:19:00.002+09:00

R/w exif is not that trivial. Good news is there's a exiv2 library out there, we can use. Bad news is it's not that easy to compile it and run the samples on windows.

My work environment due to some drm system can only be in windows. And because of cross platform extension reason, I can not use vcproj provided by exiv2. Here's the process i tried.

Plan to use eclipse cdt as the ide, so install eclipse and mingw 5.1.13 and MSYS-1.0.10 as eclipse recommended.
Use mingw tool chain to compile exiv2, encountered several problems

when run configure, the script can not determine the build systems

read the configure file, find the arg format cpu-manufactor-os
so set the --build arg as i686-pc-mingw32

compile again, some memory invalid access error, however, can continue run
at the last stage, configure script will create the config.status script, the config.status script will create some tmp directory.

error: config.status cannot create a temporary directory
read the config.status script, find the last part contains the error, but don't know how the $tmp is assigned.
read configure script, find that by using -n arg, config.status can be generated, but it will not automatically run config.status and generate Makefile
so use ./configure --build-i686-pc-mingw32 -n -without-zlib -disable-xmp to compile, it can generate the Makefile

make the Makefile, however problem happened in the middle, many memory invalid access ...

guess mingw's developer toolkit problem, so download the automake, autoconf and libtool and want to install these on the computer.
when compile either one of them, always has the same make problem. (note, also ./configure -n -prefix=/mingw setted.)

decide to drop mingw and turn to cygwin and have a try

turn to cygwin, download setup.exe and make sure the following libs are installed.

zlib, gettex, libiconv, expat, pkg-config (for samples)
then ./configure, no problem with the config.status could not create temporary dir. good.
then make, no problem, though wait around 5 minutes.
then make install, exiv2, is installed at /usr/local/lib (.a and pkg-config) and /usr/local/bin (dll)

start to compile the samples

in the exiv2 root dir, run the make samples, problems happen

pkg-config would not find the exiv2.pc

to solve it, copy /usr/local/lib/pkgconfig/exiv2.pc to /usr/lib/pkgconfig/

run again, it said that would not find the libexiv2.a

to solve it, export LD_LIBRARY_PATH=/usr/local/lib;$LD_LIBRARY_PATH to include the path into the library path.

compile again, everything is find now ...

run exifprint.exe

problem, no print out at cygwin console and on windows it said would not find *.dll (cygwin1.dll, ... )

to solve it, copy the dll and exe together into one directory, everything is fine now, the program can execute happily. but the solution is very ugly. still not figure out how to compile into one stand alone exe yet ... anyway so far it has been workable. good news ...

how to deal with msys 'Couldn't reserve space for cygwin's heap, Win32 error 6' problem

2009-07-22T17:41:00.003+09:00

i installed eclipse cdt to develop some cpp programs. the before beginning section of c++ development in eclipse help is quite useful, which includes necessary download links. however, when install the MSYS, i encounter a error:

'Couldn't reserve space for cygwin's heap, Win32 error 6'

After googling, lots of post was found. most of it was suspecting that cywin installation.

For me, I install the mingw on a fresh system without cygwin, so cygwin environment variable problem would not happen.

Finally, I solved it used the instructions from the MSYS official site. http://www.mingw.org/wiki/msys

Install MSYS 1.0.10 is just one step of the installation.

After following the instructions there, MSYS is workable now.

working with latex in eclipse

2009-06-17T10:51:00.002+09:00

the sourceforge project texlipse is a good plugin for eclipse for working with latex. it calls installed latex commands and integrates with preview programs. by using eclipse, the editing becomes enjoyable, since it supports spell checking, wrap up and many other familiar eclipse code editing features. create a project becomes easy also. the project website has very good documents also.

syntax ruby code to html in gnome

2009-05-28T21:05:00.002+09:00

before, when syntax codes to blog, i use some online tool or offline tool like java2html to convert codes. i tried gnu source-highlight, but not successful yet. when time goes by, i lost those tools, links and more how to use them, so i decided to create the wheel again. after googling, i found several links all of which say that ruby syntax is good at doing that.

here's the link:

windows clipboard working code and css style sheets in blogspot
kde clipboard working code

both of them create the snippets by getting code from clipboard, format it and set it back to clipboard. i found this idea really useful when posting codes in blogs. however, i could not find the gnome version since i use ubuntu when working.

so i created one and using the clipboard.rb in the previous post.

when use it, copy the code you want to syntax, and run the code, the clipboard will be changed to the formatted code. just copy it to the blog html editor. also remember to change the blog template file to include the css from previous links, thanks for the css.

require "rubygems"
gem "syntax"
require "syntax/convertors/html"
require "clipboard"

in_data = Clipboard::get
convertor = Syntax::Convertors::HTML.for_syntax "ruby"
code_html = convertor.convert(in_data)
Clipboard::set(code_html)

get and set gnome clipboard data from ruby

2009-05-28T20:52:00.006+09:00

I checked ruby gtk2 library today for operating the clipboard data in gnome. i could not find similar snippets in gnome, but in kde there are several examples.

about the ruby gtk2 code, refer to Ruby-GNOME2 official site. they create a impressing site including tutorials. the Api docs do not contain working examples which leads to create quick demo slower.

here's solution, really simple. thanks to the ruby gnome2 project:

require 'gtk2'

# copy the text from clipboard & set text to clipboard
# the code references: 
# -- http://ruby-gnome2.sourceforge.jp/ja/hiki.cgi?Gtk%3A%3AClipboard
# -- http://ruby.pastebin.com/f5ef5f028
#  also another post about xclip is also interesting
# -- http://www.ruby-forum.com/topic/137368
# the reason to create this code was to syntax ruby code to html from clipboard
# May 28, 2009 - imoldcat
class Clipboard
def self.set(text)
c = Gtk::Clipboard.get(Gdk::Selection::CLIPBOARD)
c.set_text(text.to_s)
c.store
end

def self.get
clipboard = Gtk::Clipboard.get(Gdk::Selection::CLIPBOARD)
clipboard.wait_for_text
end
end

#Example
if $0 == __FILE__ then
puts "before text: #{Clipboard::get}"
puts "set clipboard as hello ruby: #{Clipboard::set("hello ruby")}"
puts "after text: #{Clipboard::get}"
end

plot a histogram using gnuplot and ruby

2009-05-28T20:39:00.003+09:00

today i encountered a paper written problem which needs a histogram. i remembered that gnuplot has a chart called "histogram", so at first i did not worry at all. however, things are not like what i thought.

the gnuplot histogram actually is a kind of history chart ==!!. after plotting a demo and looking at those examples, i realized that. == maybe it's right since linux/unix softwares usually do one thing and do it well. gnuplot is not responsible to calculate the frequency for us.

so i calculated the frequency manually and give two columns data to gnuplot with boxes. the box chart use the middle point to show the freuqency which not i want, so we need to add some offset value which equals to half of the step in the histogram.

to use rubygnuplot, i wrote a class for reuse purpose, and paste it here. some of code referes histogram class by Kodama. The histogram class has some mistake when calculating the frequencies. In Kodama's code, the first and last frequencies are not with the same step as other ones, but the frequencies are treated the same.

the ruby code is syntaxed by a script created using ideas from this post using ruby syntax lib.

require 'gnuplot'

# the class use gnuplot to calculate frequency and draw a histogram 
# limitations: the step is the same in this code. it can not be used to draw
# histogram with different steps.
# May 28, 2009 - imoldcat
class GnuplotHistogram
 def h_class(x)
  ((x-@min)/@step).truncate
 end
 
 # array is the raw data, step is used when calculate frequency
 def initialize(array, step)
  @min = (array.min - 0.5).round
  @fix_boxes_offset = step.to_f / 2.0
  @x = []; @x << @min + @fix_boxes_offset
  max = array.max
  while @x[-1] <= (max - @fix_boxes_offset)
   @x << @x[-1] + step
  end
  @max = @x[-1]; @step = step
  @y = Array::new(h_class(@max)+1, 0)
  array.each{|i| push i}
 end
 
 def push(x)
  # if x<@min or @max<x then return; end;
  @y[h_class([[@min,x].max, @max].min)]+=1
  self
 end
 
 def plot(fig_name="histogram", fig_dir=".", x_lab="", y_lab="", i_xtics=10, 
  i_file="postscript eps color enhanced", i_size="0.5,0.5", i_style="fill solid 0.5 border")
  @y << 0
  Gnuplot.open do |gp|
   Gnuplot::Plot.new( gp ) do |plot|
    plot.term i_file
    plot.output "#{fig_dir}/#{fig_name}"
    plot.autoscale
    plot.size i_size
    plot.xrange "[#{@x[0] - @fix_boxes_offset - 1}:#{@x[-1] + @fix_boxes_offset + 1}]"
    plot.xtics "#{i_xtics}"
    plot.yrange "[0:#{@y.max * 1.1}]"
    plot.notitle
    plot.xlabel x_lab
    plot.ylabel y_lab
    plot.style  i_style
    plot.data = [  
     Gnuplot::DataSet.new([@x, @y]) { |ds|
      ds.with = "boxes"
      ds.title = "step = #{@step}"
     }
    ]
   end
  end
 end
end

#Example
if $0 == __FILE__ then
 a = [1, 2, 3, 32, 23, 43, 54, 55, 60]
 hist = GnuplotHistogram.new(a, 5)
 hist.plot(".", "test.eps", "test", "test_Y", 5)
end

when auto boxing and unboxing happens in codes using java generic

2009-02-07T05:44:00.006+09:00

auto boxing and unboxing in java converts 'int' like primitive types variables to 'Integer' like objects.
One thing is the generic type checking always let the primitive type pass!!
today i found a bug in some code due to that.

the following will compile successfully when JDK >= 1.5, since generic is introduced in 1.5. autoboxing and unboxing also.

int i = 1;
LinkedList<Integer> listOfIntSet = new LinkedList<Integer>();
if (listOfIntSet.contains(i)) {
 System.out.println("!! this had better to be found by compiler");
}

the code above can pass the compile. and be run silently in the running. =, =
the strong type language here is not strong. =, = , due to the generic implementation model in java.

this point is worth remember.

how to measure execution time in milliseconds, microseconds in cpp

2009-02-05T06:10:00.011+09:00

<time.h> only provides the way to get difftime in seconds. however, the algorithm i'm working with these days runs pretty fast sometime. in order to measure the milliseconds, i need to find some way. here's the solution, paste here for future use.

solution, use <sys/time.h> instead. there's a 'gettimeofday' funtion whose signiture is : int gettimeofday(struct timeval *tv, struct timezone *tz); the first parameter struct timeval has two members:
time_t tv_sec; //seconds
suseconds_t tv_usec; //microseconds part of the time
in most cases, microseconds is enough to be used as measurement.

#include <sys/time.h>
#include <iostream>

using namespace std;

int main()
{
    struct timeval begin, end;
gettimeofday(&begin, NULL);


    //algortihm to be measured here. ......

gettimeofday(&end, NULL);
    double diff_sec = difftime(end.tv_sec, begin.t_sec) * 1000000;
    double diff_milli = difftime(end.tv_usec, begin.tv_usec);
cout<<"diff (in milliseconds): "<<diff_sec + diff_milli<<endl;
    return 0;
}

if need to use second, the above way works too. however if we don't wanna use <sys/time.h>, but <time.h> use the method here.

get hardware details in ubuntu

2009-01-31T21:24:00.002+09:00

writing experiment part of a paper requires listing the machine specifications, however, after several minutes of navigating the menu in ubuntu, i can not find such functions like windows device manager. so, google it, here's an excellent answer. just write it down for remind. in the link, there's a handy way to use lshw to create a html file of the specs. cool.

sudo lshw -html > hwspec.html

TreeMap and HashMap

2009-01-28T23:25:00.002+09:00

The Map interface in java collection framework is pretty handy. There're several implementations of this interface, such as HashMap and TreeMap. The hashCode(), equals(), and compareTo() are several methods need to be implemented when using own classes as hash keys. However, the implementation requirement of own classes to HashMap and TreeMap are different, which I realized today.

HashMap - requires equals() and hashCode()
HashMap is a implementation of Map using the hashtable data structure, each object as the key need to implement both equals() and hashCode() methods in Object. Equals() is to be used when get(key) is invoked. Since one key is associated with a list of values in the hashtable data structure, to find the exact object in the list, equals() is needed. hashCode() method is to be used by system libraries as hash key. Different object will be assigned to different hash lists. HashCode's implementation has several rules, such as different calls on the same object, the hashCode value should be the same. If two obejcts are equal according to equals(), then their hashCode() should be the same also, and etc.

The best hashCode() usually gives different objects different values according to their members. Thus the access cost can be reduced to O(1). However, such a kind of hashCode sometimes is really hard to find or even impossible to be found. For example, in Java, the cpp <util> Pair class is not provided as a standard class, assuming our pair class has two integer members, say x and y, how to write a hash function, f(x,y), guarantees different x and y having different f(x,y) ? Normal way to implement this is f(x,y) = x * big_prime + y, though the access cost will increase when different values of x, y increases.

If we don't want to implement the equals and hashCode, but wanna use Map interface, one possible solution is use TreeMap.

TreeMap - requires compareTo()
TreeMap is a implementation of Map using red black tree according to jdk doc. So the access time is O(lgN), sometimes it's better than the normalized hashMap. More important, we don't need to implement equals and hashCode. The equivalence check is done by calling compareTo twice.

remote control ubuntu from windows xp

2009-01-25T23:16:00.002+09:00

these days, the weather in seoul is continuously snowing. also since the lunar new year period, few people are in the lab. the machine i'm working on is a ubuntu desktop. i plan to work in the dorm using the laptop. after searching google, the post is really useful. i failed to use vnc service in ubuntu before, even though i've set the preference of remove control. after read the post, i found the reason was lack of SSH tunnel setting. =, = anyway, everything works well now.

getline and eof

2009-01-24T10:14:00.002+09:00

cpp's library is pretty tricky. today encounter anther problem. when use getline and eof together, if the code is not carefully written, may result in errors. here's the example.

ifstream input(some_file, ios::in)
string line;
if (!getline(input, line).eof()) {
cout<<line<<endl;
}

pretty simple. the code will print all the lines in the file. the tricky part is the getline(..).eof().
file1:
a
b
c[EOF]

the result of print file1 using that code will be:
a
b

the last line 'c' will not be printed, since the [EOF] is in the last line. maybe that's the reason that the ruby interpertaor requires the ruby file should contain a empty last line with EOF. almost half a day is spent on this part when porting a ruby code to cpp code. the same file result in different results. however, the file reading part has been ignored for testing, since i thought it's easy. the fact is not.

vector & iterator combination hell in stl cpp

2009-01-23T06:28:00.002+09:00

since iterator is implemented as out iterator pattern in stl, a vector object can not update those iterators referencing some part of it. when the vector is resized, all the iterators will becomes invalid, and thus cause the segment fault. =, = writing cpp is really trying. you should know the detail of its library model, which however requires time to be got familiar with. comparing with java, although the model is similar, however, jvm can produce much more meaningful information other than only a 'segment fault' which can be caused by many kinds of mistakes.

and gdb is really handy. at least we can easily know which line happens 'segment fault' with very basic gdb instructions. today stuck in this part again, however, the debug time is much less comparing with last time. it seems to me that time and experience are important factors for a professionals, at least developers.

a interesting test about right major

2009-01-22T21:28:00.048+09:00

Here's the link.

You Scored as Psychology/Sociology

You should strongly consider majoring (or minoring) in Psychology, Sociology, or related majors (e.g., Counseling, Industrial-Organizational (I-O) Psychology, Social Work, or other social science majors).

It is possible that the best major for you is your 2nd, 3rd, or even 5th listed category, so be sure to consider ALL majors in your OTHER high scoring categories (below). You may score high in a category you didnt think you would--it is possible that a great major for you is something you once dismissed as not for you. The right major for you will be something 1) you love and enjoy and 2) are really great at it.

Consider adding a minor or double majoring to make yourself standout and to combine your interests. Psychology and Sociology are both great minors to add to any major. Please post your results in your myspace/blog/journal.

Psychology/Sociology 80%

Physics/Engineering/Computer 75%

History/Anthropology/LiberalArts 75%

Education/Counseling 69%

HR/BusinessManagement 69%

Mathematics/Statistics 69%

Accounting/Finance/Marketing 63%

English/Journalism/Comm 63%

French/Spanish/OtherLanguage 63%

Biology/Chemistry/Geology 63%

Religion/Theology 56%

PoliticalScience/Philosophy 50%

Visual&PerformingArts 44%

Nursing/AthleticTraining/Health 44%

btw, it seems the html table tag works not very well with blogger. if the article contains some table with no format settings, there'll be a large number of empty lines?

ruby's array delete

2009-01-22T16:34:00.002+09:00

It seems that the delete method of array class uses different conventions comparing with other methods. The '!' usually used after a method in ruby means that the method will change the status of the object. However, 'delete' of array is against that rule. I encountered the problem today, and spent some time on debugging. Pretty annoying. Also, copy is copying reference by default, not by value comparing with c++.

the next example shows it more clearly:
irb> a = [1,2,3]
=> [1,2,3]
irb> a.delete 1
=> [2,3]
irb> a
=> [2,3]

to calculate the cdf and inverse cdf of normal distribution in c++

2009-01-15T09:05:00.002+09:00

Also in the synthetic data generation, it requires calculating inverse cdf.
Without using boost or heavy libraries, here's the links to the codes.

calculate the cdf :
http://www.sitmo.com/doc/Calculating_the_Cumulative_Normal_Distribution

inverse cdf:
http://home.online.no/~pjacklam/notes/invnorm/

A good library for generating synthetic data set

2009-01-15T05:41:00.002+09:00

I'm working on generating synthetic data set for an algorithm these days. The generation model changes a lot, including the distributions we select. To implement a distribution efficiently and accurately is not that easy. The synthetic data generation code was originally maintained in ruby. However, in my limited knowledge, there were few random number distribution codes available in ruby. Recently I found the Colt lib, which provides a set of Open Source Libraries for High Performance Scientific and Technical Computing in Java, as they declare in the introduction. This library is supported by CERN - European Organization for Nuclear Research. Pretty handy.

the copy constructor makes people crazy

2008-10-31T20:06:00.002+09:00

in c++, it always copies those values. this makes the pointer management really tricky. just like the last month post, value is by default. when return value happens, means all the pointers to the old value are invalid. = -b this time, the object there're some self indexes structures. since each object will not share such structure, the structure is declared as objects also. to visit this structure, i also need to introduce some orders. the orders are maintained as pointers to the indexes structure. After 2 hours debug, i found when the copy happens, all the index structure orders becomes invalid. = -. T, T cxx

pointer to a vector element, caution !

2008-10-29T06:36:00.002+09:00

have a pointer to a vector element, then just push_back element to the vector, then the pointer will be invalid when vector enlarge the content.

e.g:

vector<A> vec;

vec.push_back(A(1));

A* p_a = &vec[0];

vec.push_back(A(2));

vec.push_back(A(3));

...

then try to access p_a, it will report wild pointer error ...

2 hours debug ... T, T

being too flexible sometime means disaster.

convert int to string, cpp

2008-10-28T04:30:00.002+09:00

http://notfaq.wordpress.com/2006/08/30/c-convert-int-to-string/#

In order to convert an int (or any other numeric type, e.g., float, double, etc.) to string, you can use:

#include <sstream>

int i = 5;

std::string s;

std::stringstream out;

out << i;

s = out.str();

about exporting query results in mysql

2008-10-23T20:37:00.001+09:00

Though dumping database can be done easily in mysql by the mysqldump command, it can not be used to export query result into files. Using sql to do data preprocessing is very useful approach in various cases. After reading the mysql cookbook, there're two ways:

1. select ... into outfile 'filename'

this is a specific sql pattern supported by the mysql. when we're work on the server machine and wanna export the file on the sever itself, then we can use this command. The file in default will be output into the data directory of mysql installation. Another thing is we need to give mysql command the file r/w privileges.

2. mysql -e 'select ....' -N dbname > filename -u user -p

if we can not use the server machine, for example accessing the server from a client machine, then this command can produce what we want. -e specified the query in quotes. -N tells mysql to produce the output file without the column names. dbname is the database name where the query run against, -u is follows the user who have privilege to access the db, -p is the password.

both commands will add the tab mark between different files and new line mark between different rows by default. there're lots of commands to change the output format, that is omitted here.

use Marshal instead of YAML when interoperability is not important

2008-10-18T13:49:00.002+09:00

YAML is slow. Even in research dataset, the data can be pretty huge, if interoperability is not that important, use Marshal.dump/load to manipulate the data. It is very fast, but the file is not readable by both people and other languages. I've some experience waiting for the YAML dumping some 50000 elements hash for more than three hours. Ater a quick marshalling, better let the YAML run again slowly when sleeping if you also want interoperability.

default is by value

2008-09-09T20:35:00.002+09:00

not like in java, everything is by default by reference. in c++, is by value. dreaming in java, but writing c++ gave me a nightmare yesterday night. the story is something like this:

class Family {
vector<Child> children;
vector<Child> getChildren() { return children; }
}

the calling code to the children in the family is by calling getChildren. then iterators are initialized like this:

vector<Child>::iterator pos = ourfamily.getChildren().begin();
vector<Child>::iterator end = ourfamily.getChildren().end();
while(pos != end) {
//print children
pos++;
}

another way to do this is like this way:

vector<Child>::iterator pos = ourfamily.getChildren().begin();
while (pos != ourfamily.getChildren().end()) {
//print children
pos++;
}

guess what? the two types of calling results in different print outs, not the orders, i mean the number of children. i thought i was wired at the beginning, is that a bug in the libstdc++ ? since, if the family is large enough, there'll be some segement fault. by checking everywhere, 3 hours passed.

in the morning, my post in some forum got a reply, the answer is :

vector<Child>& getChildren() { return children; }

because of being lack of this reference marker, the iterator pos and end are iterators of two different temporate objects !! and the second way to iterator even worse, which generate more than two temporate objects. segement fault maybe because some pointers in the iterator implementation pointed to some taken addresses. = -

DO REMEMBER: BY DEFAULT, CXX IS BY VALUE
this is really easy to be forgot if we wrote other languages for a long time. 3 hours just for a reference marker. orz...

DON'T generate too much temperate variables in c++

2008-09-09T03:09:00.002+09:00

writing too much java or ruby makes people too comfortable. the prof asked me to rewrite the implementation in c++, so i turned to this these days. last time, i used c++ was back to the 2003. still undergraduate freshman. no experience with any programming language expect some qbasic. c++ to me at that time was ok. after that turned to java, then c++ turns to me very ugly. lots of things you need to consider, memory leaks, meaningless compiler errors, one mistake leads hundres of error complaint and etc. after 5 years, c++ to me know, changed a lot. namespaces give somewhat name seperations, stl seems not that bad, boost provides many features whoever no binary which needs compilation yourself == at least 2 hours if you're lucky. after struggling several days, i still did not feel any beauty in it, though i'll still use it.

don't generate too much temperate variables in c++ since it has some copy constructor. if you defined a copy constructor, then any place the copy construction happens, the constructor will be used. but the problem is, without some knowledge about the circumenstance copy happens, the program is really tricky. programming methdology changed to agile these days. today lots of people do programming using those friendly langugages, can just refactoring without considering too much. however in c++, the situation is different. some step in refactoring leads to tricky problems, the reason of this maybe is its advantage : too many details. like this copy constructor.

the class i'm working with is some basic data class representing IP concept. at first, only id and address, like IP(0, 192.168.1.1), since i wanna use stl, for extension purpose, i override the default copy constructor. however, since the algorithm becomes complicated, some container sturcture added to the IP class, for example the history (map<int, vector<Site> > ). by forgetting modify the copy constructor, and my body reflection still in java and ruby something like that, the tricky happens:

Ip aip = system.getAllIps.findById(1)
aip.modify_history

guess what, if you print system.getAllIps.findById(1).history, no modification happened!! the sin is that aip temporate variable. in java, it is just a reference, to the real place of that object. however, the assignment operation by default will call that copy constructor and generate one brand new object to you. as i forgot that, 1 hour spent on it. after reminding some memory back to five years ago, then this was fixed. T, T

too many detail for a programming language to be industry wise used, is it a good idea or not ? maybe yes, maybe not. that's the powerness and productivity trade off in programming language design. most of application programmers who have not that much consideration on performance trend to be like those friendly. system programmer who has tough performance requirment tend to go to powerful first.

anyway i still do not like it very much. too many detail but friendless is one reason, there're more.

ubuntu, set resolution

2008-07-31T17:55:00.002+09:00

the machine in the lab is weird. though the cpu is core due 2.8Ghz, the lab members use 2x256m memory, also one of the 256m memory is broken, the left memory only 248m. = =b i used this machine as a web server using some heavy j2ee servers. in the last few months, always experiencing out of memory exception. the good news is they gave me 1g memory today. sounds great. after i plug in the memory, something weird happended. the screen resolution is not correct. after checking xorg.conf and refer to several articles in the web, i realized that ubunt failed to recognize the monitor. so the resolution at most is 1280x1024, however, the monitor i used is a standard monitor, which means the highest resolution for the 17 inch screen would be 1024x768.

the way to change the resolution is like this, modify the x configuration file, xorg.conf

1. sudo cp /etc/X11/xorg.conf /etc/X11/xorg.conf.backup
the reason the backup the xorg.conf is because X is famous for bad temper, when the configuration file is incorrect, we can not enter the gui. however, afterwards, we need to change this configuration file, we'd better backup it.

2. sudo vim /etc/X11/xorg.conf
found the Monitor section, change the monitor section according to the monitor model, the horizSync and vertRefresh should be contained somewhere in the monitor manual. if there's no manual or instruction, search the web using model number, it should be easy to find them. to define more resolution, change the Screen Section, Display sub section, add more resolution in the sub section. save the modifcation. restart X (ctrl + alt + backspace)

Section "Monitor"
Identifier "782LE"
Option "DPMS"
HorizSync 30-80
VertRefresh 56-75
EndSection

Section "Screen"
Identifier "Default Screen"
Device "Intel Corporation 82915G/GV/910GL Integrated Graphics Controller"
Monitor "782LE"
DefaultDepth 24
SubSection "Display"
Modes "1600x1200" "1280x960" "1280x1024" "1024x768" "800x600" "720x400" "640x480"
EndSubSection
EndSection

3. change the resolution, from System --> Preferences --> Screen Resolution, there'll be more resolution to choose from.

* If in the second step, the file fomrat is wrong, or the identifier is not right, the x will not be started properly, there'll be some configuration window pop out at low resolution, accept it temporaly. and copy the backup file to modified file, and restart x again, we'll back to the original state.

firefox bookmarks and bookmak sync

2008-07-19T14:53:00.002+09:00

i'm a big fan of Google Browser Sync , however after firefox 3.0, google decided not keep supporting this product, since there're several other tools with the same functionality available. the key idea is to synchronize the browser metadatas with several computers. last year, i worked on more than 4 computers, thus this function is extremely useful.

today, when i updated to firefox 2.0.0.16, all my bookmarks lost. I think it was because, the google browser sync do not support this higher version or the dataset in the server was cleared to let people change to other tools. anyway, i've got the problem. here's the solution

1. go to C:\Documents and Settings\[User Name]\Application Data\Mozilla\Firefox\Profiles\[A Dir Name]\bookmarkbackups
you can see lots of backups there. cool. pick some of them, for example a.html
2. uninstall google browser sync, restart firefox
3. Bookmarks => Organize Bookmarks ... => File ... => Import ... => a.html

done. the problem is the cookie info maybe lost, i'm not sure.
i'll install that firefox 3.0 and try molliza weaver.

actually, these kinds of bookmarks service, like del.us can do lots of applications. with the dataset, we can do recommendation, not only new pages, also new people. we can also do search, usually the bookmark are selected by the users. these webpages are more valuable than some of others. it'll be another way to do better search. just not sure whether google stops supporting this product is right or not, however, the codes of google browser sync is available on google codes. good news.

another way to solve international encoding and decoding

2008-07-17T18:10:00.002+09:00

from this site: http://www.ruby-lang.org.cn/forums/viewthread.php?tid=601
there's another solution, other than the libcharguess one

use rchardet instead, as said in the original post, this is a python char guess port.

since the libcharguess, they do not provide any binary things in the sourceforge, and the compile file in based on linux environment.
if we work on windows, install a cygwin for a quick demo program is too long. without compiling libcharguess, you can not use the ruby bind lib for libcharguess.

so from this point of view, rchardet is good solution, of course, also combine using with iconv

1. install rchardet, very trivial
gem install rchardet

that's it

2. use it
require 'rchardet'
str = ...
puts CharDet.detect(str).encoding

then the encoding can be used in iconv, refer to the post http://swimminginbits.blogspot.com/2008/06/use-iconv-libcharguess-solving.html

ubuntu pure-ftpd add user

2008-07-14T20:51:00.002+09:00

first time to add a user, log something here

1. use apt-get to install pure-ftpd
sudo apt-get install pure-ftpd pure-ftpd-common

2. create ftpuser group
sudo groupadd ftpgroup
useradd -g ftpgroup -d /dev/null -s /etc ftpuser

3. create user
sudo pure-pw useradd friends -u ftpuser -d /home/ftpusers/friends
sudo pure-pw mkdb
cd /etc/pure-ftpd/auth/
sudo ln -s ../conf/PureDB 60pdb

4. restart server
sudo /etc/init.d/pure-ftpd restart

5. create the virtual user root directory
sudo mkdir /home/ftpusers/friends
sudo chmod 777 -R /home/ftpusers/friends

done

enable access log in tomcat

2008-07-09T00:12:00.003+09:00

wanna test some anonymity networking softwares, so set up a tomcat, the question is how to enable access log. after reading tomcat definitive guide, 2ed, here comes the answer. btw, that book is really good, also covers tomcat 6.

uncomment these in server.xml of tomcat will enable the access log functionalities.

directory="logs" prefix="localhost_access_log."
suffix=".txt" pattern="common" resolveHosts="false"/>

next step. try tor. one problem of tor is the limited tor node on the internet. what if all people who use tor can be a node, that'll be great.

column store seems more and more popular

2008-07-08T08:31:00.002+09:00

last year, did some survey about column store. mainly about vertica, however, the trail of vertica need human verification, it's hard to get that using a graduate student profile. the c-store paper seems charming, and vertica goes further. with column store other than row store, the compression techniques can be used more vastly and deeply, since usually lots of values in the same column have similar values or valuable rules, which is a good property for compression. the retrieve efficiency will be boosted also, since columns stored together, thus only needed column in the result will be returned, other columns will keep untouched. most olap apps are read dominant. so, column techniques are good for this case. though the insertion, modification and deletion will take more time in column. sybase iq used these technology long ago, however, no big influence in the industry.

sofa, one size fit all, is a question or marketing strategy, proposed by mike stonebreaker to advertise his vertica in vldb last year, makes the industry looking that seriously again. sofa, key idea is nowadays the modern relational databases are becoming so huge, sql specification also hugs lots of rarely used, company specific features, these unnecessary features are not required by all kinds of applications. the db research fields are big fan to make any new data model be stored in the database, like store xml in relational database, store objects in relational database, even rdf and etc. the main reason is to reuse all the techniques in relational database. however, how much cost we payed for these kinds of one size fit all solutions. you can say, this sofa is a kind of marketing strategy by vertica, since business need stable and mature software.

last year vldb, the best paper was using column store to store rdf format, by mit dna who is one key contributor in vertica and mit c-store. though idea is simple, pretty charming. most influence ideas are always simple. behind these kinds of facts, maybe we can learn something, how to choose the right time, how to use old ideas to develop new techniques when the environment changes. nod.

Something frustrating in activerecord

2008-06-29T17:06:00.003+09:00

yep, another data mining project started. during the last few months, i managed to work out the algorithm. the professor got me some real dataset in some company, now it's the time to try the new bees. as usual, try to dump the plain txt file into mysql and write ruby code to handle them, the main reason is ruby is pretty agile language. i mean you can modify it without lots of effort. just write the code and go. everything works well, like speak a language. pretty satisfied. data analysis work in my last few projects only use activerecord and did not use rails at all. the problem is activerecord 2.1.0 seems can only be a plain java bean like class, and can not have any self defined methods in it ?? i hope no. but why recently at least two times, my class which extends ActiveRecord::Base which i added some instances methods, the 'save' and 'create' in ActiveRecord just do not work at all. it always pops out some errors like this :

transaction.rb line xxx, no delete method for NilClass

I've no idea how to deal with this error yet. in my memory when i still work on web development, the activerecord did not have such errors. oops = - . my current solution is comment out all the self defined code only left activereocd specifed code there. like has_many, belongs_to, set_table_name. after the data import work, dump the database, and then add what code i want, and begin analysis. actually no solution in this case.

Start working on XMP

2008-06-25T21:08:00.003+09:00

Though I've some meta data experience since 2 years ago, these days are my first chance to work on something directly related to meta. XMP = Extensible Meta data Platform , which is an open source meta data standard and toolkit. With it, we can embed meta data in the file, and query, modify and delete them when the file created, updated, being operated and etc. This BSD based standard is published by Adobe lead group. W3C said that it is important standard on its semantic web stacks, however they wanna work out something newer than XPM. Adobe disagree that the community need another one. Still reading the specification about it, seems interesting.

use iconv & libcharguess solving international encoding problem

2008-06-19T02:26:00.002+09:00

though last post said that the ruby on ubuntu was frustrating. anyhow, there's a old saying that if it is gold, it will shine. using iconv and libcharguess really solved the problem. the steps are as follows

1. get libcharguess at sourceforge
2. go to the directory ./configure
3. make install
4. make check, if nothing outputs, you're done
5. sudo make install,
6. get charguess at ruby-lang
7. go to the directory, cp libcharguess-dir/cpp/charguess.h to the ungzip directory
8. ruby -r mkmf extconf.rb, if mkmf missing, refer to the last post
9. ruby sample.rb, if something end with -JP print out, we've done

by getting these two libraries. the 'charguess' can be required can used sweetly.
require 'charguess'

somestring_encoding = CharGuess::guess somestring

if somestring is latin1 maybe, the somestring_encoding will be nil , otherwise it will be the guessing encoding. use the encoding and target encoding in the Iconv constructor:

converted_string = Iconv.new(target_encoding, somestring_encoding).iconv(somestring)

the string can be converted well. cool.

ruby on ubuntu really makes people frustrating, at least me

2008-06-19T02:23:00.001+09:00

still working on that ftp search. the first task was crawl several ftp file information and make an index. however, the directory of some ftp including non latin1 encodings. when save that into mysql which i've already configure the table to be utf8, it always can not interpret correctly. so, i turn to ruby cookbook. the first section i've found asking me to add $KCODE='u' and require 'jcode' at the beginning of the source code. i did it, but did not work. i guess maybe the ftp site return the string, its encoding need to be changed to utf8. so i turned to the second section in ruby cookbook, it told me that use iconv lib. well sounds good. but i don't know the encoding which ftp site returned. well, well, well, the great book told me you can use libcharguess, which is an sourceforge hosted open source library. also to use that library you need to install the ruby extensions. these two things come from different sites.

that's only the first part of the story. when i get both libraries, make install both of them. however the ruby extension gave me some extconf.rb file to use, since in ruby, make file sometimes is like that way. run it, = = require 'mkmf' error, that one is not found. - = how would that happen ... amazingly google gives me one harvard law school blog, which including some illustration, the mkmf is in the ubuntu ruby1.8-dev package. we need to install that first. - = . after that, the make file finally successfully created. - -

Net::FTP of ruby on windows, and ubuntu: No such file or directory - /tmp/mysql.sock

2008-06-18T22:00:00.001+09:00

I haven't used rails for a long time, since the major was slightly changed, from software engineering which is more practical to data mining. these days, i wanna build the community ftp search site for friends, so i turn to ruby again. i really like coding with it. the first thing was very strange that

Net::FTP of ruby seems not working very well on windows, look at the code

require 'net/ftp'
Net::FTP.open('host', 'user', 'pass') do |ftp|
puts ftp.pwd # this works
ftp.list do |file| # list and its alias, 'ls', 'dir', does not work
puts file
end
end

the pwd works pretty well, then the 'list' or 'ls', 'dir' just stuck there, there's little resources about that online. surprisingly, the same code works prefect well on ubuntu. So, i think it's the problem of the windows distribution of ruby.

since the piece of code works well on ubuntu, the next thing was making the rails run on the machine. the mysql.socket in ubuntu at default is not under /tmp/mysql.socket, the file is located at /var/run/mysqld/mysqld.socket. the location of this file can be get from /etc/mysql/my.cnf . by pointing this file at activerecord configure, everything should be all right.

the last thing i wanna mention is, the activerecord 2.1.0 to me seems having some strange behavior on the ubuntu, the following code

def Test
set_table_name 'test'
attri_reader :id, :text # this line has problem

def initialize
@text = 'somevalue' # this value will not store
end
end

when i call

Test.new.save

the error will occur: activerecord transaction blablabla 'delete' is not defined

but the same code runs perfectly on windows

is there anything wrong with my linux ruby settings ? or the distribution itself has something wrong ?
the activerecord i installed it by

sudo apt-get rubygems
gem install rails --include-dependencies

mysql 5 create table attribute name can not be 'group' ?

2008-06-18T18:00:00.003+09:00

it's pretty strange that the create table clause in mysql 5, can not use the 'group' as the attribute name. i tried that on ubuntu today. the way to fix the query was to change 'group' to other names.

maybe javassist will help the activerecord java implementation

2008-06-16T21:28:00.001+09:00

The last course project in adb was a simple jdbc task. I wanna try something new. The first thing, i did was to read the JDBC 4.0 related new features. However, quite disappointed about that. The whole EoD (ease of development) chapter in JSR 221 was deleted at the final stage when it published the final version.

Removed the EoD chapter, targeting for JDBC.next removing the following:
remove the methods: Connection.createQueryObject, DataSource.createQueryObject, DatabaseMetaData.providesQueryObjectGenerator, CommonDatasource.getQueryObject Generator
remove the interfaces, enums and annotations: DataSet, BaseQuery, DataSetResolver, QueryObjectGenerator, ConflictingRow, DataSetSyncStatus, GeneratedKeys, AutogeneratedKeys,
ResultColumn, Select, Update, SQLDataSetSyncException, SQLRuntimeException, QueryObjectFactory

This is really ironic, the JSR 221 objective is

This specification seeks to improve Java application access to SQL data stores by the provision of ease-of-development focused features and improvements at both the utility and API level.

I still don't understand why it happens. The original specification allowed us doing something like this. There's a student table in the db, the bean class is Student, the Controller class is StudentController which implements the BaseQuery interface. By provide the sql, without writing detailed JDBC code, we can have its implementation on the fly. Pretty cool, actually.

class Student {

    int id;

    @ResultColumn("last_name") String lastName;

    @ResultColumn("first_name") String firstName;

}



interface StudentController extends BaseQuery {

    @Select("select * from student where id = ?1")

    DataSet<Student> getStudentById(int id);

    @Update("delete from student")

    boolean deleteAllStudents();

}


StudentController studentController = con.createQueryObject(StudentController.class);

DataSet students = studentController.getStudentById(1);

for (Student student : students) {

    //operate student

}

So, if this does not work, what about making it work ? I did it last week. The effort was to implement an active record design pattern in java. The current work has not support relations between entities yet. Maybe if there's some time, I'll try to refine this work. The objective to write this post was to say maybe javassit will help the implementation. By using it, the framework can add method to the object on the fly. The dynamic behavior will let the framework more ease of use.

nod. The last thing is GluonJ seems an interesting aop framework released from the same lab who invented the javassit. Try to look at that if possible.

Something about charts

2008-06-16T21:07:00.000+09:00

After finishing implementing bide, i wanna draw some charts to present the log data. At first, try to use excel 03. However, the line chart drawn by that is hard to configure. I failed to let the x and y axes drawn dots according to their values. So, turn to some chart apis, which I planned to get familiar with at least two years ago. = =

The first thing I dig into was Google Chart API. However too many http parameters to me. After half an hour, I decided to get something else. The next thing google search engine told me was gchartrb, which was a wrapper api of google chart written in ruby. Sounds good. The example is worth reading.

I also try to get to know JFreeChart, which was another popular choice for java developers. There's one thing I wanna mention was this vector diagram. Maybe I will used it someday, since
this seems a good data presentation style for some clustering algorithm. log it here.