The DELILAH Project is a program that traces semantical relations between words in all languages. To do that, it crawls the internet and looks for statistical datapairs of words and all the data is then displayed in the form of a graph. It was initially written is pure Java, but, due to the poor management of processes, we migrated it to an interressant cocktail : PHP and SQL on the server part and C# on the feeder part.

In the image you can see that Schwarzenegger is strongly linked to Arnold, but is only weakly linked to California. That means that "a lot" of pages that deals with Schwarzenegger also deals with Arnold and California but only "a few" of pages about California actually are about Schwarzenegger.

It was initially written by Alban Galland and me under the supervision of Professor Miller in the undergraduate course taught by Professor Jean Jacques Levy at the Ecole Polytechnique.

Actually, DELILAH is composed of two parts:

• The parser (developped in C#) is designed to crawl the web. It asks the server for a topic that hasn't been already searched, googleize on it and retrieve the content of all the pages given. It then performs a statistical analysis to extract the words that are strongly linked to this subject. Lastly, it returns a list of linked topics to the server.
• The server (developped in PHP/MySQL) is designed to pretty-print the data and to provide some interactivity. When the user asks for a topic, it retrieves all pertinent information from the base and uses Graphviz to display an image of it.

Let's say Google gives you ten URLs refering to the topic shark. Delilah will retrieve each page and will squeeze all HTML from thoses pages. Afterwards, it will list all words that are physically near from each occurence of shark. The more a given word is present, the heavier its weight will be.

Afterwards, you must get rid of some words that always appears (e.g. the, a, one, is, they...). Those words are easy to blacklist because, indeed, they always appear. Thus if a word is linked to more than 10 percent of the subjects, it is a junction word of the language.

As I already said, it initially was a project, so you can take a look at the report if you want. Unfortunately it is in french and (worse!) it deals with the JAVA version of the program. That's why the old java version is still available here to understand basic concepts.

Of course, we'll be happy to give it for free to anybody. It is not totally finalized yet, so I prefer not uploading an everchanging program: Please send me a mail.