~helmut/debian-dedup.git
9 years agoimportpkg: declare boring hashes for ssdeep ssdeep
Helmut Grohne [Sat, 9 Mar 2013 21:49:00 +0000 (22:49 +0100)]
importpkg: declare boring hashes for ssdeep

9 years agoimportpkg: support ssdeep hash
Helmut Grohne [Fri, 8 Mar 2013 15:33:37 +0000 (16:33 +0100)]
importpkg: support ssdeep hash

9 years agogeneralize ImageHash to StoredHash
Helmut Grohne [Fri, 8 Mar 2013 15:33:09 +0000 (16:33 +0100)]
generalize ImageHash to StoredHash

9 years agoREADME: explain queries
Helmut Grohne [Thu, 7 Mar 2013 07:12:01 +0000 (08:12 +0100)]
README: explain queries

9 years agoREADME: added interesting query
Helmut Grohne [Wed, 6 Mar 2013 14:36:49 +0000 (15:36 +0100)]
README: added interesting query

9 years agowebapp: added /source/<pkg> page
Helmut Grohne [Tue, 5 Mar 2013 07:39:06 +0000 (08:39 +0100)]
webapp: added /source/<pkg> page

9 years agowebapp: helper function function_combination
Helmut Grohne [Tue, 5 Mar 2013 07:38:39 +0000 (08:38 +0100)]
webapp: helper function function_combination

9 years agoimportpkg: source header may contain a version
Helmut Grohne [Tue, 5 Mar 2013 07:21:13 +0000 (08:21 +0100)]
importpkg: source header may contain a version

9 years agowebapp: fix index template
Helmut Grohne [Mon, 4 Mar 2013 17:53:23 +0000 (18:53 +0100)]
webapp: fix index template

Apparently not all browsers understand <a ... /> in all rendering modes.

9 years agowebapp: use caching table "shared" for /binary page
Helmut Grohne [Mon, 4 Mar 2013 17:49:54 +0000 (18:49 +0100)]
webapp: use caching table "shared" for /binary page

9 years agowebapp: generate /comparison pages in constant-space
Helmut Grohne [Mon, 4 Mar 2013 12:49:22 +0000 (13:49 +0100)]
webapp: generate /comparison pages in constant-space

9 years agoimportpkg: record the source package relationship
Helmut Grohne [Mon, 4 Mar 2013 10:44:24 +0000 (11:44 +0100)]
importpkg: record the source package relationship

9 years agoupdate_sharing: wrong database name
Helmut Grohne [Sat, 2 Mar 2013 21:33:39 +0000 (22:33 +0100)]
update_sharing: wrong database name

9 years agoadd sharing table
Helmut Grohne [Sat, 2 Mar 2013 21:29:04 +0000 (22:29 +0100)]
add sharing table

The sharing table is a cache for the /binary web pages. It essentially
contains the numbers presented. This caching table is not automatically
populated. It needs to be reconstructed after every (group of) package
imports.

9 years agoupdate README
Helmut Grohne [Sat, 2 Mar 2013 20:46:47 +0000 (21:46 +0100)]
update README

 * Tell about schema.sql.
 * Explain WAL.

9 years agomove fetchiter from webapp to dedup.utils
Helmut Grohne [Sat, 2 Mar 2013 20:24:18 +0000 (21:24 +0100)]
move fetchiter from webapp to dedup.utils

9 years agomove sql schema to a separate file
Helmut Grohne [Sat, 2 Mar 2013 20:18:14 +0000 (21:18 +0100)]
move sql schema to a separate file

9 years agoadded html form to main page
Helmut Grohne [Sat, 2 Mar 2013 10:25:53 +0000 (11:25 +0100)]
added html form to main page

Thanks to Jan Luehr for doing the work.

9 years agowebapp: open database cursor lazily
Helmut Grohne [Mon, 25 Feb 2013 10:56:09 +0000 (11:56 +0100)]
webapp: open database cursor lazily

Makes things more correct when using Application in multiprocessing
context.

9 years agowebapp: pass database to Application class
Helmut Grohne [Mon, 25 Feb 2013 10:52:05 +0000 (11:52 +0100)]
webapp: pass database to Application class

9 years agoREADME: another interesting query
Helmut Grohne [Mon, 25 Feb 2013 10:49:27 +0000 (11:49 +0100)]
README: another interesting query

9 years agoMerge branch 'crosshash'
Helmut Grohne [Mon, 25 Feb 2013 09:00:50 +0000 (10:00 +0100)]
Merge branch 'crosshash'

Conflicts in webapp.py:
 * The fetchall -> fetchiter change caused big conflicts.
 * New hash combination (image_sha512, image_sha512) added.

9 years agowebapp: complete cross hash support
Helmut Grohne [Mon, 25 Feb 2013 08:55:35 +0000 (09:55 +0100)]
webapp: complete cross hash support

9 years agoautoimport: this is not how foreign key constraints work
Helmut Grohne [Mon, 25 Feb 2013 07:55:53 +0000 (08:55 +0100)]
autoimport: this is not how foreign key constraints work

9 years agohash image contents
Helmut Grohne [Sun, 24 Feb 2013 00:03:30 +0000 (01:03 +0100)]
hash image contents

9 years agoREADME: fix mistake
Helmut Grohne [Sun, 24 Feb 2013 00:02:38 +0000 (01:02 +0100)]
README: fix mistake

9 years agoimportpkg: ignore filenames with encoding errors
Helmut Grohne [Sat, 23 Feb 2013 08:53:33 +0000 (09:53 +0100)]
importpkg: ignore filenames with encoding errors

9 years agoautoimport: log which packages are dropped
Helmut Grohne [Sat, 23 Feb 2013 08:36:15 +0000 (09:36 +0100)]
autoimport: log which packages are dropped

9 years agoautoimport: fix version check to actually work
Helmut Grohne [Fri, 22 Feb 2013 18:59:00 +0000 (19:59 +0100)]
autoimport: fix version check to actually work

Don't fail on new packages and skip versions already processed again.

9 years agoautoimport: skip old versions entirely
Helmut Grohne [Fri, 22 Feb 2013 18:55:31 +0000 (19:55 +0100)]
autoimport: skip old versions entirely

Presumably this is responsible for the blocking curl processes, since
importpkg will terminate early when processing an old version.

9 years agowebapp: add caching headers
Helmut Grohne [Fri, 22 Feb 2013 17:33:22 +0000 (18:33 +0100)]
webapp: add caching headers

9 years agowebapp: stream responses
Helmut Grohne [Fri, 22 Feb 2013 17:21:44 +0000 (18:21 +0100)]
webapp: stream responses

Maybe this gets memory usage down for large responses.

9 years agowebapp: attempt to reduce memory usage
Helmut Grohne [Fri, 22 Feb 2013 16:47:14 +0000 (17:47 +0100)]
webapp: attempt to reduce memory usage

9 years agowebapp: support matching sha512 against gzip_sha512
Helmut Grohne [Fri, 22 Feb 2013 13:12:33 +0000 (14:12 +0100)]
webapp: support matching sha512 against gzip_sha512

This covers only the /binary page. The comparison may still be empty.

9 years agoautoimport: first wait on the import
Helmut Grohne [Fri, 22 Feb 2013 06:24:05 +0000 (07:24 +0100)]
autoimport: first wait on the import

Otherwise the import zombifies and curl blocks.

9 years agomove compression functions to module dedup.compression
Helmut Grohne [Thu, 21 Feb 2013 16:33:27 +0000 (17:33 +0100)]
move compression functions to module dedup.compression

9 years agodo not track byted compiled python files
Helmut Grohne [Thu, 21 Feb 2013 16:33:06 +0000 (17:33 +0100)]
do not track byted compiled python files

9 years agomove hashing functions to module dedup.hashing
Helmut Grohne [Thu, 21 Feb 2013 16:10:54 +0000 (17:10 +0100)]
move hashing functions to module dedup.hashing

9 years agoinclude maintainer information
Helmut Grohne [Thu, 21 Feb 2013 14:31:51 +0000 (15:31 +0100)]
include maintainer information

9 years agoadded a base template to the webapp
Helmut Grohne [Thu, 21 Feb 2013 14:26:24 +0000 (15:26 +0100)]
added a base template to the webapp

9 years agoadded useful links to webapp
Helmut Grohne [Thu, 21 Feb 2013 13:35:05 +0000 (14:35 +0100)]
added useful links to webapp

9 years agoadded README
Helmut Grohne [Thu, 21 Feb 2013 07:53:06 +0000 (08:53 +0100)]
added README

9 years agorename test.py to importpkg.py
Helmut Grohne [Thu, 21 Feb 2013 07:42:44 +0000 (08:42 +0100)]
rename test.py to importpkg.py

9 years agolicense as BSD-3
Helmut Grohne [Thu, 21 Feb 2013 07:41:56 +0000 (08:41 +0100)]
license as BSD-3

9 years agofix comparison of conflicting packages
Helmut Grohne [Wed, 20 Feb 2013 21:33:54 +0000 (22:33 +0100)]
fix comparison of conflicting packages

9 years agoreduce memory usage of autoimport
Helmut Grohne [Wed, 20 Feb 2013 20:37:57 +0000 (21:37 +0100)]
reduce memory usage of autoimport

9 years agofix links in index
Helmut Grohne [Wed, 20 Feb 2013 20:24:12 +0000 (21:24 +0100)]
fix links in index

9 years agominimal index page explaining stuff
Helmut Grohne [Wed, 20 Feb 2013 20:12:58 +0000 (21:12 +0100)]
minimal index page explaining stuff

9 years agoimplement autoimport
Helmut Grohne [Wed, 20 Feb 2013 18:04:18 +0000 (19:04 +0100)]
implement autoimport

9 years agomark required packages in binary view
Helmut Grohne [Wed, 20 Feb 2013 16:14:48 +0000 (17:14 +0100)]
mark required packages in binary view

9 years agostore hard dependencies
Helmut Grohne [Wed, 20 Feb 2013 15:52:17 +0000 (16:52 +0100)]
store hard dependencies

9 years agodetermine metadata from control.tar.gz
Helmut Grohne [Wed, 20 Feb 2013 15:41:30 +0000 (16:41 +0100)]
determine metadata from control.tar.gz

9 years agoteach ArReader to read multiple entries
Helmut Grohne [Wed, 20 Feb 2013 14:55:05 +0000 (15:55 +0100)]
teach ArReader to read multiple entries

9 years agocleanup
Helmut Grohne [Wed, 20 Feb 2013 14:39:33 +0000 (15:39 +0100)]
cleanup

9 years agomany improvements
Helmut Grohne [Wed, 20 Feb 2013 14:28:04 +0000 (15:28 +0100)]
many improvements

 * multiple hashes
 * template engine
 * new table package
 * comparison view
 * hashvalue view

9 years agofirst prototype
Helmut Grohne [Wed, 20 Feb 2013 14:27:40 +0000 (15:27 +0100)]
first prototype