teknomunk
/
dex
mirror of https://git.crlf.ninja/CRLF/dex.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
prototype federated search index and web interface
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
15 Commits
1 Branch
0 Tags
536 KiB
 
 
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
Go to file
pwm 13aca142f4
add socks dependency 		10 months ago
dex add proxy config support 10 months ago
.gitignore implement json canonicalization and signatures 11 months ago
Pipfile add socks dependency 10 months ago
Pipfile.lock add socks dependency 10 months ago
README.md clarity 11 months ago
SPEC.md record site exclusion 10 months ago
config.toml implement json canonicalization and signatures 11 months ago
dex.sql implement submission endpoint 10 months ago

README.md

Dex

Dex is a prototype federated search aggregator. It will query peers and show results from them.

Deployment

Requirements

  • Redis
  • sqlite3 cli tools
  • pipenv: pipenv will install all your dependencies for you. If you have a clean environment, like a container, you can just pip install everything listed underneath the requirements section manually. Python 3.11 is all that's tested.
$ git clone https://git.crlf.ninja/CRLF/dex.git
$ pipenv install
$ pipenv shell
$ sqlite3 dex.sqlite < dex.sql
$ rq worker 2>&1 > .worker.log &  # this will start the federation worker and put it in the background
$ sanic dex.app

A simple page indexing command is provided (not a full spider)

$ pipenv shell
$ python3 -m dex index "https://example.com/page-to-index"

It will probably not throw an error, and the indexed text content will be searchable via the web interface, and by other search nodes who send a request, and who have a compatible response endpoint waiting.