about summary refs log tree commit diff stats
path: root/README.md
blob: be4c7518c6d5a0a4b7bcc16b0fd3c7f86b99a243 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
# getwtxt  [![Build Status](https://travis-ci.com/getwtxt/getwtxt.svg?branch=master)](https://travis-ci.com/getwtxt/getwtxt) [![Go Report Card](https://goreportcard.com/badge/github.com/getwtxt/getwtxt)](https://goreportcard.com/report/github.com/getwtxt/getwtxt) [![Code Climate Maintainability](https://api.codeclimate.com/v1/badges/0e48bd9002de0f84b24e/maintainability)](https://codeclimate.com/github/getwtxt/getwtxt/maintainability)

twtxt registry written in Go!

[twtxt](https://github.com/buckket/twtxt) is a decentralized microblogging platform
for hackers based on text files. The user is "followed" and "mentioned" by referencing
the URL to their `twtxt.txt` file and a nickname.
Registries are designed to aggregate several users' statuses into a single location,
facilitating the discovery of new users to follow and allowing the search of statuses
for tags and key words.

 
\[ [Installation](#installation) \]
  \[ [Upgrading](#upgrading) \]
  \[ [Configuration](#configuration) \]
  \[ [Using the Registry](#using-the-registry) \]
  \[ [Benchmarks](#benchmarks) \]
  \[ [Other Documentation](#other-documentation) \]
  \[ [Notes](#notes) \]

## Features
 [![GitHub release](https://img.shields.io/github/release/getwtxt/getwtxt.svg)](https://github.com/getwtxt/getwtxt/releases/latest)

* Easy to set up
* Uses an in-memory cache to serve requests
* Pushes to a database at a configurable interval for persistent storage
  * `leveldb (default)`
  * `sqlite3`
* More database support is in development
* Easily run behind `nginx`, `Caddy` or another HTTP server.

## Public Instances

* [twtxt.tilde.institute](https://twtxt.tilde.institute)
* [twtxt.envs.net](https://twtxt.envs.net/)

Would you like your instance listed? Submit a PR or shoot me an email :)

## Installation

I have only personally tested getwtxt on Linux, specifically:
* `Debian 9, 10/Testing, Sid`
* `Ubuntu Server 18.04LTS, 18.10, 19.04`

Build dependencies are minimal, and only include:
* `gnu make`
* `go >= 1.11`
* `git`

First, fetch the sources using `git` and jump into the directory.

```
$ git clone git://github.com/getwtxt/getwtxt.git
...
$ cd getwtxt
```

Then, check out the latest release tag.

This will be the same tag listed on the badge under the [Features](#features) section heading.

```
$ git checkout v0.0.0
```

Or, if you want to be a `git` ninja, you can use this to just check out whatever
the latest tag is:

```
$ git checkout $(git describe --tags --abbrev=0)

```

Optionally, use the `go` tool to test and benchmark the files in `svc`.

```
$ go test -v -bench . -benchmem ./...
...
...
PASS
```

Use `make` to initiate the build and install process.
```
$ make
...
$ sudo make install
```

## Upgrading

Upgrading is a fairly simple process. First, we need to commit your local changes
to the configuration file.

```
$ cp /usr/local/getwtxt/getwtxt.yml .
$ git add getwtxt.yml
$ git commit -m 'my local config'
```

Now, we need to either run `make update` or `git pull --rebase origin master`

```
$ make update
...
```

Afterwards, follow the normal instructions for building and installing.
If no configuration changes have been made since your last upgrade,
you will not need to commit them again. While `getwtxt` is pre-`1.0`, any
patch-level updates (`v0.4.x`) will not change configuration values.

Of course, you can also just back up your configuration file, then copy it
back into `/usr/local/getwtxt/` after installing the new version.

## Configuration

\[ [Proxying](#proxying) \]   \[ [Starting getwtxt](#starting-getwtxt) \]

To configure getwtxt, you'll first need to open `/usr/local/getwtxt/getwtxt.yml`
in your favorite editor and modify any values necessary. There are comments in the
file explaining each option.

If you desire, you may additionally modify the template in
`/usr/local/getwtxt/assets/tmpl/index.html` to customize the page users will see
when they pull up your registry instance in a web browser. The values in the
configuration file under `Instance:` are used to replace text `{{.Like This}}` in
the template.

### Proxying

Though getwtxt will run perfectly fine facing the internet directly, it does not
understand virtual hosts, nor does it use TLS. You'll probably want to proxy it behind
`Caddy` or `nginx` for this reason.

`Caddy` is ludicrously easy to set up, and automatically handles `TLS` certificates. Here's the config:

```caddyfile
twtxt.example.com
proxy / example.com:9001
```

If you're using `nginx`, here's a skeleton config to get you started. Don't forget to change
the 5 instances of `twtxt.example.com` to the (sub)domain you'll be using to access the registry,
generate SSL/TLS certificates using LetsEncrypt, and change the port in `proxy_pass` to whichever
port you specified when modifying the configuration file. Currently, it's set to the default port `9001`

```nginx
server {
    server_name twtxt.example.com;
    listen [::]:443 ssl http2;
    listen 0.0.0.0:443 ssl http2;
    ssl_certificate /etc/letsencrypt/live/twtxt.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/twtxt.example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://127.0.0.1:9001;
    }
}
server {
    if ($host = twtxt.example.com) {
        return 301 https://$host$request_uri;
    }
    listen 80;
    server_name twtxt.example.com;
    return 404;
}
```

### Starting getwtxt

Once you have everything configured to your needs, use `systemctl` to enable it
to run on system boot, then start the service.

```
$ sudo systemctl enable getwtxt
...
$ sudo systemctl start getwtxt
```

## Using the Registry

The following examples will all apply to using `curl` from a `Linux`, `BSD`, or `macOS` terminal.
All timestamps are in `RFC3339` format, per the twtxt registry specification. Additionally, all
queries support the `?page=N` parameter, where `N` is a positive integer, that will retrieve page
`N` of results in groups of twenty.

The example API calls can also be found on the landing page of any getwtxt instance, assuming
the admin has not customized the landing page.
* [twtxt.tilde.institute](https://twtxt.tilde.institute)

### Adding a User
Both nickname and URL are required
```
$ curl -X POST 'https://twtxt.example.com/api/plain/users?url=https://mysite.ext/twtxt.txt&nickname=FooJr'

200 OK
```

### Get All Tweets
```
$ curl 'https://twtxt.example.com/api/plain/tweets'

foo_barrington  https://foo.bar.ext/twtxt.txt  2019-03-01T09:31:02.000Z Hey! It's my first status!
...
...
```

### Query Tweets by Keyword
```
$ curl 'https://twtxt.example.com/api/plain/tweets?q=getwtxt'

foo_barrington    https://example3.com/twtxt.txt    2019-04-30T06:00:09.000Z    I just installed getwtxt!
```

### Get All Users
Timestamp reflects when the user was added to the registry.

```
$ curl 'https://twtxt.example.com/api/plain/users'

foo_barrington      https://foo.barrington.ext/twtxt.txt  2017-01-01T09:17:02.000Z
foo_barrington_jr   https://example.com/twtxt.txt         2019-03-01T09:31:02.000Z
...
...
```

### Query Users
Can use either keyword or URL.

```
$ curl 'https://twtxt.example.com/api/plain/users?url=https://example.com/twtxt.txt'

foo               https://example.com/twtxt.txt     2019-05-09T08:42:23.000Z


$ curl 'https://twtxt.example.com/api/plain/users?q=foo'

foo               https://example.com/twtxt.txt     2019-05-09T08:42:23.000Z
foobar            https://example2.com/twtxt.txt    2019-03-14T19:23:00.000Z
foo_barrington    https://example3.com/twtxt.txt    2019-05-01T15:59:39.000Z
```

### Get all tweets with mentions
Mentions are placed within a status using the format `@<nickname http://url/twtxt.txt>`
```
$ curl 'https://twtxt.tilde.institute/api/plain/mentions'

foo               https://example.com/twtxt.txt     2019-02-28T11:06:44.000Z    @<foo_barrington https://example3.com/twtxt.txt> Hey!! Are you still working on that project?
bar               https://mxmmplm.com/twtxt.txt     2019-02-27T11:06:44.000Z    @<foobar https://example2.com/twtxt.txt> How's your day going, bud?
foo_barrington    https://example3.com/twtxt.txt    2019-02-26T11:06:44.000Z    @<foo https://example.com/twtxt.txt> Did you eat my lunch?
```

### Query tweets by mention URL
```
$ curl 'https://twtxt.tilde.institute/api/plain/mentions?url=https://foobarrington.co.uk/twtxt.txt'

foo    https://example.com/twtxt.txt    2019-02-26T11:06:44.000Z    @<foo_barrington https://foobarrington.co.uk/twtxt.txt> Hey!! Are you still working on that project?e
```

### Get all Tags
```
$ curl 'https://twtxt.example.com/api/plain/tags'

foo    https://example.com/twtxt.txt    2019-03-01T09:33:04.000Z    No, seriously, I need #help
foo    https://example.com/twtxt.txt    2019-03-01T09:32:12.000Z    Seriously, I love #programming!
foo    https://example.com/twtxt.txt    2019-03-01T09:31:02.000Z    I love #programming!
```

### Query by Tag
```
$ curl 'https://twtxt.example.com/api/plain/tags/programming'

foo    https://example.com/twtxt.txt    2019-03-01T09:31:02.000Z    I love #programming!
```

## Benchmarks

* [bombardier](https://github.com/codesenberg/bombardier)

```
$ bombardier -c 100 -n 200000 http://localhost:9001/api/plain/tweets

Bombarding http://localhost:9001/api/plain/tweets with 200000 request(s) using 100 connection(s)
 200000 / 200000 [=============================================================] 100.00% 19961/s 10s

Done!

Statistics        Avg      Stdev        Max
  Reqs/sec     20006.58    2408.55   26054.73
  Latency        5.00ms     3.58ms    62.99ms
  HTTP codes:
    1xx - 0, 2xx - 200000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    39.27MB/s
```

## Other Documentation

In addition to what is provided here, additional information, particularly regarding the configuration
file, may be found by running getwtxt with the `-m` or `--manual` flags. You will likely want to pipe the output
to `less` as it is quite long.

```
$ ./getwtxt -m | less

$ ./getwtxt --manual | less
```

If you need to remove getwtxt from your system, navigate to the source directory
you acquired using `git` during the installation process and run the appropriate
`make` hook:

```
$ sudo make uninstall
```

## Notes

twtxt Information: [`twtxt.readthedocs.io`](https://twtxt.readthedocs.io)

Interested in twtxt but don't have your own server? [`github.com/LuRsT/twtxt_on_heroku`](https://github.com/LuRsT/twtxt_on_heroku)

twtxt Client Repo: [`github.com/buckket/twtxt`](https://github.com/buckket/twtxt)

Registry Specification: [`twtxt.readthedocs.io/en/latest/user/registry.html`](https://twtxt.readthedocs.io/en/latest/user/registry.html)

Special thanks to [`github.com/kognise/water.css`](https://github.com/kognise/water.css) for open-sourcing a pleasant, easy-to-use, importable stylesheet

### Contributing

All contributions are greatly appreciated! Please open an issue, submit
a pull request, anything.

If you're interested in contributing, take a look at the
github project page for getwtxt, linked below, where I've listed various items I'm
working on or plan to work on in the very near future. It'll give you some inspiration for
pull requests to submit. If you have an idea for a new feature or come across a bug,
submitting a new issue will be greatly appreciated as well.

* [getwtxt Development: the road to 1.0.0](https://github.com/getwtxt/getwtxt/projects/1)
* [open an issue](https://github.com/getwtxt/getwtxt/issues)