1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
|
* Running the programs
Should've been started by init:
- postgres
Should've been started by init. If not, do
sudo /etc/init.d/postgresql-8.3 start
- Filter port 50001, where BGPMon runs
sudo ~linus/fw.sh
- BGPMon -- runs in foreground (so you might want to to this in screen(1))
cd ~linus/bgpmon && sudo ~frank/bgpmon_v6/bgpmon
- bgpstore -- runs in foreground (so you might want to to this in screen(1))
sudo -u bgpstore ~bgpstore/bgp-logger/src/start-bgpstore.sh victoria 50001 bgpstore bgpstore bgpstore localhost
- psql -- for peeking in the database (pw = bgpstore)
psql -h localhost bgpstore bgpstore
* BGPMon
host: victoria.tug.nordu.net
telnet localhost 50000 # CLI for the logger software, pw=nordunet
nc localhost 50001 # Stream of log data.
The product is called [[http://bgpmon.netsec.colostate.edu/][BGPmon]]. There's XFB, an [[http://tools.ietf.org/html/draft-cheng-grow-bgp-xml-00][XML format]] for BGP
messages and control information, a.k.a. XFB.
There are two scripts in tumleren.pilsnet.sunet.se:~frank called
client_dom.pl and client_sax.pl. They've been modified by Frank.
There's a version 6.1 of the BGPmon sw announced on their website but
one needs userid & pw to get at it. I've contacted He Yan ([[gnus:nnimap%2Bimap.nordu.net:INBOX.sent-mail.2009-06#52][Email from
Linus Nordberg: BGPmon version]]) about that. <2009-06-22 Mon 22:23>
Turns out that what they mailed Frank was actually 6.1, which is what
we're running.
The program (bpgmon) must be run as root since it uses source port 179
for the BGP session with se-tug. Also, I had to fix a syntax error in
the config file found in Franks home directory which looked like it
should've read
<LABEL_ACTION>1</LABEL_ACTION>
so now it does just that.
* Running postgresql
** Setting up postgres accounts
$ sudo -u postgres createuser
Enter name of role to add: bgpstore
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
** Starting the server
sudo -u postgres /opt/local/lib/postgresql83/bin/postgres -D /opt/local/var/db/postgresql83/defaultdb
or
sudo -iu postgres
/opt/local/lib/postgresql83/bin/pg_ctl -D /opt/local/var/db/postgresql83/defaultdb -l logfile start
** Backing up the bgpstore table
sudo -iu postgres
time nice pg_dump bgpstore | dd bs=10M of=/var/backups/postgresql/bgpstore
With ~3.3G in /var/lib/postgresql/ I got
3039893830 bytes (3.0 GB) copied, 778.109 s, 3.9 MB/s
real 12m58.420s
** Creating index
After ~9 million records had been added to bgp_message, selecting on
prefix was really slow. We decided to create an index on that column,
which we should've done from the start.
PostgreSQL can do that without locking the table but it will take much
longer and may fail, see [[http://www.postgresql.org/docs/8.3/interactive/sql-createindex.html][docu]]. We decided that it's perfectly ok to
lose some data at this point, so bgpstore was stopped for a couple of
minutes while the index was created.
bgpstore=> create index prefix_idx on bgp_message (prefix);
CREATE INDEX
bgpstore=> \d bgp_message
Table "public.bgp_message"
Column | Type | Modifiers
----------------+-----------+----------------------------------------------------------
id | integer | not null default nextval('bgp_message_id_seq'::regclass)
timestamp | integer | not null
precision_time | smallint |
prefix | cidr | not null
label | text | not null
path | integer[] |
nexthop | inet |
bgp_octets | text | not null
Indexes:
"bgp_message_pkey" PRIMARY KEY, btree (id)
"prefix_idx" btree (prefix)
Searching for prefixes now happens within second(s) rather than a
minute. Feels much better. :-)
* db layout
<2009-06-24 Wed> Discussions with Fredrik.
- We should store one entry per prefix that has changed. This means
that one UPDATE message can result in more than one entry.
- TIMESTAMP isn't enough, there's often more than one message per
second. We'll have to store PRECISION_TIME too.
- Because of TIMESTAMP (above), we need some other primary key, like
an id with AUTO_INCREMENT. (Can we make postmodern add that?)
Update: It's called [[http://www.postgresql.org/docs/8.3/interactive/datatype-numeric.html#DATATYPE-SERIAL][SERIAL]] in PostgreSQL. Use 'bigserial' if we
anticipate more than 2^31 entries. ':col-type serial' should
probably do it.
- We've identified, from the perl program output, the following fields
to be of interest:
- TIMESTAMP (int32)
- PRECISION_TIME (int)
- WITHDRAWN (list of prefixes), possibly including label
- NLRI (list of prefixes), possibly including label
- AS_PATH (list of integers)
- NEXT_HOP (int32)
- I think we care only about UPDATE messages.
- We also should store OCTETS in OCTET_MSG when TYPE is UPDATE (2).
Just in case we find out that we missed something. This is the
complete BGP message.
* TODO
- [X] store timestamp and precision-time
- [X] store as-path
- [X] move to victoria
- [X] store nexthop
- [ ] store octets as binary, in separate table
- [ ] limit access to bgpmon stream (acl's)
- [ ] get an idea of disk footprint
- [ ] get an idea of RAM footprint
- [ ] get an idea of CPU load, sbcl and postgres
- [ ] auto-start on boot
- [ ] reconnect
* bgpstore operations
Disk used (whole system) early morning 2009-06-26: 1781468 Kb.
| date | ~ 10^6 rows | disk used (Kb) | RAM used (Kb) | sbcl VSZ/RSS |
|------------------------+-------------+----------------+---------------+--------------|
| <2009-06-26 Fri 17:12> | | 1822492 | 499964 | |
| <2009-06-27 Sat 12:30> | | 1866388 | 496028 | 536224 36420 |
| <2009-06-27 Sat 22:42> | | 1885236 | 496924 | 536224 43300 |
| <2009-06-28 Sun 07:43> | | 1899988 | 500164 | 536224 42852 |
| <2009-06-29 Mon 19:57> | | 1983632 | 499472 | 536224 41612 |
| <2009-07-01 Wed 12:36> | | 2111188 | 485120 | 536224 42840 |
| <2009-07-03 Fri 16:28> | | 2242152 | 500028 | 536224 44992 |
| <2009-07-06 Mon 00:01> | | 2392316 | 499656 | 536224 45388 |
| <2009-07-12 Sun 12:54> | 2.9 | 2831324 | 499128 | 536224 42864 |
| <2009-07-16 Thu 22:04> | 3.86 | 3330048 | 499752 | 536224 45452 |
| <2009-07-27 Mon 14:01> | 5.8 | 4064096 | 499776 | 536224 43308 |
| <2009-09-14 Mon> | 14 | 7772144 | 492344 | 536304 39980 |
| <2009-09-21 Mon> | 15.2 | 8198072 | 498712 | 536304 43296 |
| <2009-09-30 Wed> | 16.9 | 8827008 | 498468 | 536304 41608 |
| <2009-10-02 Fri> | 17.6 | 9500536 | 495644 | 536304 41428 |
| | | | | |
** <2009-08-07 Fri> bgpstore isn't running
The SBCL process probably died when server was rebooted last Tuesday
(2009-07-28) since it ran in the foreground in a screen on server.
We have 6221901 rows in the table.
Lowest and highest timestamp:
Fri Jun 26 03:58:59 UTC 2009
Tue Jul 28 13:27:29 UTC 2009
(
bgpstore=> select timestamp from bgp_message order by timestamp limit(1);
bgpstore=> select timestamp from bgp_message order by timestamp desc limit(1);
$ date -ur 1245988739
)
Restarting bgpstore:
2009-08-07 09:30:26: bgpstore started ..
* bgpview
** Development
To enable logging:
(setf hunchentoot:*access-log-pathname* #p"/tmp/weblocks-access.log"
hunchentoot:*message-log-pathname* #p"/tmp/weblocks-message.log")
|
