Development, Analysis And Research
by Andrew Johnstone
Istio EnvoyFilter to add x-request-id to all responses
In: General
17 Sep 2020A quick snippet to add an Istio EnvoyFilter to add x-request-id to all responses
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: gateway-response
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"@type": "type.googleapis.com/envoy.config.filter.http.lua.v2.Lua"
inlineCode: |
function envoy_on_request(handle)
local metadata = handle:streamInfo():dynamicMetadata()
local headers = handle:headers()
local rid = headers:get("x-request-id")
-- for key, value in pairs(handle:headers()) do
-- handle:logTrace("key:" .. key .. " <--> value:" .. value)
-- end
if rid ~= nil then
metadata:set("envoy.filters.http.lua", "req.x-request-id", rid)
end
end
function envoy_on_response(handle)
local metadata = handle:streamInfo():dynamicMetadata():get("envoy.filters.http.lua")
local rid = metadata["req.x-request-id"]
if rid ~= nil then
handle:headers():add("x-request-id", rid)
end
end
- No Comments
- Tags: envoy, istio, lua, tracing
Amazon ELB – monitoring packet count and byte size with Amazon Cloudwatch and VPC flow logs
16 Dec 2015We recently exceeded the capacity for one of our Amazon ELBs in a 60 second period. One of the points of our debrief was to monitor the network from the ELB. Amazon doesn’t provide statistics from ELBs to Amazon cloudwatch. So I came up with the following:
- Using amazons apis capture the network interfaces with attachment.instance-owner-id for “amazon-elb”.
- For each network interface create a cloudwatch metric filter with the network interface.
Running the above in a lambda function with a scheduled event every 5 minutes will create the relevant filters and cloudwatch logs. This takes the overhead out of analysing the VPC flow logs.
Amazon ELBs rotate their servers and the attached ENIs will obviously rotate, so the second part is to identify whether the ENIs have been released and clean up.
Amazon ELBs will add/remove nodes to/from DNS, which should be faster than a 5 minute period (the current minimum for a scheduled task).
As such the time taken to start graphing new ENIs added to the ELBs will not be immediate. The above graphs should help us to infer an increase in network traffic through the ELB and to observe to set sensible alarms.
- No Comments
- Tags: Cloudwatch, ELB, ENI, Lambda, VPC, VPC Flow Logs
SSL – Amazon ELB Certificates
20 Jun 2015I always forget the order to concatenate SSL certificates and to upload to Amazon ELBs as well as commands to verify SSL certificates.
Extract dates from certificate
echo | openssl s_client -connect co.photobox.com:443 2>/dev/null | openssl x509 -noout -dates notBefore=Feb 10 00:00:00 2015 GMT notAfter=Jun 9 23:59:59 2016 GMT cat 14619087repl_1.crt | openssl x509 -noout -dates
List DNS entries for SANs certificates
echo | openssl s_client -connect co.photobox.com:443 2>/dev/null | openssl x509 -noout -text | grep DNS | sed -r 's/[ ]+DNS://g;s/,/\n/g'
Extract certificate
openssl x509 -inform PEM -in xxx.crt
Concatenation of SSL Certificates for Comodo
cat COMODORSAOrganizationValidationSecureServerCA.crt COMODORSAAddTrustCA.crt AddTrustExternalCARoot.crt >> gd_bundle.crt;
Upload SSL certificates
aws iam upload-server-certificate \
--server-certificate-name sans-photobox-com-intermediate-expires-09-06-2016-v2 \
--certificate-body file://14619087repl_1.crt \
--private-key file://../co.key \
--certificate-chain file://gd_bundle.crt
- No Comments
- Tags: Amazon, ELB, SSL, X509
Automatically update Amazon ELB SSL Negotiation Policies
In: General
31 May 2015A quick script to update Amazon ELB SSL Negotiation policies for all Amazon ELBs.
The security policies are managed from Amazon and will only update the predefined security policies, choosing the latest one.
More information available on AWS predefined negotiation policies.
Sample output…
Amazon Opsworks – Dependent on Monit
17 Oct 2014I recently converted a number Amazon Opsworks instances to be time based, so that they shut down at the end of the working day.
After doing so, I kept seeing that the Opsworks life cycle events always being marked as in progress.
Every morning the instances would be marked booting and would be able to SSH into the instances and the applications would be running on them.
When I finally managed to investigate this morning I figured out why this was occurring and was a little surprised how amazon run the opsworks agent.
We have a number of custom monit configurations to ensure that our own applications are running. These do not interfere with the Amazon Opsworks monit configuration.
The development/qa team occasionally stop monit when doing development work or switching branches on these instances.
As can be seen from the following…
[UTC Oct 9 09:50:14] info : monit: generated unique Monit id 437edc13afbe8e51fde4501f9af8db8f and stored to ‘/var/lib/monit/id’
[UTC Oct 9 09:50:14] info : Starting monit daemon
[UTC Oct 9 09:50:14] info : ‘ip-10-75-15-96.eu-west-1.compute.internal’ Monit started
[UTC Oct 9 09:54:35] info : monit daemon with pid [6927] killed
[UTC Oct 9 09:54:35] info : ‘ip-10-75-15-96.eu-west-1.compute.internal’ Monit stopped
Amazon opsworks doesn’t use any form of upstart/init.d/systemd so when servers boot up its reliant on monit to be started. If the last state is monit being stopped this will fail to run opsworks and you will see the above state.
|
1
|
$ egrep -rin opsworks /etc/rc* /etc/init.d/ |
I’ve asked our development team to disable the correct processes within monit
E.g.
|
1
|
$ monit unmonitor node_checkout_process |
It would be sensible not to rely on monit and have a relevant script to spawn the process on server restart.
|
1
2
3
4
5
6
7
8
|
root@nodejs-dev-andy:/opt/aws/opsworks/current/bin# ./opsworks-agent-cli agent_reportPidfile /var/lib/aws/opsworks/pid/opsworks-agent.pid present but no matching process running - cleaning upAWS OpsWorks Instance Agent State Report:Last activity was a "configure" on 2014-09-26 18:03:22 UTCAgent Status: No AWS OpsWorks agent runningAgent Version: 325, up to date |
Hopefully Amazon will improve the way the opsworks agent is spawned.
- (1) Comment
- Tags: aws, Bash, monit, opsworks
Awk craziness: Processing log files
13 Apr 2013A while back I wrote an awk script to process a custom log file to work out how an application was performing over a period of time.
The following script takes a timestamp in field one $1 and also an arbitrary field $10 in this case.
For every 100 lines of the log file processed it would calculate the min, max, avg, stddev for rolling stats.
zcat Summary.2012-12-09.gz | grep '2012-12-09 20:[234]' | awk '
BEGIN{
size=50;
tmax = -999999999;
tmin = 999999999;
}
{ # Accumulate basic data
cnt[$10]++;
item[++n] = $10;
if ($10 > max) max = $10; if ($10 < min) min = $10;
if ($10 > tmax) tmax = $10; if ($10 < tmin) tmin = $10;
}
{
mod=NR%size;
if(NR<=size) {
count++;
} else {
sum-=array[mod];
min=max=$10;
};
total+=$10;
sum+=$10; sumsq+=$10*$10;
array[mod]=$10;
print $1" "$2 "\tvalue: " $10 "\tmin: " min "\tmax: " max "\tavg: " sum/count "\tstddev: " sqrt(sumsq/NR - (sum/NR)**2) "\t"
}
END {
printf("\n\n# Lines processed = %d\n", NR);
printf("# Total = %d\n", total);
printf("# Sqrt = %d\n", sqrt(sumsq/NR - (sum/NR)**2));
# Print Descriptive Statistics
printf("# Count = %d\n", n);
printf("# Min = %d\n", tmin);
decile = 1;
for (decile = 10; decile < 100; decile += 10) {
idx = int((decile * n) / 100) + 1;
printf("# %d%% decile = %d\n", decile, item[idx]);
if (decile == 50)
median = item[idx];
}
printf("# Max = %d\n", tmax);
printf("# Median = %d\n", median);
for (i in cnt) {
if (cnt[i] > cnt[mode])
mode = i;
}
printf("# Mode = %d\n", mode);
}
' | less
...
2012-12-09 20:20:01 value: 18 min: max: 18 avg: 5.8 stddev: 6.20967
2012-12-09 20:20:01 value: 7 min: max: 18 avg: 6 stddev: 5.68624
2012-12-09 20:20:01 value: 7 min: max: 18 avg: 6.14286 stddev: 5.27605
2012-12-09 20:20:01 value: 4 min: max: 18 avg: 5.875 stddev: 4.98592
2012-12-09 20:20:01 value: 2 min: max: 18 avg: 5.44444 stddev: 4.85595
2012-12-09 20:20:02 value: 3 min: max: 18 avg: 5.2 stddev: 4.66476
2012-12-09 20:20:02 value: 4 min: max: 18 avg: 5.09091 stddev: 4.46103
2012-12-09 20:20:02 value: 104 min: max: 104 avg: 13.3333 stddev: 27.6687
2012-12-09 20:20:02 value: 2 min: max: 104 avg: 12.4615 stddev: 26.7542
2012-12-09 20:20:02 value: 25 min: max: 104 avg: 13.3571 stddev: 25.9824
2012-12-09 20:20:03 value: 3 min: max: 104 avg: 12.6667 stddev: 25.234
2012-12-09 20:20:03 value: 3 min: max: 104 avg: 12.0625 stddev: 24.5445
2012-12-09 20:20:03 value: 3 min: max: 104 avg: 11.5294 stddev: 23.907
2012-12-09 20:20:03 value: 2 min: max: 104 avg: 11 stddev: 23.3357
2012-12-09 20:20:03 value: 6 min: max: 104 avg: 10.7368 stddev: 22.7407
2012-12-09 20:20:04 value: 4 min: max: 104 avg: 10.4 stddev: 22.2135
2012-12-09 20:20:04 value: 3 min: max: 104 avg: 10.0476 stddev: 21.7354
2012-12-09 20:20:04 value: 2 min: max: 104 avg: 9.68182 stddev: 21.3017
2012-12-09 20:20:04 value: 70 min: max: 104 avg: 12.3043 stddev: 24.1938
2012-12-09 20:20:04 value: 4 min: max: 104 avg: 11.9583 stddev: 23.7425
2012-12-09 20:20:04 value: 12 min: max: 104 avg: 11.96 stddev: 23.2628
2012-12-09 20:20:04 value: 14 min: max: 104 avg: 12.0385 stddev: 22.8144
2012-12-09 20:20:04 value: 6 min: max: 104 avg: 11.8148 stddev: 22.417
2012-12-09 20:20:04 value: 5 min: max: 104 avg: 11.5714 stddev: 22.0493
2012-12-09 20:20:04 value: 2 min: max: 104 avg: 11.2414 stddev: 21.7361
2012-12-09 20:20:04 value: 30 min: max: 104 avg: 11.8667 stddev: 21.6344
2012-12-09 20:20:05 value: 5 min: max: 104 avg: 11.6452 stddev: 21.3172
2012-12-09 20:20:05 value: 22 min: max: 104 avg: 11.9688 stddev: 21.0587
2012-12-09 20:20:05 value: 4 min: max: 104 avg: 11.7273 stddev: 20.7821
2012-12-09 20:20:05 value: 2 min: max: 104 avg: 11.4412 stddev: 20.54
2012-12-09 20:20:05 value: 25 min: max: 104 avg: 11.8286 stddev: 20.3701
2012-12-09 20:20:05 value: 27 min: max: 104 avg: 12.25 stddev: 20.2394
2012-12-09 20:20:05 value: 3 min: max: 104 avg: 12 stddev: 20.0203
2012-12-09 20:20:06 value: 2 min: max: 104 avg: 11.7368 stddev: 19.8198
2012-12-09 20:20:06 value: 1 min: max: 104 avg: 11.4615 stddev: 19.6375
2012-12-09 20:20:06 value: 2 min: max: 104 avg: 11.225 stddev: 19.4467
2012-12-09 20:20:06 value: 36 min: max: 104 avg: 11.8293 stddev: 19.5846
2012-12-09 20:20:06 value: 3 min: max: 104 avg: 11.619 stddev: 19.3968
2012-12-09 20:20:06 value: 27 min: max: 104 avg: 11.9767 stddev: 19.3096
2012-12-09 20:20:06 value: 2 min: max: 104 avg: 11.75 stddev: 19.1467
2012-12-09 20:20:07 value: 5 min: max: 104 avg: 11.6 stddev: 18.9589
2012-12-09 20:20:07 value: 2 min: max: 104 avg: 11.3913 stddev: 18.8039
2012-12-09 20:20:07 value: 5 min: max: 104 avg: 11.2553 stddev: 18.6256
2012-12-09 20:20:07 value: 4 min: max: 104 avg: 11.1042 stddev: 18.4597
2012-12-09 20:20:07 value: 2 min: max: 104 avg: 10.9184 stddev: 18.3156
2012-12-09 20:20:07 value: 4 min: max: 104 avg: 10.78 stddev: 18.1574
...
# Lines processed = 9863
# Total = 80137
# Sqrt = 15
# Count = 9863
# Min = 1
# 10% decile = 8
# 20% decile = 1
# 30% decile = 3
# 40% decile = 3
# 50% decile = 17
# 60% decile = 7
# 70% decile = 5
# 80% decile = 1
# 90% decile = 4
# Max = 140
# Median = 17
# Mode = 2
- No Comments
- Tags: awk, Linux, log processing
Elastic Search Presentation
29 Oct 2012I’ve implemented a number of sites with Elastic Search and recently had a two-day spike to implement Elastic Search for Photobox’s backoffice systems. As a result I have since wrote a presentation and presented at Photobox on Friday. I have posted the slides for the presentation here…
I wrote this presentation using rvl.io – reveal.js, however unfortunately does not allow you to nicely export the presentation to pdf or other non html formats. Best viewed full screen…
A few other interesting presentations I came across whilst writing up these slides.
- No Comments
- Tags: Elastic Search, Elastica, PHP
Requests per second from Apache logs
22 Sep 2012Whilst recently looking into usage of SSL, I needed to analyse some logs from apache to determine requests per second.
Initially I wrote a quick bash script to do this, however figured it would be handy to elaborate on this, as a result I have posted a perl script on github at apache-log-stats
total_requests=$(zcat b2f01_ApacheAccess.2011-12-05.gz | \
sort -b -k4.9,4.12 -k4.5b,4.7Mb -k4.2,4.3 -k4.14,4 | \
awk -F '[ [/:]' '{print $7"-"$6"-"$5":"$8":"$9":"$10}' | \
sort | uniq -c | sort -g | tail -n1);
echo $total_requests;
And the perl script.
andrew@andrew-OptiPlex-7010:/var/www/logs/bin$ ./apache-log-stat $(ls ../logs/b2f*_ApacheAccess.2011-12-05.gz) Processing ==================== ../logs/b2f01_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.92 variance_requests_per_second: 25.93 total_requests: 413603 stddev_requests_per_second: 5.09 max_requests_per_second: 57 ../logs/b2f02_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 6.05 variance_requests_per_second: 26.55 total_requests: 417711 stddev_requests_per_second: 5.15 max_requests_per_second: 45 ../logs/b2f03_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 6.09 variance_requests_per_second: 26.64 total_requests: 419073 stddev_requests_per_second: 5.16 max_requests_per_second: 48 ../logs/b2f04_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.07 variance_requests_per_second: 12.82 total_requests: 325374 stddev_requests_per_second: 3.58 max_requests_per_second: 29 ../logs/b2f05_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 0 variance_requests_per_second: 0 total_requests: 0 stddev_requests_per_second: 0 max_requests_per_second: 0 ../logs/b2f06_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.20 variance_requests_per_second: 13.29 total_requests: 338946 stddev_requests_per_second: 3.65 max_requests_per_second: 30 ../logs/b2f07_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.09 variance_requests_per_second: 12.92 total_requests: 330180 stddev_requests_per_second: 3.59 max_requests_per_second: 28 ../logs/b2f08_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.08 variance_requests_per_second: 12.78 total_requests: 334373 stddev_requests_per_second: 3.57 max_requests_per_second: 33 ../logs/b2f09_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.05 variance_requests_per_second: 12.66 total_requests: 327368 stddev_requests_per_second: 3.56 max_requests_per_second: 28 ../logs/b2f10_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 0 variance_requests_per_second: 0 total_requests: 0 stddev_requests_per_second: 0 max_requests_per_second: 0 ../logs/b2f11_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.51 variance_requests_per_second: 15.00 total_requests: 370115 stddev_requests_per_second: 3.87 max_requests_per_second: 35 ../logs/b2f12_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.43 variance_requests_per_second: 14.41 total_requests: 357580 stddev_requests_per_second: 3.80 max_requests_per_second: 37 ../logs/b2f13_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.47 variance_requests_per_second: 15.09 total_requests: 366275 stddev_requests_per_second: 3.88 max_requests_per_second: 36 ../logs/b2f14_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.39 variance_requests_per_second: 14.51 total_requests: 363630 stddev_requests_per_second: 3.81 max_requests_per_second: 34 ../logs/b2f15_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.40 variance_requests_per_second: 14.40 total_requests: 361463 stddev_requests_per_second: 3.80 max_requests_per_second: 41 ../logs/b2f16_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.87 variance_requests_per_second: 17.03 total_requests: 403126 stddev_requests_per_second: 4.13 max_requests_per_second: 48 ../logs/b2f17_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.90 variance_requests_per_second: 16.81 total_requests: 404777 stddev_requests_per_second: 4.10 max_requests_per_second: 37 ../logs/b2f18_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.89 variance_requests_per_second: 16.89 total_requests: 407635 stddev_requests_per_second: 4.11 max_requests_per_second: 30 ../logs/b2f19_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.82 variance_requests_per_second: 16.13 total_requests: 402808 stddev_requests_per_second: 4.02 max_requests_per_second: 36 ../logs/b2f20_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 5.84 variance_requests_per_second: 16.44 total_requests: 404247 stddev_requests_per_second: 4.05 max_requests_per_second: 35 ../logs/b2f21_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.91 variance_requests_per_second: 29.21 total_requests: 583211 stddev_requests_per_second: 5.40 max_requests_per_second: 41 ../logs/b2f22_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.86 variance_requests_per_second: 28.31 total_requests: 575122 stddev_requests_per_second: 5.32 max_requests_per_second: 38 ../logs/b2f23_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.68 variance_requests_per_second: 27.37 total_requests: 571349 stddev_requests_per_second: 5.23 max_requests_per_second: 39 ../logs/b2f24_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.76 variance_requests_per_second: 27.80 total_requests: 571728 stddev_requests_per_second: 5.27 max_requests_per_second: 37 ../logs/b2f25_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.90 variance_requests_per_second: 28.59 total_requests: 583242 stddev_requests_per_second: 5.35 max_requests_per_second: 51 ../logs/b2f26_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.87 variance_requests_per_second: 28.02 total_requests: 576603 stddev_requests_per_second: 5.29 max_requests_per_second: 36 ../logs/b2f27_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.91 variance_requests_per_second: 29.45 total_requests: 584972 stddev_requests_per_second: 5.43 max_requests_per_second: 42 ../logs/b2f28_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.66 variance_requests_per_second: 27.15 total_requests: 561073 stddev_requests_per_second: 5.21 max_requests_per_second: 38 ../logs/b2f29_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.83 variance_requests_per_second: 28.15 total_requests: 578100 stddev_requests_per_second: 5.31 max_requests_per_second: 47 ../logs/b2f30_ApacheAccess.2011-12-05.gz - avg_requests_per_second: 7.74 variance_requests_per_second: 27.62 total_requests: 574167 stddev_requests_per_second: 5.26 max_requests_per_second: 39 Total: avg_requests_per_second: 178.19 total_requests: 12507851 variance_requests_per_second: 581.99 stddev_requests_per_second: 125.99 max_requests_per_second: 1075 Requests per hour ==================== 00 # 199683 / 12507851 ( 1.60% ) var: 12308168 rqs: 55 rqm: 3328 01 121028 / 12507851 ( 0.97% ) var: 12386823 rqs: 33 rqm: 2017 02 92839 / 12507851 ( 0.74% ) var: 12415012 rqs: 25 rqm: 1547 03 79303 / 12507851 ( 0.63% ) var: 12428548 rqs: 22 rqm: 1321 04 76682 / 12507851 ( 0.61% ) var: 12431169 rqs: 21 rqm: 1278 05 96342 / 12507851 ( 0.77% ) var: 12411509 rqs: 26 rqm: 1605 06 # 141415 / 12507851 ( 1.13% ) var: 12366436 rqs: 39 rqm: 2356 07 # 245247 / 12507851 ( 1.96% ) var: 12262604 rqs: 68 rqm: 4087 08 ### 431089 / 12507851 ( 3.45% ) var: 12076762 rqs: 119 rqm: 7184 09 #### 589031 / 12507851 ( 4.71% ) var: 11918820 rqs: 163 rqm: 9817 10 ##### 662950 / 12507851 ( 5.30% ) var: 11844901 rqs: 184 rqm: 11049 11 ##### 637200 / 12507851 ( 5.09% ) var: 11870651 rqs: 177 rqm: 10620 12 ##### 739012 / 12507851 ( 5.91% ) var: 11768839 rqs: 205 rqm: 12316 13 ###### 780787 / 12507851 ( 6.24% ) var: 11727064 rqs: 216 rqm: 13013 14 ###### 779051 / 12507851 ( 6.23% ) var: 11728800 rqs: 216 rqm: 12984 15 ##### 667872 / 12507851 ( 5.34% ) var: 11839979 rqs: 185 rqm: 11131 16 ###### 750655 / 12507851 ( 6.00% ) var: 11757196 rqs: 208 rqm: 12510 17 ###### 804304 / 12507851 ( 6.43% ) var: 11703547 rqs: 223 rqm: 13405 18 ##### 740101 / 12507851 ( 5.92% ) var: 11767750 rqs: 205 rqm: 12335 19 ###### 795684 / 12507851 ( 6.36% ) var: 11712167 rqs: 221 rqm: 13261 20 ####### 973406 / 12507851 ( 7.78% ) var: 11534445 rqs: 270 rqm: 16223 21 ####### 979602 / 12507851 ( 7.83% ) var: 11528249 rqs: 272 rqm: 16326 22 ##### 724108 / 12507851 ( 5.79% ) var: 11783743 rqs: 201 rqm: 12068 23 ### 400460 / 12507851 ( 3.20% ) var: 12107391 rqs: 111 rqm: 6674 Comparison by distribution over an hour ==================== 00 #################### 199683 / 979602 ( 20.38% ) var: 779919 rqs: 55 rqm: 3328 01 ############ 121028 / 979602 ( 12.35% ) var: 858574 rqs: 33 rqm: 2017 02 ######### 92839 / 979602 ( 9.48% ) var: 886763 rqs: 25 rqm: 1547 03 ######## 79303 / 979602 ( 8.10% ) var: 900299 rqs: 22 rqm: 1321 04 ####### 76682 / 979602 ( 7.83% ) var: 902920 rqs: 21 rqm: 1278 05 ######### 96342 / 979602 ( 9.83% ) var: 883260 rqs: 26 rqm: 1605 06 ############## 141415 / 979602 ( 14.44% ) var: 838187 rqs: 39 rqm: 2356 07 ######################### 245247 / 979602 ( 25.04% ) var: 734355 rqs: 68 rqm: 4087 08 ############################################ 431089 / 979602 ( 44.01% ) var: 548513 rqs: 119 rqm: 7184 09 ############################################################ 589031 / 979602 ( 60.13% ) var: 390571 rqs: 163 rqm: 9817 10 ################################################################### 662950 / 979602 ( 67.68% ) var: 316652 rqs: 184 rqm: 11049 11 ################################################################# 637200 / 979602 ( 65.05% ) var: 342402 rqs: 177 rqm: 10620 12 ########################################################################### 739012 / 979602 ( 75.44% ) var: 240590 rqs: 205 rqm: 12316 13 ############################################################################### 780787 / 979602 ( 79.70% ) var: 198815 rqs: 216 rqm: 13013 14 ############################################################################### 779051 / 979602 ( 79.53% ) var: 200551 rqs: 216 rqm: 12984 15 #################################################################### 667872 / 979602 ( 68.18% ) var: 311730 rqs: 185 rqm: 11131 16 ############################################################################ 750655 / 979602 ( 76.63% ) var: 228947 rqs: 208 rqm: 12510 17 ################################################################################## 804304 / 979602 ( 82.11% ) var: 175298 rqs: 223 rqm: 13405 18 ########################################################################### 740101 / 979602 ( 75.55% ) var: 239501 rqs: 205 rqm: 12335 19 ################################################################################# 795684 / 979602 ( 81.23% ) var: 183918 rqs: 221 rqm: 13261 20 ################################################################################################### 973406 / 979602 ( 99.37% ) var: 6196 rqs: 270 rqm: 16223 21 #################################################################################################### 979602 / 979602 (100.00% ) var: 0 rqs: 272 rqm: 16326 22 ######################################################################### 724108 / 979602 ( 73.92% ) var: 255494 rqs: 201 rqm: 12068 23 ########################################
- (1) Comment
- Tags: Bash, perl
Hackday @ Everlution
25 Jul 2012At Everlution we did a hack day to help motivate developers and to get the team working together. We started at 8.30am and finished at 5pm. It was pretty impressive what all the teams managed to achieve, with four teams of two developers and a designer.
It was set out in a competition style event with all teams competing against each other with a defined set of points for each task achieved. The tasks were very ambitious considering we had a day to complete the task.
The key task was to analyse a webcam and determine whether a meeting room was in use or not by analysing the web cams jpeg stream.
The motivation for which is that many staff members book meeting rooms and don’t use them or randomly hijack meeting rooms.
Task sheet/Points
| # | Task | Points* |
|---|---|---|
| 1 | To build a webpage that tells the user if the boardroom is in use or not by parsing the JPEG stream emitted from the webcam. Scoring:
|
|
| 2 | To use a Face detection and recognition library to say whether one member of your team is in the Boardroom or not. |
|
| 3 | To build an MSN or Skype Bot that will answer a natural language question on whether the Boardroom is in use or not. Must be able to handle at least 3 variations of how question could be asked. Scoring:
|
|
| 4 | To build a native mobile app that will display same information. Scoring:
|
|
| 5 | To come up with a name, brand/logo and design for the service. Scoring:
|
|
| 6 | Bonus: Good team work |
|
| 7 | Bonus: Quality, reliability and elegance of solution |
|
| 8 | Sabotaging another team’s effort, being disruptive, not participating in spirit of competition, etc. |
|
With regards to “Detection system to not trigger with someone going in/out of Studio 2″. There is a door going into the boardroom and a door just outside going into another office of Everlution. So any body outside the glass should not be detected going into that office.
The implementations varied drastically between teams, some based on motion.
Team 1 – My Team:
Used a C# client application to detect motion in real time from the remote image and/or directly attached camera using AForge.
The sample client was more than enough and simply ran a timed event to push the stats across over http to a PHP based API and store within a MySQL database. The web page then polled periodically to check the availability from the MySQL database, which produces the following data set. If the motion_level was above 0.02 it would indicate the room is occupied. This histogram is indicated at the bottom of the image stream.
The main issue was that it took me till 11am to install Visual Studio. Additionally the implementation was improved by adjusting the motion detection method depending on whether the stream was sourced from the camera itself or by the image, which was updated every two seconds.
{
"boardroom" : {
"data" : {
"date_created" : "2012-07-24 00:14:19",
"name" : "boardroom",
"signal_time" : "24/07/2012 00:14:20",
"fqdn" : "HP28458242993",
"motion_level" : "0.000406901",
"detected_objects_count" : "-1",
"has_motion_been_detected" : "",
"id" : "19945",
"motion_alarm_level" : "0.015",
"flash" : "0"
},
"last_used" : {
"date_created" : "2012-07-24 00:14:19",
"name" : "boardroom",
"signal_time" : "24/07/2012 00:14:20",
"fqdn" : "HP28458242993",
"motion_level" : "0.000406901",
"detected_objects_count" : "-1",
"has_motion_been_detected" : "",
"id" : "19945",
"motion_alarm_level" : "0.015",
"flash" : "0"
},
"is_free" : true
}
}
AForge is an excellent library and provides the following…
- AForge.Imaging – library with image processing routines and filters
- AForge.Vision – computer vision library
- AForge.Neuro – neural networks computation library
- AForge.Genetic – evolution programming library
- AForge.Fuzzy – fuzzy computations library
- AForge.MachineLearning – machine learning library
- AForge.Robotics – library providing support of some robotics kits
- AForge.Video – set of libraries for video processing
In order to achieve face detection, I was going to use OpenCV, however ran out of time to fully implement. As such we implemented face.com to attempt face recognition. Unfortunately for all implementations this API failed to recognise anyone.
Additionally an MSN bot was built on put on top although did not implement any real natural language implementation.
The results in action
Team 2:
The implementation was based on PHP Imagick using edge detection by enhancing the images with edgeImage and creating composite images.
Two sets of metrics were extracted for statistical analysis from the image. For all channels the mean, stddev from each image used (The raw image, the composite). A histogram was also generated for each channel with the std dev, variance and skew.
Team 2 were to use two important metrics that they did not manage to implement, however their intention was too…
- Edge density (from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1517&context=infopapers) , and
- Image energy ( I would calculate then total amount, standard deviation etc.
- All that after pre filtering with Sobel operator – http://www.win.tue.nl/~wstahw/2IV05/seamcarving.pdf)
These stats were then passed to Quinlan’s C4.5 and provided with some training pictures with people in the room or not.
The idea behind applying edge density & energy analysis was for the algorithm to learn about the room’s layout… i.e. moving a chair would not change density nor energy a lot, but adding another one would do, so theoretically this approach should learn about objects inside the room. Filtering by edge density to detect features would remove unimportant (not dense) objects such as chairs or people being too far away.
If C4.5 used that algorithm with enough data with people and other objects, it would learn itself about how to distinguish between a face and a chair.
Team 3:
Wrote their implementation using python with bindings for OpenCV. Initially loading the image creating 3 grayscale images.
- Image 1: Last board room image
- Image 2: The current board room image
- Image 3: The AbsDiff between Image 1 and Image 2
Using ImageChops.difference the stats for the difference are stored in sqlite and divided by an average. If the condition is met it will mark the room as being occupied.
Snapshots from OpenCV can be seen below.
 |
 |
 |
 |
Team 4:
Used PHPs Imagick extension and the following function http://robert-lerner.com/imagecompare.php. They ran a series of slices whether someone was in the room and determined that if there was a 5% variance the meeting room was occupied.
The results in action
Other references
C Sharp / C / C++
Multiple face detection and recognition in real time
Playing Card Recognition Using AForge.Net Framework
OpenCV
- No Comments
- Tags: C Sharp, C#, C4.5, Image Analysis, Imagick, PHP
I previously wrote about “Installing Java & MySQL – unattended/non-interactive installation”. Whilst using slack, I have been running apt-get during postinstall rather than preinstall.
Files are transferred from slack after preinstall as such transferring any files that conflict with the packages configuration files, apt-get will prompt regardless of DEBIAN_FRONTEND=noninteractive. Not ideal if your trying to run non-interactive. You can use the -o option to circumvent this…
apt-get -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' -f -q -y php-cli
Previous example of non-interactive installations.
INSTALLER_LOG=/var/log/non-interactive-installer.log
installnoninteractive(){
sudo bash -c "DEBIAN_FRONTEND=noninteractive aptitude install -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' -f -q -y $* >> $INSTALLER_LOG"
}
assert_package() {
is_installed=$(apt-cache policy "$1" | grep 'Installed: (none)' | wc -l);
if [ "$is_installed" -eq "1" ] ; then
#apt-get install -q -y $1;
installnoninteractive $1;
fi;
}
- (1) Comment
- Tags: apt-get, Bash, Debian, non-interactive, PHP, Ubuntu
About this blog
I have been a developer for roughly 10 years and have worked with an extensive range of technologies. Whilst working for relatively small companies, I have worked with all aspects of the development life cycle, which has given me a broad and in-depth experience.
Archives
- September 2020
- December 2015
- June 2015
- May 2015
- October 2014
- April 2013
- October 2012
- September 2012
- July 2012
- January 2012
- December 2011
- June 2011
- February 2011
- January 2011
- December 2010
- July 2010
- May 2010
- April 2010
- January 2010
- December 2009
- October 2009
- June 2009
- February 2008
- July 2007
- June 2007
- April 2007
- December 2006
- October 2006
- August 2006
- July 2006
- April 2006
- March 2006
- February 2006
- January 2006
- December 2005
- November 2005
- October 2005
- August 2005
- July 2005
- June 2005
- March 2005
- June 2004
- Eugene: I've just hit exactly the same issue. Would have been be nice if Amazon mentioned it in documentatio [...]
- Ozzy: Thanks Andy. This was useful (: [...]
- shahzaib: Nice post . I have a question regarding BGP. As you said, we'll need to advertise our route from mul [...]
- Steve: Hi there… sorry that this is old, but I’m trying to use your script to check for my u [...]
- John Cannon: Hey that was really needful. Thanks for sharing. I’ll surely be looking for more. [...]
- Istio EnvoyFilter to add x-request-id to all responses
- Amazon ELB – monitoring packet count and byte size with Amazon Cloudwatch and VPC flow logs
- SSL – Amazon ELB Certificates
- Automatically update Amazon ELB SSL Negotiation Policies
- Amazon Opsworks – Dependent on Monit
- Awk craziness: Processing log files
- Elastic Search Presentation
- Requests per second from Apache logs
- Hackday @ Everlution
- apt-get unattended / non-interactive installation